Imaging system, imaging controller, imaging control method, and program

Abstract

To facilitate video changeover with less sense of discomfort in a technology of changing over video from a plurality of imaging devices.SOLUTION: An imaging system images first video, images second video while changing an angle of view, identifies video contents, selects a video connection type showing an inter-video relation in video changeover from the first video according to a result of identifying a content of the first video, determines whether or not a result of identifying a content of the second video satisfies a condition corresponding to the video connection type, and determines that changeover of the first video to the second video is possible on the basis of a result of determining that the condition is satisfied.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an imaging system, an imaging control device, an imaging control method, and a program, and more particularly to a control method for imaging devices in a system that uses multiple imaging devices to capture video. [Background technology]

[0002] In recent years, there has been an increasing demand for live streaming and video production. For example, entertainment events such as music events, plays, and sporting events are often filmed. In these cases, multiple camera devices can be used simultaneously to film from multiple viewpoints. In this case, a video to be distributed can be selected from multiple videos filmed from multiple viewpoints. This requires the use of a control device called a switcher.

[0003] There is also known a technique for automatically controlling the angle of view of at least one camera when photographing using multiple cameras. For example, Patent Document 1 discloses a technique in which, when a user specifies an arbitrary area in an image photographed by a first photographing device, a second photographing device is controlled so as to photograph an area overlapping the specified area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2019-186635 Summary of the Invention [Problem to be solved by the invention]

[0005] When switching between images using a switcher or the like, if there is no correlation between the images before and after the switch, the images will appear unnatural.

[0006] The present disclosure provides a technique for switching between images from multiple image capture devices, which facilitates image switching with less discomfort. [Means for solving the problem]

[0007] An imaging system apparatus according to an embodiment of the present disclosure includes the following configuration: The device comprises a first imaging means for capturing a first image, a second imaging means for capturing a second image while changing the angle of view, an identification means for identifying the content of the image, a type determination means for selecting a type of image splicing indicating the relationship between images when switching from the first image according to the identification result of the content of the first image, and a switching determination means for determining whether the identification result of the content of the second image satisfies a condition corresponding to the type of image splicing, and determining that the first image can be switched to the second image based on the determination result that the condition is satisfied. [Effects of the Invention]

[0008] In a technique for switching between images from a plurality of image capturing devices, it is possible to easily switch between images with little sense of incongruity. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an imaging system according to an embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the hardware configuration of an imaging system according to an embodiment. [Figure 3] FIG. 1 is a diagram showing an example of the functional configuration of an imaging system according to an embodiment. [Figure 4] FIG. 10 is a diagram showing an example of caption information. [Figure 5] FIG. 10 is a diagram showing an example of caption information. [Figure 6] 1 is a flowchart of a process performed by a manually controlled camera in one embodiment. [Figure 7] 1 is a flowchart of a process performed by an automatically controlled camera in one embodiment. [Figure 8] FIG. 2 is a diagram showing an example of the functional configuration of a type determination unit. [Figure 9] 10 is a flowchart of a video splice determination process. [Figure 10] FIG. 10 is a diagram showing an example of relevance estimation information. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] An example of the configuration of a filming system according to one embodiment will be described with reference to Fig. 1. The filming system according to this embodiment can be used for video production. The filming system according to this embodiment can realize three types of video splicing: element splicing, action splicing, and insert splicing, which will be described later.

[0012] The image capturing system according to this embodiment includes a first image capturing device and a second image capturing device. The first image capturing device and the second image capturing device are used to capture multiple viewpoint images. The first image capturing device and the second image capturing device are each capable of changing the angle of view. For example, the first image capturing device and the second image capturing device may have pan, tilt, and zoom mechanisms.

[0013] In one embodiment, the first image capturing device is a manually controlled camera 101. For example, an operator can control the manually controlled camera 101 via an operation input device 104. Specifically, the operator can specify the angle of view of the manually controlled camera 101. At this time, the manually controlled camera 101 captures an image with the angle of view specified by the operator. However, the angle of view of the manually controlled camera 101 may be changed depending on the force that the operator applies to the manually controlled camera 101. Alternatively, the first image capturing device may be automatically controlled.

[0014] In one embodiment, the second image capture device is an automatically controlled automatically controlled camera 102. The automatically controlled camera 102 can automatically determine the angle of view, as described below. That is, the automatically controlled camera 102 can receive the analysis results of the video captured by the manually controlled camera 101. Furthermore, the automatically controlled camera 102 can control the angle of view so as to obtain video that is related to the video captured by the manually controlled camera 101, based on the analysis results. However, it is not essential that the second image capture device be automatically controlled. For example, the angle of view of the second image capture device may be manually controlled.

[0015] The photography system according to this embodiment may further include an operation input device 104. The operation input device 104 is a terminal used by an operator to control the photography system. The operation input device 104 may be, for example, an input device such as a controller, or an information processing device such as a personal computer, a smartphone, or a tablet terminal. The operation input device 104 may control a change in the angle of view of the camera. For example, the operation input device 104 may send a control signal to the manually controlled camera 101 in accordance with an operation by the operator. At this time, the manually controlled camera 101 may change the angle of view in accordance with the control signal.

[0016] The operation input device 104 can also perform switching control. Switching refers to switching between images. For example, switching can be switching between images to be distributed or recorded. In this embodiment, the operation input device 104 can switch between images captured by the manually controlled camera 101 and images captured by the automatically controlled camera 102. In this specification, switching from images captured by the manually controlled camera 101 to images captured by the automatically controlled camera 102 is sometimes referred to as switching from the manually controlled camera 101 to the automatically controlled camera 102. In one embodiment, the operation input device 104 can distribute images from a camera selected by switching control to an area outside the filming system. In another embodiment, the operation input device 104 can store images from a camera selected by switching control, or store the images in a storage device outside the filming system.

[0017] Manually controlled camera 101 can transmit the analysis results of the captured video to automatically controlled camera 102 via network 103. Based on the analysis results of the video captured by manually controlled camera 101, automatically controlled camera 102 changes the angle of view of automatically controlled camera 102 so that video that does not appear strange when switched to automatically controlled camera 102. Note that the angle of view of automatically controlled camera 102 may be changed only when the angle of view of manually controlled camera 101 is changed or when there is a change in the video captured by manually controlled camera 101 due to the movement of the subject or the like.

[0018] The manually controlled camera 101, the automatically controlled camera 102, and the operation input device 104 are connected via a network 103. The type of the network 103 is not particularly limited, and may be, for example, a wired network or a wireless network. The network 103 may also be a local area network (LAN) or the Internet.

[0019] 1 shows the main components of the imaging system according to this embodiment. That is, the imaging system may include additional devices not shown. For example, more cameras may be connected to the network 103. Also, a server device connected to the network 103, separate from the operation input device 104, may have a server function for distributing video via the network 103 or a function for storing video.

[0020] 2 shows an example of the hardware configuration of the manually controlled camera 101 and / or the automatically controlled camera 102. In this embodiment, the manually controlled camera 101 and the automatically controlled camera 102 have the same hardware configuration and operate in the same manner unless otherwise specified. However, the manually controlled camera 101 and the automatically controlled camera 102 may have different hardware configurations.

[0021] The manually controlled camera 101 and the automatically controlled camera 102 each include a CPU 201 , a RAM 202 , a ROM 203 , an operation unit 204 , an output control unit 205 , a communication I / F 206 , and an imaging unit 207 .

[0022] The ROM 203 is a memory that stores a boot program executed by the CPU 201 when the manually controlled camera 101 or the automatically controlled camera 102 is started, instruction programs for executing various processes, and data used by these programs. The ROM 203 may be a readable / writable medium such as an HDD (hard disk drive) or an SSD (solid state drive).

[0023] The CPU 201 controls a motor connected to the manually controlled camera 101 or the automatically controlled camera 102 via the output control unit 205, for changing the angle of view. In this embodiment, pan, tilt, and zoom (hereinafter, PTZ) control of the manually controlled camera 101 and the automatically controlled camera 102 is performed. The CPU 201 also acquires data via the operation unit 204. The operation unit 204 can process signals received from the operation input device 104 or the like via the communication I / F 206 and transmit data indicating the processing results to the CPU 201. The CPU 201 can also output data generated by the processing to another device via the output control unit 205. The CPU 201 executes programs loaded on the RAM 202 to realize the functions shown in FIG. 3 and other camera functions.

[0024] The communication I / F 206 receives data from other devices via the network 103 and sends the received data to the CPU 201. The communication I / F 206 also transmits data generated by the CPU 201 to other devices via the network 103. The automatically controlled camera 102 also acquires the analysis results of the video generated by the manually controlled camera 101 via the communication I / F 206 and stores them in the RAM 202.

[0025] The imaging unit 207 captures video in accordance with the angle of view controlled by the output control unit 205. The imaging unit 207 of the manually controlled camera 101 captures a first video. The imaging unit 207 of the automatically controlled camera 102 captures a second video. The imaging unit 207 may have an optical system including an image sensor and a lens. The video captured by the imaging unit 207 is stored in RAM 202. The manually controlled camera 101 and the automatically controlled camera 102 can read the video data into RAM 202 and then perform video analysis or distribution processing.

[0026] The CPU 201 can load the above-mentioned programs from the ROM 203 to the RAM 202. The CPU 201 can then execute the programs loaded into the RAM 202. On the other hand, these programs may be acquired from other devices via the communication I / F 206.

[0027] FIG. 3 shows an example of the functional configuration of the manually controlled camera 101 and the automatically controlled camera 102. The following describes the processing performed by the manually controlled camera 101 and the automatically controlled camera 102 when capturing video. In the following description, the functional units shown in FIG. 3 are responsible for the processing. In this embodiment, the processing of each functional unit is realized by the CPU 201 executing a computer program. However, at least some of the functional units shown in FIG. 3 may be implemented in hardware.

[0028] The manually controlled camera 101 has an instruction receiving unit 301, an angle of view control unit 302, an image acquisition unit 303, an area division unit 304, an image identification unit 305, and a result transmission unit 306. The automatically controlled camera 102 has a result receiving unit 307, a type determination unit 308, a switching determination unit 309, a notification unit 310, the image acquisition unit 303, the area division unit 304, the image identification unit 305, and the angle of view control unit 302.

[0029] The instruction receiving unit 301 acquires a control instruction from the operation input device 104. This control instruction may include information for controlling the angle of view of the manually controlled camera 101. The information for controlling the angle of view may be, for example, information specifying a pan, tilt, or zoom position, or information indicating the amount of change in the pan, tilt, or zoom position. Hereinafter, such information will be referred to as PTZ control information. The instruction receiving unit 301 transmits the PTZ control information to the angle of view control unit 302 in accordance with the acquired control instruction.

[0030] The angle-of-view control unit 302 changes the angle of view of the camera. The angle-of-view control unit 302 can change the angle of view by controlling hardware such as a motor mounted on the camera. For example, the angle-of-view control unit 302 can control the pan, tilt, and zoom of the camera. The angle-of-view control unit 302 can perform such control in accordance with PTZ control information. The angle-of-view control unit 302 may also achieve PTZ control electronically by cutting out a portion of the image. For example, the angle-of-view control unit 302 can use electronic zoom.

[0031] The angle-of-view control unit 302 of the manually controlled camera 101 can change the angle of view of the manually controlled camera 101 in accordance with a signal received from the instruction receiving unit 301. The angle-of-view control unit 302 of the automatically controlled camera 102 can change the angle of view of the automatically controlled camera 102 in accordance with a preset algorithm. For example, the angle-of-view control unit 302 of the automatically controlled camera 102 can change the angle of view of the automatically controlled camera 102 so that it patrols within the captureable range. The angle-of-view control unit 302 of the automatically controlled camera 102 may also continuously change the angle of view of the automatically controlled camera 102. Through such control, the angle-of-view control unit 302 can search for the angle of view of the automatically controlled camera 102 so that it can switch from the manually controlled camera 101 to the automatically controlled camera 102.

[0032] The image acquisition unit 303 converts into image data the electrical signals acquired by the imaging unit 207 during the image capturing operation, and stores the acquired image data in the RAM 202 of each camera.

[0033] The region dividing unit 304 and the image identifying unit 305 identify the content of the image. In this embodiment, the region dividing unit 304 and the image identifying unit 305 of the manually controlled camera 101 identify the content of the first image captured by the manually controlled camera 101. The region dividing unit 304 and the image identifying unit 305 of the automatically controlled camera 102 identify the content of the second image captured by the automatically controlled camera 102.

[0034] In this embodiment, the region dividing unit 304 performs region dividing processing on the captured video. The region dividing unit 304 can divide the video into regions according to the position of an object in the video. For example, the region dividing unit 304 can perform region dividing for each object based on the object detection result. For example, the region dividing unit 304 can perform processing to classify objects in the video stored in the RAM 202 on a pixel-by-pixel basis. The region dividing unit 304 may detect and recognize objects in the video by image recognition using a neural network. Then, the region dividing unit 304 can divide the region into each object according to the classification result. The region dividing unit 304 can perform such region dividing processing (hereinafter, sometimes referred to as segmentation processing) for each frame of the video. However, the method of segmentation processing is not particularly limited. For example, the region dividing unit 304 may divide the video into multiple rectangular regions of the same size. Furthermore, in this embodiment, it is not essential that the region dividing unit 304 perform segmentation processing. In this case, the video identification unit 305, which will be described later, may perform captioning processing on the entire first video.

[0035] The region division unit 304 can also identify the presence or absence of a main subject in the video. In this embodiment, the region division unit 304 can determine whether a main subject, such as a person or an object, is present in each divided region according to the classification results. In this embodiment, the main subject is different from a landscape or a uniform texture. The method for determining the presence of a main subject is not particularly limited. For example, the region division unit 304 can determine the main subject based on the position, size, or semantic information of candidate objects in the input image. As a specific example, the region division unit 304 can detect a specific type of object as the main subject. As another example, the region division unit 304 can detect an object of a specific type that occupies an area larger than a threshold as the main subject. The region division unit 304 can also determine the main subject based on the distribution characteristics of each pixel in the input image. If a main subject is present in each region, the region division unit 304 can record information indicating the presence of the main subject.

[0036] The region dividing unit 304 can record the results of the segmentation process (hereinafter, sometimes referred to as segmentation information) in the RAM 202. The segmentation information includes a pair of ID information that can uniquely identify a region and information indicating the presence or absence of a main subject in this region.

[0037] The video identification unit 305 performs a process of identifying objects in each region. In this embodiment, the video identification unit 305 performs a captioning process to generate captions that describe the video for each region. The captioning process generates captions that express in sentences the names, characteristics, or actions of objects included in each region. The video identification unit 305 performs the captioning process based on the segmentation information stored in RAM 202. The method of the captioning process is not particularly limited. For example, the video identification unit 305 may generate captions based on the object recognition results obtained by the region segmentation unit 304. The video identification unit 305 may also generate captions through image recognition using a neural network. Note that in captioning processes using a neural network, English captions are often generated. However, as will be described later with reference to FIG. 4, the video identification unit 305 may also generate captions in Japanese. In this manner, the language of the captions is not limited. The video identification unit 305 may perform captioning processing in accordance with the methods described in Japanese Patent Application Publication No. 2023-128088, Japanese Patent Application Publication No. 2022-135518, Japanese Patent Application Publication No. 2021-117860, or Japanese Patent Application Publication No. 2020-512759.

[0038] Video identification unit 305 can record the result of the captioning process (hereinafter, sometimes referred to as caption information) in RAM 202. The caption information includes ID information of the area included in the segmentation information, information indicating the presence or absence of the main subject, and a set of captions.

[0039] As described above, the video identification unit 305 can identify the content of the video. The identification result is indicated by the caption information described above. For example, the caption can indicate the name (e.g., Mr. A and Mr. B) or type (e.g., person) of an object in the video. The caption can also indicate the action (e.g., playing) of the object in the video. In this way, the video identification unit 305 can identify the object in the video and the action of the object in the video. However, captioning processing is not required in the present disclosure. For example, the video identification unit 305 may identify the object in the video and the action of the object in the video by image recognition using a neural network. In one embodiment, the result transmission unit 306, described later, transmits information indicating such an identification result to the automatic control camera 102 instead of the caption information.

[0040] The result transmitting unit 306 of the manually controlled camera 101 obtains from the RAM 202 information indicating the identification result of the content of the first video (in this example, caption information) generated by the video identification unit 305, and transmits it to the automatically controlled camera 102. The result receiving unit 307 of the automatically controlled camera 102 receives the information indicating the identification result of the content of the first video (in this example, caption information) transmitted from the manually controlled camera 101, and records it in the RAM 202.

[0041] The type determination unit 308 selects a video splicing type according to the identification result of the content of the first video captured by the manually controlled camera 101. The video splicing type indicates the relationship between the videos when switching from the first video captured by the manually controlled camera 101. In this embodiment, the type determination unit 308 selects a video splicing type that can be implemented when switching from the manually controlled camera 101 to the automatically controlled camera 102. The type determination unit 308 can select a video splicing type based on at least one of an object captured in the first video, the movement of the object captured in the first video, and the presence or absence of a main subject in the first video.

[0042] In this embodiment, the type determination unit 308 acquires caption information transmitted from the manually controlled camera 101, which indicates the result of identifying the content of the first video. Then, the type determination unit 308 selects a video splice type based on the caption information. The type determination unit 308 can select a video splice type by extracting characteristic elements or movements from the video based on the caption included in the caption information and information indicating the presence or absence of a main subject. A specific method for determining the video splice type will be described later.

[0043] The switching determination unit 309 determines whether the identification result of the content of the second video captured by the automatic control camera 102 satisfies the switching condition corresponding to the video splicing type selected by the type determination unit 308. In this embodiment, the switching determination unit 309 determines whether the identification result of the content of the second video satisfies a condition based on the identification result of the first video, in addition to the video splicing type determined by the type determination unit 308. Then, based on the determination result that this condition is satisfied, the switching determination unit 309 determines that the first video can be switched to the second video. Using the method described below, the switching determination unit 309 can determine that the first video can be switched to the second video if the first video can be spliced ​​to the second video without causing a sense of discomfort to the viewer.

[0044] In this embodiment, the switching determination unit 309 determines whether the second video satisfies a condition corresponding to the type of video connection based on the caption information for each of the first video and the second video. In the following example, the switching determination unit 309 makes this determination based on the relevance estimation information generated based on the caption information for the first video and the caption information for each of the second videos. Specific conditions will be described later.

[0045] As described above, the angle-of-view control unit 302 of the automatic control camera 102 can control the angle of view of the automatic control camera 102 so that it patrols within the captureable range. For example, the angle-of-view control unit 302 can continuously (e.g., continuously or intermittently) change the angle of view of the automatic control camera 102 before the switching determination unit 309 determines that the first video image can be switched to the second video image. On the other hand, the angle-of-view control unit 302 can stop changing the angle of view of the automatic control camera 102 in response to the switching determination unit 309 determining that the first video image can be switched to the second video image. As a result, the angle of view of the automatic control camera 102 is controlled so that switching from the first video image to the second video image is possible in accordance with the video stitching type. In this way, the angle-of-view control unit 302 of the automatic control camera 102 can determine the angle of view of the automatic control camera 102 in accordance with the determination result by the switching determination unit 309.

[0046] The notification unit 310 notifies the operator that the first video can be switched to the second video in response to the switching determination unit 309 determining that the first video can be switched to the second video. For example, the notification unit 310 can notify the operation input device 104. The operator of the switching can perform the switching after checking the notification. The notification unit 310 may send a notification to the manually controlled camera 101 or another device connected via the network 103. The notification unit 310 may also notify a device external to the imaging system. In this way, the notification unit 310 can notify the operator of the switching that the angle of view of the automatically controlled camera 102 has been changed so that the manually controlled camera 101 can be switched to the automatically controlled camera 102 in any manner that allows the operator of the switching to understand.

[0047] The functional configuration of the imaging system is not limited to that shown in FIG. 3. In the example shown in FIG. 3, each functional unit is distributed between the manually controlled camera 101 and the automatically controlled camera 102. However, for example, the manually controlled camera 101 may have the type determination unit 308 instead of the automatically controlled camera 102. Furthermore, the area division unit 304 and the video identification unit 305 of the automatically controlled camera 102 may generate caption information for the first video. In this case, the manually controlled camera 101 does not need to have the area division unit 304 and the video identification unit 305. Furthermore, the operation input device 104 may have functional units for determining whether or not switching is possible, such as the area division unit 304, the video identification unit 305, the type determination unit 308, the switching determination unit 309, and the notification unit 310, and function as an imaging control device. Such an operation input device 104 can be realized by a computer including a processor and a memory. That is, the processor executes a program stored in the memory to realize the functions of each unit.

[0048] 4 and 5 show examples of caption information generated by video identification unit 305 based on the segmentation information generated by region division unit 304. Figures 4(A) and (B) and Figures 5(A) and (B) respectively show caption information generated for a specific frame of video. As shown in Figures 4 and 5, the caption information includes region ID information, information indicating the presence or absence of a main subject, and a caption.

[0049] First, an example of a method for selecting a video splice type will be described with reference to Fig. 4. In this embodiment, the video splice types include element splice, action splice, and insert splice.

[0050] Element transition is a method of transitioning videos so that the objects or types of objects in the videos are the same before and after the transition. In element transition, for example, switching is performed so that the types of components or objects present in the videos are the same before and after the transition. Therefore, when a predetermined object is detected from the first video, the type determination unit 308 can select element transition as the video transition type. Here, the predetermined object may be any object regardless of type, or may be a specific type of object. Furthermore, the predetermined object may be an object determined to be a main subject. As a specific example, if the video before the transition includes a musical instrument as a main subject, the video after the transition may also include a musical instrument as a main subject. In this case, since there is a correlation between the videos before and after the transition, the sense of discomfort felt by the viewer due to the switching can be reduced.

[0051] Action transition is a method of connecting videos so that the actions of objects in the videos are common before and after switching. In action transition, for example, switching is performed while multiple cameras are shooting a common action performed by a moving object. Therefore, when a predetermined action is detected from the first video, the type determination unit 308 can select action transition as the video transition type. Here, the predetermined action may be any action regardless of type, or may be a specific type of action. Furthermore, the action may include the behavior and movement of an object. Action transition can also connect videos semantically. This can reduce the sense of discomfort felt by the viewer due to switching.

[0052] Insert splicing is a method of splicing videos so that the main subject does not exist in either the video before switching or the video after switching. For example, if the type determination unit 308 does not detect the main subject in the first video, it can select insert splicing as the video splicing type. When performing insert splicing, the automatic control camera 102 shoots an insert video in which the main subject does not exist. With insert splicing, it is possible to splice videos so that the main subject does not change. This reduces the sense of discomfort felt by viewers when switching.

[0053] As described above, in this embodiment, the type determination unit 308 selects the video splicing type based on caption information indicating the identification result of the video content. A method for selecting the video splicing type based on caption information will be described below.

[0054] 4A shows an example of caption information generated by the video identification unit 305 of the manually controlled camera 101. The type determination unit 308 can determine the type of video splicing based on words or sentences included in the caption that describes the first video. In this embodiment, the type determination unit 308 can select an available video splicing type by identifying the part of speech of the caption.

[0055] For example, when a caption includes a noun, such as the caption ID1, the videos can be joined together so that the object indicated by the noun is the same. Therefore, in this case, the type determination unit 308 can determine that element joining is possible. Also, when a caption includes a verb, such as the caption ID1, the videos can be joined together so that the action indicated by the verb is the same. Therefore, in this case, the type determination unit 308 can determine that action joining is possible. The type determination unit 308 may determine that action joining is possible when the caption includes a verb indicating an action.

[0056] Note that type determination unit 308 may select two or more types of video transition types. For example, in the example shown in Fig. 4(A), video identification unit 305 may determine that element transition and action transition are possible.

[0057] 4(B) shows another example of caption information generated by the video identification unit 305 of the manually controlled camera 101. In one embodiment, the video identification unit 305 can determine that an insert transition is possible when the main subject is not present in any of the regions. In the example shown in FIG. 4(B), the video identification unit 305 can determine that an insert transition is possible.

[0058] In this embodiment, the region division unit 304 determines the presence or absence of a main subject for each region. Alternatively, the video identification unit 305 may determine the presence or absence of a main subject based on a caption. For example, a case in which a main subject does not exist may be when the caption does not include a common noun or proper noun representing a person. In other words, when the caption only includes a collective noun or a common noun representing an object, it can be determined that a main subject does not exist. As another example, a word representing a main subject may be registered in advance for the automatic control camera 102. In this case, the video identification unit 305 can determine that a main subject does not exist when the caption does not include the registered word.

[0059] The video identification unit 305 may select a single video splice type. On the other hand, in the above-described determination based on captions, the conditions for each of multiple video splice types may be satisfied. In this case, the video identification unit 305 may determine that all video splice types that satisfy the conditions are executable. It is also possible that the conditions are not satisfied for all video splice types. In this case, the video identification unit 305 does not need to select a video splice type. Furthermore, the types of video splice types are not particularly limited. In other words, the selectable video splice types may include video splice types other than element splice, action splice, and insert splice. Furthermore, the number of selectable video splice types may be one, or two or more.

[0060] Next, an example of the processing performed by the type determination unit 308 and the switching determination unit 309 will be described with reference to FIG. 5. FIG. 5(A) shows an example of caption information generated by the video identification unit 305 of the manually controlled camera 101. FIG. 5(B) shows an example of caption information generated by the video identification unit 305 of the automatically controlled camera 102 while the automatically controlled camera 102 is searching for an angle of view. As described above, when the type determination unit 308 receives caption information such as that shown in FIG. 5(A) from the manually controlled camera 101, it selects an executable video transition type based on the caption information. In this example, because the caption includes common nouns such as "stage" and "guitar," the type determination unit 308 determines that element transition is executable. Furthermore, because the caption includes a verb such as "play," the type determination unit 308 determines that action transition is also executable.

[0061] At this time, the type determination unit 308 can record the information referenced to determine whether element transition or action transition is feasible in RAM 202 in combination with information indicating the video transition type. This information is referred to herein as relevance estimation information. The type determination unit 308 generates the relevance estimation information based on words or sentences included in the caption describing the first video. For example, the type determination unit 308 can record words or sentences that are selection factors (determination grounds) for the video transition type. This relevance estimation information indicates words or sentences that should be included in the caption describing the second video to satisfy the switching conditions. Therefore, the relevance estimation information can be used as a condition for the angle of view that the automatic control camera 102 should capture.

[0062] For example, the type determination unit 308 can record a noun (such as the above "stage" and "guitar") included in a caption generated by the video identification unit 305, paired with information indicating an element transition. The type determination unit 308 can also record a verb (such as the above "play") included in a caption, paired with information indicating an action transition.

[0063] 10 shows an example of the relevance estimation information that the type determination unit 308 records in the RAM 202. The relevance estimation information includes an area ID, a video splice type determined by the type determination unit 308, and an estimated basis text. The estimated basis text is a word or sentence that is the basis for determining the video splice type.

[0064] In this embodiment, the automatically controlled camera 102 captures video while changing the angle of view of the automatically controlled camera 102 so that the camera 102 moves around within the captureable range. The video identification unit 305 of the automatically controlled camera 102 generates caption information such as that shown in FIG. 5(B) for the video captured by the automatically controlled camera 102.

[0065] The switching determination unit 309 compares words or sentences included in the caption describing the first video with words or sentences included in the caption describing the second video to determine whether a switching condition is met. In this embodiment, the switching determination unit 309 acquires, from the RAM 202, relevance estimation information generated by the type determination unit 308 based on caption information acquired from the manually controlled camera 101. The video identification unit 305 then performs a comparison process between the caption information generated based on the video captured by the automatically controlled camera 102 and the relevance estimation information. For example, the switching determination unit 309 can analyze the caption information generated by the video identification unit 305. Specifically, the switching determination unit 309 can search the caption for words or sentences such as common nouns, proper nouns, or verbs that match the pair of the video splice type indicated by the relevance estimation information and the estimation basis text.

[0066] For example, when the relevance estimation information indicates element splicing, the switching condition may include detecting an object detected in the first video or an object related to the object detected in the second video. To this end, the switching determination unit 309 can search for a noun matching the estimated basis text from a caption generated based on the video captured by the automatic control camera 102. Here, the case where an object related to the object detected in the first video is detected in the second video includes a case where the object detected in the first video and the object detected in the second video share a common attribute. For example, the switching determination unit 309 may determine that element splicing is possible when the estimated basis text and a word included in the caption share a common attribute. For example, "guitar," "microphone," and "drum" are not the same word, but share the common attribute of being a musical instrument. With this configuration, it is possible to determine that switching is possible when an object or its type is common between the first video and the second video. Furthermore, the switching determination unit 309 may determine that element splicing is possible when the estimated basis text and the words included in the caption are in a conceptually inclusive relationship. Such a determination can be made, for example, by referring to dictionary data prepared in advance that indicates the attributes or inclusive relationships of words.

[0067] Furthermore, when the relevance estimation information indicates a transition, the switching condition may include that a motion detected in the first video or a motion related to the motion detected in the second video is detected. To this end, the switching determination unit 309 can search for a verb matching the inferred basis text from a caption generated based on the video captured by the automatic control camera 102. Here, the case where a motion related to the motion detected in the first video is detected in the second video includes a case where the motion detected in the first video and the motion detected in the second video have a common attribute. Furthermore, the switching determination unit 309 may determine that a transition is possible when the inferred basis text and a word included in the caption are conceptually inclusive. For example, when the caption includes a word indicating an action included in the word "play," such as "sing," the switching determination unit 309 may determine that a transition is possible.

[0068] In this example, the switching determination unit 309 determines whether the videos are connected by searching for text that matches the estimated basis text in the captions generated based on the video captured by the automatically controlled camera 102. On the other hand, the switching determination unit 309 may also perform word or sentence matching between the captions generated based on the video captured by the manually controlled camera 101 and the captions generated based on the video captured by the automatically controlled camera 102.

[0069] The caption with ID1 shown in FIG. 5(A) and the caption with ID1 shown in FIG. 5(B) share the common verb "perform." Therefore, the switching determination unit 309 can find the verb "perform" from the caption with ID1 shown in FIG. 5(B), which matches the pair "action transition" and "perform" included in the relevance estimation information shown in FIG. 10. In this case, the manually controlled camera 101 can be switched to the automatically controlled camera 102 using the action transition without creating a sense of incongruity. Therefore, the switching determination unit 309 can determine that switching from the manually controlled camera 101 to the automatically controlled camera 102 is possible. At this time, the switching determination unit 309 may record in RAM 202 a combination of the angle of view of the automatically controlled camera 102 when the video corresponding to the caption was captured and the caption information.

[0070] Note that the caption analysis and matching method may be a method other than the method of identifying parts of speech. For example, if the location information, such as "on the stage," matches, the switching determination unit 309 may determine that element transitions are possible.

[0071] On the other hand, when the relevance estimation information indicates an insert connection, the switching conditions may include the fact that the main subject is not detected in the second video. To this end, the switching determination unit 309 determines whether the caption information generated based on the video captured by the automatically controlled camera 102 indicates that the main subject is not present. When the main subject is not present in the video captured by the automatically controlled camera 102, the switching determination unit 309 can determine that an insert connection is possible.

[0072] In this embodiment, the video after switching due to insert splicing does not include a main subject. Therefore, if the relevance estimation information includes an insert splice and the region dividing unit 304 determines that the main subject is present in the second video, the caption generation and matching process for the second video captured at this angle of view can be skipped. Furthermore, the video after switching due to element splicing or action splicing often includes a main subject. Therefore, as shown in FIG. 10 , if the relevance estimation information does not include an insert splice and the region dividing unit 304 determines that the main subject is not present in the second video, the processing for the second video captured at this angle of view can be similarly skipped. As described above, the search process can be made more efficient by limiting the angle of view of the second video to be processed based on the video splicing type included in the relevance estimation information.

[0073] In this embodiment, the caption information may include captions for each of a plurality of regions. The switching determination unit 309 may determine that element transitions are possible when one of the captions for the first video and one of the captions for the second video contain a corresponding noun. The same applies to action transitions.

[0074] The processing performed by the manually controlled camera 101 in the shooting control method according to one embodiment will be described with reference to the flowchart in Fig. 6. The processing of S601 to S606 is repeated during shooting by the manually controlled camera 101. Note that the start and end of shooting by the manually controlled camera 101 and the automatically controlled camera 102 may be synchronized with the start of recording or switching of each camera, or with an instruction to start or end recording from the operation input device 104.

[0075] In S601, the instruction receiving unit 301 receives a control instruction from the operation input device 104. The instruction receiving unit 301 also outputs a signal for controlling the angle of view to the angle of view control unit 302. In S602, the angle of view control unit 302 changes the angle of view of the manually controlled camera 101 in accordance with the signal acquired from the instruction receiving unit 301. Note that if an instruction to change the angle of view is not input, the manually controlled camera 101 takes an image without changing the angle of view.

[0076] In S603, the image acquisition unit 303 acquires an image obtained by the shooting operation of the manually controlled camera 101 and outputs it to the region division unit 304. The image acquisition unit 303 may output the image to the region division unit 304 regardless of the operation of changing the angle of view. For example, the image acquisition unit 303 may continuously output the image to the region division unit 304 before and during the execution of the change in the angle of view. On the other hand, to reduce the processing load, the image acquisition unit 303 may output the image to the region division unit 304 after the change in the angle of view is completed. In this case, the image output to the region division unit 304 can be omitted while the manually controlled camera 101 is changing the angle of view.

[0077] In S604, the region dividing unit 304 performs segmentation processing on the video as described above. Furthermore, the region dividing unit 304 records the segmentation information in RAM 202. In S605, the video identification unit 305 generates a caption for each region based on the segmentation information as described above. Then, the video identification unit 305 records the caption information in RAM 202.

[0078] In S606, the result transmission unit 306 transmits the caption information to the automatically controlled camera 102. The result transmission unit 306 can transmit the caption information via the network 103. On the other hand, the method of communication between the manually controlled camera 101 and the automatically controlled camera 102 is not limited.

[0079] The process performed by the automatically controlled camera 102 in the shooting control method according to one embodiment will be described with reference to the flowchart in Fig. 7. The automatically controlled camera 102 can perform angle of view search according to this flowchart. The process shown in Fig. 7 can be started when the automatically controlled camera 102 receives caption information from the manually controlled camera 101.

[0080] In S701, the result receiving unit 307 receives the caption information transmitted from the manually controlled camera 101 and records it in the RAM 202.

[0081] In S702, the type determination unit 308 selects a video splice type based on the received caption information, and records the determination result in the RAM 202. The processing of S702 will be described later with reference to FIGS.

[0082] In S703, the angle-of-view control unit 302 changes the angle of view of the automatically controlled camera 102. The angle-of-view control unit 302 sequentially changes the angle of view of the automatically controlled camera 102 so that the entire captureable range is covered. In this way, the automatically controlled camera 102 can patrol the captureable range. To shorten the patrol time, the zoom value may be changed so that the angle of view is the widest before patrol.

[0083] In S704, the image acquisition unit 303 acquires the image captured by the automatic control camera 102 and outputs it to the area division unit 304. As in S603, the image acquisition unit 303 may output the image to the area division unit 304 regardless of the angle of view change operation. On the other hand, to reduce the processing load, the image acquisition unit 303 may output the image to the area division unit 304 after the angle of view change is complete.

[0084] In S705, the region dividing unit 304 performs segmentation processing on the video as described above, and records the segmentation information in the RAM 202. In S706, the video identification unit 305 generates a caption for each region based on the segmentation information as described above, and records the caption information in the RAM 202.

[0085] In S707, the switching determination unit 309 determines whether the video captured by the manually controlled camera 101 and the video captured by the automatically controlled camera 102 are connected, as described above. If the switching determination unit 309 determines that the videos are connected, the process proceeds to S708. If not, the process returns to S703. In this case, the angle of view of the automatically controlled camera 102 is changed, and it is determined again whether the videos are connected. Note that if the automatically controlled camera 102 receives new caption information from the manually controlled camera 101, the process may return to S701.

[0086] In S708, the switching determination unit 309 transmits a notification to the notification unit 310 indicating that switching from the manually controlled camera 101 to the automatically controlled camera 102 is possible. The notification unit 310 also transmits a notification to the operation input device 104 indicating that control of the angle of view of the automatically controlled camera 102 has been completed, so that the images of the automatically controlled camera 102 and the manually controlled camera 101 are related to each other.

[0087] An example of the processing of the type determination unit 308 in S702 will be described with reference to the flowchart of Fig. 9. Fig. 8 shows the logical configuration of the type determination unit 308. At the start of this processing, the relevance estimation information stored in the RAM 202 is initialized.

[0088] In S901 , the caption analysis unit 801 performs morphological analysis on one of the captions included in the caption information acquired from the manually controlled camera 101 .

[0089] In S902, the caption analysis unit 801 determines whether the caption includes a noun that describes an object and whether the caption includes a verb that describes an action or behavior. If the caption analysis unit 801 determines that the caption includes a noun that describes an object, the process proceeds to S903. If the caption analysis unit 801 determines that the caption includes a verb that describes an action or behavior, the process proceeds to S904.

[0090] In S903, the list recording unit 802 adds a pair of a video splice type indicating an element splice and a noun representing an object detected from the caption to the relevance estimation information held in the RAM 202. In S904, the list recording unit 802 adds a pair of a video splice type indicating an element splice and a verb representing an action or behavior detected from the caption to the relevance estimation information held in the RAM 202.

[0091] In S905, the caption analysis unit 801 determines whether analysis of all captions is complete. If there are captions remaining that have not been analyzed, the process returns to S902. Analysis of the captions for each area is performed through the loop of S902 to S905. Note that one caption may include multiple nouns, each representing an object. Also, one caption may include multiple verbs, each representing an action or movement, representing an object. Also, one caption may include both a noun representing an object and a verb representing an action or movement. In these cases, the list recording unit 802 can add multiple pairs of video splice types and estimation basis text to the relevance estimation information.

[0092] In S906, the main subject determination unit 803 refers to information included in the caption information that indicates whether or not a main subject is present in each region. If the presence of a main subject is indicated in any region, the processing in Fig. 9 ends. On the other hand, if the presence of a main subject is not indicated in any region, the processing proceeds to S907.

[0093] In S907, the list recording unit 802 adds a video splice type indicating an insert splice to the relevance estimation information held in the RAM 202.

[0094] With the above configuration, it is possible to determine whether switching from the manually controlled camera 101 to the automatically controlled camera 102 will cause an unnatural feeling when the camera is currently in the field of view of the automatically controlled camera 102. Furthermore, in the above-described embodiment, if the switching determination unit 309 determines that switching is possible, the change in the field of view of the automatically controlled camera 102 is stopped. That is, according to the above-described embodiment, the field of view of the automatically controlled camera 102 can be automatically searched for so that switching does not cause an unnatural feeling. However, searching for the field of view of the automatically controlled camera 102 is not essential. For example, a second manually controlled camera may be used instead of the automatically controlled camera 102. In this case, the camera operator can capture images while changing the field of view of the second manually controlled camera. In this case, the switching determination unit 309 can determine whether switching from the manually controlled camera 101 to the second manually controlled camera is possible. The operator can then confirm the notification from the notification unit 310 and perform the switch. This configuration also facilitates video switching with less unnatural feeling.

[0095] (Other Examples) The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit (e.g., ASIC) that realizes one or more functions.

[0096] The disclosure of this specification includes the following photography system, photography control device, photography control method, and program.

[0097] (Item 1) a first image capturing means for capturing a first image; a second image capturing means for capturing a second image while changing the angle of view; an identification means for identifying the content of the video; a type determination means for selecting an image transition type indicating a relationship between images when switching from the first image according to a result of identifying the content of the first image; a switching determination means for determining whether or not the identification result of the content of the second video satisfies a condition corresponding to the video connection type, and determining that the first video can be switched to the second video based on the determination result that the condition is satisfied; An imaging system comprising: (Item 2) Item 1. The imaging system according to item 1, wherein the identification means identifies the name or type of an object in the image. (Item 3) the type determination means, when a predetermined object is detected from the first image, selects element splicing as the image splicing type; The photographing system described in item 2, characterized in that the conditions corresponding to the element connection include that an object detected from the first image or an object related to the object is detected from the second image. (Item 4) 4. The imaging system according to any one of items 1 to 3, wherein the identification means identifies the motion of an object in the video. (Item 5) the type determination means, when a predetermined action is detected from the first video, selects an action transition as the video transition type; 5. The photographing system according to item 4, wherein the conditions corresponding to the action transition include that an action detected from the first video or an action related to the action is detected from the second video. (Item 6) 6. The photographing system according to any one of items 1 to 5, wherein the identification means identifies the presence or absence of a main subject in the video. (Item 7) the type determination means, when a main subject is not detected from the first video, selects an insert splice as the video splice type; 7. The photographing system according to item 6, wherein the conditions corresponding to the insert connection include that a main subject is not detected from the second video. (Item 8) 8. The photographing system according to any one of items 1 to 7, wherein the identification means performs processing to generate a caption that describes the image. (Item 9) Item 9. The photographing system according to item 8, wherein the type determination means determines the type of video splicing based on a word or sentence included in the caption that describes the first video. (Item 10) The photographing system described in any one of items 8 to 9, characterized in that the switching determination means determines whether the condition is met by comparing words or sentences included in the caption describing the first video with words or sentences included in the caption describing the second video. (Item 11) The photographing system described in any one of items 8 to 10, characterized in that the type determination means generates information indicating words or sentences that should be included in the caption describing the second video in order to satisfy the condition, based on words or sentences included in the caption describing the first video. (Item 12) 12. The photographing system according to any one of items 1 to 11, wherein the identification means divides the image into regions according to the position of the object in the image, and performs processing to identify the object for each region. (Item 13) Further, a control means for controlling the angle of view of the second imaging means is provided, The photographing system described in any one of items 1 to 12, characterized in that the control means stops changing the angle of view of the second photographing means in response to the switching determination means determining that the first image can be switched to the second image. (Item 14) Further, a control means for controlling the angle of view of the second imaging means is provided, 14. The photographing system according to any one of items 1 to 13, wherein the control means continuously changes the angle of view of the second photographing means before the switching determination means determines that the first image can be switched to the second image. (Item 15) 15. The photographing system according to any one of items 13 to 14, wherein the control means controls panning, tilting, and zooming of the second photographing means. (Item 16) 16. The photographing system according to any one of items 1 to 15, further comprising a notification means for notifying that the first image can be switched to the second image in response to the switching determination means determining that the first image can be switched to the second image. (Item 17) the photography system includes a first photography device and a second photography device; the first image capturing device comprises the first image capturing means and an identification means for identifying the content of the first image; The second image capturing device includes the second image capturing means, an identification means for identifying the content of the second image, the type determination means, and the switching determination means. 17. The imaging system according to any one of items 1 to 16, (Item 18) an identification means for identifying the content of the video; a type determination means for determining a type of video splicing in accordance with a result of identifying the content of the first video captured by the first image capturing means; a switching determination means for determining whether or not a result of identifying the content of a second video image captured by a second image capturing means while changing the angle of view satisfies a condition corresponding to the video stitching type, and determining that the first video image can be switched to the second video image based on a determination result that the condition is satisfied; and An imaging control device comprising: (Item 19) A photography control method performed by a photography system, comprising: identifying the content of the video; determining a type of video splicing according to a result of identifying the content of the first video captured by the first image capturing means; determining whether or not the result of identifying the content of a second video captured by a second image capturing means while changing the angle of view satisfies a condition corresponding to the video stitching type, and determining that the first video can be switched to the second video based on the determination result that the condition is satisfied; 10. A shooting control method comprising: (Item 20) 20. A program for causing a computer to execute the imaging control method according to item 19.

[0098] The present disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present disclosure. Accordingly, the following claims are appended to clarify the scope of the present disclosure. [Explanation of symbols]

[0099] 101: Manually controlled camera, 102: Automatically controlled camera, 301: Instruction receiving unit, 302: View angle control unit, 303: Video acquisition unit, 304: Area division unit, 305: Video identification unit, 306: Result transmission unit, 307: Result reception unit, 308: Type determination unit, 309: Switching determination unit, 310: Notification unit

Claims

1. a first image capturing means for capturing a first image; a second image capturing means for capturing a second image while changing the angle of view; an identification means for identifying the content of the video; a type determination means for selecting an image transition type indicating a relationship between images when switching from the first image according to a result of identifying the content of the first image; a switching determination means for determining whether or not the identification result of the content of the second video satisfies a condition corresponding to the video connection type, and determining that the first video can be switched to the second video based on the determination result that the condition is satisfied; and An imaging system comprising:

2. 2. The photographing system according to claim 1, wherein the identification means identifies the name or type of an object in the image.

3. the type determination means, when a predetermined object is detected from the first image, selects element splicing as the image splicing type; The photographing system according to claim 2 , wherein the condition corresponding to the element connection includes that an object detected from the first image or an object related to the object is detected from the second image.

4. 2. The imaging system according to claim 1, wherein said identifying means identifies the motion of an object in the video.

5. the type determination means, when a predetermined action is detected from the first video, selects an action transition as the video transition type; The photographing system according to claim 4 , wherein the condition corresponding to the action transition includes that an action detected in the first video or an action related to the action detected in the second video is detected.

6. 2. The photographing system according to claim 1, wherein said identifying means identifies the presence or absence of a main subject in the video.

7. the type determination means, when a main subject is not detected from the first video, selects an insert splice as the video splice type; 7. The photographing system according to claim 6, wherein the condition corresponding to the insert connection includes a condition that a main subject is not detected from the second video.

8. 2. The imaging system according to claim 1, wherein the identifying means performs processing to generate a caption that describes the image.

9. 9. The photographing system according to claim 8, wherein the type determining means determines the type of the video splice based on a word or sentence included in the caption that describes the first video.

10. The photographing system of claim 8, wherein the switching determination means determines whether the condition is met by comparing words or sentences contained in the caption describing the first video with words or sentences contained in the caption describing the second video.

11. The photographing system of claim 8, wherein the type determination means generates information indicating words or sentences that the caption describing the second video should contain in order to satisfy the condition, based on words or sentences contained in the caption describing the first video.

12. 2. The photographing system according to claim 1, wherein said identifying means divides the image into regions according to the position of the object in the image, and performs processing to identify the object in each region.

13. further comprising a control means for controlling the angle of view of the second imaging means; 2. The photographing system according to claim 1, wherein the control means stops changing the angle of view of the second photographing means in response to the switching determination means determining that the first image can be switched to the second image.

14. further comprising a control means for controlling the angle of view of the second imaging means; 2. The photographing system according to claim 1, wherein the control means continuously changes the angle of view of the second photographing means before the switching determination means determines that the first image can be switched to the second image.

15. 14. The photographing system according to claim 13, wherein the control means controls panning, tilting, and zooming of the second photographing means.

16. The photographing system according to claim 1, further comprising a notification means for notifying that the first image can be switched to the second image in response to the switching determination means determining that the first image can be switched to the second image.

17. the photography system includes a first photography device and a second photography device; the first image capturing device includes the first image capturing means and an identification means for identifying the content of the first image; The second image capturing device includes the second image capturing means, an identification means for identifying the content of the second image, the type determination means, and the switching determination means.

2. The imaging system according to claim 1, wherein:

18. an identification means for identifying the content of the video; a type determination means for determining a type of video splicing in accordance with a result of identifying the content of the first video captured by the first image capturing means; a switching determination means for determining whether or not a result of identifying the content of a second video image captured by a second image capturing means while changing the angle of view satisfies a condition corresponding to the video stitching type, and determining that the first video image can be switched to the second video image based on a determination result that the condition is satisfied; and An imaging control device comprising:

19. A photography control method performed by a photography system, comprising: identifying the content of the video; determining a type of video splicing in accordance with a result of identifying the content of the first video captured by the first image capturing means; determining whether or not the result of identifying the content of the second video captured by the second image capturing means while changing the angle of view satisfies a condition corresponding to the video splicing type, and determining that the first video can be switched to the second video based on the determination result that the condition is satisfied; 10. A shooting control method comprising:

20. A program for causing a computer to execute the photography control method according to claim 19.

Images