Video generation device, cutting area determination device, and program

Abstract

A video generation device (4) has: a cutting unit (42) that cuts a plurality of videos from inputted video on the basis of area information included in an inputted control signal; and a switching unit (43) that, when generating a second video from among the cut plurality of videos as a partial video after generating a first video from among the plurality of videos as a partial video, generates the partial video by selecting the second video from among the plurality of videos on the basis of an identifier for allocated video that represents operation information included in the control signal and switching from the first video to the second video on the basis of a switching type that represents operation information included in the control signal.

Description

Image generation device, cutout area determination device, and program

[0001] The present invention relates to an image generating device that generates a partial image by cutting out a partial area from an image, an image cropping area determining device that detects an object from an image and determines a cropping area from the object area, and a program.

[0002] Conventionally, techniques for extracting a partial image from an image have been known. For example, various systems have been proposed that extract and display a region of interest (ROI) from a high-definition image such as a panoramic image (see, for example, Patent Documents 1 and 2).

[0003] Furthermore, ultra-high definition video, such as 8K video, has become widespread in recent years, and its widespread use is expected. 8K video has 16 times the resolution of HD (high definition) video, so even if you crop an area equivalent to 1 / 16 of the original size, HD-equivalent image quality is guaranteed.

[0004] Therefore, a technique has been proposed in which a portion of a video captured by a fixed 8K camera is cut out as a region of interest, used as a virtual camera, and the virtual camera is switched to produce program video that appears as if it were shot with multiple cameras (see, for example, Patent Documents 3 to 5). Such program video production techniques are particularly useful for news programs, talk shows, programs with on-stage demonstrations, and the like, in which performers often appear side by side.

[0005] On the other hand, in actual production sites for program footage, a multi-camera method is used, in which, for example, multiple 2K cameras are set up in a studio, multiple cameramen operate to capture multiple images, and these images are then switched to produce the program footage.

[0006] Japanese Patent Publication No. 2005-192057 Japanese Patent No. 7011728 Japanese Patent No. 6432029 Japanese Patent Publication No. 2019-129466 Japanese Patent Publication No. 2013-017071

[0007] As described above, known video production systems for producing program videos include a method that uses switching between multiple cameras installed in a studio and a method that uses switching between virtual cameras.

[0008] The production method of a program using multiple cameras requires not only multiple cameramen but also an operator to operate a switcher. On the other hand, it is possible for one cameraman to operate multiple cameras remotely using a PTZ camera, etc., but in order to operate the cameras remotely, mechanical camera control is required.

[0009] However, when using mechanical camera control, it is necessary to reproduce the same movements as when a cameraman operates the camera, which is difficult to achieve. Also, a control system such as a high-performance robot arm is required, which increases the scale of the system.

[0010] On the other hand, the methods for producing program footage using virtual cameras described in the aforementioned Patent Documents 3 to 5 are more useful than conventional methods for producing program footage using multiple cameras, in that they do not require multiple cameramen.

[0011] However, the method of producing program footage using virtual cameras is not necessarily the most optimal method from the perspective of producing content such as television programs, and the operational load is not significantly different from that of conventional program footage production methods using multiple cameras.

[0012] For this reason, there has been a demand for a method that can improve the efficiency of program video production, reduce operational burdens, and enable program video production with a small number of people. For example, there has been a demand for a method that can enable new content production expressions, such as enabling unconventional camerawork without being bound by the physical constraints of cameras.

[0013] Furthermore, when cutting out a partial image from a video, the position and size of the cut-out area must often be determined manually, which places a heavy workload on the operator. Furthermore, when the cut-out area needs to be determined in a short amount of time, a certain level of operational skill is also required.

[0014] Thus, when determining an extraction region for extracting a partial image from an image, manual operation has the problem that the process cannot be completed in a short time.

[0015] For example, consider a case where the entire area of ​​a group of performers, the alto area, the mezzo-soprano area, and the soprano area are determined as cut-out areas from a fixed-shot video of a choral competition. In this case, all cut-out areas must be determined within the time (say, just under one minute) between when the performers line up on stage and when they begin singing. Manual operation has a speed limit. For this reason, it has been desired to automatically determine the cut-out areas.

[0016] Furthermore, for example, if a specific actor is tracked and a bust shot area is always determined as the cut-out area, manual operation would be difficult to handle sudden movements of the specific actor, and there is a possibility that errors will occur in the determined cut-out area. For this reason, there has been a demand for automatic tracking of actors using object tracking technology.

[0017] Therefore, the present invention has been made to solve the above-mentioned problems, and its object is to provide an image generation device, an extraction area determination device, and a program that can reduce the workload of an operator when extracting a portion of an input image to generate a partial image.

[0018] In order to solve the above problem, the video generation device of claim 1 is a video generation device that cuts out a portion of an input video as a cut-out area based on an input control signal to generate a partial video, wherein the control signal includes area information indicating the position and size of a plurality of cut-out areas cut out by the video generation device, as well as an identifier of an assigned video to be assigned to the partial video among a plurality of videos corresponding to the plurality of cut-out areas, and operation information indicating a switching type when operating to switch the allocation of the partial video, and is characterized in comprising: a cut-out unit that cuts out the plurality of videos from the input video based on the area information included in the control signal; and a switcher unit that, when generating a first video among the plurality of videos cut out by the cut-out unit as the partial video, and then generating a second video among the plurality of videos as the partial video, selects the second video among the plurality of videos based on the identifier of the assigned video indicated by the operation information included in the control signal, and switches from the first video to the second video based on the switching type indicated by the operation information included in the control signal to generate the partial video.

[0019] The video generating device of claim 2 is the video generating device of claim 1, characterized in that when the switching type is cut, the switcher unit generates the partial video so that, when a control signal including operation information for the switching type is input, the first video is instantly switched to the second video; when the switching type is wipe, the switcher unit generates the partial video so that, when a control signal including a wipe time together with operation information for the switching type is input, the wipe gradually switches from the first video to the second video over the wipe time; and when the switching type is dissolve, the switcher unit generates the partial video so that, when a control signal including a dissolve time together with operation information for the switching type is input, the dissolve gradually switches from the first video to the second video over the dissolve time.

[0020] Further, a clipping region determination device according to claim 3 detects an object from a predetermined video and determines a plurality of clipping regions for generating a partial video from an input video based on the object, the clipping region determination device comprising: an object detection unit that detects an object from the predetermined video; an object tracking unit that tracks the object detected by the object detection unit and obtains coordinate data of the object; and a clipping region determination unit that sets a plurality of blocks from the predetermined video, designates the number of objects for each of the plurality of blocks so that a substantially equal number of objects is obtained by dividing the number of objects S by the number of blocks N, where N is an integer equal to or greater than 2, and the number of objects tracked by the object tracking unit is S (S is an integer equal to or greater than 2), and sets the plurality of blocks from the predetermined video based on the number of objects designated for each of the plurality of blocks and the coordinate data of the objects obtained by the object tracking unit, and determines the plurality of blocks as a plurality of clipping regions.

[0021] A clipping region determination device according to claim 4 detects an object from a predetermined video and determines a plurality of clipping regions for generating a partial video from an input video based on the object, the clipping region determination device comprising an object detection unit that detects an object from the predetermined video, an object tracking unit that tracks the object detected by the object detection unit and obtains coordinate data of the object, and a plurality of blocks are set from the predetermined video, the number of the blocks is N (N is an integer of 2 or more), the number of the objects tracked by the object tracking unit is S (S is an integer of 2 or more), and the numbers of objects for the first to (N-1)th blocks of the plurality of blocks are C1 to C2 N-1 Let the number of objects be C1 to C N-1 is specified in advance by the operator, the number of objects for the Nth block, C N into the following formula: and the number of objects C1 to C2 designated for each of the plurality of blocks is N-1 , C N and a cut-out area determination unit that cuts out the plurality of blocks from the predetermined video based on coordinate data of the object detected by the object tracking unit and determines the plurality of blocks as the plurality of cut-out areas.

[0022] The clipping area determination device of claim 5 is the clipping area determination device of claim 3 or 4, characterized in that, when a mask area is set in advance at a predetermined position within the predetermined image by an operator, the object detection unit detects the object from an area of ​​the predetermined image excluding the mask area.

[0023] The clipping area determination device of claim 6 is the clipping area determination device of claim 3 or 4, characterized in that, when a boundary is set in advance at a predetermined position within the specified image by an operator, the object detection unit detects the object from an area of ​​the specified image excluding the predetermined area indicated by the boundary.

[0024] The clipping area determination device of claim 7 is the clipping area determination device of claim 3 or 4, characterized in that, when a forced object area is preset by an operator at a predetermined position within the predetermined video, the object detection unit detects the object from the predetermined video, assuming that the object exists in the forced object area.

[0025] The cut-out area determination device of claim 8 is the cut-out area determination device of claim 3 or 4, characterized in that the cut-out area determination unit changes the aspect ratio of one or more of the plurality of cut-out areas so as to include or exclude a predetermined area based on the object, based on the coordinate data of the object tracking unit.

[0026] Furthermore, a program according to a ninth aspect of the present invention is characterized in that it causes a computer to function as the image generating device according to the first or second aspect of the present invention.

[0027] A tenth aspect of the present invention provides a program for causing a computer to function as the clipping region determining device according to the third or fourth aspect of the present invention.

[0028] As described above, according to the present invention, it is possible to reduce the workload of an operator when a partial video is generated by cutting out a partial area from an input video.

[0029] 1 is a schematic diagram showing an example of the overall configuration of a video generation system. FIG. 2 is a flowchart showing an example of the processing of the video generation system. FIG. 3 is a block diagram showing an example of the configuration of a video generation device. FIG. 4 is a flowchart showing an example of the processing of the video generation device. FIG. 4 is a schematic diagram showing another example of the overall configuration of a video generation system. FIG. 5 is a block diagram showing an example of the configuration of a clipping area determination device. FIG. 6 is a flowchart showing an example of the processing of the clipping area determination device. FIG. 7 is a diagram showing an example of a clipping area in the case of automatic determination. FIG. 8 is a diagram showing an example of a clipping area in the case where the number of objects is specified by an operator. FIG. 9 is a diagram showing an example of a clipping area in the case where a mask area is specified. FIG. 10 is a diagram showing an example of a clipping area in the case where a boundary is specified. FIG. 11 is a diagram showing an example of a clipping area in the case where a boundary is specified. FIG. 12 is a diagram showing an example of a clipping area in the case where a forced object area is specified. FIG. 13 is a block diagram showing an example of the configuration of a clipping control device. FIG. 14 is a flowchart showing an example of the processing of an immediate transition operation. FIG. 15 is a flowchart showing an example of the processing of a cut transition operation. FIG. 16 is a flowchart showing an example of the processing of a panning transition operation. FIG. 17 is a diagram showing an example of an area selection operation. FIG. 18 is a diagram explaining an example of a transition curve. FIG. 19 is a diagram showing an example of implementation of an input unit and a display unit in setting mode. FIG. 19 is a diagram showing examples of input video, CH1 output video, etc. FIG. 19 is a diagram showing examples of PGM output video and NEXT output video. 1 is a diagram showing an example of a display on the display unit when a panning transition has started in a performance mode; FIG. 2 is a diagram showing an example of a display on the display unit when a panning transition is in progress in a performance mode; FIG. 3 is a diagram showing an example of a PGM output video by a panning transition operation; FIG. 4 is a block diagram showing an example of the configuration of a video generation device;

[0030] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. [Video Generation System] First, the video generation system will be described. Fig. 1 is a schematic diagram showing an example of the overall configuration of the video generation system. This video generation system 1 includes an object detection device 2, a cut-out control device 3, and a video generation device 4, as well as a high-definition camera 10, an object confirmation monitor 11, a cut-out monitor 12, a multi-screen monitor 13, and an output video monitor 14. In addition to these components, the video generation system 1 may also be configured to include a recorder / player 15.

[0031] The video generation system 1 receives high-definition video (e.g., 8K video) captured by a high-definition camera 10 such as an 8K camera as input video, converts the resolution of the input video to low-resolution video, detects objects from the low-resolution video, and calculates their coordinate data.

[0032] The high-definition video may be an overall video captured by one high-definition camera 10, or an overall video created by connecting overall videos captured by two or more predetermined number of high-definition cameras 10 by stitching or the like.

[0033] The video generation system 1 determines a plurality of crop regions based on coordinate data for each object, etc., and performs crop control to crop the plurality of crop regions from the input video. The video generation system 1 crops the plurality of crop regions from the input video in accordance with the crop control, generates a plurality of partial videos, and outputs them as PGM output video or the like (e.g., 2K video).

[0034] The object detection device 2 receives the low-resolution video image whose resolution has been reduced by the video generation device 4 , detects objects from the low-resolution video image, obtains their coordinate data, and outputs the coordinate data for each object to the extraction control device 3 .

[0035] The object detection device 2 displays the area of ​​the detected object, etc., on the object confirmation monitor 11 together with the low-resolution image.

[0036] The cut-out control device 3 receives the low-resolution video from the video generation device 4 and also receives the coordinate data for each object from the object detection device 2. Then, the cut-out control device 3 determines multiple regions of interest (virtual cameras) in the low-resolution video as multiple cut-out regions based on the region indicated by the coordinate data for each object and / or the region specified by the operator's operation.

[0037] The cut-out control device 3 generates a control signal including region information and operation information for the image generation device 4 to cut out an image from the high-definition image and generate a partial image based on a plurality of cut-out regions in accordance with an operation by an operator, and outputs the control signal to the image generation device 4. The cut-out control device 3 also displays the determined cut-out region on the cut-out monitor 12. Details of the region information and operation information will be described later.

[0038] The image generating device 4 receives high-definition images captured by the high-definition camera 10 , converts the resolution of the input images into low-resolution images, and outputs the low-resolution images to the object detecting device 2 and the extraction control device 3 .

[0039] The image generating device 4 receives a control signal including region information and operation information from the extraction control device 3, and extracts a plurality of images from the input image based on the control signal to generate a plurality of partial images. The image generating device 4 then outputs the plurality of partial images as PGM output image, NEXT output image, and CH1 to CH6 output images, which are main line images. Details of the image generating device 4 will be described later.

[0040] The video generating device 4 displays the low-resolution video obtained by reducing the resolution of the input video, the PGM output video, the NEXT output video, and the CH1 to CH6 output videos on the multi-screen monitor 13, and displays the PGM output video on the output video monitor 14.

[0041] When the video production system 1 is configured to include a recording / playback device 15, the recording / playback device 15 records high-definition video captured by the high-definition camera 10, plays back the recorded high-definition video in accordance with an operator's operation, and outputs the high-definition video to the video production device 4. As a result, the video is re-edited in the video production device 4 using the recorded high-definition video as a signal source.

[0042] Fig. 2 is a flowchart showing an example of processing performed by the image generation system 1 shown in Fig. 1. The image generation device 4 receives high-definition image captured by the high-definition camera 10 (step S201), reduces the resolution of the high-definition image, and generates low-resolution image (step S202).

[0043] The object detection device 2 detects the main objects to be captured from the entire low-resolution video generated by the video generation device 4, and obtains coordinate data for each object (step S203). The object coordinate data is data indicating the position and size of a rectangular area containing the object, and consists of, for example, the upper left coordinate and the lower right coordinate of the rectangular area.

[0044] The cut-out control device 3 determines a plurality of cut-out regions in the low-resolution video based on the coordinate data for each object, etc. (step S204). For example, the cut-out control device 3 determines the cut-out regions from the coordinate data for each of a predetermined number of objects so that each of the plurality of cut-out regions includes a predetermined number of objects. Also, for example, the cut-out control device 3 determines, for each of the plurality of cut-out regions, an area designated in advance in accordance with an operation by an operator as the cut-out region.

[0045] The cut-out control device 3 may extract an area that satisfies a preset cut-out condition (for example, a bust shot, two-shot, three-shot, etc. when the object is a person) from the low-resolution video based on the coordinate data for each object, and determine the extracted area as the cut-out area. The cut-out control device 3 may also fine-tune the determined cut-out area in accordance with an operator's operation. When the object moves, the cut-out control device 3 may cause the cut-out area to follow the moving object.

[0046] The cut-out control device 3 generates region information for the determined cut-out regions. In addition, the cut-out control device 3, in accordance with an operator's operation, allocates between a plurality of images corresponding to the determined cut-out regions (images a, b, ..., x cut out according to the region information in Fig. 3 described later) and PGM output images output by the image generation device 4, and generates operation information for switching (switching by cut, wipe, dissolve, etc.) between two images corresponding to the two determined cut-out regions.

[0047] When the operation information indicates that a wipe or dissolve switching operation is to be performed, the cut-out control device 3 generates operation information including the duration of the wipe or dissolve in addition to the allocation and switching information. The duration of the wipe or dissolve is the time from the start to the end of the wipe or dissolve. Then, the cut-out control device 3 generates a control signal including the region information and the operation information (step S205).

[0048] The region information indicates the position and size of each of a plurality of cut-out regions within the low-resolution video (high-definition video). The operation information indicates information regarding the allocation between the identifiers of the plurality of cut-out videos and the identifiers (of the assigned video) of the PGM output video or the like to be output (generated), and also indicates information regarding the switching type (cut, wipe, dissolve, etc.) for obtaining a predetermined video effect by switching the allocation of the PGM output video (if the switching type is wipe or dissolve, it also includes the duration of the wipe or dissolve).

[0049] The image generation device 4 extracts multiple images from the high-definition image based on the region information included in the control signal generated by the extraction control device 3. Then, based on the operation information included in the control signal, the image generation device 4 selects one image from the multiple images or combines two images to generate a PGM output image that reflects a predetermined visual effect, as well as a NEXT output image and CH1 to CH6 output images (step S206). The image generation device 4 outputs the PGM output image, etc. (step S207). Details of the processing by the image generation device 4 will be described later.

[0050] The image generating device 4 may generate only the PGM output image from the high-definition image based on the control signal and output this. The image generating device 4 may also generate one or more of the NEXT output image and the CH1 to CH6 output images from the high-definition image based on the control signal as needed and output these images.

[0051] (Video Generation Device 4) Next, the video generation device 4 shown in Fig. 1 will be described in detail. Fig. 3 is a block diagram showing an example of the configuration of the video generation device 4, and Fig. 4 is a flowchart showing an example of processing by the video generation device 4. This video generation device 4 includes a resolution conversion unit 41, a cropping unit 42, and a switcher unit 43.

[0052] The image generating device 4 receives high-definition image captured by the high-definition camera 10 (step S401). The resolution converting unit 41 reduces the resolution of the high-definition image to generate low-resolution image (step S402). The resolution converting unit 41 then outputs the low-resolution image to the object detecting device 2 and the extraction control device 3 (step S403).

[0053] The image generating device 4 receives a control signal including region information and operation information from the cutout control device 3 (step S404). The cutout unit 42 cuts out images a, b, ..., x corresponding to the multiple cutout regions from the high-definition image based on the region information (position and size of each cutout region) included in the control signal (step S405). The cutout unit 42 generates images a, b, ..., x that are 2K images by increasing or decreasing the resolution of the cutout images a, b, ..., x, and outputs these images a, b, ..., x to the switcher unit 43.

[0054] The switcher unit 43 receives the images a, b, ..., x from the cropping unit 42. Then, the switcher unit 43 generates the PGM output image, the NEXT output image, and the CH1 to CH6 output images by selecting one image from the images a, b, ..., x or by combining two images based on the operation information included in the control signal (allocation information between the image a, etc. and the PGM output image, etc., and information on the switching type of the PGM output image) (step S406).

[0055] The switcher unit 43 outputs the PGM output video, the NEXT output video, and the CH1 to CH6 output video (step S407).

[0056] For example, it is assumed that the switcher unit 43 assigns video a to the PGM output video and outputs the PGM output video of video a. In this state, it is assumed that the switcher unit 43 assigns video b to the PGM output video and inputs a control signal including operation information that sets the switching type of the PGM output video to cut.

[0057] In this case, when the switcher unit 43 receives the control signal, it generates a PGM output video of video b and outputs the PGM output video of video b so as to instantly switch from video a to video b. As a result, the PGM output video is switched from video a to video b at the timing when the control signal is received.

[0058] Also, assume that while the PGM output video of video a is being output, the switcher unit 43 assigns video b to the PGM output video, sets the switching type of the PGM output video to wipe, and inputs a control signal including operation information indicating the time.

[0059] In this case, when the switcher unit 43 receives the control signal, it generates a PGM output video so that the video a gradually switches to video b over the wipe time, and outputs the PGM output video. As a result, before the control signal is received, video a is output as the PGM output video, and a composite video of the wipe from video a to video b is output so that the video gradually switches from video a to video b during the period from the time the control signal is received until the wipe time has elapsed.

[0060] Also, assume that while the PGM output video of video a is being output, the switcher unit 43 assigns video b to the PGM output video, sets the switching type of the PGM output video to dissolve, and inputs a control signal including operation information indicating the time.

[0061] In this case, when the switcher unit 43 receives the control signal, it generates a PGM output video so that the video a gradually switches to video b over the dissolve time, and outputs the PGM output video. As a result, before the control signal is received, video a is output as the PGM output video, and during the period from the time the control signal is received until the dissolve time has elapsed, a composite video of the dissolve from video a to video b is output so that the video a gradually switches to video b.

[0062] Although the cropping unit 42 crops out images a, b, ..., x from the high-definition video based on the region information included in the control signal, it may also crop out only images corresponding to the PGM output video, etc. indicated by the allocation information in the control signal based on the operation information included in the control signal. For example, if the allocation information indicates that images a, b, and c are to be allocated to the PGM output video, NEXT output video, and CH1 output video, the cropping unit 42 will crop out only images a, b, and c from the high-definition video.

[0063] As a result, only the video corresponding to the PGM output video or the like output by the switcher unit 43 is cut out from the high-definition video, and no video is cut out that is not output from the switcher unit 43. This reduces the processing load on the cut-out unit 42. In this case, if the switcher unit 43 newly generates and outputs video d as CH2 output video, for example, the cut-out unit 42 starts cutting out video d from the high-definition video.

[0064] As described above, according to the video generation system 1, the object detection device 2 detects objects from low-resolution video obtained by converting the resolution of high-definition video, and obtains coordinate data for each object.

[0065] The cutout control device 3 determines a plurality of cutout regions in the low-resolution video based on the region indicated by the coordinate data for each object, and generates a control signal including region information and operation information.

[0066] Based on the control signal, the video generation device 4 cuts out multiple images from the high-definition video to generate multiple partial images, and outputs the multiple partial images as PGM output video, NEXT output video, and CH1 to CH6 output video, which are main line images.

[0067] This makes it possible to automatically determine the cut-out region based on the region of the object detected by the object detection device 2. Furthermore, the operator can perform an operation to determine the cut-out region by referring to the region of the object detected by the object detection device 2. This reduces the workload on the operator.

[0068] Furthermore, it is possible to centrally manage the process of determining the cut-out area in the cut-out control device 3 and the process of generating partial images cut out from high-definition video in the video production device 4. This allows one operator to perform the work of both cameraman and switcher. In other words, by using the video production system 1, it is possible to realize multi-camera production in which partial images are generated from high-definition video with a small number of operators.

[0069] Furthermore, the video production device 4 eliminates the need for a dedicated operator to operate the switcher, and as mentioned above, one operator can perform the work of both a cameraman and a switcher, thereby reducing the overall workload of the operator. Furthermore, because mechanical camera control is no longer necessary, program video can be produced on a small scale with a small number of people, without expanding the scale.

[0070] [Video Generation System / Other Examples] Next, another example of the video generation system will be described. Fig. 5 is a schematic diagram showing another example of the overall configuration of the video generation system. This video generation system 1' is configured to include a clipping region determination device 5, a clipping control device 6, and an video generation device 7.

[0071] 5, the high-definition camera 10, object confirmation monitor 11, cutout monitor 12, multi-screen monitor 13, and output video monitor 14 (as well as the recorder / player 15) shown in FIG. 1 are omitted.

[0072] Similar to the video generation system 1 shown in FIG. 1, the video generation system 1′ receives high-definition video (e.g., 8K video) as input video, converts the resolution of the input video to low-resolution video, detects objects from the low-resolution video, and determines multiple cut-out areas based on the coordinate data of the objects, etc.

[0073] The video generation system 1' performs clipping control to clip a plurality of clip regions from the input video, generates a plurality of partial videos from the input video in accordance with the clipping control, and outputs them as PGM output video or the like (for example, 2K video).

[0074] The clipping region determination device 5 inputs low-resolution video that has been reduced in resolution by the video generation device 7, detects objects from the low-resolution video, and calculates their coordinate data. The clipping region determination device 5 then determines a plurality of clipping regions in the low-resolution video based on the coordinate data for each object, and calculates coordinate data for each region (clip-out region). The clipping region determination device 5 outputs the coordinate data for each region to the clipping control device 6. Details of the clipping region determination device 5 will be described later.

[0075] The cut-out control device 6 receives the low-resolution image from the image generation device 7 and the coordinate data for each region from the cut-out region determination device 5. Then, in accordance with an operator's operation, the cut-out control device 6 generates a control signal including region information for the image generation device 7 to cut out an image from the high-definition image and generate a partial image based on the coordinate data for each region, and outputs the control signal to the image generation device 7. The cut-out control device 6 will be described in detail later.

[0076] The image generating device 7 receives a high-definition image, converts the resolution of the input image into a low-resolution image, and outputs the low-resolution image to the cutout region determining device 5 and the cutout control device 6 .

[0077] The image generating device 7 receives a control signal from the extraction control device 6, and extracts a plurality of images from the high-definition image based on the control signal to generate a plurality of partial images. The image generating device 7 then outputs the plurality of partial images as the PGM output image, NEXT output image, and CH1 to CH6 output images, which are main line images. The image generating device 7 will be described in detail later.

[0078] (Cut-out region determination device 5) Next, the cut-out region determination device 5 shown in Fig. 5 will be described in detail. Fig. 6 is a block diagram showing an example of the configuration of the cut-out region determination device 5, and Fig. 7 is a flowchart showing an example of the processing of the cut-out region determination device 5. This cut-out region determination device 5 includes a video input unit 51, an object detection unit 52, an object tracking unit 53, and a cut-out region determination unit 54.

[0079] As described above, the clipping region determination device 5 receives the low-resolution video from the video generation device 7, detects objects from the low-resolution video, and calculates their coordinate data. The clipping region determination device 5 then determines a plurality of clipping regions in the low-resolution video based on the coordinate data for each object, and outputs the coordinate data for each region to the clipping control device 6.

[0080] <Video Input Unit 51> The video input unit 51 receives a low-resolution video from the video generation device 7 and outputs the low-resolution video to the object detection unit 52 (step S701).

[0081] The video input unit 51 may input ultra-high definition fixed-shot video. However, capturing ultra-high definition video requires processing time. Therefore, it is desirable for the video input unit 51 to input video with a resolution reduced to about HD video, as long as this does not interfere with the object detection process by the object detection unit 52 in the subsequent stage. By inputting video with a reduced resolution, the capture processing time can be shortened, and the processing load on the cropping area determination device 5 can be reduced.

[0082] It is also desirable for the video input unit 51 to input video with as high a frame rate as possible. By inputting video with a high frame rate, the frame rate of the cutout region determined by the cutout region determination unit 54 at the subsequent stage also increases, shortening the update time of the coordinate data for each region output from the cutout region determination device 5. In other words, the update time of the control signal output from the cutout control device 6 at the subsequent stage also decreases, and as a result, smoothly changing PGM output video and the like can be output from the video generation device 7 at the subsequent stage.

[0083] <Object Detector 52> The object detector 52 receives the low-resolution video from the video input unit 51, detects preset objects such as people from the low-resolution video, and obtains coordinate data for each object (step S702).

[0084] Specifically, when the object detection unit 52 detects an object from the low-resolution video, it extracts a rectangular area containing the object from the low-resolution video and obtains coordinate data (e.g., the top left coordinate and the bottom right coordinate) to identify the rectangular area as the coordinate data of the object.

[0085] The object detection unit 52 outputs the low-resolution video and the coordinate data for each object to the object tracking unit 53 .

[0086] For example, the object detection unit 52 uses YOLO, which is a general-purpose object detection method, to detect learned objects such as people, dogs, cats, etc. Note that the object detection unit 52 may use any algorithm other than YOLO depending on the object to be detected.

[0087] YOLO has parameters for confidence and NMS (Non-Maximum Suppression). Confidence indicates the reliability of a detected object, and if the confidence of a detected object exceeds a predetermined threshold, the object is treated as having been successfully detected. By changing this threshold, the ease of detecting the target object can be adjusted. NMS is a process that excludes objects with low reliability when the bounding boxes of detected objects overlap. By changing the NMS parameters, the percentage of overlap that determines whether to exclude an object can be adjusted.

[0088] For details about the object detection method YOLO, please refer to the following literature: [Non-Patent Document] Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” In proc. Of the Computer Vision and Pattern Recognition (CVPR), arXiv:1506.02640, 2016

[0089] <Object Tracking Unit 53> The object tracking unit 53 receives the low-resolution video and the coordinate data for each object from the object detection unit 52. The object tracking unit 53 then assigns an ID (an index for identifying the object) to the object (step S703).

[0090] The object tracking unit 53 tracks the object of each ID as long as the object detection unit 52 successfully detects the object of that ID, and obtains coordinate data for that ID (step S704).The object tracking unit 53 then outputs the low-resolution video and the coordinate data for each ID to the cropping region determination unit 54.

[0091] By tracking objects for each ID, it is possible to prevent fluttering of the cut-out area determined by the cut-out area determination unit 54 at a later stage due to the ID switching for each frame, and it is possible to reduce errors in the cut-out area that is determined. Furthermore, the object tracking unit 53 works effectively even when a cut-out area is determined while tracking a specific moving object.

[0092] Any method such as template matching, advance learning, or online learning can be used to track the object.

[0093] <Cut-out region determination unit 54> The cut-out region determination unit 54 receives the low-resolution video and the coordinate data for each ID from the object tracking unit 53. Then, the cut-out region determination unit 54 determines a plurality of cut-out regions in the low-resolution video based on the coordinate data for each ID, etc. (step S705).

[0094] For example, the clipping region determination unit 54 automatically determines a plurality of clipping regions in accordance with the number of IDs based on the coordinate data for each ID.

[0095] Here, the low-resolution video input to the cropping region determination device 5 is assumed to be video of a choral competition in which a predetermined number of singers sing together, as shown in Fig. 8 (described later). It is assumed that four cropping regions will be determined from this video, each containing all the singers, an alto singer, a mezzo-soprano singer, and a soprano singer. The singers are assigned IDs by the object tracking unit 53.

[0096] The cut-out area determination unit 54 automatically determines, based on the coordinate data for each ID and the number of IDs, an area to accommodate all singers, an area to accommodate alto singers, an area to accommodate mezzo-soprano singers, and an area to accommodate soprano singers as cut-out areas for the low-resolution video.

[0097] Specifically, the cutout area determination unit 54 integrates the rectangular areas indicated by the coordinate data of all IDs for the area that will accommodate all the singers, and determines the integrated area as the cutout area.The cutout area determination unit 54 then obtains, for example, the minimum value (top left coordinate) and maximum value (bottom right coordinate) of the cutout area as the coordinate data of the area (cutout area).In this way, the cutout area that will accommodate all the singers is determined.

[0098] Here, when integrating the rectangular areas indicated by the coordinate data of all IDs, the cutout area determination unit 54 first determines the width and position of the upper horizontal part of the cutout area to be determined so that it includes the rectangular areas of all IDs.

[0099] The cutout region determination unit 54 then sets preset height data (height data corresponding to the width of the top) as the heights of the left and right portions of the cutout region to be determined. The cutout region determination unit 54 also determines the position of the left portion of the cutout region so that it includes the left edge of the leftmost rectangular region among all the rectangular regions of all the IDs, and determines the position of the right portion of the cutout region so that it includes the right edge of the rightmost rectangular region among all the rectangular regions of all the IDs.

[0100] Here, the height data corresponding to the width of the upper part is set in advance so that the cutout area has a ratio of, for example, 16:9 (16 horizontally and 9 vertically). The cutout area determination unit 54 may calculate the heights of the left and right parts from the width of the upper part each time so that the cutout area has a ratio of, for example, 16:9.

[0101] In this way, the cut-out area determination unit 54 determines the cut-out area that accommodates all singers by calculating the width and position of the top of the cut-out area, as well as the height and position of the left and right parts of the cut-out area.

[0102] On the other hand, the extraction region determination unit 54 identifies IDs so that the number of singers is equal (or approximately equal) for all singers in three regions, namely, a region accommodating alto singers, a region accommodating mezzo-soprano singers, and a region accommodating soprano singers, and then extracts N blocks B1 to B2 (N is an integer of 2 or more, N=3 in this example) for the region accommodating all singers. N By setting N The cutout region determining unit 54 then determines the blocks B1 to B N The coordinate data of the area (cut-out area) is calculated for each of the above. The cut-out area is determined by calculating the width and position of its upper part using the same process as in the case of the area containing all the singers mentioned above.

[0103] As a result, a segmentation region containing alto singers, a segmentation region containing mezzo-soprano singers, and a segmentation region containing soprano singers are automatically determined, and in this case, each of the three segmentation regions contains an equal (or approximately equal) number of singers.

[0104] If the total number of singers is S (S is an integer greater than or equal to 2), then blocks B1 to B N The number of people is divided equally so that S / N is satisfied. If S is not divisible by N, the number of people is divided into blocks B1 to B2. N The total number of people is S, and blocks B1 to B N Blocks B1 to B2 are divided so that the number of people in each block is approximately equal. N The number of people will be allocated.

[0105] 8 shows an example of automatically determined cutout regions. In this example, four cutout regions, that is, an entire region 1301 and blocks 1302-1, 1302-2, and 1302-3, are determined from a video (low-resolution video) of a choral competition.

[0106] In this video of the choral competition, all performers (singers and conductor) except for the piano player are assigned IDs of 1 to 17, and an ID indicating the person being tracked is displayed directly above each performer. The object tracking unit 53 assigns IDs of 1 to 17 to the singers and conductor objects detected by the object detection unit 52, tracks the performers with these IDs (total number S=17), and obtains coordinate data for each ID.

[0107] The entire area 1301 contains all singers (IDs 1 to 8, 10 to 17) and the conductor (ID 9), while block 1302-1 contains singers (IDs 1 to 6). Block 1302-2 contains singers (IDs 7, 8, 10 to 12) and the conductor (ID 9), and block 1302-3 contains singers (IDs 13 to 17).

[0108] The cropping area determination unit 54 automatically determines four cropping areas, namely the entire area 1301 and blocks 1302-1, 1302-2, and 1302-3, based on the coordinate data for each ID. For blocks 1302-1, 1302-2, and 1302-3, six people are specified for block 1302-1, six people for block 1302-2, and five people for block 1302-3, so that the number of people is approximately equal with respect to the total number of all performers S=17. Then, for each of blocks 1302-1, 1302-2, and 1302-3, the IDs belonging to that block are identified.

[0109] 8, the clipping region determination unit 54 automatically determines multiple clipping regions in accordance with the number of IDs based on the coordinate data for each ID. Alternatively, for example, the clipping region determination unit 54 may determine multiple clipping regions in accordance with the number of people in each clipping region specified in advance by an operator based on the coordinate data for each ID. The number of people in the multiple clipping regions is not automatically allocated equally (or approximately equally) but is manually allocated by the operator.

[0110] Specifically, the cutout region determination unit 54 counts the number of all IDs and sets the count value as the total number of people S. Then, after determining the entire region as the cutout region, the cutout region determination unit 54 divides the entire region into N blocks B1 to B2. N When determining the cut-out region, the first to (N-1)th regions (blocks B1 to B N-1 ) Number of people C1 to C N-1 Then, the cutout region determining unit 54 determines the last N-th region (the determined block B N ) Number of people C N is specified by the following formula: n is the nth block B n Indicates the number of people specified for

[0111] That is, the cut-out region determination unit 54 determines the number of people C1 to C2 designated in advance by the operator for each of the first to (N-1)th regions. N-1 Identify the ID of blocks B1 to B N-1 Then, the cutout region determination unit 54 determines the number of people C for the Nth region using the above formula (1). N Specify the number of people C N Identify the ID of block B N is determined as the cutout region.

[0112] As a result, block B N Regarding the number of people, the total number of people S, which is the number of all detected people, is calculated by subtracting the number of people C1 to C2 N-1The remaining number of people C obtained by subtracting N When the total number of people S obtained by the detection changes, the block B N The number of people in block B will also change. N Even if the number of people in the block B is likely to fluctuate, N Since the correct number of people is specified for N can be stably determined as the cutout region.

[0113] 9 shows an example of cropping regions when the number of objects is specified by the operator. In this example, as in FIG. 8, four cropping regions, namely, the entire region 1301 and blocks 1302-1, 1302-2, and 1302-3, are determined from the video of a choral competition. In this video of the choral competition, as in FIG. 8, IDs 1 to 17 are assigned to all performers (singers and conductor) except for the piano player.

[0114] It is assumed that the operator pre-specifies four people for the first area (block to be determined 1302-1) and seven people for the second area (block to be determined 1302-3).

[0115] Overall area 1301 accommodates a total of 17 singers (IDs 1 to 8, 10 to 17) and a conductor (ID 9), with block 1302-1 accommodating four singers (IDs 1 to 4). Block 1302-2 accommodates six singers (IDs 5 to 8, 10) and a conductor (ID 9), and block 1302-3 accommodates seven singers (IDs 11 to 17).

[0116] The clipping region determination unit 54 counts the number of IDs and sets the count value as the total number of people S = 17, and determines the entire region 1301 based on the coordinate data for each ID. Then, for the first region, the clipping region determination unit 54 identifies the IDs of four people, which is the number of people in the pre-designated first region, in order from the left side of the entire region 1301, for example, and determines block 1302-1 as the clipping region. Also, for the second region, the clipping region determination unit 54 identifies the IDs of seven people, which is the number of people in the pre-designated second region, in order from the right side of the entire region 1301, and determines block 1302-3 as the clipping region.

[0117] The cutout region determination unit 54 specifies the number of people for the third region as 6 (=17-4-7) in equation (1), identifies the remaining IDs as the IDs of the number of people, and determines block 1302-2 as the cutout region.

[0118] Here, if the conductor (ID=9) overlaps with the singer (ID=10), the object detection unit 52 may not necessarily be able to detect the object of the singer (ID=10), and detection / non-detection of the singer (ID=10) may be repeated. The total number of people S is 17 when the singer (ID=10) is detected, and 16 when the singer (ID=10) is not detected.

[0119] The number of people and areas of blocks 1302-1 and 1302-3 do not change regardless of whether the singer (ID=10) is detected or not. The number of people in block 1302-2 is specified by the remaining number of people using the above formula (1).

[0120] As a result, the number of people in block 1302-2 is set to six (=17-4-7) when a singer (ID=10) is detected, and five (=16-4-7) when a singer (ID=10) is not detected. Therefore, the three blocks 1302-1 to 1302-3 can be reliably determined as extraction regions, regardless of whether a singer (ID=10) is detected or not.

[0121] Returning to FIGS. 6 and 7, after step S705, the crop region determination unit 54 converts the aspect ratio of each of the determined crop regions in the low-resolution video to 16:9 (step S706).

[0122] For example, the cropping region determination unit 54 converts the aspect ratio by expanding or contracting the cropping region (frame) in the low-resolution video in the vertical or horizontal direction (widening or narrowing the cropping frame). In this case, the cropping region determination unit 54 may convert the aspect ratio while maintaining the line with the smallest vertical or horizontal size in the cropping region before conversion.

[0123] Here, if the object is a person, when the cutout region is expanded in the vertical direction, an unnatural space may appear above the person's head. Furthermore, when the cutout region is reduced in the vertical direction, the person's head may be missing. To address this, it is desirable to convert the aspect ratio so as to include the main part of the object (the head if the object is a person). Conversely, the aspect ratio may be converted so as to exclude parts other than the main part of the object (for example, the area above the head or the area at the feet).

[0124] Therefore, when converting the aspect ratio of a cut-out area when the object is a person, the cut-out area determination unit 54 enlarges or reduces the cut-out area in the low-resolution video only downward, without enlarging or reducing it upward, thereby making it possible to cut out an area including a person in a natural form.

[0125] Furthermore, when converting the aspect ratio while maintaining the line of the minimum size, the clipping region determination unit 54 may add a certain enlarged region to the line. This makes it possible to realize aspect ratio conversion so that a clipping region that does not feel cramped can be obtained when an object exists near the edge of the clipping region before conversion.

[0126] It is desirable that the face of a person within the cutout area is not cut off by the edges of the screen. Therefore, the cutout area can be corrected using a face detection function. The rectangular coordinates of the face area detected within the screen are referenced, and if the edge of the cutout area touches any of the face areas, the width or height of the cutout area is enlarged or reduced to adjust it so that the edge of the cutout area does not touch the face area. This makes it possible to avoid a situation where the angle of view is such that the person's face is not cut off.

[0127] The cropping region determination unit 54 outputs the coordinate data for each region after the aspect ratio conversion to the cropping control device 6 via, for example, Ethernet (registered trademark) communication (step S707).

[0128] 10 is a diagram showing an example of a cropping region when a mask region is specified. In a video of a choral competition, there may be objects that you want to exclude, such as a piano player or conductor. If the position of the object is fixed, specifying a rectangular mask region 1303 will prevent the object from being detected.

[0129] Specifically, the object detection unit 52 sets an area in the low-resolution image excluding a mask area 1303 designated in advance by an operator, and detects an object such as a person from this area.

[0130] The mask area 1303 is specified at any position on the low-resolution video screen using a bounding box. Multiple mask areas 1303 may be specified. Furthermore, the mask areas 1303 can be freely added, moved, or deleted by the operator.

[0131] 10, a mask area 1303 is specified in the video of the choral competition where the conductor is present. The entire area 1301 is determined as the cut-out area. Furthermore, for blocks 1302-1, 1302-2, and 1302-3, six people are specified for block 1302-1, five people for block 1302-2, and five people for block 1302-3, so that the number of performers is approximately equal with respect to the total number of performers S=16, and these are determined as the cut-out areas.

[0132] By specifying this mask area 1303, the conductor is excluded from the object detection target in the object detection unit 52, and the mask area 1303 does not affect the cut-out area determination process performed by the cut-out area determination unit 54.

[0133] 11 shows an example of a cropped area when a boundary is specified. In a video of a choral competition, noise other than the object to be detected is likely to occur at the edges of the screen, such as the audience seats in front of the stage and the ceiling. By specifying a boundary 1304 for setting mask areas on the top, bottom, left, and right of the video, objects at the edges of the screen are prevented from being detected.

[0134] Specifically, the object detection unit 52 sets an area in the low-resolution image excluding a predetermined mask area indicated by a boundary 1304 designated in advance by an operator, and detects an object such as a person from that area.

[0135] The boundary 1304 is specified at any position on the low-resolution image screen by a vertical or horizontal boundary line. The boundary 1304 may also be specified by a diagonal line. Furthermore, the boundary 1304 can be freely added, moved, or deleted by an operator.

[0136] In the example shown in FIG. 11, by specifying a boundary 1304, the area to the left of the boundary 1304 (an area of ​​about 1 / 10 of the left edge of the screen) is masked, and the entire area 1301 and blocks 1302-1, 1302-2, and 1302-3 are determined as the cropped area.

[0137] As a result, the piano player existing on the left side of the boundary 1304 is excluded from the object detection target by the object detection unit 52, and is excluded from the tracking target by the object tracking unit 53. Therefore, the piano player is not included in the cut-out region determined by the cut-out region determination unit 54.

[0138] Fig. 12 is a diagram showing an example of a cutout region when a boundary is specified, similar to Fig. 11, and shows an example in which the boundary 1304 shown in Fig. 11 is moved to the right. As in the example shown in Fig. 11, by specifying the boundary 1304, the region to the left of the boundary 1304 is masked, and the entire region 1301 and blocks 1302-1, 1302-2, and 1302-3 are determined as the cutout region.

[0139] As a result, a performer existing in the area to the left of the boundary 1304 will be excluded from the object detection target in the object detection unit 52, and will also be excluded from the tracking target in the object tracking unit 53. Therefore, the cut-out area determined by the cut-out area determination unit 54 will not be located to the left of the boundary 1304, but will be located to the right of the boundary 1304.

[0140] 13 is a diagram showing an example of a cropping area when a forced object area is specified. For example, if people overlap, the number of objects may not be counted correctly. In such cases, the object is forced to be set to prevent errors in counting the number of objects.

[0141] Specifically, the object detection unit 52 treats a forced object area 1305 in the low-resolution image, which has been designated in advance by the operator, as an area in which an object exists, and detects the object from the low-resolution image including the forced object area 1305.

[0142] The forced object region 1305 is designated at any position on the low-resolution video screen by a bounding box. A plurality of forced object regions 1305 may be designated. Furthermore, the forced object region 1305 can be freely added, moved, or deleted by the operator.

[0143] In the example shown in Fig. 13, a mandatory object area 1305 is specified in a predetermined location in the upper left of the video of a choral competition, and the entire area 1301 and blocks 1302-1, 1302-2, and 1302-3 shown in Fig. 13 are determined as the cropping areas. The entire area 1301 and block 1302-1 include the mandatory object area 1305.

[0144] By specifying this forced object region 1305, the object in the forced object region 1305 becomes a target of object detection in the object detection unit 52, and also becomes a target of tracking in the object tracking unit 53. Therefore, the forced object region 1305 is taken into consideration when determining an extraction region by the extraction region determination unit 54.

[0145] In this way, according to the clipping area determination device 5 shown in FIG. 6, the clipping area determination unit 54 automatically determines multiple clipping areas in accordance with the number of IDs, based on the coordinate data for each ID of the detected object.

[0146] Specifically, the clipping region determination unit 54 integrates rectangular regions indicated by the coordinate data of all IDs and determines the integrated region as the clipping region. In addition, the clipping region determination unit 54 divides N blocks B1 to B2 into an area that can accommodate all the people, so that the number of people is equal (or approximately equal) for all IDs. N By setting N is determined as the cutout region.

[0147] This allows the cutout area to be determined automatically without manual intervention, reducing the operator's workload and speeding up the process of determining the cutout area, resulting in increased efficiency in the production of program footage and improved production speed and quality.

[0148] (Cutout control device 6) Next, the cutout control device 6 shown in Fig. 5 will be described in detail. Fig. 14 is a block diagram showing an example of the configuration of the cutout control device 6. This cutout control device 6 includes a control unit 61, an input unit 62, a drawing unit 63, and a display unit 64.

[0149] As described above, the cut-out control device 6 receives the low-resolution video from the image generation device 7 and the coordinate data for each region from the cut-out region determination device 5. Then, in accordance with the operator's operation, the cut-out control device 6 generates a control signal including region information for the image generation device 7 to cut out an image from the high-definition video (the cut-out region in the input video) and generate a partial video based on the coordinate data for each region, and outputs the control signal to the image generation device 7.

[0150] The control unit 61 receives cut-out information (described later) from the input unit 62, and generates a control signal including region information for the image generation device 7 to cut out an image from the cut-out region in the input image and generate a partial image based on the cut-out information. The control unit 61 then outputs the control signal to the image generation device 7 and the cut-out information to the drawing unit 63.

[0151] The control signal includes area information indicating the cut-out position and size of the partial image generated by the image generation device 7 .

[0152] The input unit 62 inputs the coordinate data for each region from the cropping region determination device 5. Furthermore, the input unit 62, in accordance with the operator's operation, specifies one or more preset regions (regions of interest) based on the input coordinate data for each region and / or the coordinate data of a manually set region, and specifies one region from the one or more preset regions as a next region (region to transition to).

[0153] In this case, the input unit 62 generates cut-out information including identifiers of channels corresponding to one or more preset regions and next regions, and the specified region, and outputs the cut-out information to the control unit 61 .

[0154] 5, the channels corresponding to the multiple preset areas are the channels for output video CH1, CH2, CH3, CH4, CH5, and CH6, and the channel corresponding to the next area is the channel for output video NEXT.

[0155] In accordance with the operator's operation, the input unit 62 specifies the type and timing of the transition when the cut-out area in the input video in the video generation device 7 transitions from one of one or more preset areas to the next area, generates cut-out information including an identifier indicating the channel of the PGM output video, the type of transition, timing, cut-out area, etc., and outputs the cut-out information to the control unit 61.

[0156] The cutout region here does not refer to a cutout region determined by the cutout region determination device 5 shown in FIG. 6, but refers to a region cut out from the input video in the video generation device 7 based on a control signal.

[0157] When the control unit 61 receives the cut-out information from the input unit 62, it sequentially generates control signals including dynamic area information whose cut-out position and size change for the channels of the PGM output video generated by the video generation device 7 in accordance with the change.

[0158] The drawing unit 63 inputs the low-resolution video from the video generation device 7 and inputs the clipping information from the control unit 61. The drawing unit 63 then draws each region, such as a preset region, indicated by the clipping information as a picture together with the input low-resolution video (hereinafter referred to as video). For example, the drawing unit 63 draws one or more preset regions and / or dynamic region information as a picture together with the video. The drawing unit 63 outputs the drawn picture to the display unit 64.

[0159] The drawing unit 63 may perform predetermined scaling on the area (one or more preset areas and dynamic area information) indicated by the clipping information input from the control unit 61 so that the size of the area corresponds to the size of the video. The drawing unit 63 draws a picture by superimposing the area whose size has been changed by scaling on the video. This allows the operator to check the settings of the preset area, next area, etc., and the transition state while referring to the video, improving operability.

[0160] The display unit 64 receives the drawing from the drawing unit 63 and displays it on the screen, thereby presenting the drawing to the operator.

[0161] The input unit 62 and the display unit 64 may be configured to function as a touch panel, which allows the operator to specify the preset area and the next area by directly touching the image displayed on the screen, thereby enabling more intuitive operation.

[0162] <Types of Transition> There are three types of transitions when the cut-out area of ​​the PGM output video transitions: immediate transition, cut transition, and panning transition, as shown below.

[0163] The input unit 62 specifies the type and timing of the transition shown below in accordance with the transition operation by the operator, and generates cut-out information. Then, the control unit 61 generates a control signal including area information indicating the cut-out position and size of the area for the channel of the PGM output video based on the cut-out information, and outputs the control signal to the video generation device 7. In other words, the control unit 61 generates and outputs a control signal corresponding to one of the type and timing of the transition in accordance with the transition operation shown below for the area of ​​the PGM output video generated by the video generation device 7, thereby updating the cut-out area in the video generation device 7.

[0164] The operations performed by the transition operations are as follows: (1) An area is set by an "immediate transition operation" (for example, pressing any button in the immediate transition operation button row (PGM button row) 1206 shown in FIG. 21 , which will be described later), and then the cut-out area of ​​the image generation device 7 is immediately updated. (2) A next area is set by a "cut transition operation" (for example, pressing any button in the next area button row (NEXT button row) 1202, which will be described later), and then the cut-out area of ​​the image generation device 7 is updated from the current cut-out area to the next area by pressing a button (for example, pressing the CUT button 1203, which will be described later). (3) A next area is set by a "panning transition operation" (for example, pressing any button in the next area button row 1202, which will be described later), and then the cut-out area of ​​the image generation device 7 is updated so as to move, expand, or shrink from time to time from the current cut-out area to the next area, triggered by pressing a button (for example, pressing any button in the PAN button row 1204, which will be described later). (4) By the "transition stop operation", pressing a button (for example, the stop button 1217 described later) during the momentary panning transition stops the momentary transition in the cut-out area of ​​the image generation device 7, and the cut-out area at the time of the stop is maintained.

[0165] <Region Setting Method> Next, the setting of a preset region and a next region using the input unit 62 will be described. (1) The "Preset Region Selection" operation selects one or more preset regions. (2) The "Arbitrary Region Selection" operation selects an arbitrary region using a button, mouse, touch panel, numerical input, or the like. (3) The "Previous Region Selection" operation selects the cropped region before the start of the transition of the "Cut Transition Operation" or "Panning Transition Operation" as the previous region. (4) The "Resize" operation (e.g., pressing any button in the resize button row 1209, described later) sets a region in which the size of the currently set region is changed to a predetermined size while maintaining the center coordinates of the region. (5) The "Centering" operation (e.g., pressing the centering button (center alignment button) 1210, described later) sets a region in which the center position of the currently set region is moved to a predetermined position (e.g., the center position of the video) while maintaining the size of the region. (6) The "File Load" operation (e.g., pressing the Region File Load button 1215, described later) sets a region recorded in a file. (7) By operating "remote designation" (for example, by pressing the reconnect button 1223 of the AI ​​cut-out communication port described later), the area indicated by the coordinate data for each area input from the cut-out area determination device 5 is set. For example, when the operator operates "remote designation," the input unit 62 inputs the coordinate data for each area from the cut-out area determination device 5 and, in accordance with the operator's operation, associates each of the multiple areas corresponding to the coordinate data for each area with either a preset area or a next area. In this way, the preset area and the next area are set by the data from the external cut-out area determination device 5.

[0166] Hereinafter, an identifier for identifying a plurality of preset areas will be referred to as a "preset channel."

[0167] The input unit 62 sets a preset area after specifying a preset channel in accordance with the operator's operation. (1) "Select Default Area" sets an area with a preset center position and size (e.g., the entire screen area, an area in the center of the screen with each side being 1 / 2 of the full screen, etc.). (2) "Copy Preset Area" sets an area with the same center coordinates and size as another preset area. (3) "Select Arbitrary Area" sets an area to be arbitrarily specified by operating a button, mouse, touch panel, numerical input, etc. (4) "Resize" sets an area by changing the size of the area currently set as the preset area while maintaining the center coordinates. (5) "Centering" sets an area by moving the center position of the area currently set as the preset area to a predetermined position (e.g., the center of the video image) while maintaining the size. (6) "Load File" sets an area recorded in a file. (7) "Remote Designation" sets an area notified by communication from an external device.

[0168] 21 is a diagram showing an example of the implementation of the input unit 62 and display unit 64 in the setting mode. The setting mode will be described later. In this example, part of the functions of the input unit 62 (in addition to the functions shown in FIG. 21, an input unit 62 using a mouse, touch panel, or keyboard is also provided) and the display unit 64 are implemented as an integrated unit.

[0169] The extraction control device 6 has two operation modes: a setting mode and a production mode, which will be described later. The behavior of the input unit 62 and the display unit 64 changes depending on the operation mode. In particular, in the example of Fig. 21, the display content of the viewer 1207, the behavior when the operator touches within the viewer 1207, and the behavior when operating the keyboard change.

[0170] The setting mode is an operation mode in which preset areas are set, and the performance mode is an operation mode in which the next area is set and transition operations are performed. Referring to Fig. 21, the operation mode is switched to the setting mode by pressing the setting mode button 1211. On the other hand, the operation mode is switched to the performance mode by pressing the performance mode button 1212.

[0171] The setting mode and the production mode may be alternately switched by pressing a predetermined key on the keyboard (for example, the TAB key or the ESC key).

[0172] <Preset Channel Designation> Next, a description will be given of designating a preset channel using the input unit 62. The input unit 62 designates a preset channel in accordance with an operation by the operator.

[0173] The preset button row (PSET button row) 1201 is a button row for specifying a preset channel. In this example, the input unit 62 selectively specifies one of preset channels 0 to 9. Note that the input unit 62 may switch the operation mode to a setting mode when any button in the preset button row 1201 is pressed.

[0174] <Specifying Preset Area> Next, a description will be given of specifying a preset area using the input unit 62. The input unit 62 specifies a preset area in accordance with an operation by the operator.

[0175] The next area button row 1202 is a button row for setting the next area by specifying a preset area. In response to an operator pressing any button in the next area button row 1202, the input unit 62 sets the preset area corresponding to the preset channel numbered on the button as the next area.

[0176] The input unit 62 may set the preset region corresponding to the preset channel of a specific number as the next region in accordance with the operation of pressing a specific key on the keyboard (for example, any of the numeric keys 0 to 9) in the setting mode instead of pressing the next region button column 1202. Furthermore, the input unit 62 may set the next region in accordance with the operation of touching or clicking a mouse on a preset region frame (1208-1 to 1208-3 or 1208-6 in this example) presented in the viewer 1207 in the setting mode. Furthermore, the input unit 62 may switch the operation mode to the production mode when any of the buttons in the next region button column 1202 is pressed.

[0177] This allows any rectangular area to be specified as the next area, so that any rectangular area can be set as the destination cut-out area regardless of the area set as the preset area, thereby increasing convenience.

[0178] <Resizing> Next, a description will be given of the process of resizing the preset area or the next area by the input unit 62. The input unit 62 resizes the preset area or the next area in accordance with the operation of the operator.

[0179] When the operating mode is the setting mode, the input unit 62 changes the size of the preset area of ​​the selected preset channel to a specified size while maintaining the current center coordinates, in accordance with the operator's operation of pressing any button in the resize button row (area edit button row) 1209.

[0180] When the operation mode is the production mode, the input unit 62 changes the size of the next area to a predetermined size while maintaining the current center coordinates in response to the operator pressing any button in the resize button row 1209 .

[0181] In addition, the input unit 62 may perform the same processing as when any button in the resize button column 1209 is pressed, in response to a predetermined key operation on the keyboard (for example, simultaneous pressing of the Ctrl key and the f key, simultaneous pressing of the Ctrl key and the h key, simultaneous pressing of the Ctrl key and the q key, or simultaneous pressing of the Ctrl key and the m key).

[0182] <Centering> Next, a description will be given of the processing of centering the preset area or the next area by the input unit 62. The input unit 62 centers the preset area or the next area in accordance with the operation of the operator.

[0183] When the operating mode is the setting mode, the input unit 62, in response to the operator pressing the centering button 1210, changes the center coordinates of the preset area of ​​the selected preset channel to the center coordinates of the low-resolution image input by the extraction control device 6, while maintaining the current size of the preset area.

[0184] When the operating mode is the production mode, the input unit 62 changes the center coordinates of the next area to the center coordinates of the image input by the extraction control device 6 while maintaining the current size of the next area in accordance with the operator's operation of pressing the centering button 1210.

[0185] The input unit 62 may perform the same processing as when the centering button 1210 is pressed, in response to a predetermined key operation on the keyboard (for example, pressing the Ctrl key and the c key simultaneously).

[0186] <Deleting Preset Area> Next, a description will be given of the process of deleting a preset area by the input unit 62. The input unit 62 deletes a preset area in accordance with an operation by the operator.

[0187] When the operation mode is the setting mode, the input unit 62 deletes the preset area information of the selected preset channel in response to the operator pressing the area deletion button 1213 .

[0188] The input unit 62 may perform the same processing as when the area delete button 1213 is pressed, in response to a predetermined key operation on the keyboard (for example, pressing the Delete key).

[0189] Furthermore, when the operation mode is the setting mode, the input unit 62 deletes the information of the preset areas of all preset channels in response to the operator pressing the delete all areas button 1214 .

[0190] The input unit 62 may perform the same processing as when the Delete All Areas button 1214 is pressed, in response to a predetermined key operation on the keyboard (for example, pressing the Shift key and the Delete key simultaneously).

[0191] <File Reading and File Saving> Next, a description will be given of file reading and file saving processes performed by the input unit 62. The input unit 62 reads a file and sets a preset area in accordance with an operation by the operator. The input unit 62 also saves the set preset area in a file in accordance with an operation by the operator.

[0192] In response to the operator pressing the area file read button 1215, the input unit 62 reads the area information recorded in the file and sets the area indicated by the area information as the preset area of ​​all preset channels or the preset area of ​​the preset channel specified by the operator's operation.

[0193] In response to the operator pressing the area file save button 1216, the input unit 62 saves, in a file, information on the preset areas of all the preset channels or information on the preset area of ​​the preset channel designated by the operator's operation.

[0194] <Immediate Transition Operation> Next, a description will be given of the processing of the immediate transition operation by the input unit 62 and the control unit 61. The input unit 62 generates cut-out information including the type and timing of the transition indicating the immediate transition in accordance with the immediate transition operation by the operator. The control unit 61 generates a control signal including area information indicating the cut-out position and size of the area of ​​the partial video, which is the PGM output video cut out by the video generation device 7, based on the cut-out information, and outputs the control signal to the video generation device 7.

[0195] Fig. 15 is a flowchart illustrating an example of processing for an immediate transition operation. The immediate transition operation button column 1206 shown in Fig. 21 is a button column for executing an immediate transition operation. The input unit 62 determines whether or not the operator has pressed any button in the immediate transition operation button column 1206 (step S1501).

[0196] If it is determined in step S1501 that any button in the immediate transition operation button column 1206 has not been pressed (step S1501: N), the input unit 62 waits until the button is pressed.

[0197] If the input unit 62 determines in step S1501 that any button in the immediate transition operation button column 1206 has been pressed (step S1501: Y), the input unit 62 uses the button press as a trigger to generate cut-out information including the identifier of the channel of the PGM output video (identifier of the PGM channel), the type of transition indicating an immediate transition and the timing indicating immediate, and the preset area of ​​the preset channel numbered with the button (position and size of the preset area) (step S1502).Then, the input unit 62 outputs the cut-out information to the control unit 61.

[0198] The control unit 61 receives the cutout information from the input unit 62 and sets the position and size of the preset region to the cutout position and size of the region in the PGM channel based on the cutout information. The control unit 61 then generates a control signal including region information indicating the cutout position and size of the region in the PGM channel and outputs the control signal to the video generation device 7 (step S1503).

[0199] As a result, in the video generating device 7, an area having the cut-out position and size indicated by the area information of the control signal is cut out as a cut-out area from the input video, and the cut-out partial video is output as a PGM output video.

[0200] Therefore, when any button in the immediate transition operation button row 1206 is pressed in the cut-out control device 6, the image generating device 7 immediately reflects the preset area corresponding to that button as the cut-out area, and outputs a partial image of that cut-out area as a PGM output image.

[0201] The input unit 62 may be configured to switch the operation mode to the production mode when any button in the immediate transition operation button row 1206 is pressed.

[0202] <Cut Transition Operation> Next, a description will be given of the processing of cut transition operations by the input unit 62 and the control unit 61. The input unit 62 generates cut-out information including the type and timing of transition indicating the cut transition in accordance with the cut transition operation by the operator. The control unit 61 generates a control signal including area information indicating the cut-out position and size of the area of ​​the partial video, which is the PGM output video, cut out by the video production device 7, based on the cut-out information, and outputs the control signal to the video production device 7.

[0203] Fig. 16 is a flowchart for explaining an example of a process for a cut transition operation. The cut button 1203 shown in Fig. 21 is a button for executing a cut transition operation. The input unit 62 determines whether or not the operator has pressed the cut button 1203 (step S1601).

[0204] If it is determined in step S1601 that the cut button 1203 has not been pressed (step S1601: N), the input unit 62 waits until the cut button 1203 is pressed.

[0205] If it is determined in step S1601 that the cut button 1203 has been pressed (step S1601: Y), the input unit 62 generates, using the button press as a trigger, cut-out information including a PGM channel identifier, a transition type indicating a cut transition and a timing indicating immediacy, and a next region (position and size of the next region) (step S1602).The input unit 62 then outputs the cut-out information to the control unit 61.

[0206] The control unit 61 receives the cut-out information from the input unit 62 and sets the position and size of the next region to the cut-out position and size of the region in the PGM channel based on the cut-out information. The control unit 61 then generates a control signal including region information indicating the cut-out position and size of the region in the PGM channel, and outputs the control signal to the video generation device 7 (step S1603).

[0207] As a result, in the video generating device 7, an area having the cut-out position and size indicated by the area information of the control signal is cut out as a cut-out area from the input video, and the cut-out partial video is output as a PGM output video.

[0208] Therefore, when the cut button 1203 is pressed in the cutout control device 6, the next region is immediately reflected as a cutout region by the image generation device 7, and a partial image of the cutout region is output as a PGM output image.

[0209] The input unit 62 may switch the operation mode to the live mode when the cut button 1203 is pressed. The input unit 62 may also perform the same processing as when the cut button 1203 is pressed, in response to a predetermined key operation on the keyboard (for example, pressing the c key).

[0210] <Panning Transition Operation> Next, the processing of the panning transition operation by the input unit 62 and the control unit 61 will be described. The input unit 62 generates cut-out information including the type and timing of the transition indicating the panning transition, as well as the cut-out area, in accordance with the panning transition operation by the operator. The control unit 61 generates a control signal including area information indicating the cut-out position and size of the area of ​​the partial image that is the PGM output image cut out by the image generation device 7 based on the cut-out information, and outputs the control signal to the image generation device 7. In this case, the cut-out information and the control signal change from moment to moment from the start to the end of the panning transition. The cut-out area at the start of the panning transition is the area of ​​the current PGM output image, and the cut-out area at the end of the panning transition is the next area.

[0211] Fig. 17 is a flowchart illustrating an example of a process for a panning transition operation. The pan button row 1204 shown in Fig. 21 is a button row for executing a panning transition operation. The input unit 62 determines whether or not the operator has pressed any button in the pan button row 1204 (step S1701).

[0212] If it is determined in step S1701 that no button in the pan button column 1204 has been pressed (step S1701: N), the input unit 62 waits until the button is pressed.

[0213] If the input unit 62 determines in step S1701 that any button in the pan button row 1204 has been pressed (step S1701: Y), the input unit 62 uses the button press as a trigger to generate cut-out information including a PGM channel identifier, the type and timing of a panning transition, and an area (position and size of the area) that automatically moves, expands, or contracts from the current cut-out area (area of ​​the PGM output video) to the next area from time to time (step S1702).The input unit 62 then outputs the cut-out information to the control unit 61.

[0214] A time required for the transition (5 seconds, 10 seconds, 15 seconds, or 20 seconds in the example of FIG. 21 ) is set for each button in the pan button row 1204, and the time from the start to the end of the transition from the current cropped area to the next area is determined depending on which button is pressed. The time required for the transition may also be specified by a transition time designation slider 1205. The pan slider designation buttons included in the pan button row 1204 determine the cropped area based on the time designated by the transition time designation slider 1205 and the transition curve designated by the transition curve setting slider 1222, and perform a panning transition.

[0215] The control unit 61 receives the cut-out information from the input unit 62 and sets the position and size of the transitioning region to the cut-out position and size of the region in the PGM channel based on the cut-out information. The control unit 61 then generates a control signal including region information indicating the cut-out position and size of the region in the PGM channel, and outputs the control signal to the video generation device 7 (step S1703).

[0216] As a result, in the video generating device 7, an area having the cut-out position and size indicated by the area information of the control signal is cut out as a cut-out area from the input video, and the cut-out partial video is output as a PGM output video.

[0217] Therefore, when any button in the pan button row 1204 is pressed in the cut-out control device 6, the image generation device 7 reflects the area that transitions over a time period corresponding to the button as a cut-out area, and outputs a partial image of the cut-out area as a PGM output image. In other words, the PGM output image transitions from the area of ​​the PGM output image at the time the button was pressed to the next area by constantly moving, enlarging, or shrinking, and when the panning transition is complete, the image of the next area is output as the PGM output image, making it possible to execute a panning transition with an effect that is just as if it were achieved by panning and zooming a camera.

[0218] The input unit 62 may switch the operation mode to the performance mode when any button in the pan button row 1204 is pressed. The input unit 62 may also perform the same processing as when any button in the pan button row 1204 is pressed, in response to a predetermined key operation on the keyboard (for example, pressing any of the f key, m key, s key, v key, or space key).

[0219] 24 is a diagram showing an example of the display on the display unit 64 when a panning transition starts in the live mode. This viewer 1207 shows the display content when any button in the pan button row 1204 is pressed, when the preset region of the preset channel number 1 is set as the cut-out region (region of the cut-out region frame 1220) and the preset region of the preset channel number 3 is set as the next region (region of the next region frame 1221).

[0220] At the time when the transition starts when the button is pressed, the image of the cutout area in the cutout area frame 1220 is output from the image generating device 7 as a PGM output image.

[0221] 25 is a diagram showing an example of the display on the display unit 64 when a panning transition is in progress in the production mode, showing the panning transition from preset area number 1 to preset area number 3. This viewer 1207 shows display content including the cut-out area (area of ​​cut-out area frame 1220) when the panning transition is in progress. When the panning transition is in progress, the image of the cut-out area of ​​cut-out area frame 1220 is output from the image generation device 7 as a PGM output image.

[0222] <Panning transition cancellation> Next, a description will be given of the processing of canceling the panning transition by the input unit 62 and the control unit 61. When the operator presses the stop button 1217 for canceling the panning transition while the panning transition operation is being performed, the input unit 62 maintains the clipping information generated at the time of the panning transition cancellation operation and outputs the clipping information to the control unit 61. The control unit 61 maintains the control signal generated based on the clipping information at the time of the panning transition cancellation operation and outputs the control signal to the image generation device 7.

[0223] As a result, the transition is stopped, and the PGM output video cut out by the video generation device 7 is maintained in the state it was in when the button was pressed.

[0224] The input unit 62 may perform the same processing as when the stop button 1217 is pressed, in response to a predetermined key operation on the keyboard (for example, pressing the x key).

[0225] <Swapping of Next Region and Previous Region at Each Transition> Here, the cut-out region immediately before the start of the above-described cut transition or panning transition is referred to as the "previous region." Immediately after the end of a cut transition or panning transition, the input unit 62 may set the previous region as the new next region (setting the position and size of the previous region as the position and size of the next region as they are). In other words, the previous region, which is the region immediately before the start of the cut transition or panning transition, is set as the next region immediately after the end of a cut transition or panning transition.

[0226] <Arbitrary Area Selection> Next, a process will be described in which the input unit 62 sets the position and size of an area in accordance with a predetermined mouse operation or touch panel operation (hereinafter referred to as "area selection operation") on the viewer 1207 by the operator.

[0227] When the operation mode is the setting mode, the input unit 62 sets (and modifies) the preset region of the currently specified preset channel in accordance with the region selection operation. When the operation mode is the production mode, the input unit 62 sets (and modifies) the next region in accordance with the region selection operation.

[0228] 18 and 19 are diagrams showing examples of an area selection operation. (Setting mode: arbitrary area selection: new diagonal designation operation) The processing of a new diagonal designation operation when the operation mode is the setting mode will be described with reference to FIG. 18(1). If the start position of a drag (swipe) with the mouse is away from any area frame (e.g., 1208-6 in the example of FIG. 18(1)) of the preset area frames (e.g., 1208-1 to 1208-3 or 1208-6 in FIG. 21) by a predetermined distance or more, the input unit 62 determines that the mouse cursor is in a position unrelated to the preset area frame and is in an unselected state.

[0229] The input unit 62 sets a preset area of ​​the specified preset channel by using a rectangular area whose diagonal corners are the drag (swipe) start position and the drag (swipe) end position, and sets a preset area frame. The input unit 62 deletes the previous preset area frame of the specified preset channel.

[0230] In this case, the input unit 62 may expand the preset area in the vertical or horizontal direction to a minimum extent so that the set preset area has a predetermined aspect ratio (for example, a horizontal:vertical ratio of 16:9). That is, the input unit 62 corrects the size of the rectangular area by expanding it in either the vertical or horizontal direction so that it includes the original set preset area, and sets a new preset area with a predetermined aspect ratio. The same applies to the operations in Figures 18 (2) and (3) and Figures 19 (4) and (5) described below.

[0231] This allows the image generating device 7 to cut out the image while maintaining the aspect ratio of the image.

[0232] (Setting mode: Arbitrary area selection: New 100% magnification designation operation) The processing of the new 100% magnification designation operation when the operation mode is the setting mode will be described with reference to Fig. 18(2). If the mouse click position is away from any area frame (e.g., 1208-6 in the example of Fig. 18(2)) of the preset area frames (e.g., 1208-1 to 1208-3 or 1208-6 in Fig. 21) by a predetermined distance or more, the input unit 62 determines that the mouse cursor is in a position unrelated to the preset area frame and is in an unselected state.

[0233] The input unit 62 sets a preset region of the specified preset channel as a high-resolution rectangular region centered on the mouse cursor position, which is the click position, and sets a preset region frame. The input unit 62 deletes the previous preset region frame of the specified preset channel.

[0234] (Setting mode: arbitrary area selection: parallel movement operation) The processing of the parallel movement operation when the operation mode is the setting mode will be described with reference to Fig. 18(3). When the start position of a drag (swipe) with the mouse is within a predetermined distance (excluding within a predetermined distance from both end points or near the center of the side) from one side of any of the preset area frames (e.g., 1208-1 to 1208-3 or 1208-6 in Fig. 21), the input unit 62 determines that the mouse cursor is near the side and is in a frame selection state.

[0235] The input unit 62 sets the preset channel to the number of the preset area frame, and moves the preset area of ​​the preset channel in parallel using a vector from the drag (swipe) start position to the drag (swipe) end position to set the preset area frame. In this case, the size of the preset area is fixed.

[0236] The input unit 62 may be configured to perform translational movement by operating keys on a keyboard (for example, cursor keys) instead of (or in addition to) dragging with a mouse.

[0237] (Setting mode: Arbitrary area selection: Zoom in / out operation) Processing of zoom in / out operation (opposite side fixed) when the operation mode is the setting mode will be described with reference to Fig. 19(4). When the start position of a drag (swipe) with the mouse is within a predetermined distance from a predetermined position near the center of one side of one of the area frames (1208-6 in the example of Fig. 19(4)) of the preset area frames (for example, 1208-1 to 1208-3 or 1208-6 in Fig. 21), the input unit 62 determines that the mouse cursor is near the midpoint of the side and is in a side selection state.

[0238] The input unit 62 sets the preset channel to the number of the preset area frame, and sets the preset area of ​​the preset channel as the smallest rectangular area that includes the center of the opposite side of the one side and the drag (swipe) end position and maintains the aspect ratio, thereby providing a preset area frame. In this case, the opposite side of the preset area is fixed.

[0239] 19(5), the process of zooming in and out (fixing the diagonal vertices) when the operation mode is the setting mode will be described. When the start position of a drag (swipe) with the mouse is within a predetermined distance from one vertex of one of the preset area frames (e.g., 1208-1 to 1208-3 or 1208-6 in FIG. 21 ), the input unit 62 determines that the mouse cursor is near the vertex and is in a vertex selection state.

[0240] The input unit 62 sets the preset channel to the number of the preset area frame, and sets the preset area of ​​the preset channel as the smallest rectangular area that includes the diagonal vertex position of the one vertex and the drag (swipe) end position and maintains the aspect ratio, thereby setting the preset area frame. In this case, the diagonal vertex of the preset area is fixed.

[0241] In addition, the input unit 62 may be configured to change (enlarge or reduce) the size of the area by operating keys on the keyboard (e.g., the PageUp key and PageDown key) instead of (or in addition to) dragging with the mouse (however, the center position of the area is fixed).

[0242] (Main mode: arbitrary area selection: new diagonal designation operation) The processing of a new diagonal designation operation when the operation mode is the main mode will be described with reference to Fig. 18 (1). When the start position of a drag (swipe) with the mouse is a predetermined distance or more away from the next area frame (for example, 1221 in Fig. 24 described later), the input unit 62 determines that the mouse cursor is in a position unrelated to the next area frame and is in an unselected state.

[0243] The input unit 62 sets a next region as a rectangular region having diagonal corners defined by the drag (swipe) start position and the drag (swipe) end position, and sets a next region frame. The input unit 62 deletes the previous next region frame.

[0244] In this case, the input unit 62 may expand the set next region in the vertical or horizontal direction to a minimum extent so that the set next region has a predetermined aspect ratio (for example, 16:9). That is, the input unit 62 expands the set next region in either the vertical or horizontal direction so that it includes the set original next region, thereby setting a new next region with a predetermined aspect ratio. The same applies to the operations in FIGS. 18(2) and 18(3) and 19(4) and 19(5), which will be described later.

[0245] (Production mode: arbitrary area selection: new same-size designation operation) The processing of the new same-size designation operation when the operation mode is production mode will be described with reference to Fig. 18 (2). If the mouse click position is a predetermined distance or more away from the next area frame (for example, 1221 in Fig. 24 described later), the input unit 62 determines that the mouse cursor is in a position unrelated to the next area frame and is in an unselected state.

[0246] The input unit 62 sets the next region as a high-resolution region (rectangular region) centered on the mouse cursor position, which is the click position, and sets a next region frame. The input unit 62 deletes the previous next region frame.

[0247] (Actual mode: arbitrary area selection: parallel movement operation) The processing of the parallel movement operation when the operation mode is the actual mode will be described with reference to Fig. 18 (3). When the start position of the drag (swipe) with the mouse is within a predetermined distance from one side of the next area frame (for example, 1221 in Fig. 24 described later) (excluding within a predetermined distance from both end points or near the center of the side), the input unit 62 determines that the mouse cursor is near the side and is in a frame selection state.

[0248] The input unit 62 translates the next region along a vector from the drag (swipe) start position to the drag (swipe) end position, and sets a frame for the next region. In this case, the size of the next region is fixed.

[0249] The input unit 62 may be configured to perform translational movement by operating keys on a keyboard (for example, cursor keys) instead of (or in addition to) dragging with a mouse.

[0250] (Production mode: arbitrary area selection: zoom in / out operation) The processing of zoom in / out operation (opposite side fixed) when the operation mode is production mode will be described with reference to Fig. 19 (4). When the start position of a drag (swipe) with the mouse is within a predetermined distance from a predetermined position near the center of one side of the next area frame (for example, 1221 in Fig. 24 described later), the input unit 62 determines that the mouse cursor is near the midpoint of the side and is in a side selection state.

[0251] The input unit 62 sets the next region as the smallest rectangular region that includes the center of the opposite side of the one side and the drag (swipe) end position and maintains the aspect ratio, and sets a next region frame. In this case, the opposite side of the next region is fixed.

[0252] 19(5), the process of the zoom operation (fixed diagonal vertices) when the operation mode is the actual mode will be described. When the start position of the drag (swipe) with the mouse is within a predetermined distance from one vertex of the next region frame (for example, 1221 in FIG. 24 described later), the input unit 62 determines that the mouse cursor is near the vertex and is in a vertex selection state.

[0253] The input unit 62 sets the next region as a smallest rectangular region that includes the diagonal vertex position of the one vertex and the drag (swipe) end position and maintains the aspect ratio, and sets the next region frame. In this case, the diagonal vertex of the next region is fixed.

[0254] In addition, the input unit 62 may be configured to change (enlarge or reduce) the size of the area by operating keys on the keyboard (e.g., the PageUp key and PageDown key) instead of (or in addition to) dragging with the mouse (however, the center position of the area is fixed).

[0255] In this way, the operator can set the region of interest and the next region by performing a region selection operation (e.g., a drag or swipe operation), which allows intuitive editing of the region of interest, etc., with the same operability as operating a graphics drawing application.

[0256] Furthermore, when setting the region of interest and the next region, the input unit 62 may correct the positions of the region of interest and the next region by translating them as little as possible so as not to include areas outside the range of the region of the low-resolution video input from the video generation device 7. This makes it possible to prevent the region of the partial video to be cut out from being set outside the input video.

[0257] <Processing of Display Unit 64> Next, a description will be given of processing of the display unit 64 shown in Fig. 14. The display unit 64 displays (previews) the low-resolution video input from the video generation device 7 on the viewer 1207 shown in Fig. 21 etc.

[0258] (Setting Mode) When the operation mode is the setting mode, the display unit 64 displays a preset area frame as a rectangle superimposed on the preview display. The preset area frame may be marked with a character string or a symbol according to the number of the preset channel, and the frame color, brightness, pattern (solid line, dashed line, dotted line, chain line, multiple line, wavy line, etc.), and line width may be changed.

[0259] When the operating mode is the setting mode, the display unit 64 may highlight and display the preset area frame of the currently specified preset channel (for example, by thickening the line width, increasing the brightness, changing the pattern, etc.).

[0260] The display unit 64 may be configured to light an indicator light (tally lamp) near the button of the currently specified preset channel in the preset button row 1201 shown in FIG. 21 (in the example of FIG. 21, indicator light 1224 is lit).

[0261] Furthermore, when the operation mode is the production mode, the display unit 64 may not display the preset button row 1201 or may gray it out.

[0262] In addition, if there is a preset channel for which the same area as the next area is set as a preset area, the display unit 64 may be configured to light up an indicator light (tally lamp) near the button for that preset channel among the buttons in the next area button row 1202 (in the example of Figure 21, indicator light 1225 is lit).

[0263] Furthermore, if there is no preset channel for which the same area as the next area is set as a preset area, the display unit 64 may turn on an indicator light (tally lamp) 1218 indicating the next area.

[0264] Furthermore, if there is a preset channel for which the same area as the cut-out area (area of ​​the PGM output video) is set as the preset area, the display unit 64 may be configured to light up an indicator light (tally lamp) near the button for that preset channel among the buttons in the immediate transition operation button row 1206 (in the example of Figure 21, indicator light 1226 is lit).

[0265] In addition, if there is no preset channel in which the same area as the cut-out area (which may be a cut-out area in transition) is set as a preset area, the display unit 64 may be configured to turn on an indicator light (tally lamp) 1219 indicating other cut-out areas.

[0266] (Production Mode) When the operation mode is the production mode, the display unit 64 displays the clipping area frame 1220 in a shape such as a rectangle superimposed on the preview display, as shown in Figures 24 and 25. To distinguish the clipping area frame 1220 from other frames, a specific character string or symbol (for example, character strings such as "PGM" or "LINE") may be added to the clipping area frame 1220, or the color, brightness, pattern (solid line, dashed line, dotted line, chain line, multiple line, wavy line, etc.) or line width of the frame may be changed, or a shape like brackets may be displayed only near the four corners.

[0267] When the operation mode is the production mode, the display unit 64 displays a next area frame 1221 as a rectangle or the like, superimposed on the preview display. To distinguish the next area frame 1221 from other frames, a specific character string or symbol (for example, character strings such as "NEXT" or "PVW") may be added to the frame, or the color, brightness, pattern (solid line, dashed line, dotted line, chain line, multiple line, wavy line, etc.) or line width may be changed, or a shape like brackets may be displayed only near the four corners.

[0268] The colors of the tally lamp and frame may be different for the cut-out region (for example, red) and the next region (for example, green).

[0269] <Transition Curve> Next, a case where a cut-out region is determined in accordance with a transition curve during the panning transition operation described above will be described. As described above, when any button in the pan button row 1204 is pressed, the input unit 62 generates cut-out information including an identifier of the PGM channel, the type and timing of the transition indicating the panning transition, and the region (cut-out region) that automatically moves, expands, or contracts from the current cut-out region to the next region from time to time.

[0270] The timing of the transition is specified by the operator so that the transition amount of the position and size of the area undergoing the panning transition with respect to time follows a transition curve including, for example, a constant jerk curve.

[0271] In this case, when a button specifying a pan slider included in the pan button column 1204 is pressed, the input unit 62 determines the cut-out area that will transition during the panning transition according to the time specified by the transition time specification slider 1205 and the transition curve specified by the transition curve setting slider 1222.

[0272] The input unit 62 determines the cut-out region (center coordinates [x(t), y(t)], height v(t), and width h(t)) during the panning transition using the following formula.

[0273] Here, the center coordinates of the cutout region at the start of the transition are [x0, y0], the height is v0, and the width is h0; the center coordinates of the cutout region at the end of the transition are [x1, y1], the height is v1, and the width is h1; the total transition time from the start of the transition to the end of the transition is T, and the elapsed time from the start of the transition is t. The total transition time T is the time specified by the transition time specification slider 1205. The function f(τ) is a function that determines the transition curve, and is a monotonically increasing function in the broad sense with a domain 0≦τ<1 and a range 0≦f(τ)≦1.

[0274] The function f(τ) that defines the transition curve may be, for example, the following equation:

[0275] Furthermore, the function f(τ) that defines the transition curve may be expressed as follows:

[0276] Here, s is a parameter that changes the shape of the transition curve, and is set within the range of 0≦s≦1 by the transition curve setting slider 1222. The parameter s indicates the proportion of the total transition time T that is occupied by a section accompanied by acceleration / deceleration (a section that is not at a constant speed).

[0277] Fig. 20 is a diagram illustrating an example of a transition curve, showing the properties of the function f(τ) in equation (4). The lower part of Fig. 20 shows transition curves based on the function f(τ) when the parameter s is 0.00, 0.65, and 1.00. The horizontal axis represents time, and the vertical axis represents the position of the transition curve based on the function f(τ).

[0278] When the parameter s=0.00, the transition curve increases linearly over time, and it can be seen that the closer the parameter s is to 1, the steeper the slope of the S-curve becomes. This transition curve can be changed using the transition curve setting slider 1222, and the operator can visually confirm the curve shape and the current transition state during panning transition (see the upper right of FIG. 20, FIG. 21, etc.).

[0279] In this way, the shape of the transition curve (transition curve) can be changed by the operation of the operator, so that the agility and smoothness of the transition can be adjusted, and a variety of effects can be produced.

[0280] The upper left side of Fig. 20 shows changes in jerk, acceleration, and velocity from transition start time ta to transition end time tb. With respect to the passage of time, section A has a positive uniform jerk and positive acceleration, section B has a negative uniform jerk and positive acceleration, section C has a constant velocity, section D has a negative uniform jerk and negative acceleration, and section E has a positive uniform jerk and negative acceleration.

[0281] In other words, the section 0<t<sT / 2 is a time section of positive acceleration, the section (T-sT / 2)<t<T is a time section of negative acceleration, and the section sT / 2<t<(T-sT / 2) is a time section of constant velocity.

[0282] More specifically, section A, 0<t<sT / 4, is a section of positive uniform acceleration and positive acceleration; section B, sT / 4<t<sT / 2, is a section of negative uniform acceleration and positive acceleration; section C, sT / 2<t<(T-sT / 2), is a constant speed section; section D, (T-sT / 2)<t<(T-sT / 4), is a section of negative uniform acceleration and negative acceleration; and section E, (T-ST / 4)<t<T, is a section of positive uniform acceleration and negative acceleration.

[0283] In this way, according to the transition curve of the function f(τ) of the above-mentioned formula (4), it is possible to suppress the jerk (false acceleration) within a constant value over the entire time interval when s > 0. This makes it possible to make the acceleration curve continuous, and to provide a PGM output image that is less likely to cause a physical impact (shock) from before the transition starts to after the transition ends.

[0284] Therefore, the control unit 61 can determine a cut-out region that realizes a panning transition that is less shocking to the human eye, and as a result, cut-out that matches human intuition can be realized in the image generation device 7. Furthermore, acceleration and deceleration at the start and end of the transition enable a smooth transition with less shock, and discomfort when viewing the cut-out partial image can be reduced.

[0285] It should be noted that the larger the parameter s, the more likely it is that the driver will experience shaking due to acceleration or deceleration, so it is preferable to set the parameter s to 0.5≦s≦0.8.

[0286] Furthermore, the cut-out area during the panning transition determined by the input unit 62 is set so that both its coordinate values ​​(center coordinates [x(t), y(t)]) and size (height v(t) and width h(t)) change linearly according to a single function f(τ), as in the above formula (2), but either the coordinate values ​​or the size may change linearly according to a single function f(τ).

[0287] Furthermore, the function f(τ) used by the input unit 62 when determining the cut-out region during panning transition has the jerk characteristics shown in FIG. 20 , but it is sufficient if it has characteristics that include a section in which the absolute value of the jerk is equal to or less than a predetermined value over time.

[0288] 20 is configured to include sections A, B, C, D, and E over time, but as long as both section A and section B exist, one or more of sections C, D, and E may not exist. Also, as long as both section D and section E exist, one or more of sections A, B, and C may not exist.

[0289] That is, if the durations of sections A, B, C, D, and E are TA, TB, TC, TD, and TE, respectively, the transition curves shown in Fig. 20 are TA > 0, TB > 0, TC > 0, TD > 0, and TE > 0. Alternatively, the transition curves may be TA > 0, TB > 0, TC ≥ 0, TD ≥ 0, and TE ≥ 0, or TA ≥ 0, TB ≥ 0, TC ≥ 0, TD > 0, and TE > 0, and the curves in sections A, B, C, D, and E may be continuous over the entire section and connected in a state where they are twice differentiable.

[0290] In this way, the input unit 62 of the cut-out control device 6 specifies one or more preset areas in accordance with the operator's operation, and specifies one of these areas as the next area, which is the area to which the cut-out area will transition. Then, in accordance with the operator's operation, the input unit 62 specifies the type and timing of the transition when the cut-out area transitions from the preset area that is the transition source to the next area that is the transition destination, and generates cut-out information including an identifier indicating the channel of the PGM output video, the type and timing of the transition, the cut-out area, etc.

[0291] Based on the cut-out information, the control unit 61 generates a control signal for the image generation device 7 to cut out a partial image from the input image, which control signal includes area information indicating the cut-out position and size of the cut-out area, and outputs it to the image generation device 7.

[0292] The drawing unit 63 inputs the low-resolution image from the image generation device 7, and draws one or more areas selected from the preset area, the next area, and the area indicated by the control signal as a picture based on the cutout information. The display unit 64 displays the picture on the screen to present it to the operator.

[0293] The image generating device 7 receives a control signal from the cutout control device 6, cuts out a partial image from the input image based on the control signal, and outputs the partial image.

[0294] In this way, the type and timing of transition when transitioning from a preset area as a transition source to a next area as a transition destination is specified as an area from which a partial video is cut out in accordance with an operation by the operator. This allows the video generation device 7 to realize cutting out an area that changes from moment to moment from the video as a partial video in a manner that suits human intuition.

[0295] (Image Generation Device 7) Next, the image generation device 7 shown in Fig. 5 will be described in detail. Fig. 27 is a block diagram showing an example of the configuration of the image generation device 7. This image generation device 7 includes a resolution conversion unit 71 and a cropping unit 72.

[0296] As described above, the image generating device 7 receives a high-definition image, converts the resolution of this input image to a low-resolution image, and outputs the low-resolution image to the cutout region determining device 5 and the cutout control device 6. The image generating device 7 also receives a control signal from the cutout control device 6, cuts out a plurality of images from the input image based on the control signal to generate a plurality of partial images, and outputs the PGM output image, the NEXT output image, and the CH1 to CH6 output images.

[0297] The resolution conversion unit 71 is similar to the resolution conversion unit 41 shown in FIG. 3, and therefore a description thereof will be omitted here.

[0298] The cropping unit 72 receives the high-definition video as an input and a control signal from the cropping control device 6, and crops out the region from the input video, which is high-definition video, based on the crop position and size of the region indicated by the region information in the control signal, and outputs the cropped partial video. As described above, the control signal includes region information indicating the crop position and size of the region.

[0299] For example, the cropping unit 72 crops out a partial video from the input video, the cropping position and size of which are indicated by the area information of the control signal, based on a control signal including area information indicating the crop position and size of an area corresponding to the channel of the PGM output video, sets the partial video as the PGM output video, and outputs the PGM output video. The same applies to the NEXT output video and the CH1 to CH6 output videos.

[0300] Here, when the above-mentioned panning transition operation is performed in the cut-out control device 6, the cut-out unit 72 inputs a control signal from the cut-out control device 6 including area information indicating the cut-out position and size of the area corresponding to the channel of the PGM output video, which transitions from the first preset area at the transition source to the second preset area (next area) at the transition destination by constantly moving, expanding or contracting, and cuts out a partial video from the input video.

[0301] In other words, the cutting-out unit 72 cuts out a partial image of an area that transitions from the input image, moving, expanding or contracting from the first preset area at the source of the transition to the second preset area (next area) at the destination of the transition, and sets the partial image as the PGM output image and outputs it.

[0302] In this case, the area of ​​the partial image to be cut out may be determined by the function f(τ) shown in equations (2) to (4), and both the coordinate values ​​and the size may change linearly in accordance with the function f(τ), or either the coordinate values ​​or the size may change linearly in accordance with the function f(τ).

[0303] <Example of video output by video generation device 7> Next, an example of video output by the video generation device 7 will be described. Fig. 22 is a diagram showing an example of an input video (high definition video) and a CH1 output video, etc. Fig. 23 is a diagram showing an example of a PGM output video and a NEXT output video. Fig. 26 is a diagram showing an example of a PGM output video resulting from a panning transition operation.

[0304] The input video shown in the upper part of Fig. 22 corresponds to the display content of the viewer 1207 shown in Fig. 21. The CH1 output video, CH2 output video, CH3 output video, and CH6 output video shown in the lower part of Fig. 22 are videos that are cut out and output by the video generation device 7 based on a control signal when the preset area frames 1208-1 to 1208-3 and 1208-6 shown in Fig. 21 are set.

[0305] The input video is, for example, 8K video (resolution 7680×4320 pixels), and the CH1 output video, CH2 output video, CH3 output video, and CH6 output video are, for example, 2K video (resolution 1920×1080 pixels).

[0306] 21, preset areas are not set for preset channel numbers 4 and 5 corresponding to the CH4 output video and CH5 output video. In this case, the CH4 output video and CH5 output video for which no preset area is set may be video obtained by down-converting the entire input video, or may be a predetermined video (for example, a video of a single color of a predetermined color (blue, black, white, gray, etc.), a color bar video, a video of predetermined characters or patterns, etc.).

[0307] Furthermore, for preset channels that exist in the extraction control device 6 but do not exist in the video generation device 7, the video generation device 7 does not output individual videos like the CH1 to CH6 output videos.

[0308] The PGM output video shown on the left side of Figure 23 is a video that is cut out and output by the video generation device 7 based on a control signal when, for example, button number 1 in the immediate transition operation button row 1206 shown in Figure 21 is pressed, and the preset area of ​​the preset area frame 1208-1 is set as the cut-out area corresponding to the area of ​​the PGM output video.

[0309] In addition, the NEXT output video shown on the right side of Figure 23 is a video that is cut out and output by the video generation device 7 based on a control signal when, for example, button number 3 in the next area button row 1202 shown in Figure 21 is pressed, and the preset area of ​​the preset area frame 1208-3 is set as the next area.

[0310] The PGM output video shown in Figure 26 (1) is a video that is cut out and output by the video generation device 7 based on a control signal when the cut-out area frame 1220 shown in Figure 24 is set and the panning transition begins.

[0311] The PGM output video shown in Figures 26(2) and 26(3) is video that is cut out and output by the video generation device 7 based on a control signal during panning transition. The PGM output video in Figure 26(3) corresponds to the video of the cut-out area frame 1220 shown in Figure 25.

[0312] The PGM output video shown in Fig. 26(4) is a video that is cut out and output by the video generation device 7 based on the control signal when the panning transition is completed. This PGM output video corresponds to the video of the next area frame 1221 when the panning transition is completed in Fig. 24 and Fig. 25.

[0313] In this way, when a panning transition operation is performed, the cropping unit 72 of the video generation device 7 crops out a partial image from the input video so that the partial image moves, expands, or shrinks from the source area to the destination area from time to time in accordance with the function f(τ).

[0314] This makes it possible to extract a partial video that achieves a panning transition that is less shocking to the human eye. Therefore, when extracting a region that changes from moment to moment from a video as a partial video, it is possible to achieve extraction that matches human intuition.

[0315] As described above, according to the video generation system 1′, the cutout area determination device 5 detects objects from the low-resolution video in which the input video has been resolution-converted, calculates their coordinate data, and determines multiple cutout areas within the low-resolution video based on the coordinate data for each object, etc., and calculates the coordinate data for each area.

[0316] The cut-out control device 6 generates a control signal including area information for the image generating device 7 to cut out an image from the input image and generate a partial image based on coordinate data for each area, in accordance with an operation by an operator.

[0317] The video generating device 7 extracts a plurality of videos from the input video based on the control signal to generate a plurality of partial videos, and outputs the PGM output video, the NEXT output video, and the CH1 to CH6 output video.

[0318] This allows the operator to specify the area of ​​the partial image generated by the image generation device 7 by referring to the coordinate data for each area determined by the cut-out area determination device 5, thereby reducing the workload on the operator.

[0319] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0320] For example, the image generation device 4 of the image generation system 1 shown in FIG. 1 and the image generation device 7 of the image generation system 1′ shown in FIG. 5 are configured to input 8K high-definition images, but the input high-definition images are not limited to 8K.

[0321] 1 and the clipping region determination device 5 shown in Fig. 5 are configured to input low-resolution video, but may also input 8K high-definition video. In short, the video input to the object detection device 2 and the clipping region determination device 5 only needs to be video related to high-definition video so that an object included in the high-definition video can be detected.

[0322] Furthermore, the image generation system 1 shown in FIG. 1 is configured to include an object detection device 2, a cutout control device 3, and an image generation device 4, while the image generation system 1′ shown in FIG. 5 is configured to include a cutout area determination device 5, a cutout control device 6, and an image generation device 7.

[0323] In contrast to this, the video generation system of the modified example may include a clipping region determination device 5, a clipping control device, and a video generation device 4.

[0324] In this case, the cutout region determination device 5 receives the low-resolution image from the image generation device 4, performs the same processing as the cutout region determination device 5 shown in FIG. 5, and outputs coordinate data for each region to the cutout control device.

[0325] The cut-out control device receives coordinate data for each region from the cut-out region determination device 5 and performs some of the processing by the cut-out control device 3 shown in Fig. 1. Specifically, as shown in step S205 of Fig. 2, the cut-out control device generates a control signal including region information and operation information for the image generation device 4 to cut out an image from the input image and generate a partial image based on the coordinate data for each region (corresponding to a plurality of cut-out regions) in accordance with an operation by an operator, and outputs the control signal to the image generation device 4.

[0326] Note that a normal computer can be used as the hardware configuration of the object detection device 2, cut-out control device 3, and image generation device 4 of image generation system 1, and the cut-out area determination device 5, cut-out control device 6, and image generation device 7 of image generation system 1′. Each of the object detection device 2, cut-out control device 3, image generation device 4, cut-out area determination device 5, cut-out control device 6, and image generation device 7 is configured by a computer including a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, etc.

[0327] Each function of the object detection device 2 is realized by causing a CPU to execute a program describing the function. Similarly, each function of the cutout control device 3 is realized by causing a CPU to execute a program describing the function. Similarly, each function of the resolution conversion unit 41, cutout unit 42, and switcher unit 43 of the video generation device 4 is realized by causing a CPU to execute a program describing the function.

[0328] Furthermore, the functions of the video input unit 51, object detection unit 52, object tracking unit 53, and clipping area determination unit 54 provided in the clipping area determination device 5 are each realized by having the CPU execute a program describing these functions. Similarly, the functions of the control unit 61, input unit 62, drawing unit 63, and display unit 64 provided in the clipping control device 6 are each realized by having the CPU execute a program describing these functions. Similarly, the functions of the resolution conversion unit 71 and clipping unit 72 provided in the image generation device 7 are each realized by having the CPU execute a program describing these functions.

[0329] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be stored and distributed on storage media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memories, and can also be transmitted and received via a network.

[0330] The same applies to the cutout region determination device 5, the cutout control device, and the image generation device 4 of the modified image generation system.

[0331] 1, 1' Video generation system 2 Object detection device 3, 6 Cut-out control device 4, 7 Video generation device 5 Cut-out area determination device 10 High-definition camera 11 Object confirmation monitor 12 Cut-out monitor 13 Multi-screen monitor 14 Output video monitor 15 Recorder / playback device 41, 71 Resolution conversion unit 42, 72 Cut-out unit 43 Switcher unit 51 Video input unit 52 Object detection unit 53 Object tracking unit 54 Cut-out area determination unit 61 Control unit 62 Input unit 63 Drawing unit 64 Display unit 1201 Preset button row 1202 Next area button row 1203 Cut button 1204 Pan button row 1205 Transition time designation slider 1206 Instant transition operation button row 1207 Viewer 1208-1 to 1208-3, 1208-6 Preset area frame 1209 Resize button row 1210 Centering button 1211 Setting mode button 1212 Production mode button 1213 Delete area button 1214 Delete all areas button 1215 Load area file button 1216 Save area file button 1217 Stop button 1218, 1219, 1224, 1225, 1226 Indicator light 1220 Extraction area frame 1221 Next area frame 1222 Transition curve setting slider 1223 Reconnect button 1301 Whole area 1302-1 to 1302-3 Blocks 1303 Mask area 1304 Boundary 1305 Forced object area

Claims

1. An image generating device that cuts out a portion of an input image as a cut-out area based on an input control signal to generate a partial image, the control signal including area information indicating positions and sizes of a plurality of cut-out areas cut out by the image generating device, as well as an identifier of an assigned image to be assigned to the partial image from a plurality of images corresponding to the plurality of cut-out areas, and operation information indicating a switching type when operating to switch the allocation of the partial image, the image generating device comprising: a cut-out unit that cuts out the plurality of images from the input image based on the area information included in the control signal; and a switcher unit that, when generating a first image from the plurality of images cut out by the cut-out unit as the partial image and then generating a second image from the plurality of images as the partial image, selects the second image from the plurality of images based on the identifier of the assigned image indicated by the operation information included in the control signal, and generates the partial image by switching from the first image to the second image based on the switching type indicated by the operation information included in the control signal.

2. An image generating device as claimed in claim 1, wherein the switcher unit generates the partial image so that, when the switching type is cut, the first image is instantly switched to the second image when a control signal including operation information for the switching type is input; when the switching type is wipe, the switcher unit generates the partial image so that, when a control signal including a wipe time together with operation information for the switching type is input, the wipe gradually switches from the first image to the second image over the wipe time; and when the switching type is dissolve, the switcher unit generates the partial image so that, when a control signal including a dissolve time together with operation information for the switching type is input, the dissolve gradually switches from the first image to the second image over the dissolve time.

3. A cut-out region determination device that detects an object from a specified video and determines a plurality of cut-out regions for generating a partial video from an input video based on the object, comprising: an object detection unit that detects an object from the specified video; an object tracking unit that tracks the object detected by the object detection unit and obtains coordinate data of the object; and a cut-out region determination unit that specifies a number of objects for each of the plurality of blocks so that a substantially equal number of objects is obtained by dividing the number of objects S by the number of blocks N, where N is an integer equal to or greater than 2, and a number of blocks is set from the specified video, and specifies a number of objects for each of the plurality of blocks so that a substantially equal number of objects is obtained by dividing the number of objects S by the number of blocks N; and sets the plurality of blocks from the specified video based on the number of objects specified for each of the plurality of blocks and the coordinate data of the objects obtained by the object tracking unit, and determines the plurality of blocks as a plurality of cut-out regions.

4. A cutout region determination device that detects an object from a predetermined image and determines a plurality of cutout regions for generating a partial image from an input image based on the object, comprising: an object detection unit that detects an object from the predetermined image; an object tracking unit that tracks the object detected by the object detection unit and obtains coordinate data of the object; and a plurality of blocks are set from the predetermined image, the number of the blocks is N (N is an integer of 2 or more), the number of the objects tracked by the object tracking unit is S (S is an integer of 2 or more), and the numbers of objects for the 1st to (N-1)th blocks of the plurality of blocks are C1 to C2. N-1 Let the number of objects be C1 to C N-1 is specified in advance by the operator, the number of objects for the Nth block, C N into the following formula: The number of objects C1 to C2 designated for each of the plurality of blocks is N-1 , C N and a cut-out region determination unit that cuts out the plurality of blocks from the specified video based on coordinate data of the object detected by the object tracking unit and determines the plurality of blocks as the plurality of cut-out regions.

5. A cut-out area determination device as claimed in claim 3 or 4, characterized in that the object detection unit detects the object from an area of ​​the specified image excluding a mask area when a mask area is preset by an operator at a specified position within the specified image.

6. A cut-out area determination device as claimed in claim 3 or 4, characterized in that the object detection unit detects the object from an area of ​​the specified image excluding the specified area indicated by the boundary when a boundary is preset at a specified position within the specified image by an operator.

7. A cut-out area determination device as claimed in claim 3 or 4, characterized in that, when a mandatory object area is preset by an operator at a specified position within the specified image, the object detection unit detects the object from the specified image, assuming that the object exists in the mandatory object area.

8. A cut-out area determination device as claimed in claim 3 or 4, characterized in that the cut-out area determination unit changes the aspect ratio of one or more of the plurality of cut-out areas so as to include or exclude a specified area based on the object based on the coordinate data of the object tracking unit.

9. A program for causing a computer to function as the image generating device according to claim 1 or 2.

10. A program for causing a computer to function as the cropping region determination device according to claim 3 or 4.

Images