Automatic switching device, automatic switching method and program

Abstract

Automate video switching. [Solution] The automatic switching device comprises an acquisition unit that acquires video material of the content, a recognition unit that recognizes images or audio of people in the video material, a cut-out candidate area selection unit that calculates a role index that is an index of the role of an object in the content using the recognized image or audio of the object and uses the role index to select multiple cut-out candidate areas that include at least one object within the area and are used to cut out video from the video material, a detection unit that detects video and audio changes of objects in the video material, a cut-out area determination unit that calculates a switching index that is an index of the switching timing in each cut-out candidate area from the detection results of video and audio changes of the object in the video material and determines a cut-out area from each cut-out candidate area using the switching index of each cut-out candidate area, and a switching unit that cuts out video of the cut-out area from the video material and switches the currently output video to the cut-out video.

Description

[Technical field]

[0001] The present invention relates to an automatic switching device, an automatic switching method and a program, and more particularly to an automatic switching device, an automatic switching method and a program for automating the selection of a video to be switched. [Background technology]

[0002] Conventionally, in broadcasting station program production, multiple cameras are used, each cameraman shoots performers from his own viewpoint, and a switcher switches (selects) one image from the multiple camera images based on the intent of the program, etc., to output the image. However, the above-mentioned production system requires a lot of human resources. Therefore, in order to reduce the human resources, attempts have been made to automatically shoot using robot cameras, etc. (for example, Patent Document 1).

[0003] The technology of Patent Document 1 includes a shooting shot type setting means for setting a shooting shot type based on an external command, a shooting rule generation information storage means for storing in advance a plurality of event shooting rule generation information that associates a subject's event, a shooting shot type, and a switching shot, a shooting rule generation means for generating an event shooting rule that associates an event with a switching shot based on the set shooting shot type and the event shooting rule generation information, and a shooting shot control means for determining a shooting shot based on the external event information and the event shooting rule. The technology of Patent Document 1 also includes a function of inputting a plurality of images of a subject photographed by a plurality of robot cameras, and outputting the images to a switcher that generates one output image by switching the output of the images based on a switching command that is an external command to switch the output from one robot camera to another robot camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4741557 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the invention of Patent Document 1 requires the preparation of many robot cameras, which increases the equipment cost, and the rules for controlling the cameras must be defined in detail, and the process from shooting to switching is not fully automated.

[0006] SUMMARY OF THE PRESENT DISCLOSURE The present invention has been made in view of the above problems, and has an object to provide an automatic switching device, an automatic switching method, and a program that can automatically select a video to be switched. [Means for solving the problem]

[0007] One aspect of the present invention is an automatic switching device that includes an acquisition unit that acquires video material of a content, a recognition unit that recognizes images or audio of people in the video material, a cut-out candidate area selection unit that calculates a role index that is an index of the role of an object in the content using the recognized image or audio of the object, and uses the role index to select a plurality of cut-out candidate areas for cutting out video from the video material, the cut-out candidate areas including at least one object within an area, a detection unit that detects video and audio changes of the object in the video material, a cut-out area determination unit that calculates a switching index that is an index of the switching timing in each of the cut-out candidate areas from the detection results of the video and audio changes of the object in the video material, and determines a cut-out area from each of the cut-out candidate areas using the switching index for each of the cut-out candidate areas, and a switching unit that cuts out video of the cut-out area from the video material and switches the currently output video to the cut-out video.

[0008] One aspect of the present invention is a program for causing a computer to realize an acquisition function for acquiring video material of a content, a recognition function for recognizing an image or audio of an object in the video material, a cut-out candidate area selection function for calculating a role index that is an index of the role of an object of the content based on the image or audio of the recognized object and using the role index to select a plurality of cut-out candidate areas for cutting out video from the video material, the plurality of cut-out candidate areas including at least one object within an area, a detection function for detecting video and audio changes of an object in the video material, a cut-out area determination function for calculating a switching index that is an index of the switching timing in each of the cut-out candidate areas from the detection results of the video and audio changes of the object in the video material and determining a cut-out area from each of the cut-out candidate areas using the switching index for each of the cut-out candidate areas, and a switching function for cutting out video of the cut-out area from the video material and switching the currently output video to the cut-out video.

[0009] One aspect of the present invention is an automatic switching method in which a computer acquires video material of a content, recognizes images or audio of an object in the video material, calculates a role index that is an index of the role of the object of the content using the image or audio of the recognized object, uses the role index to select a plurality of cut-out candidate areas for cutting out video from the video material, which include one or more objects within the area, detects video and audio changes of the object in the video material, calculates a switching index that is an index of the switching timing in each of the cut-out candidate areas from the detection results of the video and audio changes of the object in the video material, determines a cut-out area from each of the cut-out candidate areas using the switching index for each of the cut-out candidate areas, cuts out video of the cut-out area from the video material, and switches the currently output video to the cut-out video.

[0010] One aspect of the present invention is an automatic switching device that includes an acquisition unit that acquires video material of a content, a recognition unit that recognizes images or audio of objects in the video material, an excision area selection unit that calculates an index taking into account the role of the object in the content and the timing of video switching based on the image or audio of the recognized object, and uses the index to select an excision area on the video material that includes at least one object, and a switching unit that excises image of the excision area from the video material and switches the currently output video to the excised image. Effect of the Invention

[0011] The present invention can automate the selection of videos to be switched. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of an embodiment of the present invention. [Diagram 2] FIG. 2 shows an example of program information in the program database 11. [Diagram 3] FIG. 3 is a diagram showing an example of a program image captured by the camera 1. As shown in FIG. [Figure 4] FIG. 4 is a diagram for explaining the operations of the cut-out candidate region selection unit 13, the cut-out candidate region selection unit 15, the switching unit 16 and the display control unit 17. As shown in FIG. [Diagram 5] FIG. 5 is a diagram for explaining the operations of the cut-out candidate region selection unit 13, the cut-out candidate region selection unit 15, the switching unit 16 and the display control unit 17. As shown in FIG. [Figure 6] FIG. 6 is a diagram for explaining the operations of the cut-out candidate region selection unit 13, the cut-out candidate region selection unit 15, the switching unit 16 and the display control unit 17. In FIG. [Figure 7] FIG. 7 is a diagram for explaining the operations of the cut-out candidate region selection unit 13, the cut-out candidate region selection unit 15, the switching unit 16 and the display control unit 17. As shown in FIG. [Figure 8] FIG. 8 is a diagram for explaining the first modification of the embodiment. [Figure 9] FIG. 9 is a block diagram of the second modification of the embodiment. [Figure 10] FIG. 10 is a diagram showing a switching video selection made by the user on the display device 3. In FIG. [Figure 11] FIG. 11 is a block diagram of the third modification of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] An embodiment of the present invention will be described.

[0014] In the following description, an example will be described in which the content is one program, the video material is program video obtained by shooting the program with a camera, and the object of the video material is the performers of the program. However, the content and video material are not limited, and for example, the content is not limited to a program, and the video material may be something that can be directly obtained, such as a video file. Furthermore, the object is not limited to people such as performers, and may be an object (animal, product, etc.) that appears in the video.

[0015] Fig. 1 is a block diagram of an embodiment of the present invention, in which 1 is a camera, 2 is a switching device, and 3 is a display device.

[0016] Camera 1 is a camera that shoots a program. Camera 1 shoots a program with a wide angle of view that captures all subjects (for example, the performers of the program). As described below, in this embodiment, the video captured by camera 1 of the program (hereinafter referred to as program video) itself, or a video cut out from a part of the program video captured by camera 1 (trimmed video) is output as a switching video to be output to the main line side. For this reason, camera 1 is preferably a 4K or 8K camera that can capture high-quality video, but is not limited to these.

[0017] The switching device 2 includes an acquisition unit 10, a program database (program DB) 11, a recognition unit 12, a cut-out candidate region selection unit 13, a detection unit 14, a cut-out region determination unit 15, a switching unit 16, and a display control unit 17. The acquisition unit 10 receives the program video and audio captured by the camera 1 .

[0018] The program database 11 is a database in which program information for a program is registered when the program is recorded or filmed. The program information includes performer information for the program's performers, and coefficients (weighting values) for the calculation formulas of the role index and switching index described below. Program information is registered for each program before the start of recording or filming of the program. This makes it possible to reflect the characteristics of each program. Note that multiple different pieces of program information may be registered for the same program. For example, program information may be registered for each corner.

[0019] Fig. 2 is an example of program information in the program database 11. The program information in the program database 11 shown in Fig. 2 is an example of program information for program X, which is a talk show. 2, information such as the image features of each performer A, B, C, and D appearing in Program X, the importance of the performer in Program X, a flag indicating the host of Program X, keywords written in the program script, and the appearance time (online time) in Program X is registered as performer information. Weighting values ​​(coefficients) used for the role index and switching index, which will be described later, are also registered.

[0020] The recognition unit 12 has a face recognition function, a bone structure determination function, and a voice recognition function. The face recognition function recognizes people (performers) in program footage using image features of the performers registered in the program database 11. It also detects the direction of gaze and facial changes (nodding and smiling) of the recognized performers and quantifies them. There is no restriction on the type of recognition and detection method, but examples include a pattern matching method and recognition using an algorithm obtained by machine learning. A representative machine learning method is deep learning, but is not limited to this.

[0021] The skeleton determination function determines the skeleton of a person appearing in a program video. Then, using the result of the skeleton determination, the positional relationship of the recognized body parts of the performer in the captured video is specified. Examples of the skeleton determination method include, but are not limited to, OpenPose, VisionPose, and tf-pose-estimation.

[0022] The voice recognition function has the function of detecting the pronunciation, sound level, and keywords of the performers from the voice of the program video. The voice recognition function may also have an identification function of identifying the performers (people) from the voice itself. Note that the voice may be obtained from a separate audio signal separate from the program video signal.

[0023] The cut-out candidate region selection unit 13 calculates a role index, which is an index of the role of each performer in the program, from the image, skeleton, or voice of the performer recognized by the recognition unit 12. Then, using the calculated role index, the cut-out candidate region selection unit 13 selects a plurality of cut-out candidate regions for cutting out video from the program video, which regions include at least one of the recognized performers.

[0024] Let us explain the role index. In a TV program, each performer usually has a role, such as the program's host (MC), main guest, reactor, etc. For example, in a talk show, there is a host who proceeds with the program, a main guest who is the central figure in the talk, and a reactor who reacts to the talk. Each performer in the program proceeds with the program while being conscious of their own role. A role index is a numerical representation of the roles of such performers.

[0025] The role index is an index that includes as parameters elements that change over time (for example, the speaker, the gaze direction of the performers, changes in the appearance of the performers (movements, nodding, smiling, etc.) and elements that are determined in advance (person importance, utterance of keywords). Parameters may also include viewer rating data, distribution data, top search keywords in search engines, etc. Below, the calculation formulas for each role index are explained using the example of a program being a talk show.

[0026] (1) Role Index (Moderator) The role index (host) is an index for identifying performers who have the role of MC (Master of ceremonies) or host from the program video. Usually, the MC and host of a program are predetermined. Therefore, performers who are registered as hosts in the program database 11 are identified from the image features of the performers recognized by the recognition unit 12. The role index (host) is calculated by substituting "1" if the performer detected by the recognition unit 12 is a host, or "0" if the performer detected by the recognition unit 12 is not a host, into the host term of the following formula for the role index (host). Role Index (Moderator) = Moderator x 1 The above-mentioned formula for the role index (chairman) is an example, and other terms may be added as appropriate. Also, the input values ​​are examples, and other values ​​may be substituted.

[0027] (2) Role indicator (topic of conversation) The role index (talk axis) is an index used to identify the performer who is the axis (center) of the talk of a program from the program footage. The axis of the talk is not limited to the main performer, and often changes as the program progresses. The role index (talk axis) is an index used to identify the performer who is the axis of the talk that changes as the program progresses. An example of the formula for calculating the role index (talk axis) is as follows: Role index (axis of conversation) = speaker x 0.5 + gaze group x 0.2 + keyword x 0.2 + person importance x 0.1

[0028] Here, the speaker is a term for considering that the performer is a speaker, and "10" is substituted for a performer who the recognition unit 12 detects talking, and "0" is substituted for a performer who the recognition unit 12 does not detect talking. The gaze set is a set value of the gaze direction of the performer detected by the recognition unit 12. For example, if one other performer is looking at the performer for whom the role index (talk axis) is calculated, 3 is substituted for the gaze set term, if two other performers are looking at the performer, 6 is substituted for the gaze set term, and if three other performers are looking at the performer, 10 is substituted for the gaze set term. The keyword is a term given when a performer speaks a keyword, and the words uttered by the performer are keywords registered in the program database 11. If the keyword is important, 10 is substituted for the keyword term, if the keyword is semi-important, 5 is substituted for the keyword term, and if it is not a keyword, 0 is substituted for the keyword term. For the person importance, the person importance of the performer recognized by the recognition unit 12 is read from the program database 11 and substituted into the person importance term. The coefficients of each term are registered in the program database 11 in the program information for each program. The above-mentioned formula for the role index (talk axis) is an example, and other terms may be added as appropriate. The input values ​​are also an example, and other values ​​may be substituted.

[0029] (3) Role Index (Reactor) Among the roles of program performers, there are performers (reactors) who are responsible for reacting (replying or nodding) to the talk of the host or the main topic of the conversation. The role of this performer (reactor) also often changes as the program progresses. The index for identifying this performer (reactor) is the role index (reactor). The formula for calculating the role index (reactor) is as follows: Role index (reactor) = nod x 0.3 + person facing the axis of the conversation x 0.2 + keyword x 0.2 + smile rate x 0.2 + person importance x 0.1

[0030] Here, a nod is a term given as one of the reactions to the talk of the host or the main topic of the talk, and a performer whose nod is detected by the recognition unit 12 is given a value from 0 to 10 according to the degree of nodding, and the value is substituted for the nod term. A performer facing the main topic of the talk is a term given to a performer facing the main topic of the talk, and if the recognition unit 12 detects that the performer is facing the main topic of the talk, "10" is substituted for the performer facing the main topic of the talk, and if not, "0" is substituted for the performer facing the main topic of the talk. When a performer utters a word, the word is a keyword registered in the program database 11, and if the keyword is important, 10 is substituted for the keyword term, if the keyword is semi-important, 5 is substituted for the keyword term, and if it is not a keyword, 0 is substituted for the keyword term. For the person importance, the person importance of the performer recognized by the recognition unit 12 is read from the program database 11 and substituted for the person importance term. The coefficient of each term is registered in the program information for each program in the program database 11. The above-mentioned formula for the role index (reactor) is an example, and other terms may be added as appropriate. Also, the input values ​​are examples, and other values ​​may be substituted.

[0031] The cut-out candidate area selection unit 13 calculates the role index for each performer as needed. Then, a predetermined number (e.g., 3) of performers with high indices for each role index are selected. Then, an area is determined so that the selected performers are included in a predetermined size (e.g., aspect ratio 16:9). Each area selected in this manner is hereinafter referred to as a cut-out candidate area. Note that if the value of each role index does not exceed a certain threshold, the cut-out candidate area selected at the time of the previous calculation of the role index may be maintained. Note that the cut-out size, etc. can be freely determined depending on the display size, etc. of the device providing the video.

[0032] In addition, the cut-out candidate area selection unit 13 may be configured to receive the recognition results of the face recognition function, voice recognition function, etc. by the recognition unit 12, and automatically superimpose (overlay) subtitles such as the names of identified performers on the image of the cut-out candidate area.

[0033] Similar to the recognition unit 12, the detection unit 14 has a face recognition function, a bone structure determination function, an object identification function (including identification of color, shape, etc.), and a voice recognition function. The detection unit 14 detects changes in the video and audio of each performer recognized by the recognition unit 12 in the program video. For example, what is detected is the identification of the performer, the performer's opening remarks, the movements of the presenter, the line of sight of each performer, etc. The detection results of each performer in the program video are output to the clipping region determination unit 15.

[0034] The cut-out region determination unit 15 calculates a switching index for each cut-out candidate region using the detection result of the detection unit 14. Then, the cut-out region is determined using the switching index for each cut-out candidate region. Here, the switching index is an index for determining a switching video to be switched from among the videos of the multiple cut-out candidate regions. The switching index is an index including, as parameters, elements that change over time (for example, the speaker, the gaze of the performers, changes in the appearance of the performers (movement, nodding, smiling, etc.), audio elements (the timing when a certain speaker finishes speaking, changes in the number of speakers, volume), and adjustment of the on-time. In addition, the parameters may include viewer rating data, distribution data, top search keywords of a search engine, etc. As with the role index, the calculation formula for the switching index will be described below using the case where the program is a talk show as an example. An example of a calculation method for the switching index is shown. (Switching index) = Starting a conversation x 0.7 + Presenter behavior x 0.2 + Gazing at the camera x 0.1 + Time adjustment

[0035] Here, the opening of speech is a term for taking into consideration the beginning of speech of a performer, and 10 is substituted for the opening of speech item when the opening of a performer is detected in the cut-out candidate area, and 0 is substituted for the opening of speech item when the opening of a performer is not detected in the cut-out candidate area. The emcee's action is a term for taking into consideration the action of the emcee, and 10 is substituted for the emcee's action item when the action of the emcee is detected in the cut-out candidate area including the emcee, and 0 is substituted for the emcee's action item when the action of the emcee is not detected. The gaze set is a set value of the gaze direction of the performer detected by the detection unit 14. For example, if one other performer is looking at a performer in the cut-out candidate area, 3 is substituted for the gaze set item, if the other two performers are looking at the performer, 6 is substituted for the gaze set item, and if the other three performers are looking at the performer, 10 is substituted for the gaze set item. The appearance time adjustment is the time difference between the appearance time registered in the program database 11 and the actual on-the-fly time.

[0036] The above-mentioned formula for the switching index is an example, and other terms may be added as appropriate. The input values ​​are also an example, and other values ​​may be substituted.

[0037] If the highest value among the calculated switching indexes of each cut-out candidate region exceeds a predetermined threshold, the cut-out region determination unit 15 determines the cut-out candidate region having the highest switching index value as the cut-out region.

[0038] Furthermore, in addition to calculating the switching index, the cut-out region determination unit 15 stores switching inhibition conditions. If the image of the cut-out candidate region having the highest switching index value at the time of switching satisfies the switching inhibition conditions, the cut-out region determination unit 15 maintains the current cut-out region and does not determine a new cut-out region. Alternatively, the cut-out candidate region having the next highest switching index value is determined to be the cut-out region.

[0039] The switching prohibition condition is a rule for preventing switching that may cause discomfort to the viewer, such as when a predetermined time has not elapsed since switching, etc. Furthermore, the conditions include when the cut-out region determination unit 15 is configured to detect a difference between the image of the current cut-out region and the image of the cut-out candidate region, when the image of the new cut-out region is a jump shot of the image of the previous cut-out region, when the image of the new cut-out region is almost the same as the image of the current cut-out region in size, angle, etc., when the image of the new cut-out region is simply increased in people who do not affect the index of the image of the current cut-out region, etc.

[0040] The switching unit 16 cuts out the video of the cut-out region from the program video, and outputs it as a switching video.

[0041] The display control unit 17 displays on the display device 3 the program video, the video of each cut-out candidate region, and the currently output video (switching video).

[0042] The display device 3 is a display that outputs each video output from the display control unit 17. The display device 3 may be a display having not only a display function but also a touch panel function, such as a tablet terminal.

[0043] Next, the operation of the switching device 2 of this embodiment will be described. In the following explanation of the operation, it is assumed that program X is a talk show and that four people, performers A, B, C, and D, appear in program X. It is also assumed that the program information of program X shown in Fig. 2 is registered in the program database 11. It is also assumed that the role index and switching index are calculated using the above-mentioned calculation formulas.

[0044] First, the acquisition unit 10 of the switching device 2 acquires a program video captured by the camera 1. At the opening of the program, it is assumed that the acquired program video is output as a switching video from the switching unit 16 as shown in FIG.

[0045] The recognition unit 12 recognizes the performers in the program video. The performers recognized in the program video are performer A, performer B, performer C, and performer D. The recognition results for each performer are cut out and output to the candidate region selection unit 13.

[0046] The cut-out candidate area selection unit 13 calculates the role index of the recognized performer at every predetermined time. Then, the cut-out candidate area selection unit 15 calculates the switching index of each cut-out candidate area and determines the cut-out area. The switching unit 16 cuts out the image of the cut-out area from the program image and outputs the cut-out image as a switching image. The operations of the cut-out candidate area selection unit 13, the cut-out candidate area selection unit 15, the switching unit 16, and the display control unit 17 at each time will be described below. The cut-out candidate area selected by the cut-out candidate area selection unit 13 is one cut-out candidate area with the largest role index value that exceeds a predetermined threshold (first threshold) for each role index. Furthermore, if there are overlapping cut-out candidate areas, the area including the performer with the next largest index is selected as the cut-out candidate area. Furthermore, the aspect ratio of the cut-out candidate area is set to 16:9, and it is set to one bust-up shot.

[0047] 1. Operation at time (t) (1) Calculation of role index by the cutout candidate region selection unit 13 Assume that the program video at time (t) is as shown in Fig. 4. In the program video of Fig. 4, the cut-out candidate region selection unit 13 calculates each role index of each performer using the detection results of the recognition unit 12 (each performer A, B, C, D and performer A's pronunciation of "Today's guest is Mr. C."), and selects the region including the performer with the highest value of each role index as the cut-out candidate region.

[0048] (1-1) Role Index (Moderator) According to the program database 11, the host is performer A. Then, the role indexes of each performer are as follows: · Role index of performer A (host) = host x 1 =1×1 =1 Performer B's role index (host) = host x 1 =0×1 =0 Performer C's role index (host) = host x 1 =0×1 =0 · Role index of performer D (host) = host x 1 =0×1 =0

[0049] Here, the first threshold value of the role index (host) is set to 0, and the role of performer A with the largest role index (host) value exceeding 0 is regarded as the host, and an extraction candidate region P1 including performer A is selected.

[0050] (1-2) Role indicators (topics of conversation) The recognition unit 12 detects each of the performers A, B, C, and D, as well as performer A's pronunciation of "Today's guest is Mr. C," that performer A is the speaker, and that performer A pronounces the keyword "Mr. C." Using this detection result, the cutout candidate region selection unit 13 calculates the role index (talk axis) of each performer. The role index (talk axis) of each performer is as follows:

[0051] · Role index of performer A (axis of conversation) = speaker x 0.6 + gaze group x 0.3 + keyword x 0.5 + person importance x 0.8 =10×0.6+0×0.3+10×0.5+8×0.8 =6+2+6.4=14.4 · Role index of performer B (axis of conversation) = speaker x 0.6 + gaze group x 0.3 + keyword x 0.5 + person importance x 0.8 =0×0.6+0×0.3+0×0.5+6×0.8=4.8 Participant C's role index (axis of conversation) = speaker x 0.6 + gaze group x 0.3 + keyword x 0.5 + person importance x 0.8 =0×0.6 +0×0.3+0×0.5+10×0.8=8 · Role index of performer D (axis of conversation) = speaker x 0.6 + gaze group x 0.3 + keyword x 0.5 + person importance x 0.8 =0×0.6+0×0.3+0×0.5+5×0.8=4.0

[0052] Here, the first threshold value of the role index (talk axis) is set to 5, and among the role index (talk axis) values ​​that exceed 5, the performer with the largest value is performer A. However, performer A is considered to be the moderator with a role index (host), and cut-out candidate region P1 is set. Next, performer C, who has a large role index (talk axis), is considered to be the axis of the talk, and cut-out candidate region P2 including performer C is selected.

[0053] (1-3) Role Index (Reactor) The recognition unit 12 recognizes each of the performers A, B, C, and D, performer A's pronunciation of "Today's guest is Mr. C," that performer A is the speaker, that performer A pronounces the keyword "Mr. C," and the smile rate of each performer. Using this detection result, the cut-out candidate region selection unit 13 calculates the role index (reactor) of each performer. The role index (reactor) of each performer is as follows:

[0054] · Role index (reactor) of performer A = nod x 0.3 + performer facing the target x 0.2 + keyword x 0.2 + smile rate x 0.2 + person importance x 0.1 =0×0.3+0×0.2+10×0.2+2×0.2+8×0.1 =2+0.4+0.8=3.2 · Role index of performer B (reactor) = nod x 0.3 + performer facing the target x 0.2 + keyword x 0.2 + smile rate x 0.2 + person importance x 0.1 =0×0.3+0×0.2+0×0.2+6×0.2+6×0.1 =1.2+0.6=1.8 · Role index of performer C (reactor) = nod x 0.3 + performer facing the target x 0.2 + keyword x 0.2 + smile rate x 0.2 + person importance x 0.1 =0×0.3+0×0.2+0×0.2+2×0.2+10×0.1 =0.4+1=1.4 · Role index of performer D (reactor) = nod × 0.3 + performer facing the target × 0.2 + keyword × 0.2 + smile rate × 0.2 + person importance × 0.1 =0×0.3+0×0.2+0×0.2+6×0.2+5×0.1 =1.2+0.5=1.7

[0055] Here, the first threshold value of the role index (reactor) is set to 5, and among the role indexes (reactor) exceeding 5, the role of performer B with the largest value is deemed to be reactor, and an extraction candidate region P3 including performer B is selected. The above-mentioned cut-out candidate regions P1, P2, and P3 are shown in FIG.

[0056] (2) Calculation of switching index by the cutout region determination unit 15 The cut-out region determination unit 15 calculates the switching index of each cut-out candidate region, and determines the cut-out candidate region having the highest switching index value among the switching indexes exceeding the second threshold as the cut-out region. At time (t), the detection unit 14 detects the speech of each of the performers A, B, C, D and performer A, that performer A is the speaker, and the behavior and eye contact of performer A (host). It also accumulates the cumulative online time of each performer. Then, the switching indexes of the cut-out candidate regions P1, P2 and P3 at time (t) are as follows:

[0057] Switching index of the cutout candidate region P1 = opening speech x 0.7 + MC behavior x 0.2 + eye contact x 0.1 + (scheduled on-time of performers - cumulative on-time of performers) =10×0.7+10×0.2+0×0.1+(15-0) =7+2+15=24 Switching index of the extraction candidate region P2 = opening speech x 0.7 + MC behavior x 0.2 + eye contact x 0.1 + (scheduled on-time of performers - cumulative on-time of performers) =0×0.7+0×0.2+0×0.1+(15-0)=15 ·Switching index of the cut-out candidate area P3 = Speech × 0.7 + MC operation × 0.2 + Gaze convergence × 0.1 + (Scheduled on-air time of the performer - Cumulative on-air time of the performer) = 0 × 0.7 + 0 × 0.2 + 0 × 0.1 + (10 - 0) = 10

[0058] Here, assuming that the second threshold is 10, the cut-out area determination unit 15 tentatively determines the cut-out candidate area P1, which has the largest switching index among the switching indices exceeding the second threshold, as the cut-out area S among the cut-out candidate areas P1, P2, and P3. Further, it is determined whether the cut-out candidate area P1 meets the switching prohibition condition. Here, assuming that the switching prohibition condition is that a predetermined time (z hours: z < t) has not elapsed since the most recent switching, since z hours have elapsed since the most recent switching, the cut-out candidate area P1 does not meet the switching prohibition condition. Therefore, as shown in FIG. 4, the cut-out candidate area P1 is determined as the cut-out area S.

[0059] The switching unit 16 cuts out the video of the cut-out area S from the program video and outputs it as the switching video.

[0060] The lower part of FIG. 4 is an example of the screen displayed by the display control unit 17 on the display device 3. FIG. 4 shows the control screen displayed on the display device 3 at time (t). The control screen displays the program video captured by the camera 1, the videos of the respective cut-out candidate areas P1, P2, and P3, and the current switching video (the video of the cut-out area S).

[0061] 2. Operations at time (t + m) The operations at time (t + m) when time m has elapsed since time (t) will be described.

[0062] (1) Calculation of the role index by the cut-out candidate area selection unit 13 The cut-out candidate region selection unit 13 calculates each role index of each performer from time (t) to time (t+m) using the detection results of the recognition unit 12. Here, the role indexes do not change significantly from time (t) to time (t+m), and the cut-out candidate region is the same as at time (t), as shown in FIG.

[0063] (2) Calculation of switching index by the cutout region determination unit 15 The cut-out region determination unit 15 calculates the switching index of each cut-out candidate region at time (t+m), and determines the cut-out candidate region having the highest switching index value among the switching indexes exceeding the second threshold as the cut-out region. At time (t+m), the detection unit 14 detects the opening words "Thank you for your time today" of each of the performers A, B, C, D, and C, the behavior of performer A (host), and the gathering of the eyes. In addition, the cumulative online time of each performer is accumulated. Then, the switching indexes of the cut-out candidate region P1, the cut-out candidate region P2, and the cut-out candidate region P3 at time (t+m) are as follows:

[0064] Switching index of the cutout candidate region P1 = opening speech x 0.7 + MC behavior x 0.2 + eye contact x 0.1 + (scheduled on-time of performers - cumulative on-time of performers) =0×0.7+0×0.2+0×0.1+(15-m)=15-m Switching index of the extraction candidate region P2 = opening speech x 0.7 + MC behavior x 0.2 + eye contact x 0.1 + (scheduled on-time of performers - cumulative on-time of performers) =10×0.7+0×0.2+0×0.1+(15-0)=22 Switching index of the cutout candidate region P3 = opening speech × 0.7 + MC behavior × 0.2 + eye contact × 0.1 + (scheduled on-time of performers - cumulative on-time of performers) =0×0.7+0×0.2+0×0.1+(10-0)=10

[0065] Here, assuming the second threshold is 10, the cut-out area determination unit 15 tentatively determines the cut-out candidate area P2, which has the switching index with the largest value among the switching indices whose values exceed the second threshold among the cut-out candidate areas P1, P2, and P3, as the cut-out area S. Further, it is determined whether the cut-out candidate area P2 meets the switching prohibition condition. Here, assuming that the switching prohibition condition is that a predetermined time (z hours: z < m) has not elapsed since the most recent switching, since z hours have elapsed since the most recent switching, the cut-out candidate area P2 does not meet the switching prohibition condition. Therefore, as shown in FIG. 5, the cut-out candidate area P2 is determined as the cut-out area S.

[0066] The switching unit 16 cuts out the video of the cut-out area S from the program video and outputs it as the switching video.

[0067] The lower part of FIG. 5 is an example of the screen displayed by the display control unit 17 on the display device 3. FIG. 5 shows the control screen displayed on the display device 3 at time (t + m). The control screen displays the program video captured by the camera 1, the videos of the respective cut-out candidate areas P1, P2, and P3, and the current switching video (the video of the cut-out area S).

[0068] 3. Operations at time (t + m + n) The operations at time (t + m + n) when n hours have elapsed from time (t + m) will be described.

[0069] (1) Calculation of role indicators by the cut-out candidate area selection unit 13

[0070] The cut-out candidate area selection unit 13 calculates the role indicators of each actor using the detection results of the recognition unit 12 between time (t) and time (t + m + n). Here, as shown in FIG. 6, the role indicators do not change significantly between time (t) and time (t + m + n), and the cut-out candidate areas are assumed to be the same as those at time (t).

[0071] (2) Calculation of switching indicators by the cut-out area determination unit 15

[0072] The cut-out region determination unit 15 calculates a switching index for each cut-out candidate region at time (t+m+n), and determines the cut-out candidate region having the highest switching index value as the cut-out region.

[0073] At time (t+m+n), the detection unit 14 detects the opening lines of each of the performers A, B, C, D, and B, "Mr. C is now...", the actions of performer A (host), and the gathering of gazes. It also accumulates the cumulative online time of each performer. Then, the switching indexes of each of the cut-out candidate regions P1, P2, and P3 at time (t+m) are as follows:

[0074] Switching index of the cutout candidate region P1 = opening speech x 0.7 + MC behavior x 0.2 + eye contact x 0.1 + (scheduled on-time of performers - cumulative on-time of performers) =0×0.7+0×0.2+0×0.1+(15-m)=15-m Switching index of the extraction candidate region P2 = opening speech x 0.7 + MC behavior x 0.2 + eye contact x 0.1 + (scheduled on-time of performers - cumulative on-time of performers) =10×0.7+0×0.2+0×0.1+(10-0)=17 Switching index of the cutout candidate region P3 = opening speech × 0.7 + MC behavior × 0.2 + eye contact × 0.1 + (scheduled on-time of performers - cumulative on-time of performers) =0×0.7+0×0.2+0×0.1+(15-n)=15-m

[0075] Here, assuming the second threshold is 10, the cut-out area determination unit 15 tentatively determines the cut-out candidate area P3, which has the switching index with the largest value among the switching indices whose values exceed the second threshold among the cut-out candidate areas P1, P2, and P3, as the cut-out area S. Further, it is determined whether the cut-out candidate area P3 meets the switching prohibition condition. Here, assuming that the switching prohibition condition is that a predetermined time (z hours: z < n) has not elapsed since the most recent switching, since z hours have elapsed since the most recent switching, the cut-out candidate area P3 does not meet the switching prohibition condition. Therefore, as shown in FIG. 6, the cut-out candidate area P3 is determined as the cut-out area S. The switching unit 16 cuts out the video of the cut-out area S from the program video and outputs it as the switching video.

[0076] Below FIG. 6 is an example of the screen that the display control unit 17 displays on the display device 3. FIG. 6 shows the control screen displayed on the display device 3 at time (t + m + n). The control screen displays the program video captured by the camera 1, the videos of the respective cut-out candidate areas P1, P2, and P3, and the current switching video (the video of the cut-out area S).

[0077] 4. Operations at time (t + n + m + o) The operations at time (t + n + m + o), when time o has elapsed since time (t + m + n), will be described.

[0078] (1) Calculation of role indices by the cut-out candidate area selection unit 13 The cut-out candidate area selection unit 13 calculates the respective role indices of each performer using the detection results of the recognition unit 12 between time (t + m + n) and time (t + n + m + o).

[0079] (1-1) Role index (host) · Role index of performer A (host) = host × 1 = 1 × 1 = 1 · Role index of performer B (host) = host × 1 = 0 × 1 = 0 · Role index of performer C (host) = host × 1 =0×1=0 · Role index of performer D (host) = host x 1 =0×1=0

[0080] Here, the first threshold value of the role index (host) is set to 0, and the role of performer A with the largest role index (host) value exceeding 0 is regarded as the host, and an extraction candidate region P1 including performer A is selected.

[0081] (1-2) Role indicators (topics of conversation) As shown in Fig. 7, the recognition unit 12 detects that each of the performers A, B, C, D and performer D pronounces "I'm a fan of Mr. C," that performer D is the speaker, and that performer D pronounces the keyword "Mr. C." Using this detection result, the cut-out candidate region selection unit 13 calculates the role index (talk axis) of each performer. The role index (talk axis) of each performer is as follows:

[0082] · Role index of performer A (axis of conversation) = speaker x 0.6 + gaze group x 0.3 + keyword x 0.5 + person importance x 0.8 =0×0.6+0×0.3+0×0.5+8×0.8=6.4 · Role index of performer B (axis of conversation) = speaker x 0.6 + gaze group x 0.3 + keyword x 0.5 + person importance x 0.8 =0×0.6+0×0.3+0×0.5+6×0.8=4.8 Participant C's role index (axis of conversation) = speaker x 0.6 + gaze group x 0.3 + keyword x 0.5 + person importance x 0.8 =0×0.6 +0×0.3+0×0.5+10×0.8=8.0 · Role index of performer D (axis of conversation) = speaker x 0.6 + gaze group x 0.3 + keyword x 0.5 + person importance x 0.8 =10×0.6+0×0.3+10×0.5+5×0.8 =6+5+4=15

[0083] Here, the first threshold value for the role index (talk axis) is set to 5, and the performer with the largest role index (talk axis) value exceeding 5 is performer D. Performer D is then deemed to be the talk axis, and a candidate cut-out region P4 containing performer D is selected.

[0084] (1-3) Role Index (Reactor) The recognition unit 12 recognizes the pronunciation of each of the performers A, B, C, D and performer D, "I'm a fan of Mr. C," that performer D is the speaker, that performer D pronounces the keyword "Mr. C," and the smile rate of each performer. Using this detection result, the cut-out candidate region selection unit 13 calculates the role index (reactor) of each performer. The role index (reactor) of each performer is as follows:

[0085] · Role index (reactor) of performer A = nod x 0.3 + performer facing the target x 0.2 + keyword x 0.2 + smile rate x 0.2 + person importance x 0.1 =2×0.3+10×0.2+0×0.2+5×0.2+8×0.1 =0.6+2+1+0.8=4.4 · Role index of performer B (reactor) = nod x 0.3 + performer facing the target x 0.2 + keyword x 0.2 + smile rate x 0.2 + person importance x 0.1 =3×0.3+10×0.2+0×0.2+6×0.2+6×0.1 =0.9+0.2+1.2+0.6=2.9 · Role index of performer C (reactor) = nod x 0.3 + performer facing the target x 0.2 + keyword x 0.2 + smile rate x 0.2 + person importance x 0.1 =0×0.3+10×0.2+0×0.2+8×0.2+10×0.1 =2+1.6+1=4.6 · Role index (reactor) of performer D = nod × 0.3 + performer facing the target × 0.2 + keyword × 0.2 + smile rate × 0.2 + person importance × 0.1 =0×0.3+0×0.2+10×0.2+5×0.2+5×0.1 =2+1+0.5=3.5

[0086] Here, the first threshold value for the role index (reactor) is set to 5, and the role of performer C with the largest role index (reactor) value exceeding 5 is considered to be reactor, and a candidate cut-out region P5 including performer C is selected, as shown in Figure 7. The above-mentioned cut-out candidate regions P1, P4, and P5 are shown in FIG.

[0087] (2) Calculation of switching index by the cutout region determination unit 15 The cut-out region determination unit 15 calculates a switching index for each cut-out candidate region, and determines the cut-out candidate region having the highest switching index value among the switching indexes exceeding the second threshold as the cut-out region.

[0088] The cut-out region determination unit 15 calculates a switching index for each cut-out candidate region at time (t+n+m+o), and determines the cut-out candidate region having the highest switching index value as the cut-out region.

[0089] At time (t+n+m+o), the detection unit 14 detects each of the performers A, B, C, and D, and performer D's opening line of speech, "I'm a fan of A-san," as well as the movements and eye contact of performer A (host). The cumulative online time of each performer is also accumulated. Then, the switching indexes of the cut-out candidate regions P1, P4, and P5 at time (t+m) are as follows:

[0090] Switching index of the cutout candidate region P1 = opening speech x 0.7 + MC behavior x 0.2 + eye contact x 0.1 + (scheduled on-time of performers - cumulative on-time of performers) =0×0.7+0×0.2+0×0.1+(15-m)=15-m Switching index of the cutout candidate region P4 = opening speech × 0.7 + MC behavior × 0.2 + eye contact × 0.1 + (scheduled on-time of performers - cumulative on-time of performers) =10×0.7 + 0×0.2 + 10×0.1 + (15 - 0) =7 + 1 + 15 = 23 ·Switching index of the cut-out candidate area P5 = Speech × 0.7 + MC operation × 0.2 + Gaze convergence × 0.1 + (Scheduled on-air time of the performer - Cumulative on-air time of the performer) =0×0.7 + 0×0.2 + 0×0.1 + (15 - n) = 15 - n

[0091] Here, assuming that the second threshold is 10, the cut-out area determination unit 15 tentatively determines the cut-out candidate area P4, which has the largest switching index among the switching indexes exceeding the second threshold among the cut-out candidate areas P1, P4, and P5, as the cut-out area S. Further, it is determined whether the cut-out candidate area P4 meets the switching prohibition condition. Here, assuming that the switching prohibition condition is that a predetermined time (z hours: z < o) has not elapsed since the most recent switching, since z hours have elapsed since the most recent switching, the cut-out candidate area P4 does not meet the switching prohibition condition. Therefore, as shown in FIG. 7, the cut-out candidate area P4 is determined as the cut-out area S.

[0092] The switching unit 16 cuts out the video of the cut-out area S from the program video and outputs it as the switching video.

[0093] The lower part of FIG. 7 is an example of the screen displayed by the display control unit 17 on the display device 3. FIG. 7 shows the control screen displayed on the display device 3 at time (t + m + n + o). The control screen displays the program video captured by the camera 1, the videos of the respective cut-out candidate areas P1, P4, and P5, and the current switching video (the video of the cut-out area S).

[0094] In this way, if there is a change in the program video for each time, each role index is calculated. If each role index exceeds the first threshold, a cut-out candidate area is selected. The switching index of each cut-out candidate area is calculated. If the switching index exceeds the second threshold and does not meet the switching prohibition condition, the cut-out area is determined and the switching video is determined.

[0095] In this embodiment, in order to automatically determine switching images, a role index is introduced to automatically select cut-out candidate areas, which are areas of images that are candidates for switching images, from images that have been shot by multiple cameramen based on their respective viewpoints. Furthermore, in order to determine one switching image from images of multiple cut-out candidate areas, a switching index, which is the viewpoint of the switcher, is introduced, so that an optimal cut-out area can be determined from multiple cut-out candidate areas, and switching can be automated.

[0096] In the above embodiment, the case where one camera is used to shoot a program has been described. However, this is not limiting, and multiple cameras may be provided. In this case, the program video from each camera may be subjected to the above-mentioned operation for each camera. The advantage of providing multiple cameras is that it is possible to obtain program video in which the performers of the program are shot from different directions, so that the output video finally output as switching video becomes more diverse.

[0097] In addition, the calculated role index and switching index can be used in other data analyses, and further, each parameter from which the role index and switching index are calculated can also be used in other data analyses.

[0098] <First Modification of the Embodiment> A first modified example of the embodiment of the present invention will be described.

[0099] In the above embodiment, for ease of explanation, the cut-out candidate region is a bust shot region including one performer, but the cut-out candidate region is not limited to this.

[0100] The recognition unit 12 performs bone structure determination in addition to face recognition, and is capable of recognizing the bone structure and relative positions of each performer. Therefore, by using each person's bone structure, when there is only one performer, it is possible to set areas corresponding to various shots (FF, BS, WS, KS, etc.) such as long shots that show the entire performer.

[0101] Furthermore, by using the bone structure and positional relationships of each performer, it is possible to set a group shot area including two or more performers, as shown in Fig. 8. For example, the position information of the performers is clustered on the video (image) plane, the performers are grouped into combinations with low variance, and a cut-out candidate area for the group shot of each group is set. Note that, for group shots, it is also possible to avoid images in the selected cut-out candidate areas that are jump shots of each other.

[0102] In this way, rather than selecting an area of ​​a specific shot containing one performer as the cut-out candidate area, it is possible to select the various shots or group shots described above as the cut-out candidate area.

[0103] When such a cut-out candidate area is selected, the role index and switching index may be calculated for each performer, or in the case of a group shot, the role index and switching index may be adjusted according to the number of people included in the area. Furthermore, the cut-out candidate area can be adjusted according to the resolution of the switching video.

[0104] In addition, when adding a group shot area including multiple performers to the cut-out candidate area, a specific switching prohibition condition may be set. For example, if a cut-out area includes one performer A and the image of the next cut-out area is simply an image of performer A and other performers, a switching prohibition condition may be set so that switching is not performed.

[0105] <Modification 2 of the embodiment> In the above-described embodiment, an example in which all the images to be switched are automated has been described. However, there are cases in which the user wishes to change the switching image selected by the switching device 2 at his / her discretion. In this regard, the second modification of the embodiment describes an example in which the images to be switched are automated and the image of the candidate region to be cut out can be determined by the user. 9 is a block diagram of a second modification of the embodiment. In the second modification of the embodiment, a switching device 2 is provided with a switch video selection unit 18.

[0106] When the display device 3 is a display equipped with a touch panel, the switch image selection unit 18 is configured to receive a user's selection of an image desired from among the images of the cut-out candidate area displayed on the display device 3, and to set the image of the cut-out candidate area selected by the user as the switching image as the current switching image, instead of an automatically selected switching image.

[0107] Fig. 10 is a diagram showing a selection of a switching video by a user on the display device 3. Fig. 10 shows a case where the user selects the video of performer A from the video of the cut-out candidate area, instead of the video of performer D currently being output as the switching video. By adopting such a configuration, it is possible to take into account the switching preferences of the user.

[0108] <Modification 3 of the embodiment> A third modification of the embodiment will be described. The switching device 2 can also be realized by a computer having a processor such as a CPU or GPU that operates according to a program. FIG. 11 is a block diagram of the third modification of the embodiment.

[0109] The computer includes a processing unit 100 and a recording unit 110. The processing unit 100 performs overall control of the operation of the switching device 2 based on programs, data, and the like stored in the storage unit 110. The functions of the processing unit 100 can be realized by electronic components, such as a microprocessor such as a CPU or a GPU, an ASIC, and an IC memory. The processing unit 100 includes, as main functional processes, an acquisition process 101, a recognition process 102, a cut-out candidate region selection process 103, a detection process 104, a cut-out region determination process 105, a switching process 106, a display control process 107, and a switch image selection process 108. The acquisition process 101 corresponds to the acquisition unit 10, the recognition process 102 corresponds to the recognition unit 12, the cut-out candidate area selection process 103 corresponds to the cut-out candidate area selection unit 13, the detection process 104 corresponds to the detection unit 14, the cut-out area determination process 105 corresponds to the cut-out area determination unit 15, the switching process 106 corresponds to the switching unit 16, the display control process 107 corresponds to the display control unit 17, and the switch image selection process 108 corresponds to the switch image selection unit 18.

[0110] Programs for implementing various functions of the computer, data used during execution of these programs, etc. are stored in advance or temporarily stored each time processing is performed in the recording unit 110. The storage unit 110 can be realized by, for example, a RAM, a ROM, an IC memory such as a flash memory, a magnetic disk such as a hard disk, an optical disk such as a CD-ROM or a DVD, etc.

[0111] <Fourth Modification of the Embodiment> In the above-described embodiment, the role index is used to select the cut-out candidate regions, and the switching index of each cut-out candidate region is used to determine the cut-out region. However, a calculation formula for a single index that combines the role index and the switching index may be used.

[0112] When creating a calculation formula for one index, it is possible to delete terms that overlap between the role index and the switching index, or to add a term that absorbs errors that occur when the role index and the switching index are combined.

[0113] When a switching video is determined based on such a single index, the process of selecting a cut-out candidate region is eliminated, and one cut-out region for the switching video is determined.

[0114] <Fifth Modification of the Embodiment> In the above embodiment, an example in which a switching index is calculated at a predetermined time interval has been described. However, it is not always possible to calculate a switching index at an appropriate timing. Therefore, in the fifth modification of the embodiment, an example in which a switching index is calculated when a video satisfies a certain condition will be described.

[0115] In the fifth modification of the embodiment, in order to calculate the switching index at an appropriate timing, a switching index calculation trigger condition that serves as a trigger for calculating the switching index is provided. Examples of the switching index calculation trigger condition include the following.

[0116] (1) A sudden change in the audio level of the program video For example, when a performer finishes talking, the audio level of the program video drops suddenly. In such a case, the currently output switching video is often switched to another video. Meanwhile, when such a change occurs, the role index of each cut-out candidate region should also change significantly. Therefore, the drop in the audio level to a specified threshold is used as a trigger to calculate the switching index of each cut-out candidate region and determine a new cut-out region.

[0117] In addition, when the overall or individual audio level rises, such as when multiple people make a call or when one person suddenly speaks loudly, the currently output switching video is often switched to another video. On the other hand, when such a change occurs, the role index of each cut-out candidate region should also change significantly. Therefore, the drop in the audio level to a certain threshold is used as a trigger to calculate the switching index of each cut-out candidate region and determine a new cut-out region.

[0118] (2) Sudden changes in movement during program footage When a TV program performer suddenly moves, the video often switches to that performer's video. When such a change occurs, the role index of each cut-out candidate region should also change significantly. Therefore, when a sudden change in the video (such as a sudden change in the motion vector) is detected, the switching index of each cut-out candidate region is calculated and a new cut-out region is determined.

[0119] By calculating a switching index when such a switching index calculation trigger condition is satisfied, a switching video can be selected at an appropriate timing.

[0120] In addition, when the switching index calculation trigger condition is satisfied and the switching index of each cut-out candidate region is calculated, the switching index value may be low overall, and the cut-out region may not be determined. For example, when one performer finishes talking, other performers do not speak. In such a case, a predetermined video of a host or a group shot may be used as the switching video.

[0121] <Sixth Modification of the Embodiment> A sixth modified example of the embodiment of the present invention will now be described. In the above embodiment, an example has been described in which the region with the largest role index value is selected for each cut-out candidate region. However, as an exception, a pattern may be provided in which a region is selected as a cut-out candidate region regardless of the role index value.

[0122] For example, if a group shot area including two performers in a group of four has a high role index and that area is selected as a candidate area for cut-out, the group shot areas of the remaining two performers may also be selected as candidate areas for cut-out, regardless of the role index values.

[0123] In addition, if a group shot (e.g., four people) of guests seated on the upper level of a tiered stage has a high role index and that area is selected as a candidate area for clipping, the area of ​​a group shot (e.g., four people) of guests seated on the lower level of the tiered stage may also be selected as a candidate area for clipping, regardless of the value of the role index.

[0124] In addition, when one performer bursts out laughing and other performers also laugh, it is possible to select not only the candidate cut-out region that includes only the performer who laughed, which has a high role index, but also the entire region that includes the performer who laughed and the other performers who laughed, as the candidate cut-out region.

[0125] Although the present invention has been described above with reference to preferred embodiments, the present invention is not necessarily limited to the above-described embodiments, and can be modified and embodied in various ways within the scope of the technical concept thereof. [Explanation of symbols]

[0126] 1 Camera 2 Switching Device 3 Display device 10 Acquisition Department 11 Program Database (Program DB) 12 Recognition part 13 Extraction candidate region selection section 14 Detection section 15 Extraction area determination section 16 Switching section 17 Display control section 18 Switch image selection section 100 Processing section 110 Storage section

Claims

1. An acquisition unit for acquiring video material of the content; a recognition unit for recognizing an image or a voice of a person in the video material; a cut-out candidate region selection unit that calculates a role index, which is an index of a role of the object in the content, based on an image or sound of the recognized object, and selects at least one cut-out candidate region, which includes at least one object within a region, for cutting out a video from the video material, using the role index; a detection unit that detects a video or audio change that occurs over time in an object in the video material; a cut-out region determination unit that calculates a switching index, which is an index of switching timing in each of the cut-out candidate regions, from a detection result of a video or audio change that changes over time in the object in the video material, and determines a cut-out region from each of the cut-out candidate regions using the switching index of each of the cut-out candidate regions; a switching unit that cuts out an image of a cut-out region from the image material and switches the currently output image to the cut-out image; An automatic switching device comprising:

2. The role index has at least one parameter including the importance of the object, the presence or absence of a sound produced by the object, a keyword uttered by the object, voice information of the object, a gaze group of the object, and a change in the external shape of the object; The cut-out candidate region selection unit selects the cut-out candidate region so as to include within the region an object in the video material whose role index exceeds a first predetermined value.

2. The automatic switching device of claim 1.

3. the switching index has at least one of the following parameters: pronunciation or voice information of an object that changes over time, a motion of an object that changes over time, a line of sight direction of an object that changes over time, and an on-time of an object; The cut-out region determination unit determines, as the cut-out region, a cut-out candidate region having a highest switching index that exceeds a second predetermined value.

3. An automatic switching device according to claim 1 or 2.

4. On the computer, An acquisition function for acquiring video material of the content; A recognition function that recognizes the image or sound of an object in the video material; a cut-out candidate region selection function that calculates a role index, which is an index of a role of an object in the content, based on an image or sound of a recognized object, and selects at least one cut-out candidate region for cutting out a video from the video material, the cut-out candidate region including at least one object using the role index; a detection function for detecting a change in a video or audio of an object in the video material over time; a cut-out region determination function that calculates a switching index, which is an index of switching timing in each of the cut-out candidate regions, from a detection result of a video or audio change that changes over time in the object in the video material, and determines a cut-out region from each of the cut-out candidate regions using the switching index of each of the cut-out candidate regions; a switching function for cutting out an image of a cut-out area from the image material and switching the currently output image to the cut-out image; A program to achieve this.

5. The computer Obtaining video material for content, Recognizing the image or sound of an object in the video material; calculating a role index, which is an index of a role of the object of the content, based on an image or sound of the recognized object, and using the role index to select at least one cut-out candidate region for cutting out a video from the video material, the cut-out candidate region including one or more objects within the region; Detecting a video or audio change that occurs over time in an object in the video material; calculating a switching index, which is an index of switching timing in each of the cut-out candidate regions, from a detection result of a video or audio change that changes over time in the object in the video material; and determining a cut-out region from each of the cut-out candidate regions using the switching index for each of the cut-out candidate regions; A video of the cut-out area is cut out from the video material, and the currently output video is switched to the cut-out video. Automatic switching method.

Images