Device and method

Abstract

The purpose of the present invention is to provide a device and a method with which a break that does not hinder viewing of a video can be determined.　A video distribution server 100 comprises: an acquisition unit 101 that acquires frames constituting a video; and a selection unit 102 that selects, among the frames, frames (a plurality of specific frames) serving as shot division points, on the basis of feature changes between adjacent frames, by using a shot transition detection model based on TransNetV2, and determines, as a break, a frame serving as a shot division point satisfying a break condition among the selected frames (the plurality of specific frames). In the present disclosure, the break is a recommended insertion position for an advertisement, but may be for something other than an advertisement. The break also may be a position indicating a break which simply indicates a change in the content of the video.

Description

Apparatus and method

[0001] The present invention relates to an apparatus and method for determining segments of a moving image.

[0002] Patent Document 1 describes that when the video advertisement playback time is reached, an advertisement is read and distributed. That is, the device described in Patent Document 1 includes means for playing a video program file transmitted from a program distribution server, means for measuring the playback time of the video program file, means for pausing playback of the video program file when the advertisement insertion time is reached, means for outputting a video advertisement file distribution request to the advertisement distribution server, means for playing the transmitted video advertisement file, and means for resuming playback of the video program file when playback of the video advertisement file is completed.

[0003] Japanese Patent Application Laid-Open No. 2003-244677

[0004] However, since advertisements are not inserted with consideration given to the content of the video, there are cases where advertisements are viewed during important scenes.

[0005] Therefore, an object of the present invention is to provide an apparatus and method that can determine divisions that do not interfere with the viewing of a video.

[0006] The present invention comprises an acquisition unit that acquires multiple frames that make up a moving image, a selection unit that selects multiple specific frames using a feature change detection model that has learned feature changes between frames, and a determination unit that selects, from the specific frames, specific frames that satisfy a division condition as divisions.

[0007] According to the present invention, it is possible to determine breaks that do not interfere with the viewing of a video.

[0008] FIG. 1 is a diagram illustrating a system configuration of a video distribution system according to the present disclosure. FIG. 2 is a block diagram illustrating a functional configuration of a video distribution server 100 according to the present disclosure. FIG. 3 is an explanatory diagram illustrating a first approach (video analysis approach) for determining an advertisement insertion position. FIG. 4 is a diagram illustrating an overview of a process for detecting shot division points using TransNetV2. FIG. 5 is a diagram illustrating an advertisement insertion recommendation position determined by the video analysis approach. FIG. 6 is an explanatory diagram illustrating a second approach (sound analysis approach) for determining an advertisement insertion position. FIG. 7 is a flowchart illustrating a process for performing scene division in the operation of the video distribution server 100 according to the present disclosure. FIG. 8 is a flowchart illustrating a process for performing shot division and scene division in the operation of the video distribution server 100 according to the present disclosure. FIG. 9 is a flowchart illustrating a process for determining an advertisement insertion position based on sound analysis results. FIG. 10 is a flowchart illustrating an approach using sound analysis. FIG. 11 is a diagram illustrating management of processing numbers and frame numbers. FIG. 12 is a diagram illustrating an example of a hardware configuration of a video distribution server 100 according to an embodiment of the present disclosure.

[0009] The present disclosure will be described with reference to the accompanying drawings. Whenever possible, the same parts are designated by the same reference numerals and redundant description will be omitted.

[0010] FIG. 1 is a diagram illustrating the system configuration of a video distribution system according to the present disclosure. As shown in the diagram, a video distribution server 100 distributes videos to a user terminal 200. Advertisements are inserted between frames that satisfy the ad insertion conditions in the distributed video. The video distribution server 100 detects frames that satisfy the conditions for inserting an advertisement and enables distribution of a video with an advertisement inserted before or after that frame. In the present disclosure, the video distribution server 100 is configured to determine the ad insertion position in a video and distribute the video, but the distribution process is not essential. It is sufficient for the video distribution server 100 to be configured as a device that determines the segmentation of the video, and a separate video distribution device may be configured to distribute the video using the determined segmentation.

[0011] 2 is a block diagram showing the functional configuration of the video distribution server 100 of the present disclosure. As shown in the figure, the video distribution server 100 includes an acquisition unit 101, a selection unit 102, a sound analysis unit 103, a presentation unit 104, a working data storage unit 105, a video DB 106, and a video distribution unit 107.

[0012] The acquisition unit 101 is a part that acquires all frames that make up a moving image from the moving image DB 106 and the working data storage unit 105 .

[0013] The selection unit 102 is a part that selects frames that satisfy a division condition from all frames of the acquired video. The selection unit 102 selects shot division points (specific frames) using a shot transition detection model such as TransNetV2 as the division condition. In this disclosure, a shot division point refers to a frame where a shot changes. Also, in this disclosure, a shot refers to a change in a captured screen in a video, and refers to a change in a camera or a scene captured by a camera.

[0014] Furthermore, the selection unit 102 selects frames that will be scene division points by taking into account the time series, using a relationship detection model that detects the mutual relationship between multiple frames, such as Trans4mer, for the multiple frames selected as shot division points.

[0015] In this disclosure, TransNetV2 and Trans4mer are used, but the deep learning models are not limited to these convolution neural networks (CNNs), and other deep learning models may also be used. TransNetV2 is a deep learning model (CNN) that detects changes and transitions in local features in videos, and Trans4mer is a deep learning model (CNN) that uses a self-attention mechanism to detect correlations between multiple images (frames in this disclosure). In this disclosure, the deep learning models are not limited to these models, and other deep learning models with similar functions may also be applied.

[0016] Furthermore, the selection unit 102 determines the frame into which a predetermined image or video such as an advertisement should be inserted based on the selected frame and the frame acquired by the sound analysis unit 103, and determines the frame that will be the final insertion position.

[0017] The sound analysis unit 103 is a part that analyzes time periods that include sounds such as conversations from the video acquired by the acquisition unit 101 and acquires the time periods.

[0018] The presentation unit 104 is a part that presents to an operator or the like the advertisement insertion recommended position that has been determined as a scene break by the selection unit 102. The presentation is performed by displaying or outputting to an external terminal. Note that although the recommended position is referred to as the recommended position, the position may also be a determined position.

[0019] The working data storage unit 105 stores frame numbers that are candidates for advertisement insertion positions, the frames (images), and processing numbers as working information. As will be described later, the processing numbers indicate the chronological order of the frames. This working information may also be stored in the video DB 106.

[0020] The video DB 106 is a part that stores videos to be distributed, information indicating the insertion position of advertisements (for example, frame numbers), and advertisements. The information indicating the insertion position of this video DB 106 is information that has finally been determined as a break and is information diverted from the working data storage unit 105.

[0021] The video distribution unit 107 distributes videos and advertisements stored in the video DB 106. When distributing, the video distribution unit 107 refers to insertion positions stored in the video DB 106, inserts advertisements into the positions, and distributes the advertisements.

[0022] Next, a method for determining the commercial insertion position, which is a division, will be described below. Fig. 3 is an explanatory diagram showing a first approach (shot division point analysis approach) for determining the commercial insertion position.

[0023] FIG. 3A is a diagram showing frames constituting a moving image. In the present disclosure, the frames constituting a moving image are configured at 30 frames / s. Frames f1 to f7 are frames showing the same or almost the same image. Frames f8 to f10 are frames of an image different from frame f1 and the like. Frame fx is also a frame of a different image. The acquisition unit 101 acquires all frames constituting these moving images. The selection unit 102 performs shot transition detection on all these frames using TransNetV2, and selects the detected frames as shot division point frames.

[0024] In the figure, the selection unit 102 selects frame f8 as the shot division point in TransNetV2, taking into consideration the frames before and after the target frame.

[0025] 3B shows frames selected as shot division points. As shown in the figure, frames f1, f8, fx, and fx2 have been selected. The selection unit 102 then performs scene division point detection on these frames using Trans4mer, and selects the detected frames.

[0026] In the figure, the selection unit 102 determines that frames f1 and f8 are related based on the time series information of the frames, but determines that frames f8 and fx are not related, using scene transition detection by Trans4mer. In this case, Trans4mer detects the following frame, frame fx, as an unrelated frame.

[0027] 3C shows frames indicated by scene division points that leave only the first frame f1 of the scene and frame fx after frame f8, etc. In this way, frames where there is a large change in the frame (scene) are detected as unrelated.

[0028] Here, an overview of the shot division point detection process by TransNetV2 will be described. FIG. 4 is a diagram showing an overview of the shot division point detection process by TransNetV2. As shown in the figure, TransNetV2 uses a sliding window W to identify several frames (two frames in the figure) before and after the target frame T1 (the target frame for feature change determination). TransNetV2 then uses these frames to output a score indicating the possibility that the target frame T1 is a frame corresponding to a shot division point. This score is expressed between 0 and 1 and indicates the possibility that it corresponds to a shot division point.

[0029] If the score is equal to or greater than a threshold value (e.g., 0.7), the target frame T1 can be determined to be a shot division point. Figures 4(a) to 4(d) show that as the target frame T1 is shifted to the target frame T4, the sliding window W (i.e., the frames before and after the target frame T) also shifts. In Figures 4(a) to 4(c), a value equal to or less than a threshold value, such as 0.1, is output as the score for the target frames T1 to T3, and in Figure 4(d), a score (e.g., 0.9) is output that indicates that the target frame T4 is a shot division point, and the target frame T4 is determined to be a shot division point.

[0030] FIG. 5 shows an overview of the process of detecting scene division points using Trans4mer. As shown in the figure, frames that serve as shot division points are arranged in chronological order. Trans4mer uses sliding windows W1 and W2 of different sizes to output scores indicating the likelihood that target frames T and T1 correspond to scene division points. FIG. 5( a) shows an overview of the process using the sliding window W1. The sliding window W1 identifies the target frame T (the frame whose relevance is to be determined) and two frames before and after the target frame T. FIG. 5( b) shows an overview of the process using the sliding window W2. The sliding window W2 identifies one frame before and one frame after the target frame T. The sliding windows W1 and W2 are windows of different sizes. That is, the number of frames identified by the sliding windows W1 and W2 is different.

[0031] Trans4mer uses sliding windows W1 and W2 of different sizes to output scores indicating the likelihood that each corresponds to a scene division point for a target frame T. The selection unit 102 selects a target frame that corresponds to a scene division point from among the multiple target frames T, based on a score obtained by adding up the scores of the sliding windows W1 and W2. In the present disclosure, the selection unit 102 shifts each of the sliding windows W1 and W2 by one frame (i.e., shifts the target shot division point and the frames before and after it), and selects, as scene division points, the frames with the highest G scores from among the multiple shot division points.

[0032] Note that the size of the sliding window in both Figures 4 and 5 is an example. Figure 4 is small for convenience of drawing, but in reality, it is better to use five frames before and after the target frame T1, etc. Also, as in Figure 5, sliding window W1 uses five frames before and after the target frame T, etc., and sliding window W2 uses three frames before and after the target frame T, etc. In the present disclosure, sliding window W1 identifies a large number of frames using sliding window W2.

[0033] The TransNetV2 and Trans4mer are trained in advance using training videos and shot division points and scene division points specified by an operator using the training videos. The shot division points and scene division points may differ depending on the type of video. For example, the content expressed in frames may differ between live-action dramas and animations, so it is best to train them accordingly.

[0034] TransNetV2 is trained so that the score for a specified target frame (shot division point, target frame for feature change determination) taking into account the frames before and after it becomes 1. Trans4mer is trained so that the score for a specified target frame (scene division point, target frame for relevance determination) for sliding windows W1 and W2 taking into account the frames before and after it becomes 1.

[0035] 6 is a diagram showing recommended positions for advertisement insertion determined by a video analysis approach. As shown in the figure, frames f1, fx, and fy1 are recommended positions for advertisement insertion. Note that the positions where images or videos such as advertisements are actually inserted are before these frames. The frame numbers of the frames selected in this way are presented by the presentation unit 104 as recommended positions for advertisement insertion.

[0036] 7A and 7B are explanatory diagrams showing a second approach (sound analysis approach) for determining an advertisement insertion position. Fig. 7A shows what kind of sound overlaps with each frame.

[0037] The sound analysis unit 103 performs sound analysis on the sound data included in the video using sound analysis AI, and can analyze what types of sounds are included in each time period, such as time periods when music is playing, time periods when people are talking, time periods when there is no sound, and other time periods. This sound analysis AI is a known technology. For example, an algorithm called inSpeechSegmenter is known.

[0038] The selection unit 102 determines whether to exclude a frame that the sound analysis unit 103 has determined to contain the sound of conversation from candidates for an insertion position.

[0039] 7B is a diagram showing frames that have been removed from the candidates for insertion positions based on sound analysis. As shown in the figure, if the selection unit 102 determines that the selected frames f1, fx, and fy1 are frames in which conversation occurs, it performs processing to remove the frames from the candidates for insertion positions. In the present disclosure, frame fy1 is determined to be a frame in which conversation occurs, and is therefore removed from the candidates for insertion positions.

[0040] Next, the operation of the video distribution server 100 of the present disclosure will be described with reference to Figs. 8 to 10. Fig. 8 is a flowchart showing the process of dividing shots and dividing scenes in the operation of the video distribution server 100 of the present disclosure. Fig. 9 is a flowchart showing the process of determining an advertisement insertion position based on the sound analysis result. Fig. 10 is a flowchart showing an approach using sound analysis.

[0041] First, a description will be given using FIG. 8 . The acquisition unit 101 acquires all frames (e.g., n frames) constituting a video from the video DB 106 (S101). Then, the selection unit 102 extracts the i-th frame from all frames, setting the initial value i = 1, and determines whether i < n (S102). If the result of step S102 is YES, the selection unit 102 extracts x frames before and after the first frame (i = 1, i is the frame number) for the n-th frame. The selection unit 102 inputs this target frame (frame number i) and x frames before and after it into TransNetV2, which outputs a score indicating the possibility that frame i (the target frame for feature change determination) is a shot division point. Note that if x previous frames cannot be obtained, such as when i = 1, a black frame is used as a dummy frame. Note that similarly, if x subsequent frames cannot be obtained, a dummy frame is used.

[0042] The selection unit 102 then determines whether the TransNetV2 score is greater than the threshold value a (S103). If the result is YES, the selection unit 102 stores the (i+1)th frame and its frame number in the working data storage unit 105 (S104). The selection unit 102 then increments i=i+1 and processes the next frame number (S105). These steps from S102 to S105 are repeated for all frames. The working data storage unit 105 stores m selected frames (i.e., frames determined to be shot division points) and their frame numbers.

[0043] Next, the acquisition unit 101 acquires m frames from the working data storage unit 105 (S106). The selection unit 102 then arranges the acquired m frames in chronological order and performs a process to determine which frames are scene division points. It is assumed that the selection unit 102 assigns new processing numbers to each frame and frame number in this process. As shown in FIG. 11A, the working data storage unit 105 stores data for processing, and stores processing numbers, frame numbers, and frames (images) in association with each other. The selection unit 102 assigns processing numbers prior to similarity determination and performs a determination process on the scores output by Trans4mer according to the processing numbers thus managed.

[0044] The selection unit 102 then selects a frame, sets the initial value j = 1, and determines whether j < m (S107). If the result of step S107 is YES, the selection unit 102 inputs the frame with the first processing number (j = 1, j is the processing number) (frame j is the frame to be determined for relevance judgment) and y frames before and after that frame into Trans4mer, outputting the score of the first (jth) frame. Similarly, the selection unit 102 inputs the frame with the first (jth) processing number (frame j is the frame to be determined for relevance judgment) and z frames before and after that frame into Trans4mer, and calculates the score of the first (jth) frame. The selection unit 102 then sums the calculated scores (S108). In the present disclosure, y > z.

[0045] The selection unit 102 stores the first (jth) frame, its frame number, and the total score in the working data storage unit 105 (S109). Then, the selection unit 102 increments j = j + 1 to process the frame with the next processing number (S110). These steps S107 to S109 are repeated for the m frames (frames determined to be shot division points) acquired in step S106. The working data storage unit 105 stores p frames, their frame numbers, and scores. When all steps are completed (S107: NO), the selection unit 102 selects the top G frames as scene division points (S111).

[0046] In some cases, the intervals between the G frames selected as scene division points are short. In such cases, it is preferable to select frames that are spaced apart by a predetermined time or more.

[0047] Next, the process of acquiring frames without conversation will be described. Fig. 9 is a flowchart showing the process of acquiring frames based on a comparison with frames with conversation acquired by another process. The acquisition unit 101 acquires q frames from the working data storage unit 105 (S117). These q frames are assigned new processing numbers in order, starting from 1. Fig. 11(b) shows a table with processing numbers assigned to them, which are used when processing shot division points and scene division points for comparison with sound.

[0048] The selection unit 102 determines whether s≦q for the frame with processing number s, with an initial value of s=1 (S118). If the result is YES, the selection unit 102 determines whether the frame contains no conversation (S118, S119). The selection unit 102 determines whether the frame contains conversation by comparing the frame with a conversation-containing frame (frame number) obtained from a separately acquired voice analysis result.

[0049] If the frame does not contain any conversation, the selection unit 102 stores the frame and its frame number in the working data storage unit 105 (S120).

[0050] The selection unit 102 sets the processing number s=s+1 and makes a judgment on the next frame (S121).

[0051] By repeatedly performing steps S118 to S120, the selection unit 102 can acquire, for each scene, a position that does not interrupt the conversation as the advertisement insertion position. In the present disclosure, the presentation unit 104 presents the frame and frame number stored in the working data storage unit 105 to the operator as the advertisement insertion recommended position (S122). The operator can confirm the position. The operator can correct the insertion position (frame number) as necessary by operating the presentation unit 104.

[0052] Next, a process for determining frames based on sound analysis will be described. This process illustrates a method for acquiring frames to be compared in the process S119. FIG. 10 is a flowchart illustrating the process for determining frames based on sound analysis according to the present disclosure. The acquisition unit 101 acquires sound data from a video (S201). The sound analysis unit 103 acquires frames of conversational portions from the sound data superimposed on the video acquired by the acquisition unit 101 using a known algorithm, such as voice analysis AI. That is, the sound analysis unit 103 samples the sound data and determines whether the sound is conversation, music, or other sounds in sampling units. Here, the sampling unit is 22,000 Hz, but is not limited to this. The sound analysis unit 103 then selects frames from a time period containing conversation (S202). The sound analysis unit 103 stores the selected frames in the working data storage unit 105 (S203).

[0053] In this way, it is possible to identify frames where advertisements will be inserted. The advertisement insertion positions identified in the present disclosure can be inserted without overlapping video scenes or conversations and without affecting video viewing.

[0054] In the present disclosure, it is important that the selection unit 102 selects at least shot division points, and that it selects, as a division point, a shot division point that satisfies a predetermined condition from the selected shot division points, for example, that the shot division point does not include sound.

[0055] Next, the effects of the video distribution server 100 of the present disclosure will be described. The video distribution server 100 of the present disclosure includes an acquisition unit 101 that acquires frames constituting a video, and a selection unit 102 that uses a shot transition detection model based on TransNetV2 to select frames (plurality of specific frames) that serve as shot division points based on changes in features between adjacent frames among the frames, and determines, as a division point, a frame from among the selected frames (plurality of specific frames) that serves as a shot division point and satisfies a division condition. In the present disclosure, a division point refers to a position recommended for inserting an advertisement, but may be something other than an advertisement. Alternatively, the division point may simply refer to a position indicating a division that marks a change in the content of the video.

[0056] The division conditions in the present disclosure include whether or not a frame that is a shot division point contains a sound that is not suitable as a division point, whether or not there is a scene change, and so on.

[0057] With this configuration, the video distribution server 100 can select frames that separate video segments. These segments can be used to insert advertisements or other information. Furthermore, by determining the video segments, various video distribution services can be provided that utilize these segments.

[0058] In the present disclosure, a selection unit 102 inputs a target frame (target frame T in FIG. 3 , which corresponds to the target frame for feature change determination) and at least one frame located before or after the target frame (target frame for feature change determination) from among multiple frames (specific frames) that serve as shot division points into TransNetV2 (a feature change detection model), and selects a frame (specific frame) that will serve as a shot division point based on the score (output) output. If the score is equal to or greater than a threshold, the selection unit 102 selects the target frame as a shot division point.

[0059] This configuration allows us to select frames that serve as shot point division points using TransNetV2, which allows us to detect shot transitions with higher accuracy than by comparing frames and determining their similarity.

[0060] The selection unit 102 further uses Trans4mer (relevance detection model) that has learned the relevance between frames to select, from among multiple frames (specific frames) detected using TransNetV2, frames that are not relevant to one frame (frame for relevance determination) as scene division points (non-relevant frames).

[0061] If the non-related frame satisfies the delimiter condition, the selection unit 102 selects it as a delimiter.

[0062] This configuration allows the detection of frames that are scene division points by using Trans4mer, which detects unrelated frames (frames where a scene change occurs) from among multiple frames selected as shot division points, making it possible to detect frames that are scene division points, thereby detecting appropriate positions for inserting advertisements.

[0063] The specific process for detecting scene division points is as follows: The selection unit 102 inputs a target frame T (relevance determination target frame) identified by a sliding window W1 from a plurality of frames (specific frames) arranged in chronological order, and N frames located before and after the target frame, into Trans4mer (relevance detection model), and selects a frame (specific frame) based on the score (first output).

[0064] Similarly, the selection unit 102 inputs a target frame T (relevance determination target frame) and M frames located before and after the target frame, identified by the sliding window W2, from among multiple frames arranged in chronological order, into Trans4mer (a relevance detection model) and obtains a score (second output) as its output. The selection unit 102 sums the scores (first output and second output) obtained using the sliding windows W1 and W2 and selects a frame based on the summed score. In the present disclosure, M is assumed to be smaller than N.

[0065] This configuration allows accurate determination of video scene division points using a relevance detection model such as Trans4mer. By changing the number of frames specified using sliding windows W1 and W2, long and short scenes can be effectively detected. While this disclosure uses both sliding windows W1 and W2, this is not a limitation, and scores using either one of the windows may be used.

[0066] Furthermore, the selection unit 102 obtains scores from the relevance detection model for all of the shot division points (specific frames) each time the target frame (relevance determination target frame) is shifted and changed. Then, the selection unit 102 selects a frame (non-relevant frame) that will be a scene division point from the shot division points (specific frames) based on the highest score (highest information) among the scores (information based on the first output and the second output) for all of the shot division points (specific frames).

[0067] The Trans4mer score indicates the reliability of a scene change, and the top G frames with the highest scores indicate frames with low relevance to the frames before and after them, which indicates that they are suitable frames as scene segmentation points.

[0068] Generally, in videos such as dramas, scenes may change to show alternate characters during conversations. When the above process is used, if the characters change repeatedly within a certain period of time (for example, the duration of a conversation), it may be appropriate to group these scenes together as a single scene to determine the division point. Trans4mer can also appropriately group these scenes together and determine the division point as a scene division point.

[0069] The video distribution server 100 of the present disclosure further includes a sound analysis unit 103 that analyzes the sound generation state in the video. If the frame of the acquired shot division point and the sound generation state coincide with each other, the selection unit 102 does not select the frame of the shot division point as a division point. For example, the sound generation state may be the state of conversation.

[0070] If the frame contains dialogue, it can be determined that it is in the middle of a scene or a scene, and is not a suitable frame for dividing a scene. In the present disclosure, by not determining such shot division points and scene division points as division points, it is possible to prevent interruptions to viewing due to the insertion of advertisements, etc. In addition to dialogue, music and other sounds intended to prevent the scene from changing can also be included.

[0071] The device and method of the present disclosure have the following configuration.

[0072] [1] An apparatus comprising: an acquisition unit that acquires a plurality of frames that constitute a moving image; a selection unit that selects a plurality of specific frames from the plurality of frames using a feature change detection model that has learned feature changes between frames; and a determination unit that selects, from the plurality of specific frames, a specific frame that satisfies a delimitation condition as a delimiter.

[0073] [2] The device according to [1], wherein the segmentation condition is at least one of a condition based on whether the specific frame includes a sound that is not suitable as a segmentation, or a condition based on whether the specific frame indicates a scene change.

[0074] [3] The device according to [1] or [2], wherein the selection unit inputs a target frame for feature change determination, which is one frame extracted from the plurality of frames, and at least one other frame positioned before or after the target frame for feature change determination, into the feature change detection model, and selects the plurality of specific frames based on the output thereof.

[0075] [4] The device described in [3], wherein the selection unit obtains an output from the feature change detection model each time the target frame for feature change determination is shifted and changed, and selects the target frame for feature change determination as the specific frame if the output from the feature change detection model satisfies an acquisition condition.

[0076] [5] The device according to [4], wherein the selection unit further uses a relevance detection model that has learned the relevance between frames to select, from the plurality of specific frames, specific frames that are not relevant to the frame for which relevance is to be determined as non-relevant frames.

[0077] [6] The device described in [5], wherein the selection unit inputs the relevance determination target frame, which is one specific frame extracted from the plurality of specific frames arranged in chronological order, and N other specific frames arranged before and after the relevance determination target frame, into the relevance detection model, and selects the non-relevant frame from the plurality of specific frames based on a first output thereof.

[0078] [7] The device described in [6], wherein the selection unit further inputs the frame to be determined for relevance judgment and M other specific frames located before and after the frame to be determined for relevance judgment, which are extracted from the plurality of specific frames arranged in chronological order, into the relevance detection model, and selects the non-relevant frame from the plurality of specific frames based on a second output and the first output, which are outputs of the relevance detection model, and M is smaller than N.

[0079] [8] The device described in [7], wherein the selection unit obtains outputs from the relevance detection model for all of the plurality of specific frames each time the frame to be determined for relevance is shifted and changed, and selects the non-relevant frame from the specific frames based on higher-order information among information based on the first output and the second output for all of the plurality of specific frames.

[0080] [9] The device according to any one of [1] to [8], further comprising a sound analysis unit that analyzes a sound generation state in the video, wherein the determination unit does not select the specific frame as the delimiter if the specific frame includes the sound generation state.

[0081]

[10] A method comprising: an acquisition step of acquiring a plurality of frames constituting a moving image; a selection step of selecting a plurality of specific frames from the plurality of frames using a feature change detection model that has learned feature changes between frames; and a determination step of selecting, as a segment, a specific frame from the plurality of specific frames that satisfies a segmentation condition.

[0082] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (e.g., via wire, wirelessly, etc.) and these multiple devices. The functional block may also be realized by combining the single device or multiple devices with software.

[0083] Functions include, but are not limited to, judgment, determination, assessment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0084] For example, the video distribution server 100 according to an embodiment of the present disclosure may function as a computer that performs processing of the video distribution method of the present disclosure. Fig. 12 is a diagram illustrating an example of the hardware configuration of the video distribution server 100 according to an embodiment of the present disclosure. The video distribution server 100 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

[0085] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the video distribution server 100 may be configured to include one or more of the apparatuses shown in the figure, or may be configured to exclude some of the apparatuses.

[0086] Each function of the video distribution server 100 is realized by loading specific software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data in the memory 1002 and storage 1003.

[0087] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the above-mentioned selection unit 102, sound analysis unit 103, etc. may be realized by the processor 1001.

[0088] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these programs. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, the selection unit 102 and the sound analysis unit 103 may be implemented by a control program stored in the memory 1002 and running on the processor 1001, and similar implementations may be used for other functional blocks. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0089] The memory 1002 is a computer-readable recording medium and may be configured, for example, by at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing the video distribution method according to one embodiment of the present disclosure.

[0090] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other appropriate medium including at least one of memory 1002 and storage 1003.

[0091] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, or a communication module. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc. to realize at least one of frequency division duplex (FDD) and time division duplex (TDD). For example, the video distribution unit 107 described above may be realized by the communication device 1004. The communication device 1004 may be implemented with a transmission unit and a reception unit that are physically or logically separated.

[0092] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0093] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0094] The video distribution server 100 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0095] The notification of information is not limited to the aspects / embodiments described in the present disclosure and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI) and Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB) and System Information Block (SIB))), other signals, or a combination thereof. Furthermore, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0096] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0097] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0098] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0099] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0100] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0101] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0102] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0103] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0104] Note that terms described in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of a channel and a symbol may be a signal (signaling). Furthermore, a signal may be a message. Furthermore, a component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.

[0105] Furthermore, the information, parameters, etc. described in the present disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information. For example, a radio resource may be indicated by an index.

[0106] The names used for the above-described parameters are not intended to be limiting in any way. Furthermore, the mathematical expressions using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

[0107] In this disclosure, the terms "Mobile Station (MS)," "user terminal," "User Equipment (UE)," "terminal," and the like may be used interchangeably.

[0108] A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

[0109] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0110] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using one or more wires, cables, and / or printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0111] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0112] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0113] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0114] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0115] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0116] 100...video distribution server, 200...user terminal, 101...acquisition unit, 102...selection unit, 103...sound analysis unit, 104...presentation unit, 105...working data storage unit, 106...video DB, 107...video distribution unit.

Claims

1. An apparatus comprising: an acquisition unit that acquires multiple frames that make up a video; a selection unit that selects multiple specific frames from the multiple frames using a feature change detection model that has learned feature changes between frames; and a determination unit that selects, from the multiple specific frames, specific frames that satisfy a separation condition as separations.

2. The device according to claim 1, wherein the segmentation condition is at least one of a condition based on whether the specific frame contains a sound that is not suitable as a segmentation, or a condition based on whether the specific frame indicates a scene change.

3. The device described in claim 1, wherein the selection unit inputs a target frame for feature change determination, which is one frame extracted from the plurality of frames, and at least one other frame located before or after the target frame for feature change determination, into the feature change detection model, and selects the plurality of specific frames based on the output thereof.

4. The device described in claim 3, wherein the selection unit obtains an output from the feature change detection model each time the frame for feature change determination is shifted and changed, and selects the frame for feature change determination as the specific frame if the output from the feature change detection model satisfies an acquisition condition.

5. The device according to claim 4, wherein the selection unit further uses a relevance detection model that has learned the relevance between frames to select, from among the plurality of specific frames, specific frames that are not relevant to the frame for which relevance is to be determined as non-relevant frames.

6. The device described in claim 5, wherein the selection unit inputs the frame to be determined for relevance judgment, which is one specific frame extracted from the plurality of specific frames arranged in chronological order, and N other specific frames arranged before and after the frame to be determined for relevance judgment, into the relevance detection model, and selects the non-relevant frame from the plurality of specific frames based on a first output thereof.

7. The device described in claim 6, wherein the selection unit further inputs the frame to be determined for relevance judgment and M other specific frames located before and after the frame to be determined for relevance judgment, which are extracted from the plurality of specific frames arranged in chronological order, into the relevance detection model, and selects the non-relevant frame from the plurality of specific frames based on a second output and the first output, which are outputs of the relevance detection model, and wherein M is smaller than N.

8. The device described in claim 7, wherein the selection unit obtains outputs from the relevance detection model for all of the plurality of specific frames each time the frame to be determined for relevance is shifted and changed, and selects the non-relevant frame from the specific frames based on higher-order information among information based on the first output and the second output for all of the plurality of specific frames.

9. The device according to claim 1, further comprising a sound analysis unit that analyzes the state of sound generation in the video, wherein the determination unit does not select the specific frame as the delimiter if the specific frame contains the state of sound generation.

10. A method comprising: an acquisition step of acquiring a plurality of frames constituting a moving image; a selection step of selecting a plurality of specific frames from the plurality of frames using a feature change detection model that has learned feature changes between frames; and a determination step of selecting, as a division, a specific frame from the plurality of specific frames that satisfies a division condition.

Images