Scene detection device, scene detection method, and scene detection program

Abstract

To provide a technology that can automatically detect scenes from a video.SOLUTION: A scene detection device 30 includes a processing unit 32 that calculates an image quality evaluation index value between an arbitrary frame in a video and a neighboring frame of the arbitrary frame, and determines the image identity between the arbitrary frame and the neighboring frame on the basis of the magnitude of the image quality evaluation index value.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a scene detection device, a scene detection method, and a scene detection program. [Background technology]

[0002] There is a technique for dividing video scenes (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6557592 Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 only divides video scenes.

[0005] The present disclosure has been made in view of the above, and aims to provide a technology that can automatically detect scenes from video. [Means for solving the problem]

[0006] A scene detection device according to one aspect of the present disclosure includes a processing unit that calculates an image quality evaluation index value between an arbitrary frame in a video and a neighboring frame of the arbitrary frame, and determines the image identity between the arbitrary frame and the neighboring frame based on the magnitude of the image quality evaluation index value.

[0007] A scene detection device according to one embodiment of the present disclosure includes a processing unit that calculates an image quality evaluation index value for an arbitrary frame in a first video and an arbitrary frame in a second video, and determines the image identity between the arbitrary frame in the first video and the arbitrary frame in the second video based on the magnitude of the image quality evaluation index value.

[0008] A scene detection device according to one aspect of the present disclosure includes a processing unit that calculates the similarity between an arbitrary frame in a video and a neighboring frame of the arbitrary frame using an image quality evaluation index, and detects a transition scene within the video based on the similarity.

[0009] A scene detection method according to one aspect of the present disclosure is a scene detection method performed by a scene detection device, which calculates an image quality evaluation index value between an arbitrary frame in a video and a neighboring frame of the arbitrary frame, and determines the image identity between the arbitrary frame and the neighboring frame based on the magnitude of the image quality evaluation index value.

[0010] A scene detection method according to one embodiment of the present disclosure is a scene detection method performed by a scene detection device, which calculates an image quality evaluation index value for an arbitrary frame in a first video and an arbitrary frame in a second video, and determines the image identity between the arbitrary frame in the first video and the arbitrary frame in the second video based on the magnitude of the image quality evaluation index value.

[0011] A scene detection method according to one aspect of the present disclosure is a scene detection method performed by a scene detection device, which uses an image quality evaluation index to calculate the similarity between an arbitrary frame in a video and a frame adjacent to the arbitrary frame, and detects a transition scene within the video based on the similarity.

[0012] A scene detection program according to an aspect of the present disclosure causes a computer to function as the scene detection device. [Effects of the Invention]

[0013] According to the present disclosure, a technology that can automatically detect scenes from video can be provided. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a scene detection system. [Figure 2] FIG. 2 is a diagram showing a processing flow of the scene detection method. [Figure 3]FIG. 3 is a diagram showing an image of calculation of the SSIM value in each video. [Figure 4] FIG. 4 is a diagram showing an example of detecting a scene change frame. [Figure 5] FIG. 5 is a diagram showing an image of detecting the same scene between videos. [Figure 6] FIG. 6 is a diagram showing an image of calculation of the similarity between videos. [Figure 7] FIG. 7 is a diagram showing a first example of a search for neighboring frames. [Figure 8] FIG. 8 is a diagram showing a second example of a search for neighboring frames. [Figure 9] FIG. 9 is a diagram showing an application example 1 of the detection result of the same scene. [Figure 10] FIG. 10 is a diagram showing an application example 2 of the detection result of the same scene. [Figure 11] FIG. 11 is a diagram illustrating an example of the hardware configuration of a scene detection device. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0016] [Summary of this disclosure] In order to grasp scene transitions in video, there is a demand for a technology that can automatically detect switching scenes and identical scenes from video using a computer. Therefore, this disclosure discloses a technology that determines the identity of images using an image quality evaluation index.

[0017] [Scene detection system configuration] 1 is a diagram showing an example of the configuration of a scene detection system 1 according to this embodiment. The scene detection system 1 includes a video distribution device 10, a video reception device 20, and a scene detection device 30. These devices are connected to each other via a communication network 40 so that they can communicate with each other.

[0018] Video distribution device 10 is a device that distributes video to video receiving device 20 via communication network 40. Video distribution device 10 is, for example, a computer device with a camera, or a television video transmitter for terrestrial digital broadcasting.

[0019] Video receiving device 20 is a device that receives video from video distribution device 10 via communication network 40. Video receiving device 20 is, for example, a computer device or a television set with a built-in digital tuner.

[0020] The video distribution device 10 and the video receiving device 20 may be devices capable of processing video on a frame-by-frame basis. The number of video distribution devices 10 and the number of video receiving devices 20 may each be two or more. The number of distributed videos (first videos) and the number of received videos (second videos) may also each be two or more. One video distribution device 10 may distribute multiple videos, and one video receiving device 20 may receive multiple videos.

[0021] In this embodiment, the above-described video distribution device 10 and video receiving device 20 are further equipped with a scene detection device 30. The scene detection device 30 uses an image quality evaluation index to determine the identity (similarity) of images within the distributed video, within the received video, and between the distributed video and the received video, and detects video switching scenes and identical scenes.

[0022] The scene detection device 30 is, for example, a server device connected to a communication network 40. The scene detection device 30 may operate within the video distribution device 10 or the video reception device 20.

[0023] The communication network 40 is, for example, the Internet. The communication network 40 may be wireless or wired.

[0024] [Scene detection device function] The scene detection device 30 includes a receiving unit 31, a processing unit 32, a display unit 33, a receiving unit 34, and a storage unit 35.

[0025] The receiving unit 31 has a function of receiving the video delivered from the video delivery device 10 to the video receiving device 20.

[0026] The receiving unit 31 has a function of receiving the video that the video receiving device 20 has received from the video distribution device 10.

[0027] The processing unit 32 has a function of calculating an image quality evaluation index value between an arbitrary frame in the video and a neighboring frame of the arbitrary frame for each of the distributed video and the received video, and determining the identity of the images between the arbitrary frame and the neighboring frame based on the magnitude of the calculated image quality evaluation index value.

[0028] The processing unit 32 has a function of calculating an image quality evaluation index value for an arbitrary frame in the distributed video and an arbitrary frame in the received video for the distributed video and the received video, and determining the image identity between the arbitrary frame in the distributed video and the arbitrary frame in the received video based on the magnitude of the calculated image quality evaluation index value.

[0029] In this way, the identity of the images between frames is determined, so that scene change and the same scene can be detected. Specifically, the processing unit 32 has the following functions.

[0030] The processing unit 32 has a function of calculating the similarity between an arbitrary frame in the video and a neighboring frame of the arbitrary frame using an image quality evaluation index for each of the distributed video and the received video, and detecting (searching for) a switching scene (scene switching frame) within the video based on the calculated similarity.

[0031] The neighboring frames of a given frame are, for example, frames within a specified range, frames at a specified sampling interval, and frames within a specified range and at a specified sampling interval.

[0032] The processing unit 32 has a function of calculating the similarity between one scene change frame in the detected distributed video and multiple scene change frames in the detected received video for the distributed video and the received video, and detecting (identifying) identical scenes in the distributed video and the received video based on the comparison results of each similarity.

[0033] The processing unit 32 has a function of calculating the similarity between scene change frames by further using frames adjacent to the scene change frame.

[0034] The neighboring frames of the scene change frame are, for example, frames within a specified range, frames at a specified sampling interval, and frames within a specified range and at a specified sampling interval.

[0035] The processing unit 32 has a function of calculating a video delay time between the distributed video and the received video based on the elapsed playback time up to the same scene.

[0036] The processing unit 32 has a function of setting the first identical scene in the distributed video and the received video as the start scene of the video in each video.

[0037] The display unit 33 has a function of displaying the results of searching for scene changes and the results of identifying identical scenes on the screen.

[0038] The receiving unit 34 has a function of receiving a search range and a sampling interval designated by the user in order to select neighboring frames of an arbitrary frame and neighboring frames of a scene change frame.

[0039] The storage unit 35 has a function of storing data to be processed by the scene detection device 30.

[0040] [Image quality evaluation index] An example of an image quality evaluation index is called SSIM (Structural Similarity), which is an index that evaluates changes in image quality between images by comparing three elements: brightness, contrast, and structure of the image.

[0041] SSIM can be calculated using existing calculation methods. If the SSIM value of the nth image and the n+1th image is equal to or greater than a threshold, the two images can be determined to be indistinguishable, high-quality images (images of the same scene). On the other hand, if the SSIM value is less than the threshold, the two images can be determined to be low-quality images (images of different scenes).

[0042] In this embodiment, SSIM is used to detect scene changes within the same video and to detect the same scene between two videos.

[0043] [Scene detection method] 2 is a diagram showing a processing flow of the scene detection method, which is executed by the scene detection device 30.

[0044] Step S1; The receiving unit 31 receives, from the video delivering device 10, the delivered video that has been delivered and played by the video delivering device 10. The receiving unit 31 receives, from the video receiving device 20, the received video that has been received and played by the video receiving device 20.

[0045] Step S2; In step S2, the SSIM values ​​of an arbitrary frame and neighboring frames in each of the distributed video and the received video are calculated, and the identity of the images is determined based on the magnitude of the calculated SSIM values.

[0046] Specifically, as shown in FIG. 3, the processing unit 32 calculates the SSIM values ​​of an arbitrary frame and a nearby frame in each of the distributed video and the received video, and determines the similarity between the arbitrary frame and the nearby frame based on the calculated SSIM values.

[0047] For example, if the SSIM value is large, the processing unit 32 determines that the similarity between the given frame and the neighboring frame is high, and if the SSIM value is small, the processing unit 32 determines that the similarity between the given frame and the neighboring frame is low.

[0048] The neighboring frames are one or more frames selected from within a certain search range centered on the arbitrary frame.

[0049] Step S3; The processing unit 32 determines that a scene is a continuous scene when the similarity between an arbitrary frame and a nearby frame is high for each of the distributed video and the received video, and determines that a scene is a transition scene when the similarity between the arbitrary frame and a nearby frame is low.

[0050] Thereafter, the processing unit 32 sequentially treats all frames from the start of the video to the end of the video as the above-mentioned arbitrary frames, thereby searching for a switching scene (scene switching frame) from each of the distributed video and the received video, as shown in FIG.

[0051] Step S4; In step S4, the SSIM values ​​of an arbitrary frame in the distributed video and an arbitrary frame in the received video are calculated, and the identity of the image in the distributed video and the image in the received video is determined based on the magnitude of the calculated SSIM values.

[0052] Specifically, the processing unit 32 selects one arbitrary scene change frame together with neighboring frames from within the distributed video, as shown in Fig. 5. Also, the processing unit 32 selects all or part of a plurality of arbitrary scene change frames from within the received video together with neighboring frames.

[0053] The neighboring frames are one or more frames selected from within a certain search range centered around an arbitrary scene change frame.

[0054] Then, the processing unit 32 calculates the similarity between each of the frames in the distributed video and the frames in the received video. For example, as shown in Fig. 6, the processing unit 32 calculates the similarity between the first frames in each video, the similarity between the second frames, ..., the similarity between the ninth frames, and sets the sum of all the calculated similarities as the similarity of the fifth frame (an arbitrary scene change frame). This similarity can also be calculated using SSIM.

[0055] Thereafter, the processing unit 32 identifies the scene change frame included in the frame group of the received video with the highest similarity as the same frame as the scene change frame included in the frame group of the distributed video.

[0056] Step S5; Finally, the processing unit 32 outputs the identified identical frame (scene change frame) as the same scene between the distributed video and the received video.

[0057] [Search for nearby frames] In steps S2 and S4, nearby frames are selected from within a certain search range. This certain search range is set to a large value with a certain degree of width in order to improve the accuracy of detecting scene transitions and identical scenes. However, it takes time to search for nearby frames, and it also takes time to detect scene transitions and identical scenes. Therefore, we will explain a method for reducing the amount of calculation required for searching nearby frames.

[0058] (Search example 1) A method of specifying a search range smaller than the preset fixed search range is conceivable, for example, by specifying a smaller distance (e.g., frame number, time) from any frame in step S2 or any scene change frame in step S4.

[0059] In this case, the receiving unit 34 receives the small search range specified by the user and stores it in the storage unit 35. The processing unit 32 searches for nearby frames from the specified small search range, rather than from a preset fixed search range.

[0060] For example, as shown in Fig. 7, the processing unit 32 searches for neighboring frames within a specified search range of frames 3 to 7. By limiting the search range for neighboring frames in this way, the calculation time required to search for neighboring frames can be reduced.

[0061] (Search example 2) A method of specifying a sampling interval (for example, a frame number interval or a time interval) smaller than a preset search range is conceivable.

[0062] In this case as well, the receiving unit 34 receives the small sampling interval designated by the user and stores it in the storage unit 35. The processing unit 32 searches for nearby frames not within a predetermined fixed search range but from the designated small sampling interval.

[0063] For example, the processing unit 32 searches for neighboring frames every two frames, as shown in Fig. 8. Since the number of neighboring frames is thinned out by sampling in this way, the calculation time required to search for neighboring frames can be reduced.

[0064] [Application example of identical scene detection results] (Application example 1) One possible example is when you want to know the delay time of video distribution or correct the delay time of video distribution in a web conference, etc. As shown in Fig. 9, the processing unit 32 sets the playback time of each video start frame of the distributed video and the received video to "0", and then determines the difference in playback time (= |time a - time b|) between the frames of the same scene, and outputs this difference as the delay time of one video relative to the other video.

[0065] (Application example 2) Consider a case where video is distributed during a web conference. As shown in Fig. 10, the processing unit 32 sets the first frame of the same scene after the start of the web conference as the start scene of the distributed video (video within the web conference video). In the case of application example 2, there is no screen sharing of materials between the video distribution device 10 and the video receiving device 20, and the first screen change is limited to when video is distributed.

[0066] [effect] According to this embodiment, the scene detection device 30 calculates the SSIM value between an arbitrary frame in the video and a neighboring frame of the arbitrary frame for each of the distributed video and the received video, and determines the identity of the image between the arbitrary frame and the neighboring frame based on the magnitude of the calculated SSIM value, thereby providing a technology that can automatically detect switching scenes and identical scenes.

[0067] Furthermore, according to this embodiment, the scene detection device 30 calculates the SSIM value of an arbitrary frame in the distributed video and an arbitrary frame in the received video for the distributed video and the received video, and determines the identity of the image of an arbitrary frame in the distributed video and an arbitrary frame in the received video based on the magnitude of the calculated SSIM value, thereby providing a technology that can automatically detect switching scenes and identical scenes.

[0068] Furthermore, according to this embodiment, the scene detection device 30 calculates the similarity between an arbitrary frame in the video and a neighboring frame of the arbitrary frame using SSIM for each of the distributed video and the received video, and detects a transition scene from within the video based on the calculated similarity, thereby providing a technology that can automatically detect transition scenes.

[0069] Furthermore, according to this embodiment, the scene detection device 30 calculates the similarity between one scene change frame detected in the distributed video and multiple scene change frames detected in the received video for the distributed video and the received video, and detects identical scenes between the distributed video and the received video based on the comparison results of each similarity, thereby providing a technology that can automatically detect identical scenes.

[0070] [others] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the present disclosure.

[0071] The scene detection device 30 of the present embodiment described above can be realized, for example, by using a general-purpose computer system including a CPU 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906, as shown in Fig. 11. The memory 902 and the storage 903 are storage devices. In the computer system, the CPU 901 executes a predetermined program loaded onto the memory 902, thereby realizing each function of the scene detection device 30.

[0072] The scene detection device 30 may be implemented by one computer. The scene detection device 30 may be implemented by multiple computers. The scene detection device 30 may be a virtual machine implemented on a computer. The program for the scene detection device 30 may be stored in a computer-readable recording medium such as a HDD, SSD, USB memory, CD, or DVD. The computer-readable recording medium is, for example, a non-transitory recording medium. The program for the scene detection device 30 may also be distributed via a communication network. [Explanation of symbols]

[0073] 1 Scene Detection System 10 Video distribution equipment 20 Video receiving device 30 Scene detection device 31 Receiving unit 32 Processing section 33 Display section 34 Reception Department 35 Storage section 40 Communication Network 901 CPU 902 memory 903 Storage 904 Communication equipment 905 Input Device 906 Output Device

Claims

1. a processing unit that calculates an image quality evaluation index value between an arbitrary frame in a video and a neighboring frame of the arbitrary frame, and determines image identity between the arbitrary frame and the neighboring frame based on the magnitude of the image quality evaluation index value; A scene detection device comprising:

2. a processing unit that calculates an image quality evaluation index value of an arbitrary frame in the first video and an arbitrary frame in the second video, and determines image identity between the arbitrary frame in the first video and the arbitrary frame in the second video based on the magnitude of the image quality evaluation index value; A scene detection device comprising:

3. a processing unit that calculates a similarity between an arbitrary frame in a video and a neighboring frame of the arbitrary frame using an image quality evaluation index, and detects a transition scene from the video based on the similarity; A scene detection device comprising:

4. The processing unit A scene detection device as described in claim 3, which calculates the similarity between one scene change frame in the first video and multiple scene change frames in the second video for the first video and the second video, and detects identical scenes in the first video and the second video based on the comparison results of each similarity.

5. The processing unit The scene detection device according to claim 4 , further comprising: a frame adjacent to the scene change frame to calculate each of the similarities.

6. The neighboring frames of the arbitrary frame are 4. The scene detection device according to claim 3, wherein the frames are within a specified range or at a specified sampling interval.

7. The neighboring frames of the scene change frame are:

6. The scene detection device according to claim 5, wherein the frames are within a specified range or at a specified sampling interval.

8. The processing unit The scene detection device according to claim 4 , wherein the video delay time between the first video and the second video is calculated based on the elapsed playback time up to the same scene.

9. The processing unit 5. The scene detection device according to claim 4, wherein the first identical scene in the first video and the second video is set as the start scene of the video in each video.

10. In a scene detection method performed by a scene detection device, calculating an image quality evaluation index value between an arbitrary frame in a video and a neighboring frame of the arbitrary frame, and determining whether the images of the arbitrary frame and the neighboring frame are identical based on the magnitude of the image quality evaluation index value; Scene detection methods.

11. In a scene detection method performed by a scene detection device, calculating an image quality evaluation index value between an arbitrary frame in the first video and an arbitrary frame in the second video, and determining image identity between the arbitrary frame in the first video and the arbitrary frame in the second video based on the magnitude of the image quality evaluation index value; Scene detection methods.

12. In a scene detection method performed by a scene detection device, calculating a similarity between an arbitrary frame in a video and a neighboring frame of the arbitrary frame using an image quality evaluation index, and detecting a transition scene from the video based on the similarity; Scene detection methods.

13. 10. A scene detection program that causes a computer to function as the scene detection device according to claim 1.

Images