Detecting scene transitions in video footage

Abstract

The present invention provides concepts for detecting scene transitions in video footage. One example includes, for each of a plurality of different frames of the video footage, obtaining a value of a confidence metric associated with an object detected in the frame. Scene transitions in the video footage are then detected based on the obtained values ​​of the confidence metric.

Description

Background Art

[0001] The present invention relates generally to video processing, and in particular to detecting scene transitions in video footage.

[0002] Scene detection is a subset of video processing that provides indexing and searching of video footage. Transitions between scenes (such as "dissolves" or "cross-fades") pose difficulties for existing scene detection algorithms because the low-level information (e.g., color histograms, brightness, contrast) used to identify scene changes does not generally generalize well over a wide range of video footage. For example, it may be difficult to detect a dissolve transition between scenes with similar contrast, brightness, color, etc.

[0003] Furthermore, existing scene detection algorithms are typically standalone and need to be run independently of other video processing tasks. Therefore, existing scene detection methods may exhibit significant processing requirements for large videos. Therefore, there is a need to address the above-mentioned problems in the art. Summary of the invention

[0004] Viewed from a first aspect, the invention provides a computer-implemented method for detecting scene transitions in a video sequence, the method comprising: obtaining, for each of a plurality of different frames of the video sequence, a value of a confidence metric associated with a detected object in the frame; and detecting a scene transition in the video sequence based on the obtained value of the confidence metric.

[0005] From another aspect, the present invention provides a computer program product, comprising a computer-readable storage medium having program instructions, which can be executed by a processing unit so that the processing unit performs a method for detecting scene transitions in video footage when running on a computer network, wherein the method includes the following steps: for each frame of a plurality of different frames of the video footage, obtaining a value of a confidence metric associated with a detected object in the frame; and detecting scene transitions in the video footage based on the obtained value of the confidence metric.

[0006] Viewed from another aspect, the present invention provides a system for detecting scene transitions in video footage, the system comprising one or more processors and one or more tangible storage media storing programming instructions for execution by the one or more processors, the programming instructions comprising instructions for: obtaining, via an interface, for each of a plurality of different frames of the video footage a value of a confidence metric associated with a detected object in the frame; and detecting scene transitions in the video footage based on the obtained values ​​of the confidence metric.

[0007] Viewed from another aspect, the present invention provides a computer program product for detecting scene transitions in video footage, the computer program product comprising a computer readable storage medium readable by a processing circuit and storing instructions executable by the processing circuit to perform a method for performing the steps of the present invention.

[0008] Viewed from another aspect, the invention provides a computer program stored on a computer readable medium and loadable into the internal memory of a digital computer, comprising software code portions for executing the steps of the invention when the program is run on the computer.

[0009] The invention also relates to a computer program product comprising a computer readable program code enabling a processor of a system or a plurality of processors of a network to implement a method. The invention also relates to a computer system comprising at least one processor and such a computer program product, wherein at least one processor is adapted to execute the computer readable program code of the computer program product. The invention also relates to a system for detecting scene transitions in video footage. The invention seeks to provide a computer implemented method for detecting scene transitions in video footage.

[0010] The invention also seeks to provide a computer program product comprising a computer program code for implementing the proposed method when executed by a processing unit.

[0011] The invention also seeks to provide a processing system suitable for executing the computer program code.

[0012] The present invention also seeks to provide a system for detecting scene changes in video footage.

[0013] According to one aspect of the present invention, a computer-implemented method for detecting scene transitions in video footage is provided. The method includes, for each of a plurality of different frames of the video footage, obtaining a value of a confidence metric associated with an object detected in the frame. The method also includes detecting a scene transition in the video footage based on the obtained value of the confidence metric.

[0014] According to another aspect of the present invention, a system for detecting scene transitions in video footage is provided. The system includes an interface component configured to obtain, for each of a plurality of different frames of the video footage, a value of a confidence metric associated with an object detected in the frame. The system also includes a detection component configured to detect scene transitions in the video footage based on the obtained value of the confidence metric.

[0015] According to another aspect of the present invention, a computer program product for detecting scene transitions in video footage is provided. The computer program product comprises a computer-readable storage medium having program instructions embodied therewith, the program instructions being executable by a processing unit to cause the processing unit to perform a method according to the proposed embodiment.

[0016] According to another aspect of the present invention, a processing system is provided, comprising at least one processor and a computer program product according to an embodiment. The at least one processor is adapted to execute computer program code of the computer program product. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which:

[0018] Figure 1 is a block diagram of an example system in which aspects of the illustrative embodiments may be implemented;

[0019] Figure 2 is a simplified block diagram of an exemplary embodiment of a system for detecting scene transitions in video footage;

[0020] Figure 3 illustrates an example of how confidence values ​​associated with detected objects vary between frames of video footage in accordance with an exemplary embodiment; and

[0021] Figure 4 is a simplified block diagram of an exemplary embodiment of a system for detecting scene transitions in video footage. DETAILED DESCRIPTION

[0022] It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to represent the same or similar components.

[0023] In the context of the present application, where the embodiments of the present invention constitute a method, it should be understood that the method can be a process for being executed by a computer, that is, a computer-implementable method. Therefore, the various steps of the method can reflect various parts of a computer program, such as various parts of one or more algorithms.

[0024] In addition, in the context of the present application, a system may be a single device or a collection of distributed devices suitable for performing one or more embodiments of the method of the present invention. For example, a system may be a personal computer (PC), a server, or a collection of PCs and / or servers connected via a network such as a local area network, the Internet, etc., so as to collaboratively perform at least one embodiment of the method of the present invention.

[0025] Concepts for detecting scene transitions in video footage are presented. In particular, it is proposed that scene transitions in video footage can be detected based on values ​​of confidence measures associated with detected objects at different frames of the video footage. In other words, changes in confidence values ​​of detected objects between frames of a video can be used to identify the presence of a scene transition in the video.

[0026] The inventors propose the concept of analyzing the confidence of detected objects between frames of a video to infer or identify the presence of a transition in the video. Therefore, the proposed embodiments can be provided as an extension to an existing video processing pipeline that detects objects in a video and indicates an associated confidence value.

[0027] This proposal can eliminate the need for manual tuning (which is currently a requirement for many conventional methods). For example, the proposed embodiment can be integrated with a known cost function method for detecting scene transitions to increase the robustness of the method in detecting scene transitions. The embodiment can also be implemented with an existing frame-by-frame object detector.

[0028] It is proposed that the values ​​of confidence metrics associated with detected objects of different frames of a video footage can be analyzed in order to detect scene transitions in the video footage. For example, the associated confidence scores of detected objects between frames can be compared in order to identify scene transitions in a relatively straightforward manner. Unlike conventional methods that do not interpret fade-type scene transitions in an efficient and robust manner, embodiments can reliably detect fade-type scene transitions.

[0029] The proposed concept can exploit high-level semantic information to provide an effective scene transition detection method, and this can be integrated with existing video processing methods / algorithms.

[0030] For example, the inventor proposes that, for each of a plurality of different frames of a video sequence, a value of a confidence metric associated with an object detected in the frame can be obtained. Then, scene transitions in the video sequence can be detected based on the obtained values ​​of the confidence metric.

[0031] Embodiments may therefore provide the advantage that they are robust to fade transitions between scenes with similar low-level pixel information.

[0032] Another advantage that the proposed concept(s) may provide is that embodiments may not require inspection of raw video frames. Instead, embodiments may employ the output(s) of conventional object detectors. As a result, embodiments may be used in conjunction with pre-existing video processing algorithms that have been configured to detect objects in video footage and provide confidence values ​​associated with the detected objects. Thus, these embodiments may utilize information provided by object detection algorithms / components and thus complement existing / traditional video processing algorithms / systems.

[0033] Thus, the proposed embodiments enable easy integration and dynamic use of existing and / or new object detection algorithms during video processing.

[0034] Thus, the proposed embodiments may provide concepts for modifying and / or supplementing the functionality of video processing algorithms. For example, the embodiments may be used to integrate new or additional scene transition algorithms into existing video processing systems. The embodiments may therefore help provide improved or extended functionality to existing video processing implementations. The embodiments may therefore provide modified and / or extended scene transition detection functionality by utilizing information about detected objects in video footage, thereby increasing the value of video processing systems. The embodiments may therefore help provide improved or extended functionality to existing scene detection and / or video processing implementations.

[0035] In an embodiment, detecting a scene transition based on the obtained confidence metric value may include: analyzing the obtained confidence metric value to identify a trend or pattern in the confidence metric value across a plurality of different frames of a video sequence; and identifying a scene transition in the video sequence based on the identified trend or pattern in the confidence metric value. For example, identifying a scene transition may include: comparing the identified trend or pattern in the confidence metric value with a trend of a predetermined pattern; and identifying a scene transition in the video sequence based on the comparison result. Therefore, the proposed embodiment may employ relatively simple data / trend analysis techniques, thereby minimizing the cost and / or complexity of implementation.

[0036] In addition, embodiments may also classify identified scene transitions based on identified trends or patterns in confidence values. Thus, embodiments may not only detect the presence of a scene transition, but may also identify the type or category of the detected scene transition. For example, a smooth, gradual decrease in confidence values ​​associated with a detected object over successive frames of a video may be used to infer that a scene transition is a "fade" transition, while a sudden step change in confidence values ​​associated with a detected object over successive frames of a video may be used to infer that a scene transition is a "cut" or "quick vanish" transition.

[0037] In some embodiments, detecting a scene transition based on the value of the obtained confidence metric may include determining the presence of a scene transition in the video footage in response to the value of the obtained confidence metric decreasing over consecutive frames of the video footage. In other words, it is proposed that when a scene slowly disappears or fades out as part of a transition, the value of the confidence metric associated with the detected object in the scene will also decrease in unison with the slow disappearance or fading out of the scene. Thus, a relatively simple data value analysis technique may be implemented to determine the presence of a scene transition in the video footage.

[0038] In an embodiment, detecting a scene transition based on the value of the obtained confidence metric may include determining the presence of a scene transition in the video footage in response to the value of the obtained confidence metric increasing over consecutive frames of the video footage. It is suggested that when a scene disappears or fades away as part of a transition, the value of the confidence metric associated with the detected object in the scene will also increase in unison with the disappearance / fading away of the scene. Again, relatively simple data value analysis techniques may thus be implemented to determine the presence of a scene transition in the video footage.

[0039] Some embodiments may further include: comparing the value of the obtained confidence metric to a threshold; and identifying at least one of the beginning and the end of a scene transition based on the comparison result. As an example, identifying the end of a scene transition may include: determining a frame of the video footage for which the value of the obtained confidence metric decreases from a value that exceeds the threshold to a value that does not exceed the threshold; and determining the end of the scene transition based on the determined frame. Similarly, identifying the beginning of a scene transition may include: determining a frame of the video footage for which the value of the obtained confidence metric increases from a value that does not exceed the threshold to a value that exceeds the threshold; and determining the beginning of the scene transition based on the determined frame. Therefore, a simple data value comparison technique can be used to accurately infer the beginning and / or end of a scene transition.

[0040] In some embodiments, obtaining a value of the confidence metric may include: obtaining a first value of the confidence metric associated with a detected object in a first frame of the video footage; and obtaining a second value of the confidence metric associated with the detected object in a second subsequent frame of the video footage. The first frame and the second frame of the video footage may be, for example, consecutive frames of the video footage.

[0041] Figure 1 2 is a block diagram of an example system 200 in which aspects of the illustrative embodiments may be implemented. System 200 is an example of a computer, such as a client in a distributed processing system, in which computer usable code or instructions implementing the processes of the illustrative embodiments of the invention may be located. For example, system 200 may be configured to implement an interface component and a detection component according to an embodiment.

[0042] In the depicted example, system 200 employs a hub architecture that includes a north bridge and memory controller hub (NB / MCH) 202 and a south bridge and input / output (I / O) controller hub (SB / ICH) 204. Processing unit 206, main memory 208, and graphics processor 210 are connected to NB / MCH 202. Graphics processor 210 may be connected to NB / MCH 202 via an accelerated graphics port (AGP).

[0043] In the depicted example, a local area network (LAN) adapter 212 is connected to the SB / ICH 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a hard disk drive (HDD) 226, a CD-ROM drive 230, a universal serial bus (USB) port and other communication ports 232, and a PCI / PCIe device 234 are connected to the SB / ICH 204 via a first bus 238 and a second bus 240. PCI / PCIe devices may include, for example, Ethernet adapters, add-on cards, and PC cards for notebook computers. PCI uses a card bus controller, while PCIe does not. ROM 224 may be, for example, a flash basic input / output system (BIOS).

[0044] HDD 226 and CD-ROM drive 230 are connected to SB / ICH 204 via second bus 240. HDD 226 and CD-ROM drive 230 may use, for example, an integrated drive electronics (IDE) or serial advanced technology attachment (SATA) interface. Super I / O (SIO) device 236 may be connected to SB / ICH 204.

[0045] An operating system runs on processing unit 206. The operating system coordinates and provides control over system 200 (or Figure 2 The client computer may be a commercially available operating system. An object-oriented programming system, such as Java TM A programming system that can run with the operating system and provide Java programs executed on the system 200 TM A call from a program or application to the operating system. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and / or its affiliates.

[0046] As a server, system 200 may be, for example, eServerTM System P5 system A computer system that runs a high-level interactive execution Operating system or Operating system. System 200 may be a symmetric multiprocessor (SMP) system including multiple processors in processing unit 206. Alternatively, a single processor system may be employed. IBM eServer, System p5, AIX are trademarks of International Business Machines Corporation registered in many jurisdictions throughout the world. The registered trademarks are used under a sublicense from the Linux Foundation, an exclusive licensee of Linus Torvalds, the owner of the trademarks worldwide. Instructions for the operating system, programming system, and applications or programs are located on storage devices such as HDD 226 and may be loaded into main memory 208 for execution by processing unit 206. Similarly, one or more message handling programs according to embodiments may be suitable for storage by storage devices and / or main memory 208.

[0047] The processes of the illustrative embodiments of the present invention may be performed by processing unit 206 using computer usable program code that may be located in a memory such as, for example, main memory 208 , ROM 224 , or in one or more peripheral devices 226 and 230 .

[0048] Such as Figure 2 The bus system shown as the first bus 238 or the second bus 240 may include one or more buses. Of course, the bus system may be implemented using any type of communication structure or architecture that provides for the transmission of data between different components or devices attached to the structure or architecture. Figure 1 The communication unit of the modem 222 or the network adapter 212 may include one or more devices for transmitting and receiving data. The memory may be, for example, the main memory 208, the ROM 224 or a memory such as a memory card. Figure 1 The cache found in NB / MCH202.

[0049] Those of ordinary skill in the art will understand that Figure 1 The hardware in the can vary depending on the implementation. Figure 1 In addition to or in place of the hardware described in the foregoing, other internal hardware or peripheral devices such as flash memory, equivalent non-volatile memory, or optical disk drives may be used. Furthermore, the processing of the exemplary embodiments may be applied to a multi-processor data processing system in addition to the previously mentioned systems without departing from the scope of the present invention.

[0050] Furthermore, system 200 may take the form of any of a variety of different data processing systems, including a client computing device, a server computing device, a tablet computer, a laptop computer, a telephone or other communication device, a personal digital assistant (PDA), etc. In some illustrative examples, system 200 may be a portable computing device that is configured with flash memory to provide non-volatile memory for storing, for example, operating system files and / or user-generated data. Thus, system 200 may be essentially any known or later developed data processing system without architectural limitation.

[0051] Reference now Figure 2 , depicts a simplified block diagram of an exemplary embodiment of a system 200 (labeled system 300 ) for detecting scene transitions in video footage 310 .

[0052] System 200 includes an interface component 320 configured to obtain, for each of a plurality of different frames of a video footage, a value of a confidence metric (ie, a confidence value) associated with an object detected in the frame.

[0053] In this example, the interface component 320 is configured to obtain a first value of a confidence metric associated with a detected object in a first frame of the video footage, and obtain a second value of the confidence metric associated with the detected object in a second subsequent frame of the video footage. In other words, the interface component 320 obtains multiple values ​​of the confidence metric associated with the detected object in a corresponding plurality of consecutive frames of the video footage.

[0054] Here, the interface component 320 obtains the confidence value from the video processor 330. The video processor 330 of this example is a conventional video processing system that is configured to process the received video footage and, as a result of the processing, detect objects in the video footage and determine associated confidence values. In doing so, the video processor 330 may employ any one or more of the known and available object detection algorithms that determine confidence values ​​associated with the detected objects.

[0055] The system 200 for detecting scene transitions in video footage 310 also includes a detection component 340. The detection component 340 is configured to detect scene transitions in the video footage based on the obtained values ​​of the confidence measure.

[0056] exist Figure 2In the example of , the detection component 340 includes an analysis component 350 and a processor 360. The analysis component 350 is configured to analyze the obtained confidence metric values ​​to identify trends or patterns in the values ​​of the confidence metric for a plurality of different frames of the video footage. The processor 360 is then configured to identify scene transitions in the video footage based on the identified trends or patterns in the confidence metric values.

[0057] By way of example only, in response to the obtained confidence metric value decreasing over consecutive frames of the video footage, the processor 360 of the detection component 340 determines the presence of a scene transition in the video footage 310. In response to the obtained confidence metric value increasing over consecutive frames of the video footage, the processor 360 of the detection component 340 determines the presence of a scene transition in the video footage 310.

[0058] exist Figure 2 In the example of , the detection component 340 also includes a comparison unit 370 configured to compare the value of the obtained confidence metric with a threshold value. The detection component 340 is configured to identify at least one of the beginning and the end of the scene transition based on the comparison result obtained by the comparison unit 370. For example, to identify the beginning of the scene transition, the detection component 340 is configured to determine a frame of the video footage for which the value of the obtained confidence metric increases from a value that does not exceed the threshold value to a value that exceeds the threshold value. Similarly, to identify the end of the scene transition, the detection component 340 is configured to determine a frame of the video footage for which the value of the obtained confidence metric decreases from a value that exceeds the threshold value to a value that does not exceed the threshold value.

[0059] By way of further explanation, the concept of analyzing the values ​​of the obtained confidence measure to detect scene transitions in a video sequence according to the proposed embodiment will now be described with reference to another example.

[0060] Steps (i) to (viii) below describe an example of a video sequence where a video clip of a dog and a cat on a couch (i.e., a first scene) transitions to a shot of a person outdoors with his car and motorcycle (i.e., a second scene). The proposed embodiment is configured to detect the transition between these two scenes.

[0061] (i) An existing frame-by-frame convolutional neural network (CNN) is used to detect objects and associated confidence values ​​in a video sequence. As an example, the detected objects and associated confidence values ​​for the first and second frames of the video sequence are determined as detailed in Table 1 below:

[0062]

[0063] Table 1

[0064] (ii) Use a sliding window approach (e.g., with batch size 100 and window size 5) to iterate over all object detector frames to obtain batch results. Figure 3 An example of how a confidence value associated with a detected object changes between frames of a video footage is shown. In the depicted example, a percentage value indicates a confidence value (i.e., a value of a confidence metric) associated with a detected object for a frame of the video footage. To represent changes in confidence values, arrows are depicted with varying shades of grayscale gradients, with lighter shades representing lower confidence values. As an example, the confidence value associated with a detected dog decreases from a value of 90% in frame #1 to approximately 47% in frame #57. The decrease in confidence value as the frames of the video progress is represented by the changing fill of the arrow from darker shades (for frame #1) to lighter shades (for frame #57).

[0065] (iii) Using the position information provided by the output of the object detector, objects in the scene are given unique IDs and tracked frame by frame. Note that if multiple objects of the same type appear, a different ID is issued for each object. Individual objects can be tracked using a perceptual estimate of the expected movement of each object type (e.g., based on the class label predicted by the object detector). This can be calculated before iterating over all frames.

[0066] (iv) If Figure 3 As shown in the gradient shading in , the confidence value associated with each unique object is monitored frame by frame.

[0067] (v) If a new set of objects appears (e.g., objects with associated confidence values ​​exceeding a threshold of 30%), and this matches a concurrent decrease in the confidence of previous objects currently on the screen, then it is inferred that this indicates the start of a fade transition (e.g., from Figure 3 45 in the image).

[0068] (vi) If the confidence of the new object continues to increase, while the confidence of the old object continues to decrease beyond the 30% threshold, this point indicates the end of the fade transition ( Figure 3 60 in the image).

[0069] (vii) If at any point any of the above conditions (v) and (vi) fail, then since there is no fade transition, the start and end positions are removed from the transient memory. Note that the assigned unique object ID information can be used to determine whether the same object is visible.

[0070] (viii) This process is repeated for all batches of frames in the video footage.

[0071] From the above examples, it will be appreciated that the proposed embodiments may infer a transition from the values ​​of the obtained confidence metric crossing a threshold on consecutive frames of a video. For example, the timing of a decrease in the confidence value associated with a first detected object and an increase in the confidence value associated with a second detected object coinciding or corresponding may indicate a fade transition (e.g., Figure 3 In addition, the start and / or end points of a transition may be identified based on frames having a confidence value exceeding a predetermined threshold (which may be configurable).

[0072] As a further example, Figure 4 As shown, embodiments may include a computer system 70, which may form part of a networked system 7. For example, a system for detecting scene transitions in video footage may be implemented by the computer system 70. Components of the computer system / server 70 may include, but are not limited to, one or more processing arrangements, such as a processor or processing unit 71, a system memory 74, and a bus 90 coupling various system components including the system memory 74 to the processing unit 71.

[0073] The system memory 74 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 75 and / or buffer memory 76. The computer system / server 70 may also include other removable / non-removable, volatile / non-volatile computer system storage media. In this case, each may be connected to the bus 90 via one or more data medium interfaces. The memory 74 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the proposed embodiment. For example, the memory 74 may include a computer program product having a program that can be executed by the processing unit 71 to cause the system to perform a method for detecting scene transitions in video footage according to the proposed embodiment.

[0074] A program / utility 78 having a set (at least one) of program modules 79 may be stored in the memory 74, which generally perform the functions and / or methods of the proposed embodiments to detect scene transitions in video footage.

[0075] The computer system / server 70 may also communicate with one or more external devices 80, such as a keyboard, pointing device, display 85, etc.; one or more devices that enable a user to interact with the computer system / server 70; and / or any device that enables the computer system / server 70 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 72. However, the computer system / server 70 may communicate with one or more networks (such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet)) via a network adapter 73 (e.g., to transmit reconstructed content to the system or user).

[0076] In the context of the present application, where the embodiments of the present invention constitute a method, it should be understood that the method is a process for being executed by a computer, i.e., a computer-implementable method. Therefore, the various steps of the method reflect various parts of a computer program, such as various parts of one or more algorithms.

[0077] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0078] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a storage class memory (SCM), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0079] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0080] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In certain embodiments, in order to perform various aspects of the present invention, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can execute the computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions.

[0081] Various aspects of the present invention are described herein with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to embodiments of the present invention. It will be understood that each frame of the flow chart and / or block diagram and the combination of frames in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0082] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0083] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0084] Flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in flow chart or block diagram can represent the module, segment or part of instruction, and it comprises one or more executable instructions for realizing the logical function of appointment.In some alternative embodiments, the function mentioned in the frame may not occur in the order mentioned in the figure.For example, two frames shown continuously can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the function involved.It will also be noted that the combination of the frame in each frame of block diagram and / or flow chart illustration and block diagram and / or flow chart illustration can be realized by the dedicated hardware-based system of performing specified function or action or performing the combination of special hardware and computer instruction.

[0085] The description of various embodiments of the present invention has been given for the purpose of illustration, but it is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements existing in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for detecting scene transitions in video footage, the method comprising: Using a sliding window method to obtain multiple batches of frames in the video sequence; For each batch of frames in the plurality of batches of frames, performing a process comprising: In each of the plurality of different frames, identifying each detected object by a unique identifier using the position information; obtaining values ​​of confidence metrics associated with detected objects in consecutive frames; as well as Scene transitions in the video footage are detected based on the obtained values ​​of the confidence metric, wherein a gradual decrease in the values ​​of the confidence metric of the detected object in consecutive frames indicates a fade transition, and wherein an abrupt change in the values ​​of the confidence metric of the detected object in consecutive frames indicates a cut transition.

2. The method according to claim 1, wherein: Detecting a scene transition based on the obtained value of the confidence measure includes: analyzing the obtained values ​​of the confidence metric to identify trends or patterns in the values ​​of the confidence metric across the plurality of different frames in the video footage; and The scene transitions in the video footage are identified based on the identified trends or patterns in the values ​​of the confidence levels.

3. The method according to claim 2, wherein: Identifying scene transitions includes: comparing the identified trend or pattern in the confidence values ​​to a predetermined pattern of trends; and The scene transition in the video footage is identified based on the comparison result.

4. The method according to claim 2, further comprising: The identified scene transitions are classified based on the identified trends or patterns in the confidence values.

5. The method according to any one of claims 1 to 4, wherein: Detecting a scene transition based on the obtained value of the confidence measure comprises: In response to the obtained value of the confidence metric decreasing over consecutive frames of the video footage, the presence of a scene transition in the video footage is determined.

6. The method according to any one of claims 1 to 4, wherein: Detecting a scene transition based on the obtained value of the confidence measure comprises: In response to the obtained value of the confidence metric increasing over consecutive frames of the video footage, the presence of a scene transition in the video footage is determined.

7. The method according to any one of claims 1 to 4, further comprising: comparing the obtained value of the confidence measure with a threshold; as well as At least one of a start and an end of the scene transition is identified based on the comparison result.

8. The method according to claim 7, wherein: Identifying the end of the scene transition includes: determining frames in the video footage for which the obtained value of the confidence metric decreases from a value exceeding the threshold to a value not exceeding the threshold; and An end of the scene transition is determined based on the determined frame.

9. The method according to claim 7, wherein: Identifying the start of the scene transition includes: determining frames in the video footage for which the obtained value of the confidence metric increases from a value that does not exceed the threshold to a value that exceeds the threshold; and A start of the scene transition is determined based on the determined frame.

10. The method according to any one of claims 1 to 4, wherein: The values ​​for which confidence measures are obtained include: obtaining a first value of the confidence metric associated with the detected object in a first frame of the video footage; and A second value of the confidence metric associated with the detected object in a second subsequent frame of the video footage is obtained.

11. The method according to claim 10, wherein: The first frame and the second frame in the video footage are consecutive frames in the video footage.

12. A computer program product for detecting scene transitions in video footage, the computer program product comprising: A computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit to perform the method according to any one of claims 1 to 11.

13. A computer program product comprising a computer program stored on a computer readable medium and loadable into the internal memory of a digital computer, and comprising software code portions for executing the method according to any one of claims 1 to 11 when the software code portions are run on a computer.

14. A system for detecting scene transitions in video footage, the system comprising one or more processors and one or more tangible storage media storing programming instructions executed by the one or more processors, the programming instructions comprising instructions for: Using a sliding window method to obtain multiple batches of frames in the video sequence; For each batch of frames in the plurality of batches of frames, performing a process comprising: In each of the plurality of different frames, identifying each detected object by a unique identifier using the position information; obtaining, via an interface, values ​​of confidence metrics associated with detected objects in consecutive frames; as well as Scene transitions in the video footage are detected based on the obtained values ​​of the confidence metric, wherein a gradual decrease in the values ​​of the confidence metric of the detected object in consecutive frames indicates a fade transition, and wherein an abrupt change in the values ​​of the confidence metric of the detected object in consecutive frames indicates a cut transition.

15. The system of claim 14, wherein: Instructions for detection include instructions for: analyzing the obtained values ​​of the confidence metric to identify trends or patterns in the values ​​of the confidence metric across the plurality of different frames in the video footage; as well as Scene transitions in the video footage are identified based on the identified trends or patterns in the confidence values.

16. The system according to claim 14 or 15, wherein: The detection includes: In response to the obtained value of the confidence metric decreasing over consecutive frames of the video footage, the presence of a scene transition in the video footage is determined.

17. The system according to claim 14 or 15, wherein: The detection includes: In response to the obtained value of the confidence metric increasing over consecutive frames of the video footage, the presence of a scene transition in the video footage is determined.

18. The system according to claim 14 or 15, wherein: The detection includes: comparing the obtained value of the confidence measure with a threshold; and At least one of a start and an end of the scene transition is identified based on the comparison result.

19. The system of claim 18, wherein: Identifying the start of the scene transition includes: determining frames in the video footage for which the obtained value of the confidence metric increases from a value that does not exceed the threshold to a value that exceeds the threshold; and A start of the scene transition is determined based on the determined frame.

20. The system according to claim 14 or 15, wherein: The values ​​for which confidence measures are obtained include: obtaining a first value of the confidence metric associated with the detected object in a first frame of the video footage; and A second value of the confidence metric associated with the detected object in a second subsequent frame of the video footage is obtained.

Images