Reducing resource costs in visual recognition

By clustering and reclustering video frames based on similarity and visual recognition results, the method significantly reduces resource consumption while maintaining high-quality image identification in video streams.

JP7619741B2Active Publication Date: 2025-01-22INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2022554600
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-04-12
Filing Date
2021-03-17
Publication Date
2025-01-22
Estimated Expiration
2041-03-17

AI Technical Summary

Technical Problem

Visual recognition of images in video streams is resource-intensive, necessitating techniques that reduce resource utilization.

Method used

Implementing a method that coarsely segments video frames into clusters, samples frames with maximum dissimilarity, determines cluster quality, and reclusters based on visual recognition results to improve quality, thereby reducing the number of frames analyzed.

Benefits of technology

Reduces visual recognition resource usage by up to 60% compared to 1 frame per second sampling and up to 20% compared to 1 frame per shot sampling, achieving high-quality visual recognition with fewer frames.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007619741000001
    Figure 0007619741000001
  • Figure 0007619741000002
    Figure 0007619741000002
  • Figure 0007619741000003
    Figure 0007619741000003
Patent Text Reader

Abstract

Embodiments may provide techniques for identifying images that can provide reduced resource utilization due to reduced sampling of video frames for visual recognition. For example, in one embodiment, a method for visual recognition processing may be implemented in a computer system including a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method including coarsely segmenting video frames of the video stream into a plurality of clusters based on scenes in the video stream, sampling a plurality of video frames from each cluster, determining a quality of each cluster, and reclustering the video frames of the video stream to improve the quality of at least some of the clusters.
Need to check novelty before this filing date? Find Prior Art

Description

[Background technology]

[0001] The present invention relates to a technique that provides image identification that can provide a reduction in resource utilization due to a reduction in sampling of video frames for visual recognition.

[0002] Due to the growth of video data on the Internet, automated video analysis and recognition of images in video streams is becoming commonplace. Typically, machine learning (ML) techniques are used for computer vision tasks, specifically recognizing objects in images. However, visual recognition of a given image is expensive in terms of resource utilization.

[0003] Therefore, a need has arisen for techniques that provide identification of images that can provide reduced resource utilization. Summary of the Invention

[0004] Embodiments may provide techniques that provide image identification that may provide reduced resource utilization due to reduced sampling of video frames for visual recognition.

[0005] In one embodiment, a method for visual recognition processing can be implemented in a computer system including a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method including coarsely segmenting video frames of the video stream into a plurality of clusters based on scenes in the video stream, sampling a plurality of video frames from each cluster, determining a quality of each cluster, and re-clustering the video frames of the video stream to improve the quality of at least some of the clusters.

[0006] In an embodiment, coarsely segmenting the video frames of the video stream may be based on image similarity of the scene. Determining a quality of each cluster may be based on a similarity of visual recognition results of the cluster. Sampling the plurality of video frames may include sampling two frames in a cluster having a longest similarity distance or a largest dissimilarity. The method may further include repeating the sampling on the reclustered video frames, determining a quality of each cluster on the reclustered video frames, and reclustering the video frames of the video stream, where the visual recognition improves the quality of the clustering, and the reclustering improves the quality of the visual recognition. The method may further include selecting a representative image from each cluster for visual recognition once the final clusters are determined. At least a portion of the selected representative image from each cluster may represent a cluster center.

[0007] In one embodiment, a system for visual recognition processing includes a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to coarsely segment video frames of the video stream into a plurality of clusters based on scenes in the video stream, sample a plurality of video frames from each cluster, determine a quality of each cluster, and recluster the video frames of the video stream to improve the quality of at least some of the clusters.

[0008] In one embodiment, a computer program product for visual recognition processing has non-transitory computer readable storage having program instructions embodied therein, the program instructions being executable by a computer to cause the computer to perform a method including coarsely segmenting video frames of the video stream into a plurality of clusters based on scenes in the video stream, sampling a plurality of video frames from each cluster, determining a quality of each cluster, and reclustering the video frames of the video stream to improve the quality of at least some of the clusters.

[0009] The details of the present invention, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numbers and designations refer to like elements and in which: [Brief description of the drawings]

[0010] [Figure 1] 1 is an exemplary diagram of a system in accordance with an embodiment of the present technology; [Diagram 2] FIG. 2 is an exemplary flow diagram of a process for visual recognition sampling in accordance with an embodiment of the present technology. [Diagram 3] FIG. 1 is an exemplary diagram of clustering according to an embodiment of the present technology. [Figure 4] FIG. 1 is an exemplary block diagram of a computer system that may implement the processes included in the embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Embodiments may provide techniques that provide image identification that can provide reduced resource utilization due to reduced sampling of video frames for visual recognition. Embodiments may reduce the cost of using visual recognition (VR) by clustering based on image similarity and re-clustering based on similarity of VR results. As a result, VR sampling can be reduced by up to 60% compared to "1 frame per second" sampling and up to 20% compared to "1 frame per shot" sampling.

[0012] An exemplary system 100 utilizing the present technology is shown in FIG. 1. As shown in FIG. 1, digital video data 102 may include a stream or series of still images 104A-N known as frames that create a motion signature, and may also include audio data. The frequency at which successive frames are displayed is called the frame rate, expressed in frames per second (fps). The frames 104A-N may be processed by a frame analysis process 106 to perform a number of functions, such as visual or image recognition, video annotation, etc. Such functions may enable automatic summarization and organization of large amounts of video data, such as video recommendation, video surveillance, Internet platforms, etc.

[0013] The frame analysis process 106 can use machine learning (ML) techniques for video analysis. One approach is to treat video frames as still images and analyze every frame to generate a prediction of the frame's content. However, since each video contains a large number of frames, processing every video frame is computationally resource intensive even for short video clips. Also, consecutive video frames may be duplicated in content. Therefore, the frames to be analyzed can be sampled, i.e., only selected frames can be analyzed.

[0014] Common techniques for selecting sample frames to be analyzed include periodic sampling and shot detection. Periodic sampling may select one or more frames on a fixed or adjustable periodic basis, such as one frame per second. Shot detection may be used to detect when the content of a frame, or shot, of a video stream has changed significantly, and to select one or more frames based on the occurrence of such a shot change.

[0015] FIG. 2 illustrates a process 200 of the operation of the model 100 and is best viewed in conjunction with FIG. 1. The process 200 begins at 202 where the video stream 102 may be coarsely segmented based on the scenes present in the video stream 102. For example, the video stream 102 may be divided into initial coarse groups or clusters 110A, 110B, 110C, etc. based on the scenes in the video stream 102 rather than a clustering process as described below. For example, image similarity may be determined using a metric that may measure the visual similarity between images. For example, images may be represented in a feature space using traditional methods such as pixel values, color histograms, descriptors, etc., or more advanced methods such as deep representations using convolutional neural networks, etc. After generating the representations, a metric may be applied to measure how similar two images are based on the representations. This may be done, for example, using cosine similarity or using a Gaussian kernel on the L2 norm. To group images into initial coarse clusters, points may be depicted in the feature space (or an approximate reduced-dimensional equivalent), and the "distance" between points is assumed to represent dissimilarity when grouping them together into clusters.

[0016] Each scene segment can then be treated as a separate coarse or coarse cluster, with the frames in each cluster 110A, 110B, 110C, etc. being roughly similar. At 204, sampling may be performed in each cluster 110A, 110B, 110C, etc., using, for example, frames with large separation or maximum distance (dissimilarity). For example, the first frame in each cluster (or a frame near the beginning of each cluster) and the last frame in each cluster (or a frame near the end of each cluster) may be sampled. Typically, such frames with large separation or distance are the least similar frames in the cluster.

[0017] At 206, the quality of each cluster may be determined using the sampled frames by determining the similarity of the VR results of the sampled frames in each cluster. The similarity of the VR results may be, for example, a metric for how similar the concepts ("tags") from two images are. In practice, the similarity of the VR results may measure how similar two groups of words generated from images are. This may be done using techniques such as bag of words, intersection over union, word vector representation, etc.

[0018] If the quality of the cluster is good, i.e., the sampled frames (which are most likely not similar) are similar enough, the sampled frames may be representative of the cluster. For example, the quality of the cluster may be based on whether the images in the cluster have high VR result similarity. This makes it possible to determine whether the initial clustering (based on image similarity) is good enough or needs refinement. Visual recognition may then be performed only on these sampled frames. Since the sampled frames are at least similar to the other images in the cluster, the results of the visual recognition should be representative of the entire cluster. Thus, high quality visual recognition may be achieved while performing the actual visual recognition only on a relatively small sample of the frames in the video stream 102. This visual recognition may generate tags describing the sampled frames, and these tags may be used to determine the similarity of each cluster. For example, if the sampled frames are not similar enough based on the generated tags, the similarity of the images in the cluster may not be high enough.

[0019] In this case, the frames may be reclustered at 208. Clustering is a method of dividing elements (images given their representations) into groups. This may be done based on similarity measures given above. Examples of clustering approaches include K-Means, K-Mediods, spectral clustering, density-based spectral clustering, deep embedded clustering, and other advanced methods. In embodiments, a clustering method specifically tailored to the problem may be used. Such a method may be called Maximum Distance Cluster Tiling (MDCT), which may be, for example, an unsupervised learning method for efficient visual recognition of videos. In embodiments, the frames of the video may be represented in a low-level color feature space and assigned to clusters using a maximum distance constraint. Limiting the maximum distance over which frames are clustered may ensure that visual elements in the video are represented while eliminating redundancy.

[0020] Unlike traditional clustering, MDCT is motivated by the fact that two similar frames that would return the same visual tag by frame-based visual recognition will certainly be close to each other in low-level color space.Traditional clustering may cluster large, scattered groups of points that may have common features but do not necessarily contain the same visual tag.

[0021] An embodiment of MDCT may use a technique called greedy cluster agglomeration (Slavik, 1997), derived from the set covering problem, where the frames represent the universe and the sets are the frames with distances smaller than the longest. An embodiment may exploit a greedy algorithm for polynomial-time approximation of the NP-hard set covering problem, which is characterized by relatively firm bounds on optimality. For example, frames that cover the least covered frames may be selected as cluster centers and added in descending order, considering a cost function consisting of the longest distances of the frames in each cluster.

[0022] An embodiment may cluster (or recluster) the samples based on a maximum dissimilarity constraint. This is important given the goal of inferring concepts for all images in a cluster, because other methods tend to have both large and small clusters at the same time, which may break down the assumption that concepts apply to other images in the cluster. Additionally, an embodiment may perform an iterative method, continuing with more splits until there is a consensus on the sampled VR results. This can be done using MDCT, simply by changing the value of the maximum distance constraint.

[0023] For example, this automatically adjustable clustering may be performed using the longest distance between sampled frames as a criterion, and using the similarity of visual recognition results, e.g., based on tags. For example, if the sampled frames are not similar enough, the cluster may be split into two new clusters around two sampled frames. Similarly, if a sampled frame is more similar to a sampled frame in another cluster than to other sampled frames in that cluster, the frames may be re-clustered such that the more similar frames are in the same cluster. Then, 204-208 may be repeated until the quality of all or a sufficient number of clusters is sufficiently high, based on image similarity and based on visual recognition of the sampled frames during clustering. At 210, once the clusters are determined, a representative image that may represent a cluster center may be selected from each cluster for visual recognition.

[0024] Because visual recognition of the sampled frames is used to recluster the frames, the visual recognition can improve the quality of the clustering and the reclustering can improve the quality of the visual recognition.

[0025] An example of clustering according to the present technique is shown in FIG. 3, best viewed in conjunction with FIG. 2. As shown in this example, at 202, initial coarse clusters 302, 304 may be determined using, for example, image similarity measures as described above. At 204, sample frames from each initial cluster may be selected. For example, sample frames 306, 308 may be selected from cluster 302 and sample frames 310, 312 may be selected from cluster 304. In this example, sample frame 306 may be an image of a dog, sample frame 308 may be an image of a cat, or both sample frames 306 and 308 may be images of a bird. At 206, the quality of clusters 302 and 304 may be determined using, for example, VR result similarity measures as described above. In this example, it may be determined that frames 306 and 308 are not sufficiently similar and frames 310 and 312 are sufficiently similar. In this case, at 208, cluster 302 may be split into new clusters 314, 316, and cluster 304 may not be split. Process 200 may then repeat step 204 to select sample frames 318, 320 from cluster 314 and sample frames 322, 324 from cluster 316. Steps 204-208 may be repeated until a final cluster is determined, and at 210, a representative frame may be selected from the final cluster.

[0026] An exemplary block diagram of a computer system 400 on which processes involved in the embodiments described herein may be implemented is shown in FIG. 4. The computer system 400 may be implemented using one or more programmed general-purpose computer systems, such as embedded processors, systems on a chip, personal computers, workstations, server systems, and minicomputers or mainframe computers, or in a distributed networked computing environment. The computer system 400 may include one or more processors (CPUs) 402A-402N, input / output circuitry 404, a network adapter 406, and memory 408. The CPUs 402A-402N execute program instructions to perform the functions of the present communication systems and methods. Typically, the CPUs 402A-402N are one or more microprocessors, such as Intel CORE® processors. 4 illustrates an embodiment in which computer system 400 is implemented as a single multiprocessor computer system in which multiple processors 402A-402N share system resources such as memory 408, input / output circuitry 404, and network adapter 406. However, the present communication systems and methods also include embodiments in which computer system 400 is implemented as multiple networked computer systems, which may be single-processor computer systems, multiprocessor computer systems, or a mixture thereof.

[0027] The input / output circuitry 404 provides the ability to input data to and output data from the computer system 400. For example, the input / output circuitry may include input devices such as a keyboard, mouse, touchpad, trackball, scanner, analog-to-digital converter, output devices such as a video adapter, monitor, printer, and modem. The network adapter 406 interfaces the device 400 with a network 410. The network 410 may be any public or proprietary LAN or WAN, including, but not limited to, the Internet.

[0028] Memory 408 stores program instructions that are executed by CPU 402 and data that is used and processed by CPU 402 to implement the functions of computer system 400 . Memory 408 may include, for example, electronic memory devices such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), flash memory, and electromechanical memory such as magnetic disk drives, tape drives, and optical disk drives, which may use an Integrated Drive Electronics (IDE) interface or a variation or extension thereof, such as Extended IDE (EIDE) or Ultra Direct Memory Access (UDMA), or a Small Computer System Interface (SCSI)-based interface or a variation or extension thereof, such as Fast SCSI, Wide SCSI, Fast and Wide SCSI, or a Serial Advanced Technology Attachment (SATA) or a variation or extension thereof, or a Fibre Channel Arbitrated Loop (FC-AL) interface.

[0029] The contents of memory 408 may vary depending on the functions that computer system 400 is programmed to execute. In the example shown in FIG. 4, exemplary memory contents are shown that represent routines and data for an embodiment of the process described above. However, one skilled in the art will recognize that these routines, along with their associated memory contents, may not be contained in one system or device, but rather may be distributed across multiple systems or devices based on well-known engineering considerations. The present systems and methods may include any and all such arrangements.

[0030] In the example shown in FIG. 4, memory 408 may include a segmentation routine 412, a sampling routine 414, a cluster quality routine 416, a reclustering routine 418, a visual recognition routine 420, and a representative selection routine 422, video stream data 424, and an operating system 426. Segmentation routine 412 may include a software routine for coarsely segmenting a scene present in video stream data 424, for example using image similarity, as described above. Sampling routine 414 may include a software routine for performing sampling on clusters, for example using frames that have a large separation or distance from each other, as described above. Cluster quality routine 416 may include a software routine for determining the similarity of sampled frames within each cluster, for example using VR result similarity, as described above. Reclustering routine 418 may include a software routine for reclustering clusters to improve the quality of their clustering, as described above. The visual recognition routine 420 may include software routines for recognizing objects in video frames and generating descriptive tags, as described above. The representative selection routine 422 may include software routines for selecting a frame from each cluster that is representative of the images in the cluster, as described above. The video stream data 424 may include a stream or a series of still images, as described above. An operating system 426 may provide overall system functionality.

[0031] As shown in FIG. 4, the present communication system and method may include implementation on one or more systems that provide multiprocessor, multitasking, multiprocess, or multithreaded computing, or a combination thereof, as well as implementation on systems that provide only single processor, single threaded computing. Multiprocessor computing involves performing computing using more than one processor. Multitasking computing involves performing computing using more than one operating system task. A task is an operating system concept that refers to the combination of an executing program and bookkeeping information used by the operating system. Each time a program is executed, the operating system creates a new task for it. This task is like a package of the program in that it identifies the program with a task number and attaches other bookkeeping information. Many operating systems, including Linux, UNIX, OS / 2, and Windows, can run many tasks at the same time and are called multitasking operating systems. Multitasking is the ability of an operating system to run more than one executable file at the same time. Since each executable file runs in its own address space, the executables have no way to share memory. This has an advantage because it cannot damage the execution of any other programs running on the system, but the programs have no way to exchange any information through the operating system (or by reading files stored in the file system). Multiprocess computing is similar to multitasking computing, since the terms task and process are often used interchangeably, although some operating systems distinguish between the two.

[0032] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to perform aspects of the present invention. The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device.

[0033] The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge structures in grooves having instructions recorded thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, should not be construed as being themselves transitory signals such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted over wires.

[0034] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and sends the computer readable program instructions for storage in a computer readable storage medium in each computing / processing device.

[0035] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or partially on the remote computer or entirely on the remote computer. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions according to state information of the computer readable program instructions to implement aspects of the invention.

[0036] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0037] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to generate a machine such that the instructions, executed via the processor of the computer or other programmable data processing apparatus, generate means for implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium that can cause a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, and thus a computer readable storage medium having instructions stored therein may include an article of manufacture including instructions that implement aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0038] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, causing a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process.

[0039] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative embodiments, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, as well as combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or implements a combination of dedicated hardware and computer instructions.

[0040] Although specific embodiments of the present invention have been described, those skilled in the art will recognize that there are other embodiments that are equivalent to the described embodiments. It is understood, therefore, that the present invention is not limited to the specific exemplary embodiments, but rather is limited only by the scope of the appended claims.

Claims

1. 1. A computer-implemented method for visual recognition processing, the method comprising: segmenting video frames of the video stream into a plurality of clusters based on scenes of the video stream; sampling a separated number of video frames from each of said plurality of clusters; determining a quality of each of the plurality of clusters based on a concept similarity obtained by visual recognition among the plurality of sampled video frames; and clustering the video frames of the video stream based on a quality of each of the plurality of clusters to improve a quality of at least some of the clusters. The method includes:

2. The method of claim 1, wherein segmenting video frames of the video stream is based on image similarity of the scene.

3. A method as described in claim 1 or 2, wherein the sampling includes selecting the separated video frames based on dissimilarity between multiple video frames from each of the multiple clusters.

4. A computer implemented method for visual recognition processing, the method comprising: Segmenting video frames of the video stream into a plurality of clusters based on image similarity of scenes of the video stream; Sampling a plurality of video frames from each of the plurality of clusters, the plurality of video frames including two frames in the cluster that have a longest distance or a greatest dissimilarity to each other; determining a quality of each of the plurality of clusters based on a similarity of visual recognition results between the sampled plurality of video frames of each of the plurality of clusters; and clustering the video frames of the video stream based on a quality of each of the plurality of clusters to improve a quality of at least some of the clusters. The method includes:

5. sampling on the clustered video frames; determining a quality of each cluster on the clustered video frames; and Re-clustering the video frames of the video stream. and further comprising repeating The result of the visual recognition improves the quality of the clustering process, and the result of the clustering improves the quality of the next visual recognition process. The method according to claim 4.

6. The method of claim 5 , further comprising: once the final clusters are determined, selecting a representative image from each cluster for visual recognition.

7. The method of claim 6 , wherein at least a portion of the selected representative images from each cluster represents a centroid in the feature space of each cluster.

8. 1. A system for visual recognition processing, comprising: segmenting video frames of the video stream into a plurality of clusters based on scenes of the video stream; sampling a separated number of video frames from each of said plurality of clusters; determining a quality of each of the plurality of clusters based on a concept similarity obtained by visual recognition among the plurality of sampled video frames; and clustering the video frames of the video stream based on a quality of each of the plurality of clusters to improve a quality of at least some of the clusters. Run the system.

9. The system of claim 8, wherein segmenting video frames of the video stream is based on image similarity of the scene.

10. The system described in claim 8 or 9, wherein the sampling includes selecting the separated video frames based on dissimilarity between multiple video frames from each of the multiple clusters.

11. A system for visual recognition processing, comprising: Segmenting video frames of the video stream into a plurality of clusters based on image similarity of scenes of the video stream; Sampling a plurality of video frames from each of the plurality of clusters, the plurality of video frames including two frames in the cluster that have a longest similarity distance or a largest dissimilarity to each other; determining a quality of each of the plurality of clusters based on a similarity of visual recognition results between the sampled plurality of video frames of each of the plurality of clusters; and clustering the video frames of the video stream based on a quality of each of the plurality of clusters to improve a quality of at least some of the clusters. Run the system.

12. sampling on the clustered video frames; determining a quality of each cluster on the clustered video frames; and Re-clustering the video frames of the video stream. and further comprising repeating The result of the visual recognition improves the quality of the clustering process, and the result of the clustering improves the quality of the next visual recognition process. The system of claim 11.

13. The system of claim 12 , further comprising: once the final clusters are determined, selecting a representative image from each cluster for visual recognition.

14. The system of claim 13 , wherein at least a portion of the selected representative images from each cluster represent a centroid in the feature space of each cluster.

15. A computer program for visual recognition processing, the computer program causing a computer to carry out the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Image analysis device and image analysis method

    JP2009123095A

  • Index video generation device, video image search apparatus and video image search system

    JP2010102393A