IMAGE CLASSIFICATION DEVICE, IMAGE CLASSIFICATION METHOD, AND PROGRAM

The image classification device efficiently clusters facial feature vectors to quickly identify target objects in video files, addressing the inefficiencies of manual inspection by automating the process and improving search speed.

JP7580713B2Active Publication Date: 2024-11-12GROOVENAUTS INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2020158880
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-09-23
Publication Date
2024-11-12
Estimated Expiration
2040-09-23

AI Technical Summary

Technical Problem

Existing technologies require manual visual inspection of video files to identify specific individuals, which is time-consuming and labor-intensive.

Method used

An image classification device that calculates and clusters facial feature vectors to quickly identify images containing a target object within a large number of video frames, using algorithms like DBSCAN and OPTICS for efficient clustering.

Benefits of technology

Enables rapid and easy confirmation of images containing a target object by simplifying the process to a few user operations, reducing the time and effort required for manual inspection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580713000001
    Figure 0007580713000001
  • Figure 0007580713000002
    Figure 0007580713000002
  • Figure 0007580713000003
    Figure 0007580713000003
Patent Text Reader

Abstract

To provide a technique capable of quickly and easily confirming an image showing an object among a huge number of images contained in a video file.SOLUTION: An image classification apparatus 200 acquires a large number of divided images from a video file, and detects a face image for each acquired divided image. The image classification apparatus 200 calculates a facial feature vector for each detected face image and clusters the feature vector. The image classification apparatus 200 outputs the clustering result to a user terminal 100. A user can quickly grasp, for example, which divided image includes the face image of a specific person (such as a talent) by confirming the clustering result displayed on the user terminal 100.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image classification device, an image classification method, and a program. [Background technology]

[0002] 2. Description of the Related Art There is known a technique for acquiring face images of people (such as entertainers) appearing in a program from the video of the program in order to identify the people appearing in the program (see, for example, Non-Patent Document 1). According to this technology, a list of facial images of the main people in a video can be automatically generated and presented to the user, so that the user can select a program to watch by referring to the presented list of facial images. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “Technology developed to detect and display human faces and automatically generate video lists”, [online], January 10, 2008, NEC Corporation, [Retrieved July 29, 2020], Internet<http: / / www.nec.co.jp / press / ja / 0801 / 1004.html> Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology disclosed in Non-Patent Document 1, for example, in order to check which video shows a particular person (target object), the television program production staff must check and visually inspect all video files, which is a huge amount of time-consuming and cumbersome work.

[0005] The present invention has been made in consideration of the circumstances described above, and one of its objectives is to provide a technology that enables quick and easy identification of images that show an object from among the vast number of images contained in a video file. [Means for solving the problem]

[0006] An image classification device according to one embodiment of the present disclosure comprises a calculation unit that calculates a feature vector of an object image, a classification unit that generates new cluster information that associates the feature vector of a new object image with a cluster by clustering the feature vector of the object image, a storage unit that stores master cluster information that associates the feature vector of an existing object image with a cluster to which it belongs, an integration unit that integrates the new cluster information with the master cluster information and updates the master cluster information, and an output unit that outputs a result of clustering based on the updated master cluster information. Effect of the Invention

[0007] According to a specific aspect of the present invention, it becomes possible to quickly and easily check images in which a target object is shown from among a huge number of images contained in a video file. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a clustering system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal and an image classification device. [Diagram 3] FIG. 2 is a block diagram showing a functional configuration of the image classification device. [Figure 4] FIG. 1 is a schematic diagram for explaining an outline of functions of an image classification device; [Diagram 5] FIG. 1 is a schematic diagram for explaining an outline of functions of an image classification device; [Figure 6] FIG. 1 is a schematic diagram for explaining an outline of functions of an image classification device; [Figure 7] FIG. 13 is a diagram illustrating an example of a label name setting screen. [Figure 8] FIG. 13 is a diagram illustrating an example of a clustering result screen. [Figure 9] An example of the Master Cluster Results Table before and after reclustering is shown in Figure 1. [Figure 10] An example of the master cluster table before and after reclustering is shown in Figure 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described with reference to the accompanying drawings, in which like reference numerals denote like or similar structures.

[0010] A. This embodiment <System configuration> FIG. 1 is a diagram showing an example of the configuration of a clustering system 1000 according to this embodiment. The clustering system 1000 includes a user terminal 100 operated by a user, and an image classification device 200. The user terminal 100 and the image classification device 200 are capable of communicating with each other via a network N.

[0011] One or more portions of network N may be a wired network or a wireless network, which may include, by way of example and without limitation, an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular network, Integrated Service Digital Networks (ISDNs), wireless LANs, Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Bluetooth, satellite communications, or the like, or a combination of two or more thereof.

[0012] <Overview of Clustering System 1000> In the clustering system 1000 according to the present embodiment, the image classification device 200 acquires a large number of divided images from a video file, and detects a face image in each of the acquired divided images. The image classification device 200 calculates a face feature vector for each detected face image, and clusters the face feature vector. The image classification device 200 outputs the clustering result to the user terminal 100. By checking the clustering result displayed on the user terminal 100, the user can quickly understand which divided image contains the face image of a specific person (such as a celebrity). Note that, although face images are clustered in the present embodiment, images of any object contained in a video file (such as images of cars, motorcycles, airplanes, ships, birds, fish, etc.) can be clustered.

[0013] The user terminal 100 is a terminal operated by a user (for example, a program production staff member, etc.) and is configured by a personal computer, a handheld computer device, a smartphone, a mobile phone, a wearable terminal, etc. The user appropriately operates the user terminal 100 to input an instruction to execute clustering, etc.

[0014] The image classification device 200 is configured, for example, by a server computer, and executes clustering of face images, etc. (details will be described later). Note that the computer configuring the user terminal 100 and the image classification device 200 does not necessarily have to be a single unit, and may be configured by multiple computers and storages distributed over a network N.

[0015] <Hardware configuration> 2 is a diagram showing an example of the hardware configuration of the user terminal 100 and the image classification device 200. The user terminal 100 and the image classification device 200 each have a CPU (Central Processing Unit) 1, a storage device 2 such as a ROM (Read Only Memory) or RAM (Random Access Memory), a communication IF (Interface) 3 for wired or wireless communication, an input device 4 for accepting input operations, and an output device 5 for outputting information. The input device 4 is, for example, a keyboard, a mouse, a touch panel, a microphone, etc. The output device 5 is, for example, a display (screen) and / or a speaker, etc.

[0016] Hereinafter, the functional configuration and functional overview of the image classification device 200 will be described with reference to FIGS.

[0017] <Functional configuration and function overview> FIG. 3 is a block diagram showing the functional configuration of the image classification device 200, and FIGS. 4 to 6 are schematic diagrams for explaining an outline of the functions of the image classification device 200. As shown in FIG.

[0018] The image classification device 200 includes a storage unit 210, a receiving unit 220, a dividing unit 230, a detecting unit 240, a generating unit 250, a calculating unit 260, a classifying unit 270, an integrating unit 280, and an output unit 290. The reception unit 220, the division unit 230, the detection unit 240, the generation unit 250, the calculation unit 260, the classification unit 270, the integration unit 280, and the output unit 290 are realized by the CPU 1 of the image classification device 200 executing various programs stored in the storage device 2. The programs can be stored in a storage medium such as a USB memory or a CD-ROM.

[0019] The storage unit 210 is realized by the storage device 2, and stores a control program P1 for realizing clustering of face images, various video files, various data related to clustering, and the like.

[0020] The receiving unit 220 receives various instructions related to the clustering of face images in response to an operation of the user terminal 100 by the user.

[0021] The division unit 230 acquires a video file (such as a moving image file) designated by the user from the storage unit 210, identifies a number of still images from the video file, and outputs them sequentially as divided images (see S1 in FIG. 4).

[0022] The detection unit 240 detects a face and each part of the face (e.g., eyebrows, eyes, nose, mouth, etc.) for each divided image, specifies this as a face image (see S2 in FIG. 4), and obtains the coordinates of the face and each part of the face (hereinafter collectively referred to as "face coordinates"). The detection unit 240 outputs the face image and the face coordinates to the generation unit 250.

[0023] The generating unit 250 generates an image in which the face is facing forward (hereinafter, also referred to as a "frontal face image") based on the face image and face coordinates supplied from the detecting unit 240, and outputs the image to the calculating unit 260 (see S3 shown in FIG. 4). By facing the face forward, it becomes possible to improve the accuracy of face recognition. In order to generate the frontal face image (direction-changed image), for example, a method of correction based on each part of the face or a method using GAN (Generative Adversarial Networks) can be adopted.

[0024] The calculation unit 260 calculates a face feature vector for each of all the frontal face images supplied from the generation unit 250, and generates a face feature vector table TA1 (see S4 shown in FIG. 4).

[0025] In the face feature vector table TA1, a feature vector ID, an image ID, face coordinates, and a face feature vector are registered in association with each other. The feature vector ID is identification information for identifying a face feature vector, and the image ID is identification information for identifying a divided image including a face image. The face coordinates are a face image acquired by the detection unit 240. The face feature vector is a 512-dimensional feature vector of a frontal face image. Note that in the example of the face feature vector table TA1 shown in FIG. 4, two feature vectors with the same image ID (="10") and different feature vector IDs (="30", "31") are shown. This means that multiple faces (here, two) exist in one divided image.

[0026] When the calculation section 260 generates the facial feature vector table TA1, it stores it in the storage section 210 and also outputs to the classification section 270 each facial feature vector to which the feature vector ID has been added.

[0027] The classification unit 270 clusters the facial feature vectors of all the frontal face images supplied from the calculation unit 260 (see S5 in FIG. 5). Here, "clustering" refers to a process of classifying a data set of facial feature vectors into groups called clusters, and clustering algorithms such as DBSCAN (Density-based spatial clustering of applications with noise), OPTICS (Ordering Points To Identify the Clustering Structure), x-means, and G-means can be used. However, other clustering algorithms that do not require the number of classes to be determined in advance may also be adopted.

[0028] The classification unit 270 assigns a provisional cluster label (cluster to which the vector belongs) to each facial feature vector through clustering, generates a provisional cluster result table TA2 and a provisional cluster table TA3, and outputs them to the integration unit 280. In the provisional cluster result table TA2, feature vector IDs and provisional cluster labels are registered in association with each other. Note that facial feature vectors to which a negative provisional cluster label (= "-1") has been assigned (for example, a feature vector of an image showing the back of a person's head instead of a face) are treated as noise.

[0029] Meanwhile, in the provisional cluster table TA3, provisional cluster labels and face feature vectors (centroids of new clusters) of new frontal face images for which clustering has been performed this time are registered in association with each other.

[0030] When the integration unit 280 receives the provisional cluster result table TA2 and the provisional cluster table TA3 from the classification unit 270, it accesses the storage unit 210 and reads out the master cluster result table TA4 and the master cluster table TA5 (see S6 in FIG. 5). In the master cluster result table TA4, existing feature vector IDs and cluster labels (clusters to which they belong) are registered in association with each other. Meanwhile, in the master cluster table TA5, cluster labels and facial feature vectors of existing frontal face images are registered in association with each other. The master cluster result table TA4 and the master cluster table TA5 are tables generated and registered by the classification unit 270 in the clustering performed previously or previously.

[0031] The integration unit 280 compares the provisional cluster result table TA2 and the provisional cluster table (new cluster information) TA3 with the master cluster result table TA4 and the master cluster table (existing cluster information) TA5, and updates the registration contents of the master cluster result table TA4 and the master cluster table TA5 by integrating (clustering) them (S7 shown in FIG. 6). For convenience, the updated master cluster result table TA4 and master cluster table TA5 will be referred to as the master cluster result table (latest) TA4 and master cluster table (latest) TA5, respectively.

[0032] As shown in the master cluster result table (latest) TA4, the tentative cluster label “1” is updated to the cluster label “5”, the tentative cluster label “-1” is not registered due to noise, and the tentative cluster label “32” is newly added because there is no corresponding cluster label.

[0033] Also, as shown in the master cluster table (latest) TA5, the centroids of the existing clusters (i.e., the existing facial feature vectors) are updated as appropriate. Specifically, if the new facial feature vector is similar to the existing facial feature vector, the integration unit 280 updates the existing facial feature vector to the new facial feature vector. On the other hand, if the new facial feature vector is not similar to the existing facial feature vector, the integration unit 280 adds the new facial feature vector to the master cluster table TA5.

[0034] Whether a new facial feature vector is similar to an existing facial feature vector is determined based on, for example, the similarity. If the similarity exceeds a set threshold, the integration unit 280 determines that the two are similar, whereas if the similarity does not exceed the set threshold, the integration unit 280 determines that the two are not similar.

[0035] The integration unit 280 stores the master cluster result table (latest) TA4 and the master cluster table (latest) TA5 in the storage unit, and notifies the output unit 290 that the integration (clustering) has ended.

[0036] When the output unit 290 receives a request for confirmation of the clustering result from the user terminal 100, it transmits the clustering result to the user terminal 100 in response to the request. Specifically, the output unit 290 reads out the facial feature vector table TA1, the master cluster result table (latest) TA4, the master cluster table (latest) TA5, and the divided images from the memory unit 210, and displays the label name setting screen and the clustering result screen on the display of the user terminal 100.

[0037] <Label name setting screen> FIG. 7 is a diagram showing an example of the label name setting screen G1. Since each cluster label is randomly assigned by the image classification device 200, it is difficult for a user to remember which cluster label is set to which person's face. Therefore, in this embodiment, an easy-to-understand name (cluster identification information) can be registered for each cluster label so that the user can easily search for each cluster.

[0038] As shown in FIG. 7, the label name setting screen G1 has a label name registration field provided alongside the cluster label and face image. The user sequentially registers label names while checking the cluster labels and face images. For example, if the person is a celebrity, the user registers the celebrity's stage name (such as "ANDY" or "JACK") as the label name. When the reception unit 220 receives a label name input by the user, it outputs the label name to the integration unit 280. When the integration unit 280 receives the label name from the reception unit 220, it stores in the storage unit 210 a master cluster name table TA6 in which the cluster labels and the label names are associated with each other (S8 shown in FIG. 6).

[0039] In the master cluster name table TA6 shown in Fig. 6, the same label name "JJ" is registered for cluster labels "39" and "40". This is because the same person may, for example, be "with makeup" and "without makeup". If you simply want to search for a person, information on whether or not they are wearing makeup is not necessary, so it is sufficient to register only the name "JJ" as shown in Fig. 6. However, if you want to search by distinguishing whether or not they are wearing makeup (for example, if you want to search for "JJ" with makeup on), you can register "JJ (with makeup)" and "JJ (without makeup)" in the master cluster name table TA6.

[0040] <Clustering results screen> FIG. 8 is a diagram showing an example of the clustering result screen G2. The cluster name, the image IDs of the detected divided images, the divided images in thumbnail format, and the URLs of the divided images are displayed in association with each other on the clustering result screen G2.

[0041] As shown in FIG. 8, for each face detected in a face image on a divided image, a frame F framing the face and cluster-related information C for that face are displayed. Here, "cluster-related information" refers to information related to a cluster, such as a feature vector ID or a cluster label. Note that since it would be difficult to confirm the face image if the cluster-related information were displayed directly on top of the face image, it is displayed in a speech bubble, for example, by hovering the mouse over the information. The example shown in FIG. 8 illustrates a case where multiple face images exist in one divided image, but there may also be a case where a single face image exists.

[0042] (When searching for a specific person) The user selects a label name of a particular person (for example, "JACK") from among a plurality of label names NM displayed in a pull-down menu shown in FIG.

[0043] When such a selection operation is performed, a split image including the face image of the specific person "JACK" is displayed on the clustering result screen G2. In this way, the user can quickly check the split image in which the face of the desired person is shown by performing the simple operation of selecting the label name.

[0044] In the above example, it is assumed that a label name set by the user is used to search for a segmented image showing the face of a desired person, but a search may also be performed using a cluster label instead of a label name.

[0045] Furthermore, an image may be used to search for a segmented image showing a face of a desired person. For example, a user may select a face image of a desired person (e.g., "TARO") stored in the user terminal 100 or the like, and instruct the image classification device 200 to search for segmented images including a face that matches this face.

[0046] Next, we will explain the case where the user instructs reclustering after checking the clustering result screen. In the following, we assume that after checking the faces belonging to the clusters, twin brothers ("ICHIRO" and "JIRO") are classified into the same cluster because their faces look similar.

[0047] FIG. 9 is a diagram illustrating the master cluster result table TA4 before and after reclustering. As shown in A of FIG. 9, in the master cluster result table TA4 before reclustering, the same cluster (cluster label = "25") is set for feature vector ID = "36" ("ICHIRO") and feature vector ID = "37" ("JIRO"). Since "ICHIRO" and "JIRO" are set to the same cluster, the user operates the user terminal 100 to set a stricter threshold for cluster label = "25" and instructs execution of reclustering. This "threshold" is a threshold for determining whether or not a cluster is similar to an existing cluster (here, cluster label = "25"), and can be set and changed appropriately for each cluster by the user operating the user terminal 100.

[0048] When the classification unit 270 of the image classification device 200 receives an instruction to perform reclustering from the user terminal 100, it performs reclustering on the face feature vector corresponding to the cluster label="25". Note that the operations after the reclustering by the classification unit 270 are the same as the operations of S5 to S8 described above, and therefore the explanation will be omitted.

[0049] If the classification unit 270 determines that "ICHIRO" and "JIRO" belong to different clusters as a result of the reclustering, the integration unit 280 sets a new cluster label (="26") for "JIRO". As a result, as shown in B of Fig. 9, in the master cluster result table (latest) TA4 after the reclustering, a new cluster label "26" is set for the feature vector ID "37" ("JIRO"). On the other hand, the cluster label "25" for the feature vector ID "36" ("ICHIRO") is maintained.

[0050] FIG. 10 is a diagram illustrating the master cluster table TA5 before and after reclustering. As is clear from comparing Figure 10A with Figure 10B, as a result of reclustering, the 512-dimensional feature vector of "JIRO" (feature vector ID = "37") is registered for the new cluster label = "26" in the master cluster table (latest) TA5. On the other hand, for cluster label = "25", the 512-dimensional feature vector of "ICHIRO" (feature vector ID = "36") that has already been registered is maintained.

[0051] As described above, according to this embodiment, a user can quickly and easily check the split images that show the face of a desired person from among the vast number of split images contained in a video file, by simply selecting a label name that the user has set.

[0052] B. Modifications It should be noted that the present invention is not limited to the above-described embodiment, and can be embodied in various other forms without departing from the scope of the present invention. Therefore, the above-described embodiment is merely illustrative in all respects and should not be interpreted as being restrictive.

[0053] For example, by using object tracking technology based on image processing, it may be possible to track the face of a person specified by the user (object), or to learn the background and scene contained in the segmented images, so that it is possible to search for the background and scene. In addition, in this embodiment, a frontal face image with the face facing forward is generated in order to improve the recognition accuracy of the face (object), but the present invention is not limited to this. For example, a right-facing face image with the face facing right or a left-facing face image with the face facing left may be generated to calculate a feature vector. Of course, the feature vector may be calculated not only by using a face image in one pose, but also by using face images in multiple poses (for example, face images in three poses: facing forward, facing right, and facing left). This makes it possible to further improve the recognition accuracy of the face.

[0054] C.Other In this specification, a "unit" does not simply mean a physical configuration, but also includes cases where the processing performed by the "unit" is realized by software. In addition, the processing performed by one "unit" or device may be realized by two or more physical configurations or devices, and the processing performed by two or more "units" or devices may be realized by one physical means or device.

[0055] The steps in each process described above in this specification can be arbitrarily changed in order or executed in parallel as long as no inconsistency occurs in the process content. [Explanation of symbols]

[0056] 1000...clustering system, 100...user terminal, 200...image classification device, 210...memory unit, 220...reception unit, 230...division unit, 240...detection unit, 250...generation unit, 260...calculation unit, 270...classification unit, 280...integration unit, 290...output unit, P1...control program, TA1...facial feature vector table, TA2...temporary cluster result table, TA3...temporary cluster table, TA4...master cluster result table, TA5...master cluster table, TA6...master cluster name table, G1...label name setting screen, G2...clustering result screen.

Claims

1. A calculation unit for calculating a feature vector of an object image; a classification unit that generates new cluster information in which the feature vector of the new object image is associated with a cluster by clustering the feature vector of the object image; a storage unit that stores master cluster information that associates a feature vector of the existing object image with a cluster to which the object image belongs; an integration unit that integrates the new cluster information with the master cluster information and updates the master cluster information; an output unit that outputs a result of clustering based on the updated master cluster information; An image classification device comprising:

2. 2. The image classification device according to claim 1, wherein the integration unit determines whether a feature vector of the new object image is similar to a feature vector of the existing object image, and if it determines that they are similar, updates the feature vector of the existing object image included in the master cluster information to the feature vector of the new object image, whereas if it determines that they are not similar, adds the feature vector of the new object image to the master cluster information.

3. The output unit outputs a result of the clustering to a user terminal; The image classification device according to claim 1 , further comprising a reception unit configured to receive cluster identification information for identifying the cluster to which the image belongs from the user.

4. The image classification device according to claim 3 , wherein the output unit displays, as a result of the clustering, the object image shown in the divided image and cluster-related information regarding the cluster to which the object image belongs, on a display device of the user's terminal.

5. A generator for generating a direction-changed image in which a direction of the object is changed based on the object image, The image classification device according to claim 1 , wherein the calculation unit calculates a feature vector of the orientation-changed image.

6. A calculation step of calculating a feature vector of the object image; A classification step of clustering the feature vectors of the object images to generate new cluster information in which the feature vectors of the new object images correspond to the clusters to which they belong; an integration step of integrating the new cluster information with existing master cluster information in which the feature vector of the object image is associated with a cluster to which the object image belongs, and updating the master cluster information; an output step of outputting a clustering result based on the updated master cluster information; Image classification methods including:

7. On the computer, A calculation step of calculating a feature vector of the object image; A classification step of clustering the feature vectors of the object images to generate new cluster information in which the feature vectors of the new object images correspond to the clusters to which they belong; an integration step of integrating the new cluster information with existing master cluster information in which the feature vector of the object image is associated with a cluster to which the object image belongs, and updating the master cluster information; an output step of outputting a clustering result based on the updated master cluster information; A program for executing.

Citation Information

Patent Citations

  • Image processing apparatus, image processing method, and program

    JP2011134115A

  • Data classification device and data classification method

    JP2013242810A

  • Representative image display device and representative image selection method

    WO2010041377A1

  • Face comparison device, method, and recording medium

    WO2015146101A1