Image data classification device, image data classification method executed by the image data classification device, and image data classification program

Abstract

To provide an image data classification apparatus configured to cause a storage unit to keep holding face characteristic data of members of a specific group.SOLUTION: An image data classification apparatus includes: a storage unit which stores multiple pieces of face characteristic data classified into multiple clusters; and a control unit which calculates, for each of the multiple pieces of face characteristic data, a product of weight of one of the multiple pieces of face characteristic data and the number of pieces of face characteristic data of one cluster to which the one piece of face characteristic face data belongs, out of the multiple clusters. The control unit deletes, based on the product for each of the multiple pieces of face characteristic data, at least one of the multiple pieces of face characteristic data, from the storage unit.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an image data classification device, Executed by the image data classification device Image data classification method and image data classification program Regarding the topic. [Background technology]

[0002] An image data classification device is being developed that classifies and stores a plurality of facial feature data extracted from a plurality of image data into a plurality of clusters. In the image data classification device, as the number of facial feature data increases, the amount of calculation required for classification also increases.

[0003] Specifically, in order to classify N pieces of facial feature data into multiple clusters, the distances between the N pieces of facial feature data must be calculated. Therefore, the amount of calculation required for classification is N 2 The order will be as follows.

[0004] As a result, the response speed of the image classification device to the user's operation of the operation unit becomes slower or the battery consumption of the image classification device increases, which is particularly problematic when trying to implement the functions of an image data classification device on a communication terminal such as a smartphone, which has a smaller memory capacity and lower computing power than a large computer.

[0005] One common approach to solving this problem is to set an upper limit on the number of facial feature data that the image data classification device can store. With this approach, if the amount of facial feature data stored in the storage unit reaches the upper limit and the image classification device acquires more facial feature data, the oldest stored facial feature data is deleted from the storage unit. In other words, a so-called FIFO (First-In / First-Out) method is adopted. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2011 / 089884 Summary of the Invention [Problem to be solved by the invention]

[0007] The above-described FIFO image data classification device may be used to take a photo containing many faces of other people, such as a group photo. In this case, when the amount of facial feature data stored in the storage unit reaches an upper limit and the image data classification device acquires new facial feature data from the group photo, the previously acquired facial feature data is deleted from the storage unit by the number of newly acquired facial feature data.

[0008] For example, facial feature data of members of a particular group, such as a family, may be lost. As a result, the accuracy of the image data classification device's ability to determine whether image data contains facial data of members of a particular group may be reduced. In addition, because all facial feature data of the group's composition has been lost, the image data classification device may no longer be able to determine whether a person in a photograph is a member of a particular group.

[0009] As can be seen from the above, when facial feature data of a large number of people, including people other than members of a specific group such as a family, is acquired, a problem arises in that the facial feature data of members of a specific group such as a family cannot be continuously stored in the memory unit.

[0010] The present disclosure has been made in consideration of the above-mentioned problems. An object of the present disclosure is to provide an image data classification device that can continuously store facial feature data of members of a specific group in a storage unit; Executed by the image data classification device An object of the present invention is to provide an image data classification method and an image data classification program. [Means for solving the problem]

[0011] The image data classification device of the present disclosure includes a memory unit that classifies and stores a plurality of facial feature data into a plurality of clusters, and a control unit that calculates, for each of the plurality of facial feature data, the product of the weight of one of the plurality of facial feature data and the number of facial feature data in one cluster to which the one of the plurality of clusters belongs, and the control unit deletes at least one of the plurality of facial feature data from the memory unit based on the product of each of the plurality of facial feature data.

[0012] The image data classification method disclosed herein classifies and stores multiple facial feature data into multiple clusters, calculates the product of the weight of one of the multiple facial feature data and the number of facial feature data in one cluster to which the one facial feature data belongs, for each of the multiple facial feature data, and the control unit deletes at least one facial feature data from the multiple facial feature data based on the product of each of the multiple facial feature data.

[0013] The image data classification program of the present disclosure is an image data classification program for causing a computer to operate as a memory unit that classifies and stores multiple facial feature data into multiple clusters, and a control unit that calculates, for each of the multiple facial feature data, the product of the weight of one of the multiple facial feature data and the number of facial feature data in one cluster to which the one facial feature data belongs, and the control unit deletes at least one of the multiple facial feature data based on the product of each of the multiple facial feature data. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a diagram showing a configuration of an image data classification system according to a first embodiment. [Figure 2] 2 is a block diagram illustrating an internal configuration of a communication terminal (external device) according to the first embodiment. FIG. [Figure 3]2 is a block diagram illustrating the internal configuration of a server (image data classification device) according to the first embodiment. FIG. [Figure 4] 1 is a flowchart illustrating an image data classification method according to the first embodiment. [Figure 5] FIG. 10 is a block diagram illustrating the internal configuration of a communication terminal (external device) according to a second embodiment. [Figure 6] FIG. 10 is a block diagram illustrating the internal configuration of a server (image data classification device) according to a second embodiment. [Figure 7] 10 is a flowchart illustrating an image data classification method according to the second embodiment. [Figure 8] FIG. 11 is a block diagram illustrating the internal configuration of a communication terminal (image data classification device) according to a third embodiment. [Figure 9] 11 is a flowchart illustrating an image data classification method according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an image data classification device, an image data classification method, and an image data classification program according to embodiments of the present disclosure will be described with reference to the drawings. Note that in the drawings, the same or equivalent elements are designated by the same reference numerals, and redundant description will not be repeated.

[0016] (Embodiment 1) The image data classification device, image data classification method, and image data classification program according to the first embodiment will be described with reference to FIGS.

[0017] Fig. 1 is a diagram showing the configuration of an image data classification system according to this embodiment. As shown in Fig. 1, the image data classification system according to this embodiment includes a communication terminal 100 as an external device and a server 200 as an image data classification device. The communication terminal 100 is, for example, a communication terminal called a smartphone. The server 200 is a device managed by a business that provides an image data classification service. The communication terminal 100 and the server 200 are connected to each other so as to be able to communicate with each other via an electrical information and communication network 300 called the Internet.

[0018] 2 is a block diagram illustrating the internal configuration of communication terminal 100 (external device) according to the present embodiment. As shown in FIG. 2, communication terminal 100 (external device) includes image data acquisition unit 101, terminal control unit 102, terminal communication unit 103, and operation unit 104.

[0019] The image data acquisition unit 101 is a camera capable of acquiring image data consisting of still images or moving images. The terminal control unit 102 causes the terminal communication unit 103 to transmit the image data acquired by the image data acquisition unit 101 to the server 200. The terminal control unit 102 is a computer called a processor.

[0020] The terminal communication unit 103 is an antenna or the like for transmitting image data via radio waves. In this embodiment, the operation unit 104 is a touch panel of a smartphone, but it may also be a physical operation unit of a push button type. The operation unit 104 may be any unit as long as it can perform an operation for setting weights of facial feature data, which will be described later.

[0021] 3 is a block diagram illustrating the internal configuration of server 200 (image data classification device) according to this embodiment. As shown in FIG. 3, server 200 (image data classification device) includes device storage unit 201, device control unit 202, and device communication unit 203.

[0022] The device storage unit 201 stores one or more pieces of facial feature data for each of a plurality of clusters. The device storage unit 201 is a memory called a database. The facial feature data is stored as vector data. The device storage unit 201 also stores an image data classification program for executing an image data classification process, which will be described later.

[0023] The device control unit 202 is a computer called a processor. Based on an image data classification program stored in the device storage unit 201, the device control unit 202 extracts facial feature data from image data acquired via the device communication unit 203. Based on the image data classification program, the device control unit 202 determines to which cluster the extracted facial feature data belongs, and stores the determined facial feature data in one of the multiple clusters in the device storage unit 201. In this embodiment, an example is shown in which M pieces of facial feature data are stored in each of clusters 1 to N, but the number of facial feature data stored in each cluster may be any number. The facial feature data belonging to each cluster in the device storage unit 201 is stored in association with a "weight" set in advance according to the image data classification program.

[0024] Furthermore, device control unit 202 receives weight numerical information capable of identifying a "weight" corresponding to specific facial feature data from communication terminal 100 (external device) via device communication unit 203 through operation of operation unit 104. In this case, device control unit 202 associates the received "weight" with the specific facial feature data and stores it in device storage unit 201, instead of the "weight" set in accordance with the image data classification program.

[0025] The device communication unit 203 is an antenna or the like that receives image data contained in radio waves, and receives image data from the communication terminal 100 (external device) and transmits it to the device control unit 202.

[0026] Figure 4 is a flowchart for explaining the image data classification method of this embodiment. As shown in Figure 4, terminal control unit 102 of communication terminal 100 (external device) executes the process described in step S1. Furthermore, device control unit 202 of server 200 (image data classification device) executes the process described in steps S2 to S8 based on an image data classification program. Specifically, the image data classification method of this embodiment is as follows.

[0027] In step S1, the terminal control unit 102 causes the image data acquisition unit 101 to acquire image data. The terminal control unit 102 causes the terminal communication unit 103 to transmit the image data acquired by the image data acquisition unit 101 to the server 200 (image data classification device).

[0028] The device communication unit 203 of the server 200 (image data classification device) receives the image data and transmits the received image data to the device control unit 202. In step S2, the device control unit 202 detects the position of the face image data in the acquired image data and cuts out the face image data from the image data.

[0029] In step S3, device control unit 202 extracts facial feature data from the facial image data cut out from the image data. In step S4, device control unit 202 stores the image data and facial feature data in device storage unit 201. In step S5, device control unit 202 classifies the extracted facial feature data into one of a plurality of clusters based on an image data classification program.

[0030] At this time, device control unit 202 associates the facial feature data classified into one cluster with a weight for that facial feature data based on the image data classification program, and stores the associated weight in device storage unit 201. This weight is a value that has been pre-stored in device storage unit 201 based on the image data classification program. However, there are cases where a "weight" corresponding to image data is transmitted from communication terminal 100 (external device). In this case, device control unit 202 associates the "weight" transmitted from communication terminal 100 (external device) with the facial feature data corresponding to the image data, instead of the pre-set weight, and stores the associated weight in device storage unit 201.

[0031] In step S6, device control unit 202 determines whether the total amount of facial feature data stored in device storage unit 201 exceeds a predetermined amount. If the total amount of facial feature data stored in device storage unit 201 does not exceed the predetermined amount in step S6, device control unit 202 executes the processing of step S1. The amount of facial feature data stored in device storage unit 201 may be specified by the number of pieces of facial feature data, or may be specified by the storage capacity (number of bytes) of the facial feature data, etc.

[0032] On the other hand, in step S6, there is a case where the total amount of facial feature data stored in device storage unit 201 exceeds a predetermined amount. In this case, in step S7, device control unit 202 calculates, for each of the plurality of facial feature data stored in device storage unit 201, the "product" of the weight of one selected facial feature data and the number of facial feature data in one cluster to which the selected facial feature data belongs. The number of facial feature data is the number of one or more pieces of facial image data included in one cluster to which the facial feature data corresponding to the weight belongs. In step S8, among the plurality of facial feature data stored in device storage unit 201, facial feature data for which the aforementioned "product" is smallest is deleted from the cluster and device storage unit 201.

[0033] The device control unit 202 may, for example, delete from the device storage unit 201 at least one facial feature data whose importance corresponding to the respective products is lower than a predetermined value. In this case, if the above-mentioned products are positive numerical values, the larger the product, the higher the importance may be, or alternatively, the larger the product, the lower the importance may be. This allows at least one facial feature data whose importance is lower than a predetermined value to be preferentially deleted from the device storage unit 201. In this case, the device control unit 202 may, for example, delete only one facial feature data whose importance corresponding to the above-mentioned product is the lowest value from the device storage unit 201. This allows facial feature data with the lowest importance to be preferentially deleted from the device storage unit 201.

[0034] According to the image data classification method of the present embodiment, an upper limit is set for the amount of facial feature data stored in the device storage unit 201. Under such circumstances, the device control unit 202 may acquire new facial feature data in excess of the upper limit. In this case, the device control unit 202 calculates the number of facial feature data included in each of the plurality of clusters. The device control unit 202 also reads a weight corresponding to each piece of facial feature data from the device storage unit 201. Furthermore, for each piece of facial feature data stored in the device storage unit 201, the device control unit 202 multiplies the number of pieces of facial feature data included in a cluster to which the facial feature data belongs by the weight of the piece of facial feature data. The weight is set in advance based on some criterion. The device control unit 202 then deletes the facial feature data with the smallest multiplied value from the device storage unit 201.

[0035] According to the server 200 (image data classification device) of this embodiment, facial feature data of members of a specific group, such as a family, can be continuously stored in the device storage unit 201. Therefore, the server 200 (image data classification device) can continuously maintain the function of determining whether or not a person included in image data is a member of the specific group. The effect of this will be described in detail below.

[0036] 4, according to the image data classification method of the present embodiment, communication terminal 100 only captures the image data, and server 200 extracts facial feature data using the received image data and performs subsequent processing. This makes it possible to provide an image data storage service similar to Google Photos. Furthermore, because the resources within communication terminal 100 (external device) are hardly used, it is possible to reduce battery consumption and heat generation of communication terminal 100 as an external device.

[0037] The method for setting the weights will now be described with an example.

[0038] 1. A weighting method in which a smaller weight is assigned to facial feature data whose corresponding image data was captured on an older date and time. Even when there is a large number of facial feature data belonging to the same cluster (i.e., it can be assumed that the facial feature data belongs to the user's family), there is a high possibility that there is a large difference between facial feature data captured at an older date and time and the current facial feature data. In particular, facial feature data may differ significantly between a person's childhood and adolescence. Therefore, this weighting allows for preferential deletion of facial feature data corresponding to image data captured earlier (image data when family members were infants or toddlers), which is likely to be unnecessary for determination using the current facial feature data. This reduces the amount of calculation required for comparing facial feature data, while improving the accuracy of determination using facial feature data.

[0039] More specifically, this method is as follows. For example, a group of image data capturing images of three family members, a father, a mother, and a child, is likely to contain facial image data of the three family members. When facial feature data of another person Z is added to a group of facial feature data corresponding to this group of image data, the total number of facial feature data included in a cluster to which one of the facial feature data in the group belongs is multiplied by the weight of that one of the facial feature data. This multiplication is performed for each of the group of facial feature data. Then, the facial feature data with the smallest multiplied value is deleted from the device storage unit 201.

[0040] According to this method, if the weight is a positive value, the product of the following two types of facial feature data, the first facial feature data and the second facial feature data, becomes large. The first facial feature data is facial feature data corresponding to image data captured at a relatively old date and time, but is facial feature data corresponding to image data including three people (father, mother, and child) that contains a relatively large amount of facial feature data. The second facial feature data is facial feature data corresponding to image data that does not include anyone other than other person Z, but is facial feature data corresponding to image data of other person Z that was captured at a relatively recent date and time. These first facial feature data and second facial feature data are likely to remain in the device storage unit 201.

[0041] On the other hand, facial feature data corresponding to image data of other person A, which has an old photographing date and time and has a small number of facial feature data belonging to one cluster, is deleted from the device storage unit 201 relatively early.

[0042] According to the above method, even if the amount of facial feature data that can be stored in the device storage unit 201 is limited, facial feature data of members of a group of features such as a family is maintained.

[0043] On the other hand, even for the facial feature data of family members, for example, there are large differences between the facial feature data of a child when he or she was young and the facial feature data of a child now, so there is a high possibility that these facial feature data will be classified into different clusters. Also, since the facial feature data of a child when he or she was young is taken at an older date and time, the value of the multiplication described above will be relatively small. Therefore, by sequentially adding the facial feature data of current family members, facial feature data that is not suitable for determination using facial feature data can be naturally deleted. In other words, facial feature data that is suitable for determination using facial feature data can be retained.

[0044] 2. A weighting method that assigns a larger weight to facial feature data having a larger ratio of a partial area of ​​facial image data to the entire area of ​​the image data (the ratio of the number of pixels in the partial area of ​​facial image data to the total number of pixels in the image data). A large ratio of the facial image data area to the entire image data area indicates that the photographer of the image data is paying attention to the subject. In other words, it is highly likely that the subject is a close friend, such as a family member. Furthermore, when the ratio of a partial area of ​​facial image data to the entire image data area is large, the amount of information that can be extracted as facial feature data from the image data is large. Therefore, the image data can be considered preferable for use in a judgment that compares facial feature data with each other. This method allows facial feature data extracted from image data that is preferable for use in such a judgment to be preferentially retained in the device storage unit 201. This reduces the amount of calculation required for comparing facial feature data with each other, while improving the accuracy of judgments that use facial feature data.

[0045] 3. A weighting method in which a smaller weight is assigned to facial feature data corresponding to image data with a larger rotation angle of the face direction relative to a reference direction (note that the reference direction is assumed to be the direction perpendicular to the planar area of ​​the image data). The direction in which facial image data in image data faces may be rotated (by roll, pitch, or yaw) relative to the direction of a reference axis (reference direction) within the camera's angle of view. In this case, facial feature data extracted from the image data may be difficult to compare with facial feature data extracted from image data including face image data facing forward, which has a zero rotation angle relative to the direction of the reference axis within the camera's angle of view. Furthermore, facial feature data extracted from such image data may lack information necessary for facial feature data due to factors such as one eye being hidden. Therefore, facial feature data extracted from such image data with a large rotation angle relative to the reference direction may be inappropriate for determination using facial feature data. Therefore, an algorithm is used to estimate the rotation angle of the direction in which the face faces faces relative to the reference axis of the camera's angle of view. The rotation angle of the direction in which the face faces faces relative to the reference axis of the camera's angle of view is calculated, and weighting is performed such that the larger the rotation angle, the smaller the weight assigned.

[0046] In this embodiment, the reference direction is assumed to coincide with the roll, pitch, or yaw rotation axis in a three-axis coordinate system. For each of roll, pitch, and yaw, the rotation angle of the direction in which the face faces relative to the reference axis may be calculated, and a smaller weight may be assigned to facial feature data for which the multiplication value of these three rotation angles is larger. Alternatively, a smaller weight may be assigned to facial feature data for which only one of the roll, pitch, and yaw rotation angles is larger. This also allows facial feature data suitable for determination using facial feature data to remain in the device storage unit 201. Therefore, the accuracy of determination using facial feature data can be improved while reducing the amount of calculation required for comparing facial feature data.

[0047] It should be noted that the weighting method described above is merely an example. The weighting method used in the image data classification method of this embodiment is not limited to the above weighting method. At least two of the three weighting methods described above may be used in combination. Furthermore, the product of at least two of the weights obtained by each of the three methods described above may be used as the final weight.

[0048] Additionally, the aforementioned "weight" may be set by the user's operation of operation unit 104. In this case, the weight is specified as a numerical value by operation of operation unit 104, transmitted from communication terminal 100 (external device) to server 200 (image data classification device), and stored in device storage unit 201 in association with the facial feature data. This allows the user to select a specific person or group that the user wishes to continue to store in device storage unit 201.

[0049] (Embodiment 2) An image data classification device, an image data classification method, and an image data classification program according to the second embodiment will be described with reference to Figures 5 to 7. Note that the following description will not repeat the same points as in the first embodiment. The image data classification device, the image data classification method, and the image data classification program according to the present embodiment differ from the image data classification device, the image data classification method, and the image data classification program according to the first embodiment in the following points.

[0050] Fig. 5 is a block diagram for explaining the internal configuration of communication terminal 100 (external device) according to the present embodiment. As shown in Fig. 5, the internal configuration of communication terminal 100 (external device) according to the present embodiment is almost the same as the internal configuration of communication terminal 100 (external device) according to the first embodiment shown in Fig. 2.

[0051] The communication terminal 100 (external device) of this embodiment differs from the communication terminal 100 (external device) of embodiment 1 in that the terminal control unit 102 extracts facial feature data from image data and causes the terminal communication unit 103 to transmit the extracted facial feature data.

[0052] Fig. 6 is a block diagram for explaining the internal configuration of server 200 (image data classification device) of this embodiment. As shown in Fig. 6, the internal configuration of server 200 (image data classification device) of this embodiment is almost the same as the internal configuration of server 200 (image data classification device) of embodiment 1 shown in Fig. 3.

[0053] The server 200 (image data classification device) of this embodiment differs from the server 200 (image data classification device) of embodiment 1 in that the device communication unit 203 receives facial feature data and the device control unit 202 stores the received facial feature data in the device memory unit 201.

[0054] Fig. 7 is a flowchart for explaining the image data classification method of this embodiment. As shown in Fig. 7, communication terminal 100 (external device) executes the processes described in steps S1 to S3. Furthermore, device control unit 202 of server 200 (image data classification device) executes the processes described in steps S4 to S8. Specifically, the image data classification method of this embodiment is as follows.

[0055] In step S1, the terminal control unit 102 causes the image data acquisition unit 101 to acquire image data. In step S2, the terminal control unit 102 detects the position of facial image data in the acquired image data and cuts out the facial image data. In step S3, the terminal control unit 102 extracts facial feature data from the facial image data cut out from the image data. The terminal control unit 102 causes the terminal communication unit 103 to transmit the facial extraction data extracted by the terminal control unit 102 to the server 200 (image data classification device).

[0056] The device communication unit 203 of the server 200 (image data classification device) receives the facial feature data and transmits the received facial feature data to the device control unit 202. In step S4, the device control unit 202 stores the received facial feature data in the device storage unit 201. The processing from step S5 onwards in this embodiment is the same as the processing from step S5 onwards in embodiment 1, and therefore description of those steps will not be repeated.

[0057] The server 200 (image data classification device) of this embodiment can also continuously store facial feature data of members of a specific group in the device storage unit 201. Therefore, it is possible to continuously maintain the function of determining whether or not a person included in image data is a member of a specific group.

[0058] According to the image data classification method of the present embodiment, the steps up to the extraction of facial feature data are executed by communication terminal 100, and the facial feature data is managed by server 200 (image data classification device). In this case, it is difficult for server 200 to restore a facial image corresponding to the facial feature data managed by server 200. Therefore, according to server 200 (image data classification device) of the present embodiment, the user of communication terminal 100 does not need to transmit image data including facial image data outside communication terminal 100. Therefore, the privacy of persons included in image data stored in communication terminal 100 can be protected.

[0059] (Embodiment 3) Next, an image data classification device, an image data classification method, and an image data classification program according to embodiment 3 will be described. Note that the following description will not repeat the same points as in embodiment 1 or 2. This embodiment differs from embodiment 1 or 2 in that the image data classification device is communication terminal 400, and the image data classification method is executed only in communication terminal 400.

[0060] 8 is a block diagram for explaining the internal configuration of communication terminal 400 (image data classification device) of this embodiment. As shown in FIG. 8, communication terminal 400 (image data classification device) includes image data acquisition unit 401, terminal control unit 402, terminal storage unit 403, and operation unit 404.

[0061] The image data acquisition unit 401 is a camera capable of acquiring still images or moving images. The image data acquisition unit 401 transmits the acquired image data to the terminal control unit 402.

[0062] The terminal control unit 402 extracts facial feature data from image data based on an image data classification program. The terminal control unit 402 determines to which cluster the extracted facial feature data belongs based on the image data classification program, and stores the determined facial feature data in one of multiple clusters in the terminal storage unit 403.

[0063] The device storage unit 403 is a memory called a database. Facial feature data is stored as vector data. The device storage unit 403 stores one or more pieces of facial feature data for each of a plurality of clusters. The device storage unit 403 also stores an image data classification program for executing an image data classification process, which will be described later.

[0064] In this embodiment, the operation unit 404 is a touch panel of a smartphone, but may be a physical operation unit of a push button type. The operation unit 404 may be any unit as long as it can perform an operation for setting the weight of facial feature data, which will be described later.

[0065] 9 is a flowchart for explaining the image data classification method of the present embodiment. As shown in FIG. 9, terminal control unit 402 of communication terminal 400 (image data classification device) executes all of the processes from step S1 to step S8.

[0066] In step S1, the terminal control unit 402 causes the image data acquisition unit 401 to acquire image data. In step S2, the terminal control unit 402 detects the position of facial image data in the acquired image data and cuts out the facial image data. In step S3, the terminal control unit 402 extracts facial feature data from the image data. In step S4, the terminal control unit 402 stores the image data and facial feature data in the terminal storage unit 403. In step S5, the terminal control unit 402 classifies the extracted facial feature data into one of multiple clusters.

[0067] At this time, the terminal control unit 402 associates the facial feature data classified into one cluster with the weight of that facial feature data based on the image data classification program, and stores the associated data in the terminal storage unit 403. However, if a "weight" corresponding to the image data is set by operating the operation unit 404, the terminal control unit 402 associates the "weight" with the facial feature data, and stores the associated data in the terminal storage unit 403.

[0068] In step S6, the terminal control unit 402 determines whether the total amount of facial feature data stored in the terminal storage unit 403 exceeds a predetermined amount. If the total amount of facial feature data stored in the terminal storage unit 403 does not exceed the predetermined amount in step S7, the terminal control unit 402 executes the processing of step S1.

[0069] On the other hand, in step S6, there is a case where the total amount of facial feature data stored in the device storage unit 403 exceeds a predetermined amount. In this case, in step S7, the terminal control unit 402 calculates the product of the weight of one piece of facial feature data and the number of pieces of facial feature data in one cluster to which that piece of facial feature data belongs, for each of the plurality of facial feature data stored in the device storage unit 403. In step S8, the facial feature data with the smallest product among the plurality of facial feature data stored in the device storage unit 403 is deleted from the cluster and the device storage unit 403.

[0070] The communication terminal 400 (image data classification device) of this embodiment can also continuously store facial feature data of members of a specific group in the device storage unit 201. Therefore, it is possible to continuously maintain the function of determining whether or not a person included in image data is a member of a specific group.

[0071] As described in the first and second embodiments above, the database constituting the storage unit that stores facial feature data may be provided in the server 200 via the electrical information communication network 300, rather than in the communication terminal 400. In this case, even if facial feature data of a member of a particular group, such as a family, is discovered while previewing captured image data, it is difficult to realize real-time utilization of the determination results, such as immediately focusing on the facial image data corresponding to the facial feature data. However, according to the image data classification method of the present embodiment, all of the image data classification processing is executed in the communication terminal 400. Therefore, the determination results using the facial feature data can be utilized in real time in the communication terminal 400.

[0072] The characteristic configurations of the image data classification device, image data classification method, and image data classification program of the present disclosure and the effects obtained thereby are summarized below.

[0073] (1) The image data classification device (server 200, communication terminal 400) includes a storage unit (device storage unit 201, terminal storage unit 403) and a control unit (device control unit 202, terminal control unit 402). The storage unit classifies and stores a plurality of facial feature data into a plurality of clusters. The control unit (device control unit 202, terminal control unit 402) calculates, for each of the plurality of facial feature data, the product of the weight of one of the plurality of facial feature data and the number of facial feature data in one cluster to which the one of the plurality of clusters belongs. The control unit (device control unit 202, terminal control unit 402) deletes at least one of the plurality of facial feature data from the storage unit (device storage unit 201, terminal storage unit 403) based on the product of each of the plurality of facial feature data.

[0074] According to this, when a large amount of facial feature data of a large number of persons, including persons other than members of a specific group such as a family, is acquired, the facial feature data of persons other than members of the specific group can be preferentially deleted from the storage unit. In other words, the image data classification device can continue to store the facial feature data of members of the specific group in the storage unit. As a result, the image data classification device can continue to maintain the function of determining whether a person included in image data is a member of a specific group such as a family.

[0075] (2) The control unit (device control unit 202, terminal control unit 402) may delete at least one piece of facial feature data whose importance corresponding to each of the aforementioned products is lower than a predetermined value from the storage unit (device storage unit 201, terminal storage unit 403). According to this configuration, at least one piece of facial feature data whose importance is lower than a predetermined value can be preferentially deleted from the storage unit (device storage unit 201, terminal storage unit 403).

[0076] (3) The control unit (device control unit 202, terminal control unit 402) may delete the facial feature data having the lowest importance corresponding to the product from the storage unit (device storage unit 201, terminal storage unit 403). This configuration allows the facial feature data with the lowest importance to be preferentially deleted from the storage unit (device storage unit 201, terminal storage unit 403).

[0077] (4) The control unit (device control unit 202, terminal control unit 402) may calculate the above-mentioned product when the amount of multiple pieces of facial feature data exceeds a predetermined amount. This configuration makes it possible to limit the amount of facial feature data stored in the memory unit (device memory unit 201, terminal memory unit 403) to a predetermined amount or less. This makes it possible to prevent the facial feature data from excessively occupying the memory unit (device memory unit 201, terminal memory unit 403).

[0078] (5) The earlier the capture dates of the multiple image data corresponding to each of the multiple facial feature data, the smaller the weights may be. This configuration allows for preferential deletion of facial feature data that is not effective for determination using facial feature data. This allows for improved accuracy of determination using facial feature data while reducing the amount of calculations performed by the image data classification device (device storage unit 201, terminal storage unit 403).

[0079] (6) The higher the ratio of the facial image area to the total image area of ​​each of the plurality of image data corresponding to the plurality of facial feature data, the larger the weight may be. With this configuration, facial feature data that is effective for determination using facial feature data can be retained as much as possible in the storage unit. Therefore, it is possible to improve the accuracy of determination using facial feature data while reducing the amount of calculations in the image data classification device (device storage unit 201, terminal storage unit 403).

[0080] (7) The larger the rotation angle of the face direction relative to the reference direction of each of the plurality of image data corresponding to each of the plurality of facial feature data, the smaller the weight may be. In this case, the rotation angle is the angle of rotation about at least one of roll, pitch, and yaw in a three-axis coordinate system. This configuration allows facial feature data that is effective for determination using facial feature data to be retained as much as possible in the storage unit. Therefore, the amount of calculation required by the image data classification device can be reduced while improving the accuracy of determination using facial feature data.

[0081] (8) The weights may be set by the user through operation of the operation unit 104, 404. This configuration allows the user to select a specific person that the user wants to keep stored in the storage unit (device storage unit 201, terminal storage unit 403).

[0082] (9) The control unit (device control unit 202) may receive multiple image data captured by the external device (communication terminal 100) and extract multiple pieces of facial feature data from each of the multiple pieces of image data. This configuration makes it possible to provide an image data storage service. Furthermore, since little power is used in the external device (communication terminal 100) to use the facial feature data, battery consumption and heat generation of the external device (communication terminal 100) can be reduced.

[0083] (10) The control unit (device control unit 202) may receive multiple pieces of facial feature data from an external device (communication terminal 100). With this configuration, there is no need to transmit image data from the external device (communication terminal 100) to the image data classification device (server 200), so the confidentiality of the image data in the external device (communication terminal 100) can be maintained. Therefore, the personal privacy of the user of the external device (communication terminal 100) can be protected.

[0084] (11) The image data classification device (communication terminal 400) may further include an image data acquisition unit 401 that acquires a plurality of image data. The control unit (terminal control unit 402) may extract a plurality of facial feature data from each of the plurality of image data acquired by the image data acquisition unit 401. With this configuration, the facial feature data can be immediately used in the image data classification device (communication terminal 400).

[0085] (12) The image data classification method is realized by the operation of the image data classification device (server 200, communication terminal 400) described above.

[0086] (13) The image data classification program is intended to cause the computer to operate as the aforementioned memory unit (device memory unit 201, terminal memory unit 403) and the aforementioned control unit (device control unit 202, terminal control unit 402). [Explanation of symbols]

[0087] 100 communication terminal 104 Operation section 200 servers 201 Device storage section 202 Device control section 401 Image data acquisition unit 402 Terminal control unit 403 Terminal memory section 404 Operation section

Claims

1. a storage unit that classifies a plurality of facial feature data into a plurality of clusters and stores the clustered data; a control unit that calculates, for each of the plurality of facial feature data, a product of a weight of one of the plurality of facial feature data and a number of facial feature data in one cluster to which the one of the plurality of facial feature data belongs, the control unit deletes at least one facial feature data from the plurality of facial feature data based on the product of each of the plurality of facial feature data. Image data classification device.

2. the control unit deletes from the storage unit the at least one facial feature data whose importance corresponding to each of the products is lower than a predetermined value; 2. The image data classification device according to claim 1.

3. the control unit deletes the one facial feature data having the lowest importance value corresponding to the product from the storage unit.

3. The image data classification device according to claim 1.

4. the control unit calculates the product when the amount of the plurality of facial feature data exceeds a predetermined amount.

4. The image data classification device according to claim 1.

5. The older the time at which each of the plurality of image data corresponding to each of the plurality of facial feature data was captured, the smaller the weight.

5. The image data classification device according to claim 1.

6. the weight increases as the ratio of a face image area to the entire image area of ​​each of the plurality of image data corresponding to the plurality of facial feature data increases; 6. The image data classification device according to claim 1.

7. the weight is smaller as the rotation angle of the direction of the face relative to the reference direction of each of the plurality of image data corresponding to the plurality of facial feature data is larger; The rotation angle is an angle of rotation about at least one of roll, pitch, and yaw in a three-axis coordinate system.

7. The image data classification device according to claim 1.

8. The weight may be set by a user operating an operation unit.

8. The image data classification device according to claim 1.

9. the control unit receives a plurality of image data captured by an external device and extracts the plurality of facial feature data from the plurality of image data, respectively; 9. The image data classification device according to claim 1.

10. the control unit receives the plurality of facial feature data from an external device; 9. The image data classification device according to claim 1.

11. further comprising an image data acquisition unit that acquires a plurality of image data; the control unit extracts the plurality of facial feature data from the plurality of image data acquired by the image data acquisition unit, 9. The image data classification device according to claim 1.

12. Classifying and storing a plurality of facial feature data into a plurality of clusters; For each of the plurality of facial feature data, a product of a weight of one of the plurality of facial feature data and a number of facial feature data of one cluster to which the one of the plurality of clusters belongs is calculated; deleting at least one facial feature data from the plurality of facial feature data based on the product of each of the plurality of facial feature data; An image data classification method executed by an image data classification device.

13. Computer, a storage unit that classifies a plurality of facial feature data into a plurality of clusters and stores the clustered data; a control unit that calculates, for each of the plurality of facial feature data, a product of a weight of one of the plurality of facial feature data and a number of facial feature data in one cluster to which the one of the plurality of facial feature data belongs; An image data classification program for operating as the control unit deletes at least one face feature data from the plurality of face feature data based on the product of each of the plurality of face feature data. Image data classification program.

Images