Image processing device, control method for image processing device, program

Abstract

To solve the problem in which: it may be difficult to provide a learning model having a good balance between the accuracy of face recognition and low execution load.SOLUTION: An information processing apparatus communicates with a terminal device performing image recognition of a photographed image and performs learning of an image recognition model in a neural network for the image recognition, and the information processing apparatus has: receiving means that receives an image transmitted from the terminal device; recognition object image discrimination means that discriminates a recognition object image from the image received by the receiving means; training data acquisition means that acquires training data from an image database on a server; and learning means that learns the image recognition model to discriminate the recognition object image by using the training data.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image processing apparatus for recording a learning model.

Background Art

[0002] In recent years, a deep learning method based on a convolutional neural network (CNN) model has been known. In order to perform face recognition using such a learning model, a large-scale database having a large number of labeled face images is required.

[0003] For example, Patent Document 1 describes a method of generating a plurality of training images from an original teacher image and using them as teacher data in order to solve the problem that a large number of labeled face images are required for training a CNN model.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] A CNN model learned with a large amount of teacher data can recognize face images of a large number of people with high accuracy. However, such a CNN model generally has a large data size and requires a high computing power for the CPU for recognition processing.

[0006] On the other hand, there is a need to perform face recognition in an edge device such as glasses (eyeglass-type wearable device). However, in general, an edge device often does not have sufficient machine power for a CNN model that can recognize a large number of faces as described above.

[0007] Therefore, it is conceivable to create a CNN model that requires little computing power but has low facial recognition capabilities, which can be installed on edge devices, and then install it on those devices.

[0008] For example, in environments where wireless communication is possible, the glasses could capture facial images and send them to a cloud server, where the cloud server could then recognize the faces and perform facial recognition based on the high accuracy of the facial recognition results. In environments where wireless communication is not possible, the glasses could capture facial images and perform facial recognition using a low-precision CNN model installed on the glasses.

[0009] However, CNN models with poor facial recognition capabilities have the problem of not being able to perform effective facial recognition for the user.

[0010] In view of the circumstances described above, the present invention aims to provide a learning model that balances facial recognition accuracy with low execution load. [Means for solving the problem]

[0011] An information processing system that communicates with a terminal device that performs image recognition on captured images and trains a neural network image recognition model for the image recognition, comprising: a receiving means for receiving face images transmitted from the terminal device; a discriminating means for determining whether the face images received by the receiving means are to be trained; and the information processing system that processes the face images corresponding to the face images determined by the discriminating means to be to be recognized, along with the corresponding person's name information. system An information processing system comprising: a means for acquiring training data to obtain training data from a database; a means for training an image recognition model to recognize a person from images determined to be the learning target using the training data; and a means for transmitting the trained image recognition model to the terminal device, wherein the determination means determines that images of faces received from the terminal device a certain number of times or more within a unit period are to be recognized, and does not determine that images of faces not received within that period are to be recognized. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a learning model that balances facial recognition accuracy with low execution load. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram of the entire system in the first embodiment. [Figure 2] This is a sequence diagram in the first embodiment. [Figure 3] This is a flowchart of the person recognition server in the first embodiment. [Figure 4] This is a flowchart of the attribute information server in the first embodiment. [Figure 5] This is a flowchart of the management server in the first embodiment. [Figure 6] This is a flowchart of the training server in the first embodiment. [Figure 7] This is a flowchart of the person recognition glasses in the first embodiment. [Modes for carrying out the invention]

[0014] The embodiments for carrying out the present invention will be described in detail below with reference to the attached drawings.

[0015] The embodiments described below are merely examples of means for realizing the present invention, and may be modified or changed as appropriate depending on the configuration of the apparatus to which the present invention is applied and various conditions. Furthermore, the embodiments can be combined as appropriate.

[0016] [First Embodiment] <Configuration of each device> Figure 1 is a block diagram showing the overall configuration of a human face image recognition system using the image recognition system of the present invention.

[0017] Here, as an example of each device constituting the face image recognition system, a server PC and smart glasses will be described, but the devices are not limited to these. For example, the device may be an information processing device such as a portable PC, a tablet PC, or a media player.

[0018] First, the learning server A100 will be described.

[0019] The control unit A101 controls each part of the learning server A100 according to the input signal and the program described later. Note that instead of the control unit A101 controlling the entire device, a plurality of hardware may share the processing to control the entire device.

[0020] The non-volatile memory A102 is an electrically erasable and recordable non-volatile memory, and stores programs and the like executed by the control unit A101 described later.

[0021] The working memory A103 is used as a buffer memory for temporarily holding learning data, an image display memory for the display unit A105, a working area of the control unit A101, and the like.

[0022] The operation unit A104 is used to receive instructions from the user for the learning server A100. The operation unit A104 includes, for example, the power button of the learning server A100, a keyboard, and a mouse.

[0023] The display unit A105 displays learning data, displays a GUI (Graphical User Interface) for interactive operations, and the like. Note that the display unit A105 does not necessarily have to be built into the learning server A100. The learning server A100 can be connected to an external display device and only needs to have at least a display control function for controlling the display of the display unit A105.

[0024] The learning target determination unit A106 determines whether or not the face image data recorded on the recording medium A110 described later is a learning target.

[0025] The face image learning unit A107 performs learning processing on face image data recorded on the recording medium A110, which will be described later. In this embodiment, a neural network is used and a machine learning algorithm is used to generate a learning model (image recognition model) that recognizes face images recorded on the recording medium A110. The neural network is used to predict output values ​​from input values, and by learning training data consisting of actual input values ​​and actual output values ​​in advance, it is possible to estimate output values ​​for new input values. In this embodiment, it is assumed that a database of training data created from any face images is recorded on the recording medium A110, which will be described later.

[0026] The recording medium A110 can record face image data and training data. The recording medium A110 may be configured to be detachable from the learning server A100, or it may be built into the learning server A100. In other words, the learning server A100 only needs to have a means to access the recording medium A110.

[0027] The communication unit A120 is an interface that performs wireless LAN communication, for example, compliant with the IEEE 802.11 standard. Wireless LAN communication enables wireless communication with the access point. Furthermore, higher-level protocols such as TCP / IP enable data transmission and reception between the access point and devices connected to the cloud network. The communication unit A120 communicates with the person recognition glasses B100 via the internet C100.

[0028] The above is a description of the learning server A100.

[0029] Next, we will describe the B100 person recognition glasses, which are an example of a terminal device.

[0030] The control unit B101 controls various parts of the person recognition glasses B100 according to the input signals and the program described later. Alternatively, instead of the control unit B101 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0031] The non-volatile memory B102 is an electrically erasable and recordable non-volatile memory that stores programs and other data executed by the control unit B101, as described below.

[0032] The working memory B103 is used as a buffer memory for temporarily holding training data, as well as as memory for displaying images in the display unit B105 and as a work area for the control unit B101.

[0033] The control unit B104 is used to receive instructions from the user for the person recognition glasses B100. The control unit B104 includes, for example, the power button and operation buttons for the person recognition glasses B100.

[0034] The display unit B105 displays the recognition results of the person recognition unit B106 (described later), as well as a GUI (Graphical User Interface) for interactive operation. Note that the display unit B105 does not necessarily need to be built into the person recognition glasses B100. The person recognition glasses B100 can be connected to an external display device and only need to have a display control function to control the display of the display unit B105.

[0035] The person recognition unit B106 performs recognition processing on face image data recorded on the recording medium B110, which will be described later. The learning model transmitted from the learning server A100 is used for the recognition processing.

[0036] The facial image acquisition unit B107 is a digital camera. In this embodiment, it is mounted facing the front of the glasses and constantly captures images with the same field of view as what the user is seeing.

[0037] The face image recognition unit B108 recognizes the face portion from the image captured by the face image acquisition unit B107 and extracts it as a face image. If multiple faces are recognized in the image, each face portion is extracted as a separate face image.

[0038] The recording medium B110 can record facial image data. The recording medium B110 may be configured to be detachable from the person recognition glasses B100, or it may be built into the person recognition glasses B100. In other words, the person recognition glasses B100 only needs to have means to access the recording medium B110.

[0039] Communication unit B120 is an interface that performs wireless LAN communication, for example, compliant with the IEEE 802.11 standard. Wireless LAN communication enables wireless communication with the access point. Furthermore, higher-level protocols such as TCP / IP enable data transmission and reception between the access point and devices connected to the cloud network. Communication unit B120 communicates with the learning server A100 via the internet C100.

[0040] The above is a description of the B100 person recognition glasses.

[0041] <System Overview> Next, with reference to Figure 2, an overview of the face image recognition system in this embodiment will be described.

[0042] Figure 2 is a sequence diagram showing the flow of this embodiment. In the following description, the numbering of the components of each device will be the same as that described in Figure 1.

[0043] The learning server A100 will be broken down and explained according to its role into the person recognition server A100a, attribute information server A100b, management server A100c, and training server A100d.

[0044] The person recognition server A100a has a database that links facial images with personal name information, and can search for personal name information from facial images, and vice versa.

[0045] The attribute information server A100b has a database that links personal name information with attribute information, and it is possible to search for attribute information from personal name information, and vice versa.

[0046] The management server A100c stores the name information, facial image, and attribute information obtained through the search in a linked manner, and can select and pass the appropriate information to each server.

[0047] The training server A100d uses the training data provided by the management server A100c to train a CNN model that infers personal names from facial images.

[0048] Here, personal name information refers to a person's name and the unique ID information that identifies that individual.

[0049] Here, attribute information refers to information about an individual's attributes, such as age, gender, nationality, height, weight, occupation, affiliation, hobbies, etc.

[0050] In step S201, the person recognition glasses B100 transmit the image captured by the user's operation to the person recognition server A100a. At this time, the user operation may be a clear image taken via the operation unit B104 by the user, or it may simply be the position of the face image within the field of view of the face image capturing unit B107.

[0051] In step S203, the person recognition server A100a determines whether the face image received in step S201 is a target for training. If it is a target, the process proceeds to step S204. If it is not a target, the process is interrupted.

[0052] In step S204, the person recognition server A100a searches for the name information of the person whose face image was determined to be a target for recognition in step S203.

[0053] In step S205, the person recognition server A100a sends the person name information retrieved in step S204 to the attribute information server A100b.

[0054] In step S206, the attribute information server A100b searches for attribute information of the person's name received in step S205. Alternatively, without selecting attribute information, the system may proceed to step S213 and search for training face images using only the face images transmitted from the person recognition glasses B100.

[0055] In step S207, attribute information server A100b sends the attribute information retrieved in step S206 to management server A100c.

[0056] In step S208, the management server A100c saves the attribute information received in step S207.

[0057] Here, the operations in steps S201 to S208 may be repeated for a predetermined period of time.

[0058] In step S209, the management server A100c selects the attribute information to be searched from the attribute information saved in step S208.

[0059] The choice here is that you can either search all saved attribute information, or search only the attribute information that is most frequently shared among all saved attribute information.

[0060] Alternatively, the selected search target attribute information may be sent to the person recognition glasses B100 at this time, and the user may decide whether or not to include it in the search.

[0061] In step S210, the management server A100c sends the attribute information selected in step S209 to the attribute information server A100b.

[0062] In step S211, the attribute information server A100b searches for person name information corresponding to the attribute information received in step S210.

[0063] The above steps S209 to S211 are repeated for each attribute piece of information to be searched.

[0064] In step S212, attribute information server A100b transmits the person name information retrieved in step S211 to person recognition server A100a.

[0065] In step S213, the person recognition server A100a searches for a face image that corresponds to the person name information received in step S212, and uses it as a training face image.

[0066] In step S214, the person recognition server A100a sends the teacher face image retrieved in step S213 to the management server A100c.

[0067] In step S215, the management server A100c saves the teacher's face image received in step S214.

[0068] The above steps S212 to S215 are repeated for the number of people involved.

[0069] In step S216, the management server A100c sends all the teacher face images saved in step S215 to the training server A100d. (Method for acquiring teacher data) In step S217, the training server A100d trains the network model using all the training facial images received in step S216. (Training method) In step S218, the training server A100d sends a training completion notification to the person recognition glasses B100.

[0070] In step S219, the person recognition glasses B100 sends a request to the training server A100d to acquire a learning model.

[0071] In step S220, the training server A100d sends the trained model to the person recognition glasses B100.

[0072] In step S221, the person recognition glasses B100 save the learned model received in step S220.

[0073] <Operation of each device> Next, we will explain the detailed operation of the person recognition server A100a in realizing the process shown in Figure 2, referring to Figure 3.

[0074] Figure 3 is a flowchart showing the operation of the person recognition server A100a in this embodiment. The processes shown in this flowchart are realized by the control unit A101 of the learning server A100 controlling each part of the learning server A100 according to the input signals and program.

[0075] In step S301, the control unit A101 determines whether it has received a face image from the person recognition glasses B100 via the communication unit A120. If it has received an image, it proceeds to step S303. If it has not received an image, it proceeds to step S306. (Receiving means) In step S303, the learning target determination unit A106 determines whether the face image received in step S301 is a learning target. If it is a learning target, the process proceeds to step S304. If it is not a learning target, the process returns to step S301. Here, the criteria for determining whether or not it is a learning target is that the face image has been received from the person recognition glasses B100 "a certain number of times or more" "within a unit period." In this embodiment, for example, a face image received 10 or more times a day is determined to be a recognition target as it represents a person who is frequently encountered. (Recognition target image determination means) In step S304, the control unit A101 performs a name information search on the face image received in step S301.

[0076] In step S305, the control unit A101 transmits the person name information retrieved in step S304 to the attribute information server A100b.

[0077] In step S306, the control unit A101 determines whether it has received a name information notification from the attribute information server A100b. If it has received it, it proceeds to step S307. If it has not received it, it returns to step S301.

[0078] In step S307, the control unit A101 searches for a face image that corresponds to the person name information received in step S306.

[0079] In step S308, the control unit A101 sends the face image found in step S307 to the management server A100c.

[0080] In step S309, the control unit A101 determines whether it has received a termination command from the user via the operation unit A104. If a termination command is received, the process is terminated. If no termination command is received, the process returns to step S301.

[0081] The above describes the operation of the person recognition server A100a in this embodiment.

[0082] Next, we will explain the detailed operation of attribute information server A100b in implementing the process shown in Figure 2, referring to Figure 4.

[0083] Figure 4 is a flowchart showing the operation of attribute information server A100b. The processes shown in this flowchart are realized by the control unit A101 of learning server A100 controlling each part of learning server A100 according to input signals and programs.

[0084] In step S401, the control unit A101 determines whether it has received name information from the person recognition server A100a. If it has received it, it proceeds to step S402. If it has not received it, it proceeds to step S404.

[0085] In step S402, the control unit A101 performs an attribute information search associated with the name information received in step S401. (Attribute information search means) In step S403, the control unit A101 sends the attribute information retrieved in step S402 to the management server A100c.

[0086] In step S404, the control unit A101 determines whether it has received an attribute information notification from the management server A100c. If it has received it, it proceeds to step S405. If it has not received it, it returns to step S401.

[0087] In step S405, the control unit A101 searches for personal name information corresponding to the attribute information received in step S404.

[0088] In step S406, the control unit A101 transmits the person name information retrieved in step S405 to the person recognition server A100a.

[0089] In step S407, the control unit A101 determines whether it has received a termination command from the user via the operation unit A104. If a termination command is received, the process is terminated. If no termination command is received, the process returns to step S401.

[0090] The above describes the operation of the attribute information server A100b in this embodiment.

[0091] Next, we will explain the detailed operation of the management server A100c in implementing the process shown in Figure 2, referring to Figure 5.

[0092] Figure 5 is a flowchart showing the operation of the management server A100c. The processes shown in this flowchart are realized by the control unit A101 of the learning server A100 controlling each part of the learning server A100 according to the input signals and program.

[0093] In step S501, the control unit A101 determines whether it has received attribute information from the attribute information server A100b. If it has received it, it proceeds to step S502. If it has not received it, it proceeds to step S505.

[0094] In step S502, the control unit A101 records the attribute information received in step S501 onto the recording medium A110.

[0095] In step S503, the control unit A101 selects the attribute information to be searched from the attribute information recorded on the recording medium A110 in step S502. This selection may involve searching all of the saved attribute information, or it may involve searching the attribute information that is most frequently common among all the saved attribute information.

[0096] In step S504, the control unit A101 transmits the attribute information selected in step S503 to the attribute information server A100b.

[0097] In step S505, the control unit A101 determines whether it has received a teacher face image from the person recognition server A100a. If it has received it, it proceeds to step S506. If it has not received it, it returns to step S501.

[0098] In step S506, the control unit A101 records the teacher's face image received in step S505 onto the recording medium A110.

[0099] In step S507, the control unit A101 transmits the teacher's face image recorded in step S506 to the training server A100d.

[0100] In step S508, the control unit A101 determines whether it has received a termination command from the user via the operation unit A104. If a termination command is received, the process is terminated. If no termination command is received, the process returns to step S501.

[0101] The above describes the operation of the management server A100c in this embodiment.

[0102] Next, we will explain the detailed operation of training server A100d in implementing the process shown in Figure 2, referring to Figure 6.

[0103] Figure 6 is a flowchart illustrating the operation of training server A100d. The processes shown in this flowchart are realized by the control unit A101 of learning server A100 controlling various parts of learning server A100 according to input signals and programs.

[0104] In step S601, the control unit A101 determines whether it has received a teacher's face image from the management server A100c. If it has received it, it proceeds to step S602. If it has not received it, it returns to step S601. (Teacher data acquisition means) In step S602, the face image learning unit A107 creates a learning model using the training face images received in step S601 as training data. (Learning method) In step S603, the control unit A101 records the learning model learned in step S602, along with the creation date and time information, onto the recording medium A110.

[0105] In step S604, the control unit A101 sends a notification of completion of the learning model to the person recognition glasses B100 via the communication unit A120.

[0106] In step S605, the control unit A101 determines whether it has received a notification from the person recognition glasses B100 requesting the acquisition of a learned model via the communication unit A120. If it has received the notification, it proceeds to step S606. If it has not received the notification, it returns to step S605.

[0107] In step S606, the control unit A101 transmits the learning model to the person recognition glasses B100 via the communication unit A120.

[0108] In step S607, the control unit A101 determines whether it has received a termination command from the user via the operation unit A104. If a termination command is received, the process is terminated. If no termination command is received, the process returns to step S601.

[0109] The above describes the operation of the training server A100d in this embodiment.

[0110] Next, we will explain the detailed operation of the person recognition glasses B100 in realizing the process shown in Figure 2, referring to Figure 7.

[0111] Figure 7 is a flowchart showing the operation of the person recognition glasses B100 in this embodiment. The processes shown in this flowchart are realized by the control unit B101 of the person recognition glasses B100 controlling each part of the person recognition glasses B100 according to the input signals and program.

[0112] In step S701, the face image acquisition unit B107 acquires an image. Image acquisition is performed when the user gives an acquisition command via the operation unit B104. The acquired image is sent to the face image recognition unit B108. The acquisition operation may be performed only once when instructed by the user, or it may be performed continuously when the user gives an operation as a start signal.

[0113] In step S708, the face image recognition unit B108 recognizes the face portion in the image sent from the face image acquisition unit B107 in step S701 and extracts it as a face image. The extracted face image is recorded on the recording medium B110. If face images with the same field of view are recognized consecutively, recording to the recording medium B110 may be stopped.

[0114] In step S702, the control unit B101 transmits the face image recorded on the recording medium B110 to the person recognition server A100a via the communication unit B120. Once the transmission is complete, the transmitted face image may be deleted from the recording medium B110.

[0115] In step S703, the control unit B101 determines via the communication unit B120 whether it has received a notification from the training server A100d indicating the completion of the learning model. If it has received the notification, it proceeds to step S704. If it has not received the notification, it returns to step S703.

[0116] In step S704, the control unit B101 determines whether it has received a request from the user to acquire a learning model via the operation unit B104. If a request is received, the process proceeds to step S705. If no request is received, the process returns to step S704.

[0117] In step S705, the control unit B101 sends a learning model acquisition request notification to the training server A100d via the communication unit B120.

[0118] In step S706, the control unit B101 determines whether it has received the learning model from the training server A100d via the communication unit B120. If it has received it, it proceeds to step S707. If it has not received it, it returns to step S706.

[0119] In step S707, the control unit B101 records the learning model received in step S706 onto the recording medium B110. Then, the control unit B101 adopts the learning model recorded on the recording medium B100 as the learning model to be used by the person recognition unit B106.

[0120] The above is a description of the operation of the person recognition glasses B100 of this embodiment.

[0121] As described above, this embodiment provides a learning model that can recognize a person designated by the user as a target for recognition, as well as people who are similar to that person.

[0122] Furthermore, although this embodiment describes the training data database as being stored on the recording medium A110 of the learning server A100, it may also be created from any facial images available on the internet via services such as search engines and social networking services.

[0123] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

Claims

1. An information processing system that communicates with a terminal device that performs image recognition on captured images and trains a neural network image recognition model for the purpose of image recognition, Receiving means for receiving a facial image transmitted from the terminal device, A determination means for determining whether or not the facial image received by the receiving means is a learning target, The aforementioned discrimination means includes a training data acquisition means that acquires a face image corresponding to the face image determined to be the target of recognition, and a person name information corresponding to that face image, from the database of the information processing system as training data. A learning means for training an image recognition model to recognize a person from an image determined to be the target of learning using the aforementioned training data, A transmission means for transmitting the trained image recognition model to the terminal device. It has, The aforementioned discrimination means is characterized in that it identifies facial images received from the terminal device a certain number of times or more within a unit period as recognition targets, and does not identify facial images that do not meet this criterion as recognition targets.

2. The information processing system according to claim 1, characterized in that the database stores image data received from the Internet.

3. The information processing system according to claim 1, characterized in that the learning means performs learning of an image recognition model when the receiving means receives an image from the terminal device.

4. The information processing system according to claim 1, characterized in that the learning means performs learning of an image recognition model at regular intervals.

5. The training data obtained from the aforementioned database includes attribute information of individuals. The information processing system according to any one of claims 1 to 4, characterized in that the attribute information includes any of age, gender, nationality, height, weight, occupation, affiliation, or hobbies.

6. A control method for an information processing system that communicates with a terminal device that performs image recognition on captured images and trains a neural network image recognition model for the said image recognition, A receiving step of receiving a facial image transmitted from the terminal device. A determination step to determine whether the face image received in the above receiving step is a learning target, The aforementioned discrimination step involves acquiring training data from the database of the information processing system to obtain training data, which includes training data for the face image determined to be the target of recognition in the aforementioned discrimination step, and training data for the face image and the corresponding person's name information. A learning process in which an image recognition model is trained to recognize a person from an image determined to be the target of learning using the aforementioned training data, A transmission step of transmitting the learned image recognition model to the terminal device. It has, The control method for an information processing system is characterized in that, in the discrimination step, images of faces that have been received from the terminal device a certain number of times or more within a unit period are identified as recognition targets, and images of faces that do not meet this criteria are not identified as recognition targets.

7. A computer-readable program for which a computer operates as one of the means of the information processing system described in any one of claims 1 to 5.

Images