Information processing device, information processing method, and program

Abstract

The present invention prevents a decrease in identification performance even when a usage environment is changed. An information processing device in accordance with an embodiment of the present invention comprises one or more input devices, an extraction unit, an identification unit, an environment information acquisition unit, an analysis unit, and a re-registration unit. The one or more input devices acquire input information. The extraction unit uses a person feature model to extract, from the input information, a feature amount that indicates a feature of a person. The identification unit identifies the person by comparing the feature amount and a feature amount that is indicated in a person dictionary. The environment information acquisition unit acquires environment information from the input information. The analysis unit analyzes a change in the environment information and the accuracy of identification of the person by the identification unit. The re-registration unit performs control for re-registering the person dictionary when there is a change in the environment information or when the accuracy of identification is less than an accuracy threshold value.

Description

Information processing device, information processing method, and program

[0001] An embodiment of the present invention relates to an information processing device, an information processing method, and a program.

[0002] There are known techniques for identifying people that prevent performance degradation due to changes over time or changes in physical condition, etc. For example, there is a known method for updating personal information when identification fails by having the user input personal characteristic information.

[0003] International Publication No. 2008 / 018136

[0004] Wan et al. , Generalized End-to-End Loss for Speaker Verification, ICASSP 2018, pp. 4879-4883, 2018 Synder et al. , X-Vectors: Robust DNN Embeddings for Speaker Recognition, ICASSP 2018, pp. 5329-5333, 2018Ioffe, Probabilistic linear discriminant analysis, ECCV, Part IV, LNCS 3954, pp. 531-542, 2006

[0005] However, with conventional technology, it has been difficult to prevent a decline in identification performance when the usage environment changes.

[0006] An information processing device according to an embodiment includes one or more input devices, an extraction unit, an identification unit, an environmental information acquisition unit, an analysis unit, and a re-registration unit. The one or more input devices acquire input information. The extraction unit uses a person feature model to extract features indicating characteristics of a person from the input information. The identification unit identifies the person by comparing the features with features indicated in a person dictionary. The environmental information acquisition unit acquires environmental information from the input information. The analysis unit analyzes changes in the environmental information and the accuracy of the identification of the person by the identification unit. The re-registration unit controls re-registration of the person dictionary when there is a change in the environmental information or when the identification accuracy is less than an accuracy threshold.

[0007] FIG. 1 is a diagram illustrating an example of the functional configuration of an information processing apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example of a person dictionary storage unit according to the first embodiment. FIG. 3 is a diagram illustrating an example of an environmental information storage unit according to the first embodiment. FIG. 4 is a flowchart illustrating an example of a person identification method according to the first embodiment. FIG. 5 is a diagram illustrating an example of the functional configuration of an environmental information acquisition unit according to Modification 1 of the first embodiment. FIG. 6 is a diagram illustrating example environmental information according to Modification 1 of the first embodiment. FIG. 7A is a diagram illustrating an example of a method of acquiring environmental information and a method of calculating a difference according to Modification 2 of the first embodiment. FIG. 7B is a diagram illustrating an example of a method of acquiring environmental information and a method of calculating a difference according to Modification 2 of the first embodiment. FIG. 8 is a diagram illustrating an example of the functional configuration of an environmental information acquisition unit according to Modification 2 of the first embodiment. FIG. 9 is a diagram illustrating an example of the functional configuration of an information processing apparatus according to the second embodiment. FIG. 10 is a diagram illustrating an example of an individual distribution model according to the second embodiment. FIG. 11 is a diagram illustrating an example of the functional configuration of the environmental information acquisition unit according to the second embodiment. FIG. 12 is a diagram illustrating an example of generating additional input information according to the second embodiment. FIG. 13 is a flowchart illustrating example 1 of a person dictionary re-registration flow according to the second embodiment. Fig. 14 is a flowchart showing example 2 of the person dictionary re-registration flow according to the second embodiment. Fig. 15 is a diagram showing an example of the functional configuration of an information processing device according to the third embodiment. Fig. 16 is a diagram showing an example of the functional configuration of an information processing device according to the fourth embodiment. Fig. 17 is a diagram showing an example of processing by a change-over-time processing unit according to the fourth embodiment. Fig. 18 is a diagram showing an example of the functional configuration of an information processing device according to a modified example of the fourth embodiment. Fig. 19 is a diagram showing an example of the hardware configuration of the information processing device according to the first to fourth embodiments.

[0008] Hereinafter, embodiments of an information processing device, an information processing method, and a program will be described in detail with reference to the accompanying drawings.

[0009] (First embodiment) In the first embodiment, an information processing device will be described that, when registering personal identification information, takes into consideration environmental information such as noise types, detects changes in the environment, and updates the personal identification information in accordance with the environment, thereby preventing performance degradation. The information processing device of the first embodiment may be any device, such as a personal computer, a smart device (e.g., a tablet, a smartphone, etc.), or a game console.

[0010] 1 is a diagram showing an example of the functional configuration of an information processing apparatus 100 according to the first embodiment. The information processing apparatus 100 according to the first embodiment includes an input device 1, a signal acquisition unit 2, an extraction unit 3, a person feature model storage unit 4, a registration unit 5, a person dictionary storage unit 6, an identification unit 7, an environment information acquisition unit 11, an analysis unit 21, a re-registration unit 31, and an environment information storage unit 41.

[0011] The input device 1 is a microphone, a camera, etc. that acquires input information. The number of input devices 1 is not limited to one, and a plurality of input devices 1 may be provided.

[0012] The signal acquisition unit 2 acquires input information from the input device 1 .

[0013] The extraction unit 3 extracts person features from the acquired input information based on the person feature model read from the person feature model storage unit 4. For example, when the input device 1 is a microphone or the like and the signal included in the input information is audio, the extraction unit 3 performs feature extraction on the acoustic signal input from the microphone at each time. In this case, the extracted features include MFCC (Mel-Frequency Cepstrum Coefficients) and Mel filter bank features.

[0014] Furthermore, the extraction unit 3 generates an embedding vector based on the extracted feature amount using i-vector, d-vector (Non-Patent Document 1), x-vector (Non-Patent Document 2), or a method derived from these.

[0015] Furthermore, when the input device 1 is a camera or the like, the extraction unit 3 may extract, as the person feature, a feature obtained from an image.

[0016] The person feature model storage unit 4 stores a person feature model used for extracting feature amounts. Specifically, the person feature model storage unit 4 stores parameters of a person feature model for extracting person features.

[0017] The registration unit 5 stores, for example, an average vector of person feature amounts of a plurality of frames obtained in a unit time in the person dictionary storage unit 6 as a person dictionary indicating person features.

[0018] FIG. 2 is a diagram illustrating an example of the person dictionary storage unit 6 of the first embodiment. The person dictionary storage unit 6 of the first embodiment includes an id and person features. The id is identification information (e.g., a unique value) that identifies each row of data. The person dictionary indicates the average vector of the above-mentioned person features. The example of FIG. 2 illustrates a case where person A is registered in two places and person B is registered in one place. When person A uses the information processing device 100, for example, person A may select one of two person dictionaries for person A depending on the place where person A uses the device. Furthermore, for example, when selecting a person dictionary, person A may be asked to speak, and the person dictionary with the highest score (the person dictionary with the highest similarity to person A's voice) may be selected.

[0019] 1 , the identification unit 7 calculates the similarity between the person dictionary stored in the person dictionary storage unit 6 and the person feature amount extracted from the input signal. Methods such as cosine similarity and PLDA (Non-Patent Document 3) are used to calculate the similarity. If there is a person dictionary whose similarity exceeds a predetermined threshold, the identification unit 7 notifies that the person identified by the person dictionary has been identified.

[0020] The environmental information acquisition unit 11 acquires environmental information indicating the environment of the information processing device 100, and stores the acquired environmental information in the environmental information storage unit 41. Details of the method for acquiring environmental information will be described later in Modification 1 of the first embodiment.

[0021] FIG. 3 is a diagram showing an example of the environmental information storage unit 41 of the first embodiment. The environmental information storage unit 41 of the first embodiment includes an id, noise, and microphone. The description of the id is the same as that of FIG. 2 and will be omitted. The noise and microphone are examples of environmental information of the first embodiment. The noise stores the presence or absence of noise. The microphone stores the type of microphone (for example, identification information for identifying an input device).

[0022] 1 , the analysis unit 21 includes a determination unit 22 and a monitoring unit 23. The determination unit 22 determines (detects) an environmental change from the environmental information acquired by the environmental information acquisition unit 11. The monitoring unit 23 monitors the accuracy of the identification unit 7 (person identification function).

[0023] Possible examples of environmental changes include the presence or absence of noise, differences in input devices, etc. As a method for monitoring accuracy, for example, before using the information processing device 100, the number of detections of a user registered in the person dictionary storage unit 6 is divided by the total number of detections, including the number of detections of other users who were incorrectly identified. For example, if the person dictionary of a user registered in the person dictionary storage unit 6 detects the user 10 times and another user once, the accuracy is 10 / 11.

[0024] For example, to monitor the accuracy of the person identification function using speaker identification, a method may be used in which the number of times a person speaks measured using a camera or the like is compared with the number of times the person is detected using the person dictionary registered in the person dictionary storage unit 6.

[0025] Furthermore, for example, a method of monitoring the accuracy of the person identification function by speaker identification may be used, in which a decrease in the similarity between the user's voice and the person dictionary of that user is monitored.

[0026] Furthermore, for example, the accuracy of the person identification function through speaker identification may be monitored by having the user provide feedback on the performance of the person identification function after using the information processing device 100. As a feedback method, for example, a method in which the user is asked to rate the performance on a five-point scale using an input device 1 such as a touch panel may be used. As a feedback method, for example, a method of providing a button to notify that the performance is poor, or a method of recognizing the user's facial expression using an input device 1 such as a camera may be used.

[0027] The re-registration unit 31 updates the person dictionary of the user when the user's usage environment has changed or when the identification accuracy of the user falls below a predetermined threshold as a result of analysis by the analysis unit 21. For example, a method of updating the person dictionary is to have the user input input information into the input device 1 again and update the person dictionary in the person dictionary storage unit 6.

[0028] When using the person identification function of the information processing device 100 of the first embodiment, there are two phases: a phase for registering a person dictionary in the person dictionary storage unit 6, and a phase for identifying a person using the registered person dictionary.

[0029] The phase of registering the person dictionary in the person dictionary storage unit 6 is performed by the above-described registration unit 5 before the person identification function of the information processing device 100 is used.

[0030] The phase of identifying a person using the registered person dictionary will be described with reference to FIG.

[0031] 4 is a flowchart showing an example of the person identification method according to the first embodiment. First, the information processing device 100 receives an operation input to turn on the power (step S1), and use of the information processing device 100 begins.

[0032] Next, the identification unit 7 selects a person dictionary to be used for identifying a person (user selection) (step S2). For example, the following methods (i) to (iii) are used for user selection.

[0033] (i) A method of accepting a user's selection operation from a registered person dictionary via the input device 1. (ii) A method of assuming a certain user group and narrowing down the user's behavior patterns to guess who is using the service. (iii) A method of guessing using a technology to identify a person other than the person in question.

[0034] In the example of (i), for example, if a microphone is provided as one of the input devices 1, candidate person dictionaries and their descriptions (for example, descriptions of the usage environment based on environmental information) are presented on a display device or the like, and the selection of a person dictionary is accepted by voice input from the user. Also, for example, if a keyboard and a mouse are provided as one of the input devices 1, the selection of a person dictionary is accepted by operation input from the user.

[0035] In the example of (ii), for example, if the information processing device 100 is a game console or the like, the identification unit 7 identifies the user from login information when the user logs in to the game console and selects a person dictionary for the user. Also, for example, if the information processing device 100 is installed in a vehicle or the like, the identification unit 7 uses the behavioral pattern of the user (driver) (for example, on weekdays: Person A drives, on holidays: Person B drives, etc.) as one of the factors for determining user selection.

[0036] In the example of (iii), for example, if the input device 1 is equipped with a camera, a fingerprint authentication sensor, etc., the identification unit 7 identifies the user from the captured image and fingerprint information, etc., and selects a person dictionary for the user. In the example of (iii), for example, if the person identification function by the identification unit 7 is speaker identification using voice, the user is selected using a function other than the person identification function, such as image recognition by a camera.

[0037] Next, the environmental information acquisition unit 11 acquires a portion of the environmental information (step S3). Next, the determination unit 22 determines an environmental change score indicating the magnitude of the difference (magnitude of environmental change) between the environmental information linked to the person dictionary selected in step S2 (environmental information at the time of person dictionary registration) and the environmental information acquired in step S3 (step S4). The presence or absence of a difference is determined, for example, as no difference (environment change score less than a threshold) if the same microphone is used, or no difference (environment change score less than a threshold) if the same noise type is used. Details of the environmental change score will be described in Modification 1 of the first embodiment.

[0038] If the environmental change score is greater than the threshold value (Yes at step S4), the re-registration unit 31 re-registers the person dictionary selected at step S2 (step S5).

[0039] If the environmental change score is less than the threshold value (step S4, No), the identification unit 7 identifies (identifies) the person by comparing the features extracted from the user input with the features indicated in the person dictionary (step S6).

[0040] Furthermore, the environmental information acquisition unit 11 acquires environmental information from the user input during the identification in step S6 (step S7).

[0041] Next, the analysis unit 21 (the determination unit 22 and the monitoring unit 23) analyzes the change in the environmental information (the difference between the environmental information at the time of person dictionary registration and the environmental information acquired in step S7) and the accuracy of person identification by the identification unit 7 (step S8).

[0042] If the analysis is not successful (there is a difference from the environmental information at the time of person dictionary registration, or the accuracy of person identification is less than the threshold) (step S8, No), the re-registration unit 31 re-registers the person dictionary selected in step S2 (step S9).

[0043] If the analysis is OK (there is no difference from the environmental information at the time of person dictionary registration, or the accuracy of person identification is above a threshold) (step S8, Yes), the process returns to step S6 unless the information processing device 100 is turned off (step S10, No), and if it is turned off (step S10, Yes), the process ends.

[0044] As described above, in the information processing device 100 of the first embodiment, one or more input devices 1 acquire input information. The extraction unit 3 uses a person feature model to extract feature quantities indicating the characteristics of a person from the input information. The identification unit 7 identifies a person by comparing the feature quantities with feature quantities indicated in the person dictionary. The environmental information acquisition unit 11 acquires environmental information from the input information. The analysis unit 21 analyzes changes in the environmental information and the accuracy of person identification by the identification unit 7. Then, the re-registration unit 31 controls re-registration of the person dictionary when there is a change in the environmental information or when the identification accuracy is below an accuracy threshold.

[0045] As a result, according to the first embodiment, even if the usage environment changes, it is possible to prevent a deterioration in the identification performance. For example, even if the person feature incorporates noise from the usage environment in addition to the user's voice, or even if the feature takes into account factors other than the user's features due to the frequency characteristics of the microphone being used, signal processing, etc., it is possible to prevent a deterioration in the identification performance.

[0046] (Variation 1 of First Embodiment) Next, Variation 1 of the first embodiment will be described. In the description of Variation 1, the same description as in the first embodiment will be omitted, and only differences from the first embodiment will be described. In Variation 1, a more detailed specific example of the environmental information acquisition unit 11 in FIG. 1 and an example of a method for calculating an environmental change score will be described.

[0047] 5 is a diagram illustrating an example of the functional configuration of the environmental information acquisition unit 11 according to Modification 1 of the first embodiment. The environmental information acquisition unit 11 receives input of a plurality of pieces of input information from a plurality of input devices 1, and acquires environmental information from the plurality of pieces of input information.

[0048] For example, the multiple input devices 1 include a microphone, a GPS (Global Positioning System) receiver that accepts input of GPS information, a camera that captures images, and devices that accept operational input from the user (e.g., a keyboard and a mouse).

[0049] The environmental information acquisition unit 11 of variant example 1 includes a microphone information acquisition unit 111, an SNR (Signal-to-Noise Ratio) acquisition unit 112, a noise type acquisition unit 113, a spatial information acquisition unit 114, a captured image acquisition unit 115, a GPS information acquisition unit 116, a total mora number acquisition unit 117, a registered word number acquisition unit 118, an age acquisition unit 119, and a gender acquisition unit 120.

[0050] For example, if the person identification function is a speaker identification technology, a microphone is used as the input device 1 used for speaker identification. The extraction unit 3 extracts a signal (feature) used for person identification from the acoustic signal acquired from the microphone via the signal acquisition unit 2.

[0051] The microphone information acquisition unit 111, the SNR acquisition unit 112, the noise type acquisition unit 113, and the spatial information acquisition unit 114 each acquire (estimate) environmental information using a method different from that used by the signal acquisition unit 2, and store the acquired environmental information in the environmental information storage unit 41.

[0052] Specifically, the microphone information acquisition unit 111 acquires the frequency characteristics of the microphone from the input information from the microphone. The SNR acquisition unit 112 acquires the SNR from the input information from the microphone. The noise type acquisition unit 113 acquires the type of noise from the input information from the microphone, for example, using a classifier that can classify noise. The spatial information acquisition unit 114 acquires acoustic spatial information (e.g., reverberation and reverberation time) from the input information from the microphone.

[0053] The captured image acquisition unit 115 acquires a captured image from input information from a camera. For example, the captured image is an image including a person. For example, if the camera has an iris sensor, the captured image includes iris information of the person.

[0054] The GPS information acquisition unit 116 acquires the position information of the information processing device 100 based on the GPS information from the input information from the GPS receiver.

[0055] The total mora number acquisition unit 117 acquires the total mora number from the input information input through the microphone when registering a person dictionary. Similarly, the registered word number acquisition unit 118 acquires the number of registered words from the input information input through the microphone when registering a person dictionary.

[0056] Furthermore, the age acquisition unit 119 acquires the age of the user via a keyboard, a mouse, etc. Similarly, the gender acquisition unit 120 acquires the gender of the user via a keyboard, a mouse, etc.

[0057] By acquiring the environmental information as described above, it becomes possible to detect that the environmental information differs between when the person is registered in the person dictionary and when the person is identified.

[0058] FIG. 6 is a diagram showing an example of environmental information according to Modification 1 of the first embodiment. The id column is the same as that described in FIG. 2 , and therefore will not be described again. The noise type column stores, for example, the classification results obtained by a classifier capable of classifying noise. The microphone column stores the device name of the microphone, the frequency characteristics of the microphone, and the like. The SNR column stores the average SNR of noise at the time of registration in the person dictionary as the noise level. The camera column stores captured images. The other environmental information described above is also similarly stored in the environmental information storage unit 41.

[0059] Next, an example of calculating the environmental change score will be described. For example, the difference between each environmental information item is expressed as a scalar value, and then the difference is scored using the following formula (1).

[0060] Environmental change score r = c1 * microphone information difference + c2 * spatial information difference + c3 * SNR difference + ... + ... (1)

[0061] Here, c1, c2, c3, ... are weights, and the method for determining the optimal values ​​of c1, c2, c3, ... depends on the usage requirements of the information processing device 100. If the environmental change score is greater than a threshold value (for example, 5(%)), it is determined that the environment has changed, and re-registration of the person dictionary is performed.

[0062] As an example, a method of expressing the difference between each environmental information item as a scalar value has been given, but the environmental change score may also be defined by a method using a nonlinear function such as a neural network. The method of updating (re-registration) the person dictionary can be changed depending on the combination of this environmental change score and the results of monitoring the accuracy of person identification.

[0063] 7A and 7B show examples of the method of acquiring the environmental information and the method of calculating the difference in each of Fig. 2. Figs. 7A and 7B are diagrams showing examples of the method of acquiring the environmental information and the method of calculating the difference in Modification 1 of the first embodiment. In the examples of Figs. 7A and 7B, when multiple methods of calculating the difference are possible, Example 2 of the difference is also shown.

[0064] As described above, according to the first modification, it is possible to take into consideration environmental changes by taking into account a wider variety of information in a composite manner.

[0065] 5 is merely an example, and environmental information may be acquired by other methods. For example, if the input device 1 is equipped with a sensor that acquires a person's fingerprint, information indicating the person's fingerprint may be acquired as environmental information.

[0066] (Variation 2 of First Embodiment) Next, Variation 2 of the first embodiment will be described. In the description of Variation 2, the same description as in the first embodiment will be omitted, and only differences from the first embodiment will be described. Variation 2 describes a method for estimating the part of data that could not be acquired when part of the environmental information of the environmental information acquisition unit 11 in FIG. 1 cannot be acquired.

[0067] 8 is a diagram illustrating an example of the functional configuration of the environmental information acquisition unit 11 according to Modification 2 of the first embodiment. The environmental information acquisition unit 11 according to Modification 2 includes an acquisition unit 12 and an estimation unit 13. The acquisition unit 12 acquires environmental information from input information input from the input device 1.

[0068] The estimation unit 13 estimates the environmental information by using, for example, environmental information previously acquired by the acquisition unit 12 and stored in the environmental information storage unit 41. This makes it possible to more accurately detect environmental changes and update the person dictionary.

[0069] For example, there may be cases where noise and spatial information cannot be acquired due to some reason (such as when the device (input device 1) installed in the information processing device 100 is unavailable due to a malfunction, etc.). In such cases, when some data indicating environmental information cannot be acquired, the estimation unit 13 estimates the location information indicated by the part of the data from the location information and the movement speed of the information processing device 100, and estimates the environmental information indicated by the part of the data that could not be acquired from the estimated location information. Specifically, the estimation unit 13 estimates where the user is currently located (e.g., in a car, in a crowded city, at an airport, or in a department store) from the location information and movement speed obtained from GPS information, thereby estimating environmental information indicating the type of noise and spatial information (e.g., reverberation and reverberation time).

[0070] In addition, for example, a portion of a certain continuous time-series data may be missing. When the environmental information is time-series data and some data indicating the environmental information cannot be acquired, the estimation unit 13 estimates the environmental information by interpolating the data from the acquired portion of the environmental information. For example, when data from 2 to 4 seconds of a 10-second period is missing, the estimation unit 13 interpolates the missing data by functional approximation using a linear function or the like using the data before and after. Although a method using a linear function has been given as an example of an interpolation method, a method using approximation to a nonlinear function such as a neural network, linear predictive analysis, or the like may also be used.

[0071] As another example of a method for estimating environmental information, a method is possible in which several versatile environmental information candidates are prepared, and the estimation unit 13 infers the environmental information by selecting one from the multiple environmental information candidates.

[0072] Second Embodiment Next, a second embodiment will be described. In the description of the second embodiment, the same description as in the first embodiment will be omitted, and only differences from the first embodiment will be described. In the second embodiment, a plurality of person dictionary re-registration methods (update methods) are prepared, and control for selecting one or more re-registration methods from the plurality of re-registration methods will be described.

[0073] 9 is a diagram showing an example of the functional configuration of an information processing device 100-2 according to the second embodiment. The information processing device 100-2 according to the second embodiment includes an input device 1, a signal acquisition unit 2, an extraction unit 3, a person feature model storage unit 4, a registration unit 5, a person dictionary storage unit 6, an identification unit 7, an environment information acquisition unit 11, an analysis unit 21, a re-registration unit 31, an input information storage unit 32, a generation unit 33, and an environment information storage unit 41.

[0074] In the second embodiment, an input information storage unit 32 and a generation unit 33 are further added to the configuration of the information processing device 100 of the first embodiment.

[0075] There are three methods for re-registering (updating) the person dictionary, for example:

[0076] The first method is to have the user re-input the input information via the input device 1.

[0077] The second method is for the re-registration unit 31 to request the generation unit 33 to generate input information. In this method, the generation unit 33 superimposes environmental information of the current usage environment on input information indicating the user's voice stored in the input information storage unit 32, and re-registers the person dictionary with the superimposed input information. This method eliminates the user's effort and makes it possible to automate the re-registration of the person dictionary. Examples of superimposition methods include synthesizing noise with the user's voice to achieve the current SNR, or reproducing reverberation and the like using an acoustic simulator.

[0078] For example, the second method is used when the environmental information includes voice recognition environment information obtained from input information input through a microphone. That is, when the environmental change score is greater than a threshold, the generation unit 33 generates updated data by superimposing data including the voice recognition environment information on the accumulated input information. Then, the re-registration unit controls re-registration of the person dictionary based on the updated data.

[0079] Specifically, the voice recognition environment information includes, for example, a noise type indicating the type of noise. When the environment change score is greater than a threshold, the generation unit 33 generates update data by superimposing data including noise of the noise type corresponding to the current environment on the accumulated input information.

[0080] For example, when the SNR as well as the noise type is obtained from the input information as the voice recognition environment information, if the environment change score is greater than a threshold value, the generation unit 33 generates update data by superimposing data including noise of the noise type corresponding to the current environment on the accumulated input information at a ratio corresponding to the SNR.

[0081] For example, when microphone characteristics are obtained from input information as voice recognition environment information, if the environmental change score is greater than a threshold value, the generation unit 33 generates update data by applying a filter to the accumulated input information to convert it into microphone characteristics that correspond to the current environment.

[0082] For example, when acoustic spatial information is obtained from input information as voice recognition environment information, if the environmental change score is greater than a threshold value, the generation unit 33 generates updated data by superimposing data including spatial information corresponding to the current environment on the accumulated input information.

[0083] In addition, not only in the case of voice recognition but also in the case of face recognition, even if only a camera image is obtained as environmental information, the generation unit 33 can generate update data that improves the performance of face recognition, for example, by making blurred areas of the face clearer or correcting the lighting conditions.

[0084] The third method is to change the person feature model in the person feature model storage unit 4 or re-learn by adding learning data used to re-learn the person feature model when the re-registration unit 31 determines that improvement of the person feature model that extracts person features is necessary.

[0085] Next, a method for determining whether registration has succeeded or failed when re-registering a person dictionary will be described. There are, for example, the following two methods for determining whether registration has succeeded or failed when re-registering a person dictionary.

[0086] (i) The identification unit 7 identifies a person using each of the multiple pieces of input information and the updated person dictionary, and the monitoring unit 23 monitors the accuracy of the identification. The re-registration unit 31 determines that the update is successful if the accuracy is equal to or greater than a certain threshold.

[0087] (ii) The environmental information acquisition unit 11 calculates the distribution of features for each user (see FIG. 10 ) from multiple pieces of input information using a Gaussian distribution, a von Mises distribution, or the like. FIG. 10 is a diagram showing an example of an individual distribution model according to the second embodiment. The re-registration unit 31 determines that the update of the person dictionary has been successful if the features indicated by the updated person dictionary are within the confidence interval of the individual distribution model in FIG. 10 .

[0088] The functional configuration of the environment information acquisition unit 11 for determining whether re-registration of a person dictionary has succeeded or failed by the distribution-based determination method (ii) will be described with reference to FIG.

[0089] 11 is a diagram illustrating an example of the functional configuration of the environmental information acquisition unit 11 according to the second embodiment. The environmental information acquisition unit 11 according to the second embodiment includes an individual distribution model calculation unit 34 (an additional input information generation unit 35, an embedding vector generation unit 36, and a calculation unit 37).

[0090] The additional input information generating unit 35 increases (pads) the number of pieces of input information by generating additional input information from the input information stored up to now in the input information storage unit 32. For example, when the person identification function of the identification unit 7 uses speaker identification technology, the additional input information generating unit 35 generates the additional input information by superimposing multiple types of noise on the input information with different SNRs. Also, for example, the multiple pieces of input information used for determination may include additional input information newly generated by superimposing data including speech recognition environment information on the input information.

[0091] Furthermore, for example, when the speaker identification technology does not depend on keywords, the additional input information generation unit 35 generates additional input information by dividing the speech used to generate speaker information into segments of several tens of milliseconds and randomly shuffling the segments. In other words, the multiple pieces of input information used for determination may include additional input information newly generated by rearranging the time series order of data included in the input information.

[0092] 12 is a diagram showing an example of generation of additional input information according to the second embodiment. As shown in Fig. 12, the additional input information generation unit 35 generates a plurality of new additional input information by rearranging the time series order of data included in the input information.

[0093] Returning to FIG. 11, the embedding vector generation unit 36 ​​extracts a plurality of feature quantities by generating an embedding vector indicating the feature quantity from each of a plurality of pieces of input information including the additional input information.

[0094] The calculation unit 37 calculates (estimates) an individual distribution model based on a multidimensional Gaussian distribution or the like from the extracted plurality of feature amounts, and stores the calculated model in the environmental information storage unit 41 as one piece of environmental information.

[0095] 10 has been calculated, the individual distribution model can be used thereafter in the processing of the monitoring unit 23 and the re-registration unit 31. Specifically, each time input information is entered by the user, it is determined whether or not the feature amount extracted from the input information is within the confidence interval of the individual distribution model, thereby making it possible to determine the success / failure of person identification and the success / failure of re-registration of the person dictionary.

[0096] 13 is a flowchart showing Example 1 of the person dictionary re-registration flow according to the second embodiment. First, the re-registration unit 31 determines the person dictionary update method (re-registration method) (step S21).

[0097] If the environmental change score is greater than the threshold (i.e., the difference is large), the re-registration unit 31 controls the re-registration process of the person dictionary based on the current environmental information. Specifically, first, the generation unit 33 superimposes the current environmental information on the user's input information stored in the input information storage unit 32 (step S22). Then, the registration unit 5 recalculates the person dictionary in the person dictionary storage unit 6 based on the feature amount extracted by the extraction unit 3 from the input information that matches the environment (step S25).

[0098] Furthermore, if there is insufficient registration information when registering a person (for example, if the total number of moras or registered words (or the registration time) is less than a threshold), performance may be good at the time of registration, but may deteriorate later. Therefore, for example, if there is insufficient registration information, the re-registration unit 31 receives input information from the user via the input device 1 and controls the re-registration process of the person dictionary by compensating for the insufficient registration information (steps S23 and S25).

[0099] For example, if the environmental information includes the total number of moras corresponding to the feature values ​​indicated by the person dictionary or the number of registered words corresponding to the feature values ​​indicated by the person dictionary, the flow of steps S23 and S25 can be executed. That is, when the re-registration unit 31 re-registers the person dictionary, if the total number of moras is smaller than the mora number threshold or if the number of registered words is smaller than the registered word number threshold, the re-registration unit 31 controls the re-registration of the person dictionary based on the voice re-input by the microphone.

[0100] In other cases (when there are no environmental changes and there is no shortage of registered information), the accuracy of person identification may be improved by updating (relearning) the person feature model used for extraction.

[0101] For example, as an example of an updating method for other cases in the determination of step S21, a determination as to whether to update the person feature model can be made using the above-mentioned individual distribution model. In this determination example, for example, when the user is known in advance, the distribution of the user's input information is obtained, and if the feature amount obtained from the current input information is outside the confidence interval of the distribution, the process proceeds to step S24, where the person feature model for feature extraction is updated (step S24). Then, the registration unit 5 recalculates the person dictionary in the person dictionary storage unit 6 based on the feature amount extracted using the updated person feature model (step S25).

[0102] Next, the monitoring unit 23 calculates the performance of the person identification function using the recalculated person dictionary and previously accumulated input information, and if the accuracy is equal to or greater than a threshold (performance equal to or greater than a standard), it determines that the re-registration of the person dictionary is successful (step S26, Yes), and the re-registration process ends.On the other hand, if the identification accuracy is below the threshold, the monitoring unit 23 determines that the re-registration of the person dictionary is unsuccessful (step S26, No), and the update process returns to step S21, and the recalculation of the person dictionary using another update method continues.

[0103] The example of the person dictionary re-registration flow in FIG. 13 is merely an example, and another re-registration flow may be used.

[0104] Fig. 14 is a flowchart showing a second example of the person dictionary re-registration flow according to the second embodiment. For example, as shown in Fig. 14, the success of re-registration may be determined (steps S33, S36, and S39), and the person dictionary may be recalculated (updated) in order until the re-registration is successful. In the example of Fig. 14, three recalculation patterns are tried in order, and if the re-registration is determined to be successful in any of steps S33, S36, and S39, the update is completed. If the re-registration is determined to be unsuccessful in step S39, the update is unsuccessful.

[0105] According to the second embodiment, by simply selecting a user (or estimating a user) at the start of use, for example, when identification fails, the person dictionary can be automatically updated in accordance with the usage environment without relying on user input, thereby saving the user the trouble of having to input information each time identification fails.

[0106] Note that the person dictionary can be updated by overwriting the existing one, adding newly generated features to the old one, averaging the features indicated by the old person dictionary and the newly generated features, or leaving both the features indicated by the old person dictionary and the newly generated features.

[0107] (Modification of Second Embodiment) Next, a modification of the second embodiment will be described. In the description of the modification, the same description as in the second embodiment will be omitted, and only differences from the second embodiment will be described. For example, when the person identification function by the identification unit 7 of the information processing device 100-2 is used in a car, noise and the like may change depending on whether the car is moving or not, and environmental information may frequently change significantly. In such a case, if the method of the second embodiment is used as is, re-registration of the person dictionary may occur frequently. In Example 2, a method for dealing with such a case will be described.

[0108] In a modified example, when selecting a person dictionary to be used for person identification, a plurality of person dictionaries are selected, the number of which corresponds to the number of types of environments that can frequently change (or more than that number). The identification unit 7 then performs person identification using each of the plurality of person dictionaries, and when a person is identified using one or more person dictionaries, it determines that the person has been identified.

[0109] Alternatively, the identification unit 7 may determine that a person has been identified when the same person is identified in more than half of the person dictionaries. Alternatively, instead of identifying a person using multiple person dictionaries, a method of acquiring environmental information for selecting an appropriate person dictionary at regular intervals and reselecting one person dictionary that matches the current environmental information at regular intervals may be used to identify a person.

[0110] Third Embodiment Next, a third embodiment will be described. In the description of the third embodiment, the same description as in the second embodiment will be omitted, and only differences from the second embodiment will be described. In the third embodiment, control in a case where multiple person dictionaries for one user are stored in the person dictionary storage unit 6 will be described.

[0111] 15 is a diagram showing an example of the functional configuration of an information processing device 100-3 according to the third embodiment. The information processing device 100-3 according to the third embodiment includes an input device 1, a signal acquisition unit 2, an extraction unit 3, a person feature model storage unit 4, a registration unit 5, a person dictionary storage unit 6, an identification unit 7, an environment information acquisition unit 11, an analysis unit 21, a re-registration unit 31, an input information storage unit 32, a generation unit 33, an environment information storage unit 41, and a selection unit 42.

[0112] In the third embodiment, a selection unit 42 is further added to the configuration of the information processing device 100-2 of the second embodiment.

[0113] When the information processing device 100-3 controlled using the person identification function of the identification unit 7 changes its usage environment frequently, it is conceivable that one user will register multiple person dictionaries according to the usage environment in the person dictionary storage unit 6. When the information processing device 100-3 is a portable laptop computer or the like, it is expected that the usage environment will change frequently.

[0114] When multiple person dictionaries for one person are stored in the person dictionary storage unit 6, the selection unit 42 selects a person dictionary according to environmental information. Specifically, when selecting a person dictionary during use of the information processing device 100-3, the selection unit 42 calculates an environmental change score using each piece of environmental information stored when the multiple person dictionaries were registered and the current environmental information. The selection unit 42 then selects the person dictionary with the smallest environmental change score, thereby enabling automatic selection of a person dictionary for an appropriate environment. For example, a user can simply input user identification information, such as a user name, into the information processing device 100-3, eliminating the need for the user to manually select a person dictionary.

[0115] (Fourth Embodiment) Next, a fourth embodiment will be described. In the description of the fourth embodiment, the same description as in the second embodiment will be omitted, and only differences from the second embodiment will be described. When monitoring the accuracy of person identification, even if the environmental information has not changed, the accuracy may decrease due to changes over time. In the fourth embodiment, a method for automatically updating the feature vector and person dictionary in accordance with changes over time in such a case will be described.

[0116] 16 is a diagram showing an example of the functional configuration of an information processing device 100-4 according to the fourth embodiment. The information processing device 100-4 according to the fourth embodiment includes an input device 1, a signal acquisition unit 2, an extraction unit 3, a person feature model storage unit 4, a registration unit 5, a person dictionary storage unit 6, an identification unit 7, an environmental information acquisition unit 11, an analysis unit 21, a re-registration unit 31, an input information storage unit 32, a generation unit 33, an environmental information storage unit 41, and a time-varying processing unit 51-1.

[0117] In the fourth embodiment, a time-varying processing unit 51-1 is further added to the configuration of the information processing device 100-2 of the second embodiment.

[0118] For example, the time-change processing unit 51-1 processes the time-change of the feature amounts indicated in the person's person dictionary by calculating the transition of the change in the feature amounts of the person from the input information stored in the input information storage unit 32. Then, the re-registration unit 31 controls the re-registration of the person's person dictionary based on the feature amounts after the time-change.

[0119] For example, the time-change processing unit 51-1 keeps a record of updates to the person dictionary (speaker vector) for each person, and automatically changes the speaker vector over time by analyzing how the speaker vector changes over time.

[0120] Specifically, in the fourth embodiment, when registering a person dictionary, the registration unit 5 stores environmental information, including the date of registration, in the environmental information storage unit 41 as environmental information for the person dictionary. Then, the time-varying processing unit 51-1 analyzes how the speaker vector changes over time, for example, by using a method for training an estimation model that estimates a current speaker vector from past speaker vectors for each user using a long short-term memory (LSTM) or the like. For example, this estimation model is trained as an estimation model that inputs the most recent n past person dictionaries (speaker vectors) from person dictionaries with the same environmental information into the LSTM and outputs the (n+1)th person dictionary (speaker vector).

[0121] 17 is a diagram showing an example of processing by the time-varying processor 51-1 of the fourth embodiment. In the example of Fig. 17, an example of processing of time-varying speaker information (speaker vector) of person A is shown.

[0122] (Modification of Fourth Embodiment) Next, a modification of the fourth embodiment will be described. In the description of the modification, the same description as in the fourth embodiment will be omitted, and only the differences from the fourth embodiment will be described.

[0123] 16 is a diagram showing an example of the functional configuration of an information processing device 100-5 according to a modified example of the fourth embodiment. The information processing device 100-5 according to the modified example includes an input device 1, a signal acquisition unit 2, an extraction unit 3, a person feature model storage unit 4, a registration unit 5, a person dictionary storage unit 6, an identification unit 7, an environmental information acquisition unit 11, an analysis unit 21, a re-registration unit 31, an input information storage unit 32, a generation unit 33, an environmental information storage unit 41, and a time-varying processing unit 51-2.

[0124] In this modification, the time-change processing unit 51-1 of the fourth embodiment is changed to a time-change processing unit 51-2 that is added between the extraction unit 3 and the identification unit 7. The modification differs from the fourth embodiment in that the time-change processing unit 51-2 performs processing that takes into account the influence of time-changes of a person not on the person dictionary (speaker vector) but on the feature amount to be compared with the person dictionary.

[0125] The configuration of the modified example of the fourth embodiment also provides the same effects as the fourth embodiment.

[0126] Finally, an example of the hardware configuration of the information processing apparatuses 100 (100-2, 100-3, 100-4) according to the first to fourth embodiments and the information processing apparatus 100-5 according to the modified example will be described.

[0127] 19 is a diagram showing an example of the hardware configuration of the information processing devices 100 (100-2, 100-3, 100-4) according to the first to fourth embodiments, and the information processing device 100-5 according to a modified example. The information processing devices 100 (100-2 to 100-5) include a processor 91, a main storage device 92, an auxiliary storage device 93, a display device 94, an input device 95, and a communication device 96. The processor 91, the main storage device 92, the auxiliary storage device 93, the display device 94, the input device 95, and the communication device 96 are connected via a bus 97.

[0128] The information processing devices 100 (100-2 to 100-5) may not be provided with some of the above configurations. For example, if the information processing devices 100 (100-2 to 100-5) can use the input function and display function of an external device, the information processing devices 100 (100-2 to 100-5) may not be provided with the display device 94 and the input device 95.

[0129] The processor 91 executes a program read from the auxiliary storage device 93 to the main storage device 92. The main storage device 92 is memory such as a ROM and a RAM. The auxiliary storage device 93 is a hard disk drive (HDD), a memory card, or the like.

[0130] The display device 94 is, for example, a liquid crystal display, etc. The input device 95 corresponds to one or more of the above-mentioned input devices 1. The communication device 96 is an interface for communicating with other devices.

[0131] Furthermore, for example, the programs executed by the information processing devices 100 (100-2 to 100-5) may be stored on a computer connected to a network such as the Internet, and may be provided by being downloaded via the network.

[0132] Furthermore, for example, the programs executed by the information processing devices 100 (100-2 to 100-5) may be provided via a network such as the Internet without being downloaded. Specifically, the information processing may be performed by a so-called ASP (Application Service Provider) type service, which does not transfer the programs from the server computer but realizes the processing function only by issuing execution instructions and obtaining the results.

[0133] Furthermore, for example, the programs for the information processing devices 100 (100-2 to 100-5) may be provided in a state where they are pre-installed in a ROM or the like.

[0134] The programs executed by the information processing devices 100 (100-2 to 100-5) have a modular configuration that includes functions that can be realized by the programs among the above-mentioned functional configurations. As for each function, the processor 91 reads the program from a storage medium and executes it, and the above-mentioned functional blocks are loaded onto the main storage device 92. In other words, the above-mentioned functional blocks are generated on the main storage device 92.

[0135] Note that some or all of the above-described functions may be realized by hardware such as an IC, rather than by software.

[0136] Furthermore, each function may be realized using multiple processors 91, in which case each processor 91 may realize one of the functions, or may realize two or more of the functions.

[0137] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

Claims

1. An information processing device comprising: one or more input devices for acquiring input information; an extraction unit for extracting features indicating characteristics of a person from the input information using a person feature model; an identification unit for identifying the person by comparing the features with features indicated in a person dictionary; an environmental information acquisition unit for acquiring environmental information from the input information; an analysis unit for analyzing changes in the environmental information and the accuracy of identification of the person by the identification unit; and a re-registration unit for controlling re-registration of the person dictionary when there is a change in the environmental information or when the identification accuracy is below an accuracy threshold.

2. The information processing device of claim 1, wherein when the environmental information is time-series data and some data indicating the environmental information cannot be acquired, the environmental information acquisition unit infers the environmental information by interpolating the part of the data from the acquired part of the environmental information.

3. The information processing device according to claim 1 or 2, wherein the analysis unit calculates an environmental change score indicating the magnitude of change in the environmental information, and the re-registration unit controls re-registration of the person dictionary when the environmental change score is greater than a change threshold.

4. The information processing device of claim 3, further comprising: a person dictionary storage unit for storing the person dictionary; an environmental information storage unit for storing the environmental information; and an input information storage unit for accumulating the input information, wherein the environmental information includes voice recognition environment information obtained from input information input by the microphone; and a generation unit for generating update data by superimposing data including the voice recognition environment information on the accumulated input information when the environmental change score is greater than the change threshold, wherein the re-registration unit executes a first re-registration process for re-registering the person dictionary based on the update data.

5. The information processing device of claim 4, wherein the environmental information stored in the environmental information storage unit further includes a total number of moras corresponding to the features indicated by the person dictionary, or a number of registered words corresponding to the features indicated by the person dictionary, and when re-registering the person dictionary, the re-registration unit executes a second re-registration process to re-register the person dictionary based on the voice re-input by the microphone if the total number of moras is smaller than a mora number threshold or if the number of registered words is smaller than a registered word number threshold.

6. The information processing device of claim 5, wherein the re-registration unit, when re-registering the person dictionary, executes a third re-registration process to update the person feature model by re-learning the person feature model if the environmental change score is less than the change threshold, the total number of moras is equal to or greater than the mora number threshold, and the number of registered words is equal to or greater than the registered word number threshold.

7. The information processing device described in claim 6, wherein the re-registration unit terminates the re-registration process if, after the person dictionary is re-registered, the identification accuracy is greater than an accuracy threshold, or if the features indicated by the re-registered person dictionary fall within a confidence interval of the distribution of the features of the person calculated using a plurality of pieces of input information.

8. The information processing device according to claim 7, wherein the plurality of pieces of input information include additional input information newly generated by superimposing data including the voice recognition environment information on the input information.

9. The information processing device according to claim 7, wherein the plurality of pieces of input information include additional input information newly generated by rearranging the chronological order of data included in the input information.

10. The information processing device according to claim 4, further comprising: a selection unit that, when a plurality of person dictionaries are stored in the person dictionary storage unit for one of the person, selects the person dictionary in accordance with the environmental information.

11. The information processing device according to claim 4, further comprising a time-change processing unit that processes changes over time in the features indicated in the person's person dictionary by calculating the transition of changes in the features of the person from the input information stored in the input information storage unit, wherein the re-registration unit controls the re-registration of the person's person dictionary based on the features after the change over time.

12. The information processing device according to claim 4, wherein the environmental information further includes a frequency characteristic of the microphone.

13. The information processing device of claim 1, wherein the one or more input devices include a receiver that accepts input of GPS (Global Positioning System) information, the environmental information further includes location information of the information processing device based on the GPS information, and when some of the data indicating the environmental information cannot be acquired, the environmental information acquisition unit estimates the location information indicated by the part of the data from the location information and the moving speed of the information processing device, and infers the environmental information indicated by the part of the data that could not be acquired from the estimated location information.

14. An information processing device as described in claim 1 or 2, wherein the one or more input devices include a camera that captures a captured image, and the environmental information includes at least one of a captured image including the person and iris information of the person.

15. An information processing device according to claim 1 or 2, wherein the one or more input devices include a sensor for acquiring a fingerprint of the person, and the environmental information includes information indicating the fingerprint of the person.

16. An information processing device as described in claim 1 or 2, wherein the one or more input devices include a device that accepts operational input indicating at least one of the person's gender and the person's age, and the environmental information includes at least one of the person's gender and the person's age.

17. An information processing method including the steps of: an information processing device acquiring input information; the information processing device extracting features indicating characteristics of a person from the input information using a person feature model; the information processing device identifying the person by comparing the features with features indicated in a person dictionary; the information processing device acquiring environmental information from the input information; the information processing device analyzing changes in the environmental information and the identification accuracy of the person in the identification step; and the information processing device performing control to re-register the person dictionary when there is a change in the environmental information or when the identification accuracy is below an accuracy threshold.

18. A program for causing a computer having one or more input devices for acquiring input information to function as: an extraction unit that extracts features indicating a person's features from the input information using a person feature model; an identification unit that identifies the person by comparing the features with features indicated in a person dictionary; an environmental information acquisition unit that acquires environmental information from the input information; an analysis unit that analyzes changes in the environmental information and the accuracy of the person identification by the identification unit; and a re-registration unit that controls re-registration of the person dictionary when there is a change in the environmental information or when the identification accuracy is below an accuracy threshold.

Images