Information processing device, information processing method, and program

Abstract

The invention provides an information processing apparatus which prevents degradation of identification performance even when a usage environment is changed. The system includes one or more input devices, an extraction unit, a determination unit, an environment information acquisition unit, an analysis unit, and a re-registration unit. One or more input devices obtain input information. The extraction unit extracts, from the input information, a feature amount indicating a feature of a person using a person feature model. The specifying unit specifies the person by comparing the feature amount with a feature amount represented by a person dictionary. The environment information acquisition unit acquires environment information from the input information. The analysis unit analyzes a change in the environmental information and the determination accuracy of the person by the determination unit. The re-registration unit performs control to re-register the character dictionary when there is a change in the environmental information or when the determination accuracy is less than an accuracy threshold value.

Description

Technical Field

[0001] The embodiments of the present invention relate to information processing apparatus, information processing methods, and programs. Background Technology

[0002] Previously known methods for people recognition prevent performance degradation caused by changes in time or physical condition. For example, previously known methods involve asking the user to input personal characteristics when recognition fails, thereby updating the person information in case of misidentification.

[0003] Existing technical documents Patent documents Patent Document 1: International Publication No. 2008 / 018136 Non-patent literature Non-patent literature 1: Wan et al., Generalized End-to-End Loss for Speaker Verification, ICASSP 2018, pp. 4879-4883, 2018 Non-patent literature 2: Synder et al., X-Vectors: Robust DNN Embeddings for Speaker Recognition, ICASSP 2018, pp. 5329-5333, 2018 Non-patent literature 3: Ioffe, Probabilistic linear discriminant analysis, ECCV, Part IV, LNCS 3954, pp. 531-542, 2006 Summary of the Invention

[0004] The problem that the invention aims to solve However, in existing technologies, it is difficult to prevent a decline in recognition performance when the usage environment changes.

[0005] Methods for solving problems The information processing apparatus of this embodiment includes one or more input devices, an extraction unit, a determination unit, an environmental information acquisition unit, an analysis unit, and a re-registration unit. One or more input devices acquire input information. The extraction unit uses a person feature model to extract feature quantities representing a person's characteristics from the input information. The determination unit determines the person by comparing the feature quantities with feature quantities represented in a person dictionary. The environmental information acquisition unit acquires environmental information from the input information. The analysis unit analyzes changes in the environmental information and the determination accuracy of the person. The re-registration unit controls the re-registration of the person dictionary when the environmental information changes or when the determination accuracy is less than an accuracy threshold. Attached Figure Description

[0006] Figure 1 This is a diagram illustrating an example of the functional structure of the information processing apparatus according to the first embodiment.

[0007] Figure 2 This is a diagram showing an example of the character dictionary storage unit according to the first embodiment.

[0008] Figure 3 This is a diagram illustrating an example of the environmental information storage unit in the first embodiment.

[0009] Figure 4 This is a flowchart illustrating an example of the person determination method according to the first embodiment.

[0010] Figure 5 This is a diagram illustrating an example of the functional structure of the environmental information acquisition unit in a variation of the first embodiment 1.

[0011] Figure 6 This is a diagram illustrating an example of environmental information from a variation of the first embodiment, Example 1.

[0012] Figure 7A This is a diagram illustrating an example of a method for obtaining environmental information and a method for calculating differences in a variation of the first embodiment, Example 2.

[0013] Figure 7B This is a diagram illustrating an example of a method for obtaining environmental information and a method for calculating differences in a variation of the first embodiment, Example 2.

[0014] Figure 8 This is a diagram illustrating an example of the functional structure of the environmental information acquisition unit in a variation of the first embodiment, Example 2.

[0015] Figure 9 This is a diagram illustrating an example of the functional structure of the information processing apparatus according to the second embodiment.

[0016] Figure 10 This is a diagram illustrating an example of an individual distribution model in the second embodiment.

[0017] Figure 11 This is a diagram illustrating an example of the functional structure of the environmental information acquisition unit in the second embodiment.

[0018] Figure 12 This is a diagram illustrating an example of generating additional input information according to the second embodiment.

[0019] Figure 13 This is a flowchart illustrating Example 1 of the re-registration process for the person dictionary according to the second embodiment.

[0020] Figure 14 This is a flowchart illustrating Example 2 of the re-registration process for the person dictionary according to the second embodiment.

[0021] Figure 15 This is a diagram illustrating an example of the functional structure of the information processing apparatus according to the third embodiment.

[0022] Figure 16 This is a diagram illustrating an example of the functional structure of the information processing apparatus according to the fourth embodiment.

[0023] Figure 17 This is a diagram illustrating a processing example of the time-varying processing unit in the fourth embodiment.

[0024] Figure 18 This is a diagram illustrating an example of the functional structure of an information processing apparatus according to a variation of the fourth embodiment.

[0025] Figure 19 This is a diagram illustrating examples of the hardware structure of the information processing apparatus according to the first to fourth embodiments. Detailed Implementation

[0026] Hereinafter, with reference to the accompanying drawings, the implementation methods of the information processing apparatus, information processing method and program will be described in detail.

[0027] (First Implementation) In the first embodiment, an information processing device will be described. This device considers environmental information such as noise type when registering person identification information, thereby detecting changes in the environment and updating the person identification information accordingly, thus preventing performance degradation. The information processing device in the first embodiment can be any device, such as a personal computer, a smart device (e.g., a tablet computer and a smartphone), and a game console.

[0028] [Example of functional structure] Figure 1This is a diagram illustrating an example of the functional structure of the information processing apparatus 100 according to the first embodiment. The information processing apparatus 100 according to the first embodiment includes an input device 1, a signal acquisition unit 2, an extraction unit 3, a person feature model storage unit 4, a registration unit 5, a person dictionary storage unit 6, a determination unit 7, an environmental information acquisition unit 11, an analysis unit 21, a re-registration unit 31, and an environmental information storage unit 41.

[0029] Input device 1 includes microphones and cameras, etc., for acquiring input information. Input device 1 is not limited to one; multiple devices may be used.

[0030] The signal acquisition unit 2 acquires input information from the input device 1.

[0031] The extraction unit 3 extracts character features from the acquired input information based on the character feature model read from the character feature model storage unit 4. For example, if the input device 1 is a microphone or the signal included in the input information is speech, the extraction unit 3 performs feature extraction on the audio signal input by the microphone at each moment. In this case, the extracted features are MFCC (Mel-Frequency Cepstrum Coefficients) and Mel filter features, etc.

[0032] Furthermore, the extraction unit 3 generates an embedding vector based on the extracted features using i-vector, d-vector (non-patent document 1), x-vector (non-patent document 2), and their derived methods.

[0033] Furthermore, when the input device 1 is a camera or the like, the extraction unit 3 can extract the feature quantities obtained from the image as human feature quantities.

[0034] The character feature model storage unit 4 stores the character feature model used for extracting feature quantities. Specifically, the character feature model storage unit 4 stores the parameters of the character feature model for extracting character features.

[0035] Registration unit 5, for example, stores the average vector of the character feature quantities of multiple frames obtained per unit time as a character dictionary representing the character's features in character dictionary storage unit 6.

[0036] Figure 2 This diagram illustrates an example of the person dictionary storage unit 6 according to the first embodiment. The person dictionary storage unit 6 of the first embodiment includes an ID and person feature values. The ID is identification information (e.g., a unique value) that identifies each row of data. The person dictionary represents an average vector of the aforementioned person feature values. Figure 2The example illustrates a situation where A registers in two places and B registers in one place. When A uses the information processing device 100, for example, one can be selected from two character dictionaries for A based on the location of A's use. Alternatively, for example, when selecting a character dictionary, A can be asked to speak, and the character dictionary with the highest score (the one with the highest similarity to A's voice) can be selected.

[0037] return Figure 1 The determination unit 7 calculates the similarity between the person dictionary stored in the person dictionary storage unit 6 and the person feature quantities extracted from the input signal. As a method for calculating the similarity, methods such as cosine similarity and PLDA (Non-Patent Document 3) are used. When there is a person dictionary with a similarity exceeding a predetermined threshold, the determination unit 7 notifies that a person identified by that person dictionary has been determined.

[0038] The environmental information acquisition unit 11 acquires environmental information representing the environment of the information processing device 100 and stores the acquired environmental information in the environmental information storage unit 41. Furthermore, details regarding the method for acquiring environmental information will be described later in a variation of the first embodiment.

[0039] Figure 3 This is a diagram illustrating an example of the environmental information storage unit 41 according to the first embodiment. The environmental information storage unit 41 of the first embodiment includes ID, noise, and microphone. The explanation of ID is as follows... Figure 2 The description is the same, so it is omitted. Noise and microphone are examples of environmental information in the first embodiment. The presence or absence of noise is stored in the noise. The type of microphone is stored in the microphone (e.g., identification information of the input device, etc.).

[0040] return Figure 1 The analysis unit 21 includes a judgment unit 22 and a monitoring unit 23. The judgment unit 22 judges (detects) environmental changes based on environmental information acquired by the environmental information acquisition unit 11. The monitoring unit 23 monitors the accuracy of the determination unit 7 (personnel determination function).

[0041] As examples of environmental changes, the presence or absence of noise and differences in input devices are considered. As a method for monitoring accuracy, for example, the number of detections of users registered in the character dictionary storage unit 6 before using the information processing device 100 is divided by the total number of detections, including the number of detections of other users who were incorrectly identified. For example, when a user is detected ten times through the character dictionary registered in the character dictionary storage unit 6, and another user is detected once, the accuracy is 10 / 11.

[0042] Alternatively, for example, as a method for monitoring the accuracy of a person identification function based on speaker recognition, a method can be used that compares the number of times a person speaks as measured by a camera or the like with the number of times that person is detected based on the person dictionary stored in the person dictionary storage unit 6.

[0043] Alternatively, for example, as a method for monitoring the accuracy of speaker identification based person identification functions, it is also possible to use a method that monitors the decrease in the similarity between the user's voice and the user's person dictionary.

[0044] Alternatively, for example, as a method for monitoring the accuracy of the person identification function based on speaker recognition, a method can be used that asks the user to provide feedback on the performance of the person identification function after using the information processing device 100. As a feedback method, for example, a method can be used that asks the user to evaluate the function in five stages via an input device 1 such as a touch panel. Alternatively, as a feedback method, a method can be used that provides a notification button for poor performance, or a method can be used that recognizes the user's facial expressions via an input device 1 such as a camera.

[0045] If the analysis result of the analysis unit 21 indicates that the user's usage environment has changed, or if the user's accuracy is less than a predetermined threshold, the registration unit 31 updates the user's character dictionary. For example, in the character dictionary update method, a method is used to update the character dictionary storage unit 6 by asking the user to re-enter input information into the input device 1.

[0046] When the person determination function of the information processing device 100 of the first embodiment is used, there are two stages: the stage of registering the person dictionary with the person dictionary storage unit 6, and the stage of determining a person using the registered person dictionary.

[0047] Before utilizing the person identification function of the information processing device 100, the registration unit 5 performs the stage of registering the person dictionary with the person dictionary storage unit 6.

[0048] use Figure 4 This section explains the stages of identifying individuals using the registered biographical dictionary.

[0049] Figure 4 This is a flowchart illustrating an example of the person determination method according to the first embodiment. First, the information processing device 100 receives an operation input to turn on the power (step S1) and begins to use the information processing device 100.

[0050] Next, the determination unit 7 selects the character dictionary used for character determination (user selection) (step S2). For example, the user selection uses the methods described in (i) to (iii) below.

[0051] (i) A method for a user to select a registered person dictionary via input device 1 (ii) Assuming a user group, narrow down user behavior patterns to infer who is using the method. (iii) Methods of speculation using techniques for identifying persons other than the person in question. In example (i), if a microphone is provided as one of the input devices 1, candidate characters for the character dictionary and their descriptions (e.g., descriptions of the usage environment based on environmental information, etc.) are displayed on a display device, and the selection of the character dictionary is received through voice input from the user. Alternatively, if a keyboard, mouse, or the like is provided as one of the input devices 1, the selection of the character dictionary is received through operational input from the user.

[0052] Regarding the example of (ii), for instance, if the information processing device 100 is a game console or the like, the determination unit 7 determines the user based on the login information when logging into the game console and selects the user's character dictionary. Alternatively, for instance, if the information processing device 100 is installed in a car or the like, the determination unit 7 uses the user's (driver's) behavior pattern (e.g., weekdays: driving A, weekends: driving B, etc.) as one of the criteria for user selection.

[0053] Regarding the example of (iii), for instance, if the input device 1 is equipped with a camera and a fingerprint authentication sensor, the determination unit 7 determines the user based on the captured image and fingerprint information, and selects the user's person dictionary. Another example of (iii) is when the person determination function of the determination unit 7 uses speaker recognition based on voice, and the user is selected using a function different from the person determination function, such as image recognition by the camera.

[0054] Next, the environmental information acquisition unit 11 acquires a portion of the environmental information (step S3). Then, the determination unit 22 determines the environmental change score (step S4), which represents the magnitude of the difference (the magnitude of the environmental change) between the environmental information associated with the character dictionary selected in step S2 (the environmental information when registering the character dictionary) and the environmental information acquired in step S3. The presence or absence of a difference is determined, for example, by: no difference if the same microphone is used (environmental change score less than a threshold); and no difference if the noise type is the same (environmental change score less than a threshold). Furthermore, details regarding the environmental change score will be explained in a variation of the first embodiment, Example 1.

[0055] If the environmental change score is greater than the threshold (step S4, yes), the re-registration unit 31 performs re-registration of the dictionary of people selected in step S2 (step S5).

[0056] If the environmental change score is less than the threshold (step S4, no), the determination unit 7 determines (identifies) the person by comparing the feature quantity extracted from the user input with the feature quantity represented by the person dictionary (step S6).

[0057] In addition, the environmental information acquisition unit 11 acquires environmental information from the user input during the identification process in step S6 (step S7).

[0058] Next, the analysis unit 21 (determination unit 22 and monitoring unit 23) analyzes the changes in environmental information (the difference between the environmental information when the character dictionary is registered and the environmental information obtained in step S7) and the accuracy of the determination of the character by the determination unit 7 (step S8).

[0059] If the analysis shows NG (the environmental information differs from that at the time of registration in the character dictionary, or the accuracy of the character determination is less than the threshold) (step S8, no), the re-registration unit 31 performs re-registration of the character dictionary selected in step S2 (step S9).

[0060] If the analysis is OK (there is no difference from the environmental information when the character was registered in the character dictionary, or the accuracy of the character determination is above the threshold) (step S8, Yes), as long as the information processing device 100 is not set to off (step S10, No), return to step S6, and if it becomes off (step S10, Yes), end the processing.

[0061] As described above, in the information processing apparatus 100 of the first embodiment, one or more input devices 1 acquire input information. The extraction unit 3 uses a person feature model to extract feature quantities representing the characteristics of a person from the input information. The determination unit 7 determines a person by comparing the feature quantities with feature quantities represented in a person dictionary. The environmental information acquisition unit 11 acquires environmental information from the input information. The analysis unit 21 analyzes changes in the environmental information and the determination accuracy of the person by the determination unit 7. Then, the re-registration unit 31 controls the re-registration of the person dictionary if the environmental information changes or if the determination accuracy is less than an accuracy threshold.

[0062] Therefore, according to the first embodiment, even when the usage environment changes, a decrease in recognition performance can be prevented. For example, even when the characteristic quantities of a person are mixed with noise from the usage environment in addition to the user's voice, or when the frequency characteristics of the microphone used and signal processing take into account characteristics other than the user's characteristics, a decrease in recognition performance can still be prevented.

[0063] (Modification 1 of the first embodiment) Next, a variation of the first embodiment, namely Example 1, will be described. In the description of Example 1, descriptions identical to those in the first embodiment will be omitted, and descriptions will focus on the differences from the first embodiment. In Example 1, regarding… Figure 1 The environmental information acquisition section 11 will be explained in more detail with examples of specific cases and methods for calculating environmental change scores.

[0064] Figure 5 This is a diagram illustrating an example of the functional structure of the environmental information acquisition unit 11 in a variation of the first embodiment. The environmental information acquisition unit 11 receives multiple input information from multiple input devices 1 and acquires environmental information from the multiple input information.

[0065] For example, multiple input devices 1 include a microphone, a GPS receiver for receiving GPS (Global Positioning System) information input, a camera for acquiring captured images, and devices for receiving user input (e.g., keyboard and mouse).

[0066] The environmental information acquisition unit 11 of Modified Example 1 includes a microphone information acquisition unit 111, an SNR (Signal Noise Ratio) acquisition unit 112, a noise type acquisition unit 113, a spatial information acquisition unit 114, an image acquisition unit 115, a GPS information acquisition unit 116, a total number of beats acquisition unit 117, a number of registered words acquisition unit 118, an age acquisition unit 119, and a gender acquisition unit 120.

[0067] For example, when the person identification function is speaker recognition technology, a microphone is used as the input device 1 for speaker recognition. The extraction unit 3 extracts the signal (feature quantity) used for person identification from the audio signal acquired from the microphone via the signal acquisition unit 2.

[0068] Microphone information acquisition unit 111, SNR acquisition unit 112, noise type acquisition unit 113 and spatial information acquisition unit 114 acquire (infer) environmental information respectively through a method different from that of signal acquisition unit 2, and store the acquired environmental information in environmental information storage unit 41.

[0069] Specifically, the microphone information acquisition unit 111 acquires the frequency characteristics of the microphone based on input information from the microphone. The SNR acquisition unit 112 acquires the SNR from the input information from the microphone. The noise type acquisition unit 113 acquires the noise type based on the input information from the microphone, for example, using a classifier capable of classifying noise. The spatial information acquisition unit 114 acquires the spatial information of the sound (e.g., echo and reverberation time) from the input information from the microphone.

[0070] Additionally, the image acquisition unit 115 acquires a captured image from input information from the camera. For example, the captured image may include a person. Furthermore, for example, if the camera is an iris sensor, the captured image may include the person's iris information.

[0071] The GPS information acquisition unit 116 acquires the location information of the GPS-based information processing device 100 from the input information from the GPS receiver.

[0072] In addition, the total beat count acquisition unit 117 acquires the total beat count from the input information input via the microphone when registering the character dictionary. Similarly, the registered word count acquisition unit 118 acquires the registered word count from the input information input via the microphone when registering the character dictionary.

[0073] In addition, the age acquisition unit 119 acquires the user's age via the keyboard and mouse, etc. Similarly, the gender acquisition unit 120 acquires the user's gender via the keyboard and mouse, etc.

[0074] By obtaining the environmental information described above, it is possible to detect situations where the environmental information differs between the time of registration in the character dictionary and the time of character identification.

[0075] Figure 6 This is a diagram illustrating an example of environmental information from a variation of the first embodiment, Example 1. Regarding the id column, ... Figure 2 The description is the same as above, so it is omitted. The noise type column stores, for example, the classification results of a classifier capable of classifying noise. The microphone column stores the microphone's device name and frequency characteristics, etc. The SNR column stores the average SNR of the noise during the registration of the person dictionary, representing the noise level. The camera column stores captured images. Other environmental information mentioned above is also stored in the environmental information storage unit 41.

[0076] Next, an example of calculating the environmental change score will be explained. For example, the differences in various environmental information are represented by scalar values ​​and scored using the following formula (1).

[0077] Environmental change score r = c1 × microphone information difference + c2 × spatial information difference + c3 × SNR difference + … + … (1).

[0078] Here, c1, c2, c3… are weights, and the method for determining the optimal values ​​of c1, c2, c3… depends on the usage requirements of the information processing device 100, etc. When the environmental change score is greater than a threshold (e.g., 5%), it is determined that the environment has changed, and the character dictionary is re-registered.

[0079] As an example, a method of representing differences in environmental information using scalar values ​​is given, but environmental change scores can also be defined using nonlinear functions such as neural networks. The update method (re-registration method) of the character dictionary can be changed based on a combination of this environmental change score and the accuracy monitoring results determined for the character.

[0080] Will Figure 2 Examples of methods for obtaining various environmental information and calculating differences are shown in the figure. Figure 7A and 7B . Figure 7A and 7B This is a diagram illustrating an example of a method for acquiring environmental information and a method for calculating differences according to a variation of the first embodiment 1. Furthermore, in Figure 7A and 7B In the example, considering multiple methods of calculating differences, Example 2 of the difference is also recorded.

[0081] As described above, according to Variation 1, more information can be comprehensively considered when taking into account environmental changes.

[0082] also, Figure 5 The example of the environmental information acquisition unit 11 is one example; environmental information can also be acquired through other methods. For example, if the input device 1 is equipped with a sensor that can acquire a person's fingerprint, information representing the person's fingerprint can also be acquired as environmental information.

[0083] (Modification 2 of the first embodiment) Next, a variation of the first embodiment, example 2, will be described. In the description of variation 2, descriptions identical to those in the first embodiment will be omitted, and the differences from the first embodiment will be explained. In variation 2, details where data cannot be obtained... Figure 1 The method for inferring a portion of the environmental information that cannot be obtained in the case of a portion of the environmental information in the environmental information acquisition unit 11 is explained.

[0084] Figure 8 This is a diagram illustrating an example of the functional structure of the environmental information acquisition unit 11 of Modified Example 2 of the first embodiment. The environmental information acquisition unit 11 of Modified Example 2 includes an acquisition unit 12 and an estimation unit 13. The acquisition unit 12 acquires environmental information from input information input by the input device 1.

[0085] The estimation unit 13 uses, for example, environmental information previously acquired by the acquisition unit 12 and accumulated in the environmental information storage unit 41 to estimate environmental information. As a result, environmental changes can be detected and the character dictionary updated with higher accuracy.

[0086] For example, there may be a situation where the installation machine (input device 1) of the information processing device 100 cannot acquire noise, spatial information, etc., due to some reason (e.g., it cannot be used due to malfunction). In this case, when the estimation unit 13 cannot acquire a portion of the data representing environmental information, it estimates the location information represented by that portion of the data based on the location information and the moving speed of the information processing device 100, and estimates the environmental information represented by the portion of the data that could not be acquired based on the estimated location information. Specifically, the estimation unit 13 estimates the current location (e.g., in a vehicle, in a crowd, at an airport, or in a department store, etc.) based on the location information obtained from GPS information and the moving speed, and thereby estimates environmental information representing the type of noise and spatial information (e.g., echo and reverberation time, etc.).

[0087] Furthermore, there are cases where a portion of a continuous time series data is missing. When the environmental information is time series data, if the inference unit 13 cannot obtain a portion of the data representing the environmental information, it interpolates a portion of the data based on the obtained portion of the environmental information, thereby inferring the environmental information. For example, if the inference unit 13 cannot obtain data from 2 to 4 seconds out of 10 seconds of data, it uses the preceding and following data to perform a function approximation using a linear function, thereby interpolating the missing data. As an example of an interpolation method, using a linear function is cited, but methods that approximate nonlinear functions such as neural networks, or linear predictive analysis, can also be used.

[0088] In addition, for example, as a method for inferring environmental information, consider preparing several candidates of environmental information with general applicability in advance, and the inference unit 13 selecting one of the candidates of environmental information to infer the environmental information, etc.

[0089] (Second Implementation) Next, the second embodiment will be described. In the description of the second embodiment, the descriptions that are the same as those in the first embodiment will be omitted, and the differences from the first embodiment will be described. In the second embodiment, the method for re-registering (updating) multiple character dictionaries and the control for selecting one or more re-registration methods from multiple re-registration methods will be described.

[0090] [Example of functional structure] Figure 9 This diagram illustrates an example of the functional structure of the information processing apparatus 100-2 according to the second embodiment. The information processing apparatus 100-2 according to the second embodiment includes an input device 1, a signal acquisition unit 2, an extraction unit 3, a person feature model storage unit 4, a registration unit 5, a person dictionary storage unit 6, a determination unit 7, an environmental information acquisition unit 11, an analysis unit 21, a re-registration unit 31, an input information storage unit 32, a generation unit 33, and an environmental information storage unit 41.

[0091] In the second embodiment, an input information storage unit 32 and a generation unit 33 are added to the structure of the information processing device 100 of the first embodiment.

[0092] There are three methods, for example, for re-registration (update) of the character dictionary.

[0093] The first method is to ask the user to re-enter the input information via input device 1.

[0094] The second method involves the re-registration unit 31 requesting the generation unit 33 to generate input information. This method involves the generation unit 33 overlaying environmental information of the current usage environment onto the input information representing the user's voice accumulated in the input information storage unit 32, and then using the overlaid input information to re-register the character dictionary. In this case, the re-registration of the character dictionary can be automated without inconveniencing the user. As an overlay method, there are methods such as synthesizing noise with the user's voice to obtain the current SNR, or reproducing an echo using an audio simulator.

[0095] For example, when the environmental information includes speech recognition environmental information obtained from input information input via a microphone, the second method is used. That is, when the environmental change score is greater than a threshold, the generation unit 33 overlays data containing speech recognition environmental information onto the accumulated input information, thereby generating updated data. Then, the re-registration unit controls the re-registration of the person dictionary based on the updated data.

[0096] Specifically, as environmental information for speech recognition, there are, for example, noise types representing different types of noise. When the environmental change score is greater than a threshold, the generation unit 33 superimposes noise data, including noise types corresponding to the current environment, onto the accumulated input information, thereby generating updated data.

[0097] In addition, for example, as speech recognition environment information, when SNR is obtained from the input information in addition to noise type, the generation unit 33, if the environmental change score is greater than the threshold, adds noise data including the noise type corresponding to the current environment to the accumulated input information at a ratio corresponding to SNR, thereby generating updated data.

[0098] Additionally, for example, as speech recognition environment information, when the characteristics of the microphone are obtained from the input information, the generation unit 33 applies a filter that converts the microphone characteristics corresponding to the current environment to the accumulated input information if the environmental change score is greater than a threshold, thereby generating updated data.

[0099] Additionally, for example, as speech recognition environmental information, when spatial information of the sound is obtained from the input information, the generation unit 33, if the environmental change score is greater than a threshold, superimposes data including spatial information corresponding to the current environment onto the accumulated input information, thereby generating updated data.

[0100] Furthermore, not limited to speech recognition, in the case of facial recognition, even when only camera images are obtained as environmental information, the generation unit 33 can generate updated data to improve facial recognition performance, for example, by making blurry facial features clear or by correcting lighting conditions.

[0101] The third method is a relearning method in which the re-registration unit 31, when it determines that the feature model of the extracted person needs to be improved, performs a change to the feature model of the person in the feature model storage unit 4, or adds learning data to the relearning of the feature model.

[0102] Next, the methods for determining the success or failure of re-registration in the character dictionary will be explained. For example, there are two methods for determining the success or failure of re-registration in the character dictionary.

[0103] (i) The determination unit 7 uses each of the multiple input information and the updated character dictionary to determine the character, and the monitoring unit 23 monitors the accuracy of the determination. The re-registration unit 31 determines that the update is successful when the accuracy is above a certain threshold.

[0104] (ii) The environmental information acquisition unit 11 calculates the distribution of feature quantities for each user based on multiple input information, using Gaussian distribution or von Mises distribution, etc. (refer to...) Figure 10 ). Figure 10 This is a diagram illustrating an example of the personal distribution model of the second embodiment. If the updated person dictionary represents the feature quantity... Figure 10 If the confidence interval of the individual distribution model is within the range, then the registration department 31 determines that the update of the character dictionary is successful.

[0105] Reference Figure 11 The functional structure of the environmental information acquisition unit 11, which uses the distribution determination method (ii) to determine the success or failure of the re-registration of the character dictionary, will be explained.

[0106] Figure 11 This is a diagram illustrating an example of the functional structure of the environmental information acquisition unit 11 in the second embodiment. The environmental information acquisition unit 11 in the second embodiment includes a personal distribution model calculation unit 34 (an additional input information generation unit 35, an embedding vector generation unit 36, and a calculation unit 37).

[0107] The additional input information generation unit 35 generates additional input information based on the input information previously accumulated in the input information storage unit 32, thereby increasing (expanding) the amount of input information. For example, if the person determination function of the determination unit 7 uses speaker recognition technology, the additional input information generation unit 35 changes the SNR and superimposes various noises onto the input information, thereby generating additional input information. Furthermore, for example, the multiple input information used in the determination may include newly generated additional input information by superimposing data containing speech recognition environment information onto the input information.

[0108] Additionally, for example, in cases where the speaker recognition technology is a keyword-independent method, the supplementary input information generation unit 35 segments the speech used for speaker information generation into tens of milliseconds and randomly shuffles the segmented speech to generate supplementary input information. That is, supplementary input information newly generated by rearranging the time sequence of data contained in the input information can be included among the multiple input information used for determination.

[0109] Figure 12 This is a diagram illustrating an example of generating additional input information according to the second embodiment. For example... Figure 12 As shown, the additional input information generation unit 35 generates multiple additional input information by rearranging the order of the time series of the data contained in the input information.

[0110] return Figure 11 The embedding vector generation unit 36 ​​extracts multiple features by generating an embedding vector representing a feature from each of multiple input information containing additional input information.

[0111] The calculation unit 37 calculates (infers) a personal distribution model based on a multidimensional Gaussian distribution or the like based on the extracted multiple feature quantities, and stores it as one of the environmental information in the environmental information storage unit 41.

[0112] After the above... Figure 10 In the case of calculating the personal distribution model, the personal distribution model can then be used in the processing of the monitoring unit 23 and the re-registration unit 31. Specifically, whenever there is user input of input information, by determining whether the feature quantity extracted from the input information is within the confidence interval of the personal distribution model, the success / failure of the person determination and the success / failure of the re-registration of the person dictionary can be determined.

[0113] Figure 13 This is a flowchart illustrating Example 1 of the re-registration process for the character dictionary according to the second embodiment. First, the re-registration unit 31 determines the method for updating the character dictionary (re-registration method) (step S21).

[0114] When the environmental change score exceeds a threshold (i.e., the difference is large), the re-registration unit 31 controls the re-registration process of the character dictionary based on the current environmental information. Specifically, firstly, the generation unit 33 overlays the current environmental information onto the user's input information stored in the input information storage unit 32 (step S22). Then, the registration unit 5 recalculates the character dictionary in the character dictionary storage unit 6 based on the feature quantity extracted by the extraction unit 3 from the input information that matches the environment (step S25).

[0115] Furthermore, when there is insufficient registration information during character registration (e.g., when the total number of beats or registered words (or registration time) is less than a threshold), the performance may be good at the time of registration but poor later. Therefore, for example, when there is insufficient registration information, the re-registration unit 31 receives input information from the user via the input device 1 and controls the re-registration process of the character dictionary based on the insufficient registration information (steps S23 and S25).

[0116] For example, if the environmental information includes the total number of phonations corresponding to the feature values ​​represented by the character dictionary, or the number of registered words corresponding to the feature values ​​represented by the character dictionary, then steps S23 and S25 can be executed. That is, when re-registering the character dictionary, if the total number of phonations is less than the phonation count threshold, or the number of registered words is less than the registered word count threshold, the re-registration unit 31 controls the re-registration of the character dictionary based on the voice input from the microphone.

[0117] In other cases (where there are no environmental changes and the registration information is sufficient), updating (relearning) the character feature model used to extract the character may improve the accuracy of character identification.

[0118] For example, in the determination in step S21, as an example of an update method under other circumstances, consider using the aforementioned personal distribution model to determine whether to update the person feature model. In this determination example, for example, if the user has been identified in advance, the distribution of the user's input information is calculated. If the feature quantity obtained from the current input information is outside the confidence interval of the distribution, then proceed to step S24 to update the person feature model used for feature extraction (step S24). Then, the registration unit 5 calculates the person dictionary in the person dictionary storage unit 6 based on the feature quantity extracted using the updated person feature model (step S25).

[0119] Next, the monitoring unit 23 uses the recalculated character dictionary and previously accumulated input information to calculate the performance of the character determination function. If the accuracy is above a threshold (performance above the baseline), the character dictionary re-registration is determined to be successful (step S26, Yes), and the re-registration process ends. On the other hand, if the determined accuracy is below the threshold, the monitoring unit 23 determines that the character dictionary re-registration has failed (step S26, No), and the update process returns to step S21 to continue recalculating the character dictionary based on other update methods.

[0120] also, Figure 13 The example of the re-registration process in the character dictionary is just one example; other re-registration processes can also be used.

[0121] Figure 14 This is a flowchart illustrating Example 2 of the re-registration process for the person dictionary according to the second embodiment. For example, as... Figure 14 As shown, it can be determined that the re-registration was successful (steps S33, S36, and S39), and the character dictionary can be recalculated (updated) sequentially until the re-registration is successful. Figure 14 In the example, three recalculation modes are tried in sequence. If the re-registration is determined to be successful in any of the steps S33, S36 and S39, it is set as an update completed. If the re-registration is determined to be unsuccessful in step S39, it is set as an update failure.

[0122] According to the second embodiment, the person dictionary can be automatically updated at the start of use by simply requiring user selection (or user inference). For example, if recognition fails, it can be automatically updated based on the usage environment without relying on user input. Thus, the hassle of user inputting information is eliminated whenever recognition fails.

[0123] In addition to covering the existing character dictionary, updating the character dictionary also considers methods such as adding newly generated features to the existing character dictionary. Furthermore, methods such as averaging the features represented by the existing character dictionary and the newly generated features, or retaining both the features represented by the existing character dictionary and the newly generated features, are also considered.

[0124] (A variation of the second embodiment) Next, a variation of the second embodiment will be described. In the description of the variation, descriptions identical to those in the second embodiment will be omitted, and the differences from the second embodiment will be explained. When the person determination function of the determination unit 7 of the information processing device 100-2 is used, for example, in a vehicle, environmental information such as noise levels can change frequently and drastically depending on whether the vehicle is moving. In this case, if the method of the second embodiment is used, frequent re-registration of the person dictionary may occur. In Embodiment 2, a method to address this situation will be described.

[0125] In a variation, when selecting the character dictionary for character identification, multiple character dictionaries are selected, including those representing a number (or more) of environments that may change frequently. Then, the identification unit 7 uses each of the multiple character dictionaries to identify the character, and if a character is identified through one or more character dictionaries, it determines that the character has been identified.

[0126] Alternatively, the determination unit 7 can determine that a person has been identified if more than half of the character dictionaries identify the same person. Alternatively, the following method can be used: instead of determining a person through multiple character dictionaries, environmental information for selecting an appropriate character dictionary is obtained at regular intervals, and a character dictionary matching the current environmental information is selected again at regular intervals, thereby determining the person.

[0127] (Third implementation method) Next, the third embodiment will be described. In the description of the third embodiment, the descriptions that are the same as those in the second embodiment will be omitted, and the differences from the second embodiment will be described. In the third embodiment, the control for the case where a user stores multiple character dictionaries in the character dictionary storage unit 6 will be described.

[0128] [Example of functional structure] Figure 15 This diagram illustrates an example of the functional structure of the information processing apparatus 100-3 according to the third embodiment. The information processing apparatus 100-3 according to the third embodiment includes an input device 1, a signal acquisition unit 2, an extraction unit 3, a person feature model storage unit 4, a registration unit 5, a person dictionary storage unit 6, a determination unit 7, an environmental information acquisition unit 11, an analysis unit 21, a re-registration unit 31, an input information storage unit 32, a generation unit 33, an environmental information storage unit 41, and a selection unit 42.

[0129] In the third embodiment, a selection unit 42 is added to the structure of the information processing device 100-2 in the second embodiment.

[0130] When the user environment of the information processing device 100-3, controlled by the character determination function of the determination unit 7, frequently changes, it is conceivable that a user could register multiple character dictionaries corresponding to the user environment in the character dictionary storage unit 6. In the case where the information processing device 100-3 is a portable laptop or the like, it is assumed that the user environment changes frequently.

[0131] When multiple character dictionaries are stored in the character dictionary storage unit 6 for a given character, the selection unit 42 selects a character dictionary based on environmental information. Specifically, when selecting a character dictionary using the information processing device 100-3, the selection unit 42 calculates an environmental change score using each of the environmental information stored when registering the multiple character dictionaries and the current environmental information. Then, the selection unit 42 can automatically select a character dictionary for the appropriate environment by choosing the one with the lowest environmental change score. For example, the user only needs to input user identification information such as the username into the information processing device 100-3, saving the user the trouble of manually selecting a character dictionary.

[0132] (Fourth Implementation) Next, the fourth embodiment will be described. In the description of the fourth embodiment, descriptions identical to those in the second embodiment will be omitted, and the differences from the second embodiment will be explained. In the case of accurate monitoring for character identification, even if the environmental information remains unchanged, the accuracy may decrease due to changes over time. In the fourth embodiment, a method for automatically updating the feature vector and character dictionary based on changes over time will be described in this situation.

[0133] [Example of functional structure] Figure 16 This diagram illustrates an example of the functional structure of the information processing apparatus 100-4 according to the fourth embodiment. The information processing apparatus 100-4 according to the fourth embodiment includes an input device 1, a signal acquisition unit 2, an extraction unit 3, a person feature model storage unit 4, a registration unit 5, a person dictionary storage unit 6, a determination unit 7, an environmental information acquisition unit 11, an analysis unit 21, a re-registration unit 31, an input information storage unit 32, a generation unit 33, an environmental information storage unit 41, and a time-varying processing unit 51-1.

[0134] In the fourth embodiment, a time-varying processing unit 51-1 is added to the structure of the information processing device 100-2 in the second embodiment.

[0135] For example, the time-varying processing unit 51-1 calculates the trajectory of change in a person's characteristic values ​​based on the input information accumulated in the input information storage unit 32, thereby processing the time-varying changes in the characteristic values ​​represented by the person's dictionary. Then, the re-registration unit 31 controls the re-registration of the person's dictionary based on the time-varying characteristic values.

[0136] In addition, for example, the time-varying processing unit 51-1 retains records of the updated speaker dictionary (speaker vector) according to the person, and analyzes the changes in the speaker vector caused by time-varying changes, thereby making the speaker vector change automatically over time.

[0137] Specifically, in the fourth embodiment, when registering a person dictionary, the registration unit 5 stores environmental information including the registration date as environmental information for that person dictionary in the environmental information storage unit 41. Then, the time-varying processing unit 51-1 analyzes the changes in speaker vectors caused by time-varying changes using, for example, the following method: for each user, an inference model is learned based on past speaker vectors using a model such as LSTM (Long Short-Term Memory) to infer the current speaker vector. For example, this inference model is learned as an inference model that inputs the n most recent person dictionaries (speaker vectors) from the person dictionary with the same environmental information into an LSTM and outputs the (n+1)th person dictionary (speaker vector).

[0138] Figure 17 This diagram illustrates a processing example of the time-varying processing unit 51-1 according to the fourth embodiment. Figure 17 The example shown illustrates the time-varying processing of speaker information (speaker vector) for A.

[0139] (A variation of the fourth embodiment) Next, a variation of the fourth embodiment will be described. In the description of the variation, the descriptions that are the same as those in the fourth embodiment will be omitted, and the differences from the fourth embodiment will be described.

[0140] [Example of functional structure] Figure 16 This is a diagram illustrating an example of the functional structure of the information processing apparatus 100-5, a modified example of the fourth embodiment. The modified information processing apparatus 100-5 includes an input device 1, a signal acquisition unit 2, an extraction unit 3, a person feature model storage unit 4, a registration unit 5, a person dictionary storage unit 6, a determination unit 7, an environmental information acquisition unit 11, an analysis unit 21, a re-registration unit 31, an input information storage unit 32, a generation unit 33, an environmental information storage unit 41, and a time-varying processing unit 51-2.

[0141] In a variant, the time-varying processing unit 51-1 of the fourth embodiment is replaced by a time-varying processing unit 51-2 added between the extraction unit 3 and the determination unit 7. The difference between the variant and the fourth embodiment is that the time-varying processing unit 51-2 processes the feature quantities compared with the person dictionary, rather than the person dictionary (speaker vector), taking into account the influence of the person's time-varying changes.

[0142] According to the structure of the modified example of the fourth embodiment, the same effect as the fourth embodiment can also be obtained.

[0143] Finally, examples of the hardware structure of the information processing apparatus 100 (100-2, 100-3, 100-4) in the first to fourth embodiments and the information processing apparatus 100-5 in the modified embodiment will be described.

[0144] [Examples of hardware architecture] Figure 19 This is a diagram illustrating examples of the hardware structure of the information processing apparatus 100 (100-2, 100-3, 100-4) according to the first to fourth embodiments and the modified information processing apparatus 100-5. The information processing apparatus 100 (100-2 to 100-5) includes a processor 91, a main storage device 92, an auxiliary storage device 93, a display device 94, an input device 95, and a communication device 96. The processor 91, main storage device 92, auxiliary storage device 93, display device 94, input device 95, and communication device 96 are connected via a bus 97.

[0145] Furthermore, the information processing apparatus 100 (100-2 to 100-5) may not include a portion of the above-described structure. For example, if the information processing apparatus 100 (100-2 to 100-5) can utilize the input and display functions of an external device, the information processing apparatus 100 (100-2 to 100-5) may not include the display device 94 and the input device 95.

[0146] The processor 91 executes programs read from the secondary storage device 93 into the main storage device 92. The main storage device 92 is a memory such as ROM and RAM. The secondary storage device 93 is a HDD (Hard Disk Drive) or a memory card, etc.

[0147] The display device 94 is, for example, a liquid crystal display (LCD). The input device 95 corresponds to one or more of the input devices 1 described above. The communication device 96 is an interface for communicating with other devices.

[0148] Alternatively, for example, it can be configured to provide the program executed by the information processing device 100 (100-2 to 100-5) by storing it on a computer connected to a network such as the Internet and downloading it via the network.

[0149] Alternatively, for example, it can be configured to provide the information via a network such as the Internet without downloading the program executed by the information processing device 100 (100-2 to 100-5). Specifically, it can also be configured to perform information processing through a so-called ASP (Application Service Provider) type service, which does not transmit the program from the server computer but only performs processing functions through its execution instructions and result acquisition.

[0150] Alternatively, for example, the program configured as the information processing device 100 (100-2 to 100-5) may be pre-installed in ROM or the like.

[0151] The program executed by the information processing device 100 (100-2 to 100-5) becomes a modular structure that includes functions that can also be implemented by the program, as described above. In each of these functions, as actual hardware, the processor 91 reads the program from the storage medium and executes it, thereby loading the aforementioned functional blocks onto the main storage device 92. That is, the aforementioned functional blocks are generated on the main storage device 92.

[0152] In addition, some or all of the above functions can be implemented through hardware such as ICs without using software.

[0153] Alternatively, multiple processors 91 can be used to implement the functions. In this case, each processor 91 can implement one or more of the functions.

[0154] Several embodiments of the present invention have been described, but these embodiments are merely illustrative and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope and spirit of the present invention, and are included within the scope of the invention described in the patent claims and their equivalents.

Claims

1. An information processing device, comprising: One or more input devices that acquire input information; The extraction unit uses a person feature model to extract feature quantities representing the features of a person from the input information; The determination unit determines the person by comparing the feature quantity with the feature quantity represented by the person dictionary; The environmental information acquisition unit acquires environmental information from the input information; The analysis unit analyzes the changes in the environmental information and the accuracy of the determination unit's determination of the person; as well as The re-registration department controls the re-registration of the dictionary of people when the environmental information changes or when the accuracy of the determination is less than the accuracy threshold.

2. The information processing apparatus according to claim 1, wherein, When the environmental information is time-series data, if the environmental information acquisition unit cannot acquire a portion of the data representing the environmental information, it infers the environmental information by interpolating the acquired portion of the data.

3. The information processing apparatus according to claim 1 or 2, wherein, The analysis unit calculates an environmental change score, representing the magnitude of the change in the environmental information. When the environmental change score is greater than the change threshold, the re-registration department controls the re-registration of the character dictionary.

4. The information processing apparatus according to claim 3, wherein, The one or more input devices include a microphone. The information processing device also includes: The Character Dictionary Storage Department stores character dictionaries. An environmental information storage unit stores the environmental information; and The input information storage unit accumulates the input information. The environmental information includes speech recognition environmental information obtained from input information input from the microphone; The information processing device further includes a generation unit that, when the environmental change score is greater than the change threshold, generates updated data by superimposing data containing the speech recognition environmental information onto the accumulated input information. The re-registration department performs a first re-registration process based on the updated data to re-register the person dictionary.

5. The information processing apparatus according to claim 4, wherein, The environmental information stored in the environmental information storage unit also includes the total number of musical beats corresponding to the feature quantities represented by the character dictionary, or the number of registered words corresponding to the feature quantities represented by the character dictionary. When the re-registration department re-registers the character dictionary, if the total number of phonations is less than the phonation count threshold, or the number of registered words is less than the registered word count threshold, a second re-registration process is performed based on the voice input from the microphone.

6. The information processing apparatus according to claim 5, wherein, When the re-registration department performs re-registration of the character dictionary, if the environmental change score is less than the change threshold, the total number of beats is greater than or equal to the beat count threshold, and the number of registered words is greater than or equal to the registered word count threshold, it performs a third re-registration process to update the character feature model by relearning the character feature model.

7. The information processing apparatus according to claim 6, wherein, After re-registering the person dictionary, the re-registration department terminates the re-registration process when the determination accuracy is greater than the accuracy threshold, or when the feature quantity represented by the re-registered person dictionary falls within the confidence interval of the feature quantity distribution of the person calculated using multiple input information.

8. The information processing apparatus according to claim 7, wherein, The input information includes additional input information newly generated by superimposing data including the speech recognition environment information onto the input information.

9. The information processing apparatus according to claim 7, wherein, The input information includes additional input information that is newly generated by rearranging the time series of data contained in the input information.

10. The information processing apparatus according to claim 4, further comprising: The selection unit selects a character dictionary based on environmental information when multiple character dictionaries are stored in the character dictionary storage unit for a given character.

11. The information processing apparatus according to claim 4, further comprising: The time-varying change processing unit calculates the change trajectory of the character's feature values ​​based on the input information accumulated in the input information storage unit, thereby processing the time-varying changes of the character's feature values ​​represented by the character dictionary. The re-registration department controls the re-registration of the person's dictionary based on the characteristic quantities after the change over time.

12. The information processing apparatus according to claim 4, wherein, The environmental information also includes the frequency characteristics of the microphone.

13. The information processing apparatus according to claim 1, wherein, The one or more input devices include a receiver that receives GPS information as input, which is Global Positioning System information. The environmental information also includes the location information of the information processing device based on the GPS information. When the environmental information acquisition unit cannot acquire a portion of the data representing the environmental information, it infers the location information represented by the portion of the data based on the location information and the moving speed of the information processing device, and infers the environmental information represented by the portion of the data that cannot be acquired based on the inferred location information.

14. The information processing apparatus according to claim 1 or 2, wherein, The one or more input devices include a camera for acquiring captured images. The environmental information includes at least one of a photograph containing the person and the person's iris information.

15. The information processing apparatus according to claim 1 or 2, wherein, The one or more input devices include a sensor for acquiring the person's fingerprint. The environmental information includes information representing the person's fingerprints.

16. The information processing apparatus according to claim 1 or 2, wherein, The one or more input devices include a device for receiving operational input representing at least one of the person's gender and the person's age. The environmental information includes at least one of the person's gender and the person's age.

17. An information processing method, characterized in that... include: The steps by which an information processing device acquires input information; The information processing device uses a person feature model to extract feature quantities representing the features of a person from the input information; The information processing device determines the person by comparing the feature quantity with the feature quantity represented by the person dictionary; The step of the information processing device acquiring environmental information from the input information; The information processing device analyzes the changes in the environmental information and the determination step improves the accuracy of the determination of the person; as well as The information processing device performs a step of re-registering the dictionary of people when the environmental information changes or when the accuracy of the determination is less than the accuracy threshold.

18. A program, characterized in that, To enable one or more input devices capable of acquiring input information to function as follows: The extraction unit uses a person feature model to extract feature quantities representing the features of a person from the input information; The determination unit determines the person by comparing the feature quantity with the feature quantity represented by the person dictionary; The environmental information acquisition unit acquires environmental information from the input information; The analysis unit analyzes the changes in the environmental information and the accuracy of the determination unit's determination of the person; as well as The re-registration department controls the re-registration of the dictionary of people when the environmental information changes or when the accuracy of the determination is less than the accuracy threshold.

Images