Information processing system and information processing method

Abstract

An information processing system includes: a biometric information detection unit that detects biometric information from one or more portions of a user; an estimation unit that estimates text information or voice information from the biometric information detected by the biometric information detection unit; a determination unit that determines whether the biometric information corresponds to a specific movement for controlling the system; and a control unit that controls the system based on a result of the determination by the determination unit.

Description

Technical Field

[0001] This invention relates to an information processing system that converts biometric information into text or voice information. Background Technology

[0002] In recent years, techniques have been developed to recognize spoken content using a user's voice information. Simultaneously, techniques have been developed to acquire biometric information instead of voice information, recognize facial expressions and text information from biometric information, and output the user's facial expressions and text information. NPL 1 discloses a technique for recognizing spoken content from biometric information related to skin movement during silent actions using accelerometers and angular velocity sensors.

[0003] Citation List

[0004] Patent documents

[0005] PTL 1 Japanese Patent Application Publication No. 2010-21902

[0006] Non-patent literature

[0007] NPL 1 Japan Information Processing Society Interaction 2020 "Derma: Silent Speech Interaction Using Transcutaneous Motion Perception" Summary of the Invention

[0008] Technical issues

[0009] However, in the case of NPL 1, it is necessary to control the device, for example, by pressing a button or by using a device connected to the device (such as a PC or smartphone), but control cannot be performed when the hands are not free to move relative to such a device. In device control using a combination of NPL 1 and PTL 1, each device is necessary and cannot be used when the hands are not free to move.

[0010] The purpose of this invention is to enable easy control of devices in information processing systems that convert biometric information into text or voice information.

[0011] Solution to the problem

[0012] According to a first aspect of this disclosure, an information processing system includes: a biometric information detection unit that detects biometric information from one or more parts of a user; an estimation unit that estimates text information or voice information from the biometric information detected by the biometric information detection unit; a determination unit that determines whether the biometric information corresponds to a specific movement for controlling the system; and a control unit that controls the system based on the determination result of the determination unit.

[0013] According to a second aspect of this disclosure, an information processing method includes: a biometric information detection step, the biometric information detection step detecting biometric information from one or more parts of a user; an estimation step, the estimation step estimating text information or voice information from the biometric information detected in the biometric information detection step; a determination step, the determination step determining whether the biometric information corresponds to a specific movement for controlling the system; and a control step, the control step controlling the system based on the determination result in the determination step.

[0014] Advantages of the invention

[0015] According to the present invention, in an information processing system that converts biometric information into text or voice information, it is possible to easily control the device without using another device. Attached Figure Description

[0016] Figure 1 This is a diagram illustrating the configuration of an information processing system according to an embodiment.

[0017] Figure 2 This is a flowchart illustrating the operation of an information processing system according to an embodiment.

[0018] Figure 3 This is a diagram illustrating the configuration of the information processing system based on the first example.

[0019] Figure 4 This is a schematic diagram of the detection equipment based on the first example.

[0020] Figure 5 This is a diagram illustrating the configuration of the information processing system based on the second example.

[0021] Figure 6 This is a schematic diagram of the detection equipment based on the second example.

[0022] Figure 7 This is a diagram illustrating the configuration of the information processing system based on the third example.

[0023] Figure 8This is a schematic diagram of the detection equipment based on the third example.

[0024] Figure 9 This is a diagram illustrating the configuration of the contact type sensor unit according to the third example.

[0025] Figure 10 This is a diagram illustrating the configuration of the contact type sensor unit according to the third example.

[0026] Figure 11 This is a diagram illustrating the external force calculation process in the third example.

[0027] Figure 12 This is a diagram illustrating the external force calculation process in the third example. Detailed Implementation

[0028] Embodiments of the present invention will be described in detail below. Figure 1 An overview of the information processing system of the present invention is illustrated. The information processing system converts speech content into text or speech information based on biometric information such as muscle activity during vocalization. In this disclosure, vocalization includes the act of vocalizing in which the audible sound is zero or substantially zero. Furthermore, for illustrative purposes, the act of vocalizing in which the audible sound is zero or substantially zero is specifically referred to as inaudible speech (silent speech) or silent vocalization, and the act of vocalizing that produces a normal level of audible sound is referred to as vocalization. In other words, in this disclosure, vocalization is a term that includes both silent speech or silent vocalization and vocalization.

[0029] (Configuration) The information processing system according to this embodiment is a system for estimating the content of a user's speech (including silent speech) from the user's biometric information. The information processing system according to this embodiment can also be regarded as an information conversion system that converts the user's biometric information into text information or voice information representing the content of speech.

[0030] The information processing system according to this embodiment mainly includes a detection device 100 that can be attached to a user, and an information processing device 101 configured to communicate with the detection device 100.

[0031] The detection device 100 includes a biometric information detection unit 102, a first transmitting unit 103, a first receiving unit 104, and a device output unit 105. The biometric information detection unit 102 detects biometric information from one or more parts of the user. The first transmitting unit 103 transmits the biometric information detected by the biometric information detection unit 102 to an information processing device 101. The first receiving unit 104 receives signals transmitted from the information processing device 101. The device output unit 105 provides information to the user based on the received signals from the first receiving unit 104.

[0032] Figure 1 The illustration shows a configuration in which the detection device 100 includes one biometric information detection unit; however, the detection device 100 may also include two or more biometric information detection units. Note that, in addition to biometric information, the detection device 100 may also have a mechanism for acquiring voice information. In other words, the detection device 100 may also be referred to as a biometric information acquisition unit.

[0033] The biometric information detection unit 102 is configured to include sensors for detecting biometric information related to the movement of a user's muscles, skin, and tongue. The biometric information detected by the biometric information detection unit 102 is, for example, biometric information related to the movement of the user's vocal organs. The biometric information detection unit 102 includes, for example, an accelerometer and an angular velocity sensor for detecting the movement of the user's mouth and tongue. Alternatively, the biometric information detection unit 102 may also include an accelerometer, an angular velocity sensor, and an electromyography (EMG) sensor for detecting the movement of the user's mouth and tongue. Again, alternatively, the biometric information detection unit 102 may include an accelerometer, an angular velocity sensor, and a tactile sensor for detecting the movement of the user's mouth and tongue.

[0034] An accelerometer detects acceleration and outputs data or signals corresponding to the detected acceleration. An angular velocity sensor (gyroscope sensor) detects angular velocity and outputs data or signals corresponding to the detected angular velocity. A tactile sensor outputs data or signals corresponding to the force sent to the sensor or the direction of the force. The accelerometer, angular velocity sensor, and tactile sensor in the biometric information detection unit 102 can also be integrated with each other.

[0035] The biometric information detection unit 102 is deployed at a location capable of detecting the movement of the user's mouth and tongue. For example, the biometric information detection unit 102 is deployed around the user's mouth. Specifically, the biometric information detection unit 102 can be deployed on at least any one of the user's jaw region, cheek, perioral area of ​​the mouth, throat, subauricular region, neck region, and temples. The biometric information detection unit 102 can also be deployed at locations other than those mentioned above, as long as it is capable of detecting biometric information related to the movement of the mouth and tongue.

[0036] The biometric information detection unit 102 may further include an electromyography (EMG) sensor, an ultrasound sensor, an optical sensor, a pressure sensor, etc. Specifically, the biometric information detection unit 102 may also include an accelerometer and an angular velocity sensor incorporating the EMG sensor and the pressure sensor. Additionally, a geomagnetic sensor may be incorporated into the biometric information detection unit 102. The biometric information detection unit 102 can also acquire information related to skin movement and muscle movement at the same location.

[0037] The accelerometer and angular velocity sensor in the biometric information detection unit 102 can also detect user movements other than mouth and tongue movements, such as blinking and head movements.

[0038] The biometric information detected by the biometric information detection unit 102 is used for both the processing of estimating the corresponding text or voice information and the processing of determining whether the detected biometric information corresponds to a specific movement used for system control. In other words, the biometric information to be converted into voice or text information and the biometric information used to determine whether the biometric information corresponds to a specific movement used for system control are detected by the same sensor.

[0039] The first receiving unit 104 can receive signals from the information processing device 101. The device output unit 105 outputs any one of sound, light, and vibration to the user based on the signals received by the first receiving unit 104. The device output unit 105 may include headphones, a light-emitting diode, a vibration element, etc. When headphones are included, the device output unit 105 outputs sound to the user based on the signals received by the first receiving unit 104. Similarly, when a light-emitting diode or a vibration element is included, the device output unit 105 outputs light or vibration to the user based on the signals received by the first receiving unit 104.

[0040] Information processing device 101 is a computer including a processor, memory, input / output devices, and communication devices. Information processing device 101 functions as a second receiving unit 106, a determining unit 107, a second transmitting unit 108, an estimating unit 109, and a system control unit 112 by executing programs stored in memory via the processor. Second receiving unit 106 receives biometric information transmitted from detection device 100. Determining unit 107 determines whether the biometric information received by second receiving unit 106 is a pre-set movement. Second transmitting unit 108 transmits information based on the determination result of determining unit 107 to detection device 100. Estimating unit 109 estimates text or voice information corresponding to the received biometric information. System control unit 112 controls the entire system based on the determination result of determining unit 107.

[0041] The information processing device 101 is a smartphone, a personal computer (PC), a tablet PC, etc., but is not limited to these. The information processing device 101 can also be installed in any equipment such as a camera. When the information processing device 101 is, for example, a personal computer, the text information estimated by the estimation unit 109 is sent to the display unit 110, such as a display.

[0042] The display unit 110 displays text information. The information processing device 101 may also include a display control unit (not shown) for controlling the display mode on the display unit 110.

[0043] The estimation unit 109 estimates corresponding text or speech information from the biometric information detected by the biometric information detection unit 102 using a predetermined estimation algorithm (conversion algorithm). The estimation unit 109 can also be expressed as a conversion unit that converts received biometric information into text or speech information. The estimation unit 109 can also estimate the speech information corresponding to the estimated text information. Alternatively, the estimation unit 109 can directly estimate the speech information based on the received biometric information.

[0044] The estimation unit 109 uses a predetermined estimation method when estimating text or speech information from biometric information detected by the biometric information detection unit 102. For the estimation method, a trained model (a first trained model) with an architecture configured using a neural network is used. The information processing device 101 includes a storage unit (not shown) for storing the trained model. The estimation unit 109 has the function of performing inference using the trained model. The trained model is a model generated using deep learning such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), or Transformer. The trained model can be a model derived from CNN, RNN, or Transformer, or it can be a model based on other machine learning techniques such as Support Vector Machine, Logistic Regression, Random Forest, and Hidden Markov Model. Alternatively, the estimation unit 109 may also perform estimation using a rule-based method instead of the trained model, or by using a rule-based method in addition to using the trained model.

[0045] exist Figure 1 In this embodiment, the estimation unit 109 is configured in the information processing device 101. Note that the estimation unit 109 can also be configured in the cloud.

[0046] Note that the estimation unit 109 can estimate the text or speech information of the discourse content from biometric information acquired during the phonation period (i.e., during audible phonation and silent speech). A silent speech period refers to a period during which the user performs an action that produces zero or substantially zero audible sound. An audible phonation period refers to a period during which the user performs an action that produces audible sound.

[0047] The speech information estimated by the estimation unit 109 is sent to the sound information output unit 111. The sound information output unit 111 is a speaker, and the sound information output unit 111 can reproduce the speech information. The text information or speech information estimated by the estimation unit 109 can also be displayed or reproduced at the destination by transmitting the information to another receiver via a network or the like.

[0048] The determination unit 107 determines whether the biometric information detected by the biometric information detection unit 102 corresponds to any of one or more preset specific movements used to control the system. The determination unit 107 performs this determination according to a determination algorithm, for example, using a trained model (a second trained model) with an architecture configured using a neural network. The information processing device 101 includes a storage unit (not shown) for storing the trained model. The determination unit 107 has the function of performing estimation using the trained model. The trained model is a model generated using deep learning such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), or Transformer. The trained model can be a model derived from CNN, RNN, or Transformer, or it can be a model based on other machine learning techniques such as Support Vector Machine, Logistic Regression, Random Forest, and Hidden Markov Model. Alternatively, the estimation unit 109 can also perform estimation using a rule-based method instead of the trained model, or by using a rule-based method in addition to the trained model.

[0049] Although it is determined that unit 107 is in Figure 1 The determination unit 107 is configured in the information processing device 101, but the determination unit 107 can also be configured in the detection device 100. In this case, the device output unit 105 can directly generate output to the user based on the determination result from the determination unit 107. Alternatively, it may be possible to integrate the determination unit 107 and the estimation unit 109 with each other and perform determination and estimation via a single trained model. In other words, the trained model to be used by the determination unit 107 (the first trained model) and the trained model to be used by the estimation unit 109 (the second trained model) can also be the same trained model.

[0050] The system control unit 112 controls the system based on the determination results from the determination unit 107. More specifically, when the determination unit 107 determines that biometric information detected from one or more parts of the user corresponds to any of the specific movements used to perform system control, the system control unit 112 performs system control corresponding to the movement.

[0051] "Specific movement for performing system control" (also referred to below as "specific movement") can be any movement that can be acquired by the detection device 100, and can be, for example, a movement different from the movement during phonation. Examples of "specific movement" are any one or a combination of the following: movement of the mouth, movement of the tongue, movement of the cheeks, movement of the eyeballs or eyelids, and movement of the face. Note that "specific movement" can also be a movement for uttering a predetermined keyword.

[0052] "System control" can be any control, as long as it is related to the detection device 100 or the information processing device 101. Examples of "system control" include starting or stopping the conversion of detected biometric information to text or voice information, or specifying an operating mode during the conversion. For example, when it is determined that biometric information corresponds to a first specific movement without performing estimation processing, estimation by the estimation unit 109 begins. Simultaneously, when it is determined that biometric information corresponds to a second specific movement during estimation processing, estimation by the estimation unit 109 ends. In other words, when it is determined that biometric information corresponds to a first specific movement, the system control unit 112 begins estimation by the estimation unit 109, and when it is determined that biometric information corresponds to a second specific movement, the system control unit 112 stops estimation by the estimation unit 109. Therefore, the first specific movement serves as a signal to start estimation by the estimation unit 109, and the second specific movement serves as a signal to stop estimation by the estimation unit 109. The user allows the estimation unit 109 to perform estimation control by executing either the first specific movement or the second specific movement.

[0053] Alternatively, depending on which of a number of preset specific movements the biometric information corresponds to, the system is controlled to switch to the operating mode corresponding to the movement or activate the application corresponding to the movement.

[0054] For example, when it is determined that the biometric information corresponds to a third specific movement, the system control unit 112 begins to display text information using the display unit 110, and when it is determined that the biometric information corresponds to a fourth specific movement, the system control unit 112 stops displaying the text information using the display unit 110. Alternatively, when it is determined that the biometric information corresponds to a fifth specific movement, the system control unit 112 may also begin to output voice information using the voice information output unit 111. Alternatively still, when it is determined that the biometric information corresponds to a sixth specific movement, the system control unit 112 may also stop outputting voice information using the voice information output unit 111. In other words, the system control unit 112 can control components such as the estimation unit 109, the display unit 110, and the voice information output unit 111—that is, the system—based on the type of movement based on the biometric information.

[0055] Alternatively, the information processing device 101 can be configured in the cloud. In this case, the first transmitting unit 103 of the detection device 100 sends the biometric information detected by the biometric information detection unit 102 to the cloud. The cloud performs a conversion to text or voice information based on the biometric information and sends the converted text information to the display unit 110. The display unit 110 displays the text information. Alternatively, the cloud can further convert the converted text information into voice information and send the voice information to the sound information output unit 111. Alternatively still, the cloud can directly convert the biometric information into voice information and send the voice information to the sound information output unit 111.

[0056] Communication between the detection device 100 and the information processing device 101 can be wired or wireless. In the case of wired communication, the first transmitting unit 103 and the first receiving unit 104 in the detection device 100 are wired to the second receiving unit 106 and the second transmitting unit 108 in the information processing device 101 via a USB cable or the like. In the case of wireless communication, the aforementioned units are wirelessly connected via wireless LAN communication such as Wi-Fi or short-range wireless communication such as Bluetooth (registered trademark).

[0057] When the information processing device 101 is a smartphone or tablet PC, the display unit 110 is a monitor. The audio output unit 111 consists of a smartphone, a speaker installed in the tablet PC, or a headset connected to the smartphone or tablet PC.

[0058] Note that the information processing device 101 can be considered as an information conversion device for converting biometric information detected by the biometric information detection unit 102 into text information or speech information. The information processing device 101 can also be considered as an information estimation device for estimating text information or speech information from the biometric information detected by the biometric information detection unit 102. Furthermore, the information processing device 101 may also include a processing unit that evaluates the biometric information detected by the biometric information detection unit 102 based on predetermined evaluation criteria and deletes text information or speech information corresponding to biometric information that does not meet the predetermined evaluation criteria.

[0059] (deal with) Figure 2 The illustration shows a flowchart illustrating the information processing method to be implemented by the information processing system according to this embodiment.

[0060] Step S100: The user attaches the detection device 100 to the biometric information detection unit 102 to detect the location of biometric information. The user attaches the detection device 100 to the head region, neck region, or the periphery of the head region or neck region. The biometric information detection unit 102 detects biometric information, which includes acceleration information and angular velocity information, or acceleration information and angular velocity information including at least one of electromyographic signals, tactile information, etc.

[0061] Step S101: The first sending unit 103 sends the biometric information to the information processing device 101. Accompanying information related to the time period information (time information) has been added to the biometric information. The first sending unit 103 can send the biometric information and the time period information to the information processing device 101.

[0062] Step S102: The biometric information detected by the detection device 100 is received by the second receiving unit 106 of the information processing device 101. The second receiving unit 106 sends the received biometric information to the determining unit 107.

[0063] Step S103: The determining unit 107 determines whether the biometric information corresponds to a preset specific movement (including the case where the mouth has not moved for a predetermined time period) according to a predetermined determining algorithm. When it is determined that the biometric information indicates a movement corresponding to the start of the transition, the system control unit 112 sends a signal to the device output unit 105 via the second sending unit 108 and via the first receiving unit 104 of the detection device 100, and the device output unit 105 generates a sound to notify the user of the start of the transition. The system control unit 112 also executes control to start the estimation unit 109 to estimate the subsequently acquired biometric information. At the same time, after the estimation begins, when the determining unit 107 determines that the biometric information indicates that the mouth has not moved for a predetermined time period, the system control unit 112 executes control to end the estimation performed by the estimation unit 109.

[0064] Step S104: The estimation unit 109 estimates text information, speech information, or command information from the biometric information according to a predetermined estimation algorithm. Alternatively, the estimation unit 109 may also estimate speech information from the biometric information, or it may estimate text information and then perform speech synthesis to convert the text information into speech information. The estimation unit 109 outputs the information estimated from the biometric information.

[0065] Step S105: Display unit 110 displays the text information estimated by estimation unit 109. Sound information output unit 111 outputs the speech information obtained by further converting the text information estimated by estimation unit 109.

[0066] The text information estimated by the estimation unit 109 can be stored in the storage unit of the information processing device 101. The text information can also be transmitted via a network and displayed on an external terminal.

[0067] By converting text information into speech, the speech can be reproduced and recorded at an external terminal, and the user can listen to the reproduced speech through headphones. This allows the user to verify the accuracy of the estimation. Furthermore, the speech information can be transmitted over a network and reproduced and recorded at another external terminal. By repeating this sequential process, continuous communication using biometric information becomes possible.

[0068] The information processing system according to this embodiment allows users to perform hands-free system control by moving their mouth, face, cheeks, etc., which is highly convenient for users. Furthermore, since the information processing system according to this embodiment determines the specific movements for system control using biometric information to be used for voice recognition, it is unnecessary to use a separate biometric information detection device for system control. This eliminates the need for users to use additional devices, improves convenience, and can also reduce product costs.

[0069] (First example)

[0070] Next, the information processing system according to the first example will be described. The information processing system according to this example converts sensing data (biometric information) obtained by using acceleration and angular velocity sensors attached to the user's cheeks and under-ear areas into text information or voice information.

[0071] Figure 3 An overview of the information processing system according to this example is illustrated, and Figure 4 A schematic diagram of a detection device 100 according to this example is shown. Here, the information processing device of the information processing system is in the form of a smartphone 300. The detection device 100 is powered by a battery (not shown).

[0072] like Figure 4 As shown, a user wears a detection device 100 around their neck area. The detection device 100 is a neckband type worn around the user's neck area. The detection device 100 is configured to include a first biometric information detection unit 401, a second biometric information detection unit 402, a device transmitting unit 403, a device receiving unit 404, and an output unit 405. The first biometric information detection unit 401 and the second biometric information detection unit 402 are connected to the device transmitting unit 403 via wires. The device transmitting unit 403 can transmit the biometric information detected by the first biometric information detection unit 401 and the second biometric information detection unit 402 to an external source.

[0073] Each of the first biometric information detection unit 401 and the second biometric information detection unit 402 is a 6-axis accelerometer / angular velocity sensor. A 6-axis accelerometer / angular velocity sensor is a sensor capable of measuring, for example, 3-axis translational acceleration and 3-axis angular acceleration.

[0074] The first biometric information detection unit 401 and the second biometric information detection unit 402 are attached to the user's cheek and under-ear area, respectively, via, for example, self-adhesive gel. This allows the first biometric information detection unit 401 and the second biometric information detection unit 402 to detect biometric information (such as acceleration and angular velocity). Therefore, the first biometric information detection unit 401 and the second biometric information detection unit 402 can detect information related to the movement of the user's body parts.

[0075] The biometric information detected by the first biometric information detection unit 401 and the second biometric information detection unit 402 is transmitted by the device transmission unit 403 to a smartphone 300, for example, connected via Wi-Fi. The receiving unit 301 in the smartphone 300 receives the data.

[0076] The determining unit 302 uses a trained model to determine whether the data received by the receiving unit 301 corresponds to a specific movement or a situation in which the mouth has not moved for 1 second. An example of a specific movement could be continuously closing the eyes for 0.5 seconds or longer. When it is determined that the received data corresponds to biometric information corresponding to a movement in which the eyes are closed for 0.5 seconds or longer, the determining unit 302 sends the information to the device receiving unit 404 via the transmitting unit 304. The output unit 405 provides the user with an audio signal to which the subsequent biometric information will be converted. After hearing the sound, the user performs a silent vocalization with the intention of converting the biometric information in order to estimate the biometric information. Meanwhile, after the estimation begins, when it is determined that the data received by the determining unit 302 indicates a state in which the mouth has not moved for 1 second or longer after the estimation begins, it is determined that the data acquisition for estimation is complete, and the biometric information is not sent to the estimation unit 303 until the start of the next estimation is determined.

[0077] The determination process in determination unit 302 is based on a trained model. For the generation of the trained model in this example, biometric information for each movement is obtained 30 times from five users across 20 pre-defined types of movements (such as movements involving closing the eyes for 0.5 seconds or longer). Here, 80% of all data is used as training data, and the remaining 20% ​​is used as evaluation data. For training, the training data is used to train the neural network for 500 rounds using each movement assigned a sequence number as the correct label, thereby producing the trained model.

[0078] The estimation unit 303 inputs the biometric information received by the receiving unit 301 into the trained model and estimates the corresponding text or speech information. Specifically, it estimates the text or speech information by using a total of 12-axis sensor information from 6-axis × 2 accelerometer / angular velocity sensors as input data to the trained model.

[0079] When the determining unit 302 determines that the biometric information detected from one or more parts of the user corresponds to any of the specific movements used to perform system control, the system control unit 307 performs the system control corresponding to the movement.

[0080] For the generation of the trained model in this example, biometric information during silent speech was obtained from 5000 sentences from five users. Here, 80% of all data was used as training data, and the remaining 20% ​​was used as evaluation data. For training, the training data was used to train the neural network for 2000 rounds using sentences converted to phonemes as correct labels, thereby producing the trained model. The estimation unit 303 has the function of estimating text information, including kanji and hiragana, from the estimated phoneme string. The estimation unit 303 also has the function of estimating speech information from the phoneme string.

[0081] The character information estimated by the estimation unit 303 (generated by the conversion) is displayed on a display, which is the display unit 305 of the smartphone. The voice information output unit 306 reproduces the voice information estimated by the estimation unit 303 (generated by the conversion).

[0082] (Second example)

[0083] Next, the information processing system according to the second example will be described. The information processing system according to this example converts sensing data (biometric information) obtained by using acceleration and angular velocity sensors and electromyography sensors attached to the user's cheek and jaw regions into text information or voice information.

[0084] Figure 5 An overview of the information processing system according to this example is illustrated, and Figure 6 A schematic diagram of a detection device 100 according to this example is shown. Here, the information processing device of the information processing system is in the form of a smartphone 500. The detection device is powered by a battery (not shown).

[0085] like Figure 6As shown, a user wears a detection device 100 in and around his or her neck and ears. The detection device 100 is configured to include a neckband portion worn around the user's neck area and an earphone portion. The neckband portion and the earphone portion are connected by wires. The detection device 100 may also be provided with connecting portions for attaching retractable members to the ends for wearing around the user's neck area. The detection device 100 is configured to include a first biometric information detection unit 601, a second biometric information detection unit 602, a transmitting / receiving unit 603, and a sound output unit 604. The first biometric information detection unit 601, the second biometric information detection unit 602, and the sound output unit 604 are connected to the transmitting / receiving unit 603. The transmitting / receiving unit 603 can transmit biometric information detected by the first biometric information detection unit 601 and the second biometric information detection unit 602 to an external location. Furthermore, the transmitting / receiving unit 603 can receive information determined by the determining unit 502 and information generated by the conversion by the estimating unit 506 via the transmitting unit 505.

[0086] Each of the first biometric information detection unit 601 and the second biometric information detection unit 602 integrates a 6-axis accelerometer and angular velocity sensor as well as an electromyography (EMG) sensor. The 6-axis accelerometer and angular velocity sensor is capable of measuring, for example, 3-axis translational acceleration and 3-axis angular acceleration. Triple Ag electrodes are used as electrodes for the EMG sensor. In other words, each of the first biometric information detection unit 601 and the second biometric information detection unit 602 is configured to include multiple different sensors.

[0087] The first biometric information detection unit 601 and the second biometric information detection unit 602 are respectively pressed against the user's cheek and jaw area via support members extending from the sound output unit 604. To further improve wearability, for example, the first biometric information detection unit 601 and the second biometric information detection unit 602 can also be adhered via self-adhesive gel. Therefore, the first biometric information detection unit 601 and the second biometric information detection unit 602 can detect biometric information (various accelerations and electromyography). This allows the first biometric information detection unit 601 and the second biometric information detection unit 602 to detect information related to the movement of the user's body parts.

[0088] Biometric information detected by biometric information detection units 601 and 602 is transmitted by transmission / reception unit 603 to a smartphone 500 connected via Wi-Fi, for example. The receiving unit 501 in the smartphone 500 receives the data.

[0089] The determining unit 502 uses a trained model to determine whether the biometric information received by the receiving unit 501 corresponds to a specific movement or a situation where the mouth has not moved for 1 second. Examples of specific movements could be nodding, i.e., the user's upright head movement, or the user continuously closing their eyes for 0.5 seconds or longer. When the determining unit 502 determines that the user has nodded, the smartphone 500 sends the determination result from the sending unit 505 to the sending / receiving unit 603, and the detection device 100 uses the sound output unit 604 to generate a sound to notify the user of the start of the command input mode. Simultaneously, when the determining unit 502 determines that the user has closed their eyes for 0.5 seconds or longer, the smartphone 500 sends the determination result from the sending unit 505 to the sending / receiving unit 603, and the detection device 100 uses the sound output unit 604 to generate a sound to notify the user of the start of the direct conversion mode. The movement associated with the determination of such a mode has been preset, and could also be other movements. The determining unit 502 determines the subsequent conversion mode based on the determined result. At the same time, after the conversion begins, when it is determined that the data received by the determining unit 502 indicates that the mouth has not moved for 1 second or longer, it is determined that the data acquisition for the conversion is complete, and the biometric information is not sent to the estimating unit 506 until the start of the next conversion is determined.

[0090] The determination process in determination unit 502 is performed based on a trained model. For the generation of the trained model in this example, biometric information from 20 pre-defined types of movements (such as head nodding) is obtained 30 times from five users for each movement. Here, 80% of all data is used as training data, and the remaining 20% ​​is used as evaluation data. For training, the training data is used to train the neural network for 500 rounds by using each movement assigned a sequence number as the correct label, thereby producing the trained model.

[0091] When the determining unit 502 determines that the biometric information detected from one or more parts of the user corresponds to any of the specific movements used to perform system control, the system control unit 507 performs the system control corresponding to the movement.

[0092] The estimation unit 506 inputs the biometric information received by the receiving unit 501 into the trained model and estimates the corresponding text or speech information. For example, when the operating mode is direct conversion mode, it estimates the text or speech information corresponding to the user's speech content from the received biometric information.

[0093] For the generation of the trained model in this example, biometric information during silent speech was obtained from 5000 sentences from five users. Of all the data, 80% was used as training data, and the remaining 20% ​​was used as evaluation data. For training, the training data was used to train the neural network for 2000 rounds using sentences converted to phonemes as the correct labels, thus producing the trained model.

[0094] In direct conversion mode, the content estimated by estimation unit 506 is converted from phoneme information into text information for display on a screen, which is the display unit 503 of the smartphone. Voice information output unit 504 reproduces the speech information from the phoneme information estimated by estimation unit 506. Alternatively, the speech information estimated by estimation unit 506 may be sent to a device via transmission unit 505, received by the device's transmission / reception unit 603, and reproduced by voice output unit 604. Meanwhile, in command input mode, the estimated content is used as a control command for the smartphone. For example, when the estimated content is "display email," the content of a newly received email recorded in the smartphone is displayed on display unit 503. Conversely, when the estimated content is "read email," the content of a newly received email recorded in the smartphone is synthesized into speech and output from voice information output unit 504 or voice output unit 604.

[0095] (Third example)

[0096] Next, the information processing system according to the third example will be described. The information processing system according to this example converts sensing data (biometric information) obtained by using acceleration and angular velocity sensors and contact type sensors attached to the user's cheek and jaw area into text information or voice information.

[0097] Figure 7 An overview of the information processing system according to this example is illustrated, and Figure 8 The diagram shows a schematic of the testing equipment 100.

[0098] The detection device 100 is a head-mounted type that attaches to a user's ear. The detection device 100 is configured to include a first biometric information detection unit 701, a second biometric information detection unit 702, a transmitting / receiving unit 703, and an audio information output unit 704. Each of the first biometric information detection unit 701 and the second biometric information detection unit 702 is configured with a contact-type sensor unit.

[0099] Figure 9 This is a diagram illustrating the configuration of the contact-type sensor unit 900 used in this example.

[0100] The sensor unit 900 is configured to include a circuit board 903 on which a magnetic sensor IC 901 and an inertial sensor IC 902 are mounted, an elastic member 904, a permanent magnet 905, and a contact member 906. The inertial sensor IC 902 has embedded detection units for 3-axis acceleration and 3-axis angular velocity. The contact member 906 is a member that contacts the skin.

[0101] An elastic member 904 is provided between the circuit board 903 and the contact member 906. The elastic member 904 can be deformed by an external force applied between the contact member 906 and the circuit board 907. As the elastic member 904, a soft material such as silicone gel, synthetic rubber or sponge can be used, or a metal or resin coil spring can also be used.

[0102] A permanent magnet 905 is fixed on the contact member 906 side of the elastic member 904. A magnetic sensor IC 901 detects the relative displacement with respect to the permanent magnet 905. The magnetic sensor IC 901 is an element having four magnetic field detection points. Alternatively, a 3-axis geomagnetic sensor capable of detecting the direction of the magnetic field vector can be used as the magnetic sensor IC 901. The sensor unit 900 uses the output from the magnetic sensor IC 901 to detect the three-dimensional displacement of the magnet using subsequent signal processing circuitry (not shown), and calculates the external force based on the detected displacement.

[0103] Figure 10 This is a diagram illustrating the configuration of another sensor unit 1000 used in this example. Sensor unit 1000 is an optical tactile sensor.

[0104] The sensor unit 1000 is configured to include a circuit board 1010 on which quadrant photodiodes 1007a and 1007b and an inertial sensor IC 1002 are mounted, an elastic member 1004, a light-emitting element 1009, and a contact member 1006. The inertial sensor IC 1002 has embedded detection units for 3-axis acceleration and 3-axis angular velocity. The contact member 1006 is a member that contacts the skin.

[0105] An opening member 1011 and an elastic member 1004 are disposed between the circuit board 1010 and the contact member 1006. The elastic member 1004 can be deformed by an external force applied between the contact member 1006 and the circuit board 1010.

[0106] A light-emitting element 1009, composed of LEDs, is fixed on the contact member 1006 side of the elastic member 1004. Each of the quadrant photodiodes 1007a and 1007b is a light-receiving unit having a light-receiving surface that is vertically and horizontally divided into four segments. The opening member 1011 includes openings 1030a and 1030b in the optical path between the light-emitting element 1009 and the quadrant photodiodes 1007a and 1007b. The sensor unit 1000 uses the outputs from the quadrant photodiodes 1007a and 1007b to detect the three-dimensional displacement of the light-emitting element 1009 via a subsequent signal processing circuit (not shown), and calculates the external force from the detected displacement.

[0107] refer to Figure 11 The process of detecting the amount of movement of the light-emitting element 1009 caused by external force and calculating the external force will be described. Figure 11 This is a schematic diagram illustrating the movement of the light-emitting element 1009 and the projected light spot 1040 when a vertical external force is received. The distance between the light spot pairs formed by one light-emitting element 1009 varies depending on the ratio of the distance between the light-emitting element 1009 and each of the openings 1030a and 1030b to the distance between each of the openings 1030a and 1030b and the surface of the light-receiving IC. When a force pushing the light-emitting element 1009 in the surface direction is received, the light-emitting element 1009 moves in the downward direction, and the distance between the light-emitting element 1009 and each of the openings 1030a and 1030b decreases. As a result, the distance between the centers of the light spot pairs increases. Conversely, when a force pulling the light-emitting element 1009 in the surface direction is received, the light-emitting element 1009 moves in the upward direction, and the distance between the light-emitting element 1009 and each of the openings 1030a and 1030b increases. As a result, the distance between the centers of the light spot pairs decreases.

[0108] The sensor unit 1000 can estimate the external force in the vertical direction by calculating the distance between the coordinates of the center positions of the light spot pair.

[0109] Figure 12This is a schematic diagram illustrating the movement of the light-emitting element 1009 and the projected light spot 1040 when an external force in the shear direction is received. When the elastic member 1004 receives an external force in the shear direction via the contact member 1006, the light-emitting element 1009 moves in the in-plane direction. As a result, the center of the light spot pair moves in the direction opposite to the external force. By calculating the average value of the coordinates of the center position of the light spot pair, the external force in the shear direction can be estimated. Since each of the segmented photodiodes 1007a and 1007b has a light-receiving portion divided into four segments, the sensor unit 1000 can estimate the two-dimensional external force in the in-plane direction.

[0110] The first biometric information detection unit 701 and the second biometric information detection unit 702 are pressed against the user's cheeks and chin. By bringing the first biometric information detection unit 701 and the second biometric information detection unit 702 into contact with the user's skin, the first biometric information detection unit 701 and the second biometric information detection unit 702 are able to detect biometric information.

[0111] The transmitting / receiving unit 703 is connected to the first biometric information detection unit 701 and the second biometric information detection unit 702. The transmitting / receiving unit 703 can transmit the biometric information detected by the first biometric information detection unit 701 and the second biometric information detection unit 702 to an external source. Furthermore, the transmitting / receiving unit 703 can also obtain information from the smartphone 805. The biometric information detected by the first biometric information detection unit 701 and the second biometric information detection unit 702 is wirelessly transmitted to the smartphone 805 using the transmitting / receiving unit 703.

[0112] In the application of the smartphone 805, the receiving unit 806 receives biometric information detected by the first biometric information detection unit 701 and the second biometric information detection unit 702. The determining unit 812 determines, based on a trained model, whether the received biometric information corresponds to a specific movement used to control the smartphone. For example, when the determining unit 812 determines that the biometric information corresponds to a nodding movement, the smartphone 805 transmits the determined result to the detection device 100 via the data transmission unit 807 and the send / receive unit 703. The voice information output unit 704 generates a sound to notify the user that the estimation of the biometric information will begin, while the data transmission unit 807 transmits the biometric information to the cloud 811. The estimation unit 808 on the cloud 811 begins to estimate from the biometric information to voice information using the trained model. The estimation unit 808 continues to estimate until the determining unit 812 determines that the user has not moved his or her mouth for one second or longer. When the estimation unit 808 is notified by the determination unit 812 that the mouth has not moved for 1 second or longer, the estimation unit 808 ends the estimation process.

[0113] When the determining unit 812 determines that the biometric information detected from one or more parts of the user corresponds to any of the specific movements used to perform system control, the system control unit 813 performs the system control corresponding to the movement.

[0114] In this example, the same trained model was used to determine predetermined movements for system control and to estimate speech or text information from biometric data. The trained model was generated as follows: First, biometric data during silent speech was obtained from five users for 20 movements, in addition to 5000 sentences. Then, 80% of all data was used as training data, and the remaining 20% ​​was used as evaluation data. For training, the training data was used to train the neural network for 2000 epochs, resulting in the trained model.

[0115] The voice information generated by the estimation unit 808 is transmitted to the data transmission unit 807. The voice information generated by the estimation unit 808 is then transmitted to another smartphone 810 for playback. Simultaneously, the converted voice is transmitted to a transmit / receive unit via the data transmission unit 807. The transmitted voice is played back via a sound information output unit 704, such as an earpiece, and the user confirms the conversion result.

[0116] (Other examples)

[0117] Each of the above examples is merely an example of an information processing system based on this disclosure and may be appropriately modified within the scope of the concepts of this disclosure.

[0118] For example, in the above description, detection devices of the neckband type, neckband-headphone combination type, and headband type are shown by way of example, but the detection device can also be of the glasses type, mask type, jaw mask type, collar type, ring type, etc. In the case of the ring type device, when input to the system is to be performed, the device is pressed against a part such as the face.

[0119] Furthermore, the biometric information used for estimating text or voice information and determining inputs for system control is not limited to the specific examples described above. Any biometric information can be used in the information processing according to this disclosure, as long as it can be obtained from a part of the user. For example, biometric information related to movement of the arm, leg, chest, or abdominal region can also be obtained. Moreover, the detection of biometric information can also be performed using sensors other than permanent sensors, and for example, it can be possible to capture images using a camera and use the movement of the user's body parts obtained through image analysis as biometric information.

[0120] Furthermore, although an information processing system configured to include detection equipment and information processing device has been shown by way of example, the information processing system may also be configured to include another device, or it may be an integrated system in which the detection equipment and information processing device are included in one device.

[0121] It should be noted that the various types of control described above can be performed by a piece of hardware (e.g., a processor or circuit) or by a process in other ways. The process can be distributed among multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits), thereby performing control over the entire device.

[0122] Furthermore, the processors mentioned above are processors in a broad sense, encompassing both general-purpose and special-purpose processors. Examples of general-purpose processors include central processing units (CPUs), microprocessor units (MPUs), and digital signal processors (DSPs). Examples of special-purpose processors include graphics processing units (GPUs), application-specific integrated circuits (ASICs), and programmable logic devices (PLDs). Examples of PLDs include field-programmable gate arrays (FPGAs) and complex programmable logic devices (CPLDs).

[0123] The above embodiments (including variations) are merely examples. Any configurations obtained by suitably modifying or changing some configurations of the embodiments within the scope of this disclosure are also included in this disclosure. This disclosure also includes other configurations obtained by suitably combining various features of the embodiments.

[0124] The present invention can also be implemented by supplying a program for implementing one or more functions of the above embodiments to a system or device via a network or storage medium, and by having the program read and executed by one or more processors of the computer of the system or device. The present invention can also be implemented by circuitry that implements one or more functions.

[0125] This invention is not limited to the embodiments described above, and various changes and modifications can be made thereon without departing from the spirit and scope of the invention. Therefore, the appended claims are attached to disclose the scope of the invention.

[0126] This application claims priority to Japanese Patent Application No. 2023-174025, filed on October 6, 2023, the entire contents of which are incorporated herein by reference.

[0127] [List of reference numerals]

[0128] 102 Biometric Information Detection Unit

[0129] 107 Determining Unit

[0130] 109 estimation units

[0131] 112 System control unit.

Claims

1. An information processing system, comprising: A biometric information detection unit, wherein the biometric information detection unit detects biometric information from one or more parts of a user; An estimation unit estimates text or speech information from the biometric information detected by the biometric information detection unit; A determining unit, the determining unit determining whether the biometric information corresponds to a specific movement used to control the system; as well as A control unit that controls the system based on the determination result made by the determining unit.

2. The information processing system according to claim 1, in, When it is determined that the biometric information corresponds to a first specific movement, the control unit controls the system to begin estimation of the text information or the voice information by the estimation unit based on the biometric information.

3. The information processing system according to claim 1 or 2, in, When it is determined that the biometric information corresponds to a second specific movement, the control unit controls the system to terminate the estimation of the text information or the voice information by the estimation unit based on the biometric information.

4. The information processing system according to any one of claims 1 to 3, The control unit controls the system to switch operating modes depending on which of a plurality of specific movements the biometric information corresponds to.

5. The information processing system according to any one of claims 1 to 4, The control unit controls the system to activate the application based on which of the plurality of specific movements the biometric information corresponds to.

6. The information processing system according to any one of claims 1 to 5, The specific movement is at least one of the user's facial movement, cheek movement, tongue movement, and eyeball or eyelid movement.

7. The information processing system according to any one of claims 1 to 6, The biometric information detection unit uses the same sensor to detect biometric information for estimating the voice information or the text information, as well as biometric information for determining whether the biometric information corresponds to the specific movement.

8. The information processing system according to any one of claims 1 to 7, The biometric information detection unit includes at least one of an accelerometer and an angular velocity sensor, and also includes an electromyography (EMG) sensor.

9. The information processing system according to any one of claims 1 to 7, The biometric information detection unit includes at least one of an accelerometer and an angular velocity sensor, and also includes a tactile sensor.

10. The information processing system according to any one of claims 1 to 9, The biometric information detection unit detects biometric information from at least two or more parts of the user, including the jaw region, cheek, throat, subauricular region, neck region, and temples.

11. The information processing system according to any one of claims 1 to 10, When the biometric information is determined to correspond to a third specific movement, the control unit begins to display the text information using the display unit, and when the biometric information is determined to correspond to a fourth specific movement, the control unit stops displaying the text information using the display unit.

12. The information processing system according to any one of claims 1 to 11, When the biometric information is determined to correspond to a fifth specific movement, the control unit begins to output the voice information using the voice information output unit, and when the biometric information is determined to correspond to a sixth specific movement, the control unit stops outputting the voice information using the voice information output unit.

13. The information processing system according to any one of claims 1 to 12, The estimation unit estimates the text information or the speech information from the biometric information using a first trained model, and The determining unit uses a second trained model to determine whether the biometric information corresponds to the specific movement.

14. The information processing system according to claim 13, Each of the first trained model and the second trained model is the same trained model.

15. The information processing system according to any one of claims 1 to 14, The biometric information detection unit is included in a device that can be attached to the user, and The estimation unit, the determination unit, and the control unit are included in an information processing device configured to communicate with the device.

16. The information processing system according to claim 15, The device includes an output unit that outputs the text information or the voice information estimated by the estimation unit.

17. The information processing system according to any one of claims 1 to 16, The control unit controls the system based on the type of movement, which is based on the biometric information.

18. An information processing method, comprising: A biometric information detection step, wherein the biometric information detection step detects biometric information from one or more parts of a user; An estimation step, wherein the estimation step estimates text information or speech information from the biometric information detected in the biometric information detection step; The determining step determines whether the biometric information corresponds to a specific movement used in the control system; as well as A control step, wherein the control step controls the system based on the result determined in the determining step.

19. A program for causing a computer to perform each of the steps of the information processing method according to claim 18.

Images