Information processing system and information processing method

Abstract

This information processing system comprises: a biological information detection unit that detects biological information from at least one location of a user; an inference unit that infers character information or voice information from the biological information detected by the biological information detection unit; a determination unit that determines whether or not the biological information corresponds to a specific movement for controlling the system; and a control unit that controls the system on the basis of the determination result of the determination unit.

Description

Information processing system and information processing method

[0001] The present invention relates to an information processing system that converts biometric information into text information or voice information.

[0002] In recent years, speech content has been recognized using voice information of a user. On the other hand, biometric information has been acquired instead of voice information, and facial expressions, text information, etc. of the user are recognized from the biometric information and output. Non-Patent Document 1 discloses a technology for identifying speech content from biometric information related to skin movement during silent movements using an acceleration sensor and an angular velocity sensor.

[0003] JP 2010-21902 A

[0004] Information Processing Society of Japan Interaction 2020 "Derma: Silent Speech Interaction Using Skin Movement Measurement"

[0005] However, in the case of Non-Patent Document 1, for example, it is necessary to press a button to control the device or to control it using a PC or smartphone connected to the device, which cannot be controlled in a situation where the hands are not free.Furthermore, device control by combining Non-Patent Document 1 and Patent Document 1 requires each device and cannot be used if the hands are not free to move.

[0006] An object of the present invention is to enable easy control of devices in an information processing system that converts biometric information into text information or voice information.

[0007] A first aspect of the present disclosure is an information processing system including a biometric information detection unit that detects biometric information from one or more locations on a user, an estimation unit that estimates text information or audio information from the biometric information detected by the biometric information detection unit, a determination unit that determines whether the biometric information corresponds to a specific movement for controlling a system, and a control unit that controls the system based on the determination result of the determination unit.

[0008] A second aspect of the present disclosure is an information processing method including a biometric information detection step of detecting biometric information from one or more locations on a user, an estimation step of estimating text information or audio information from the biometric information detected in the biometric information detection step, a determination step of determining whether the biometric information corresponds to a specific movement for controlling a system, and a control step of controlling the system based on the determination result of the determination step.

[0009] According to the present invention, in an information processing system that converts biometric information into text information or voice information, it is possible to easily control a device without using any other device.

[0010] FIG. 1 is a diagram illustrating a configuration of an information processing system according to an embodiment. FIG. 2 is a flowchart illustrating an operation of the information processing system according to an embodiment. FIG. 3 is a diagram illustrating a configuration of an information processing system according to Example 1. FIG. 4 is a schematic diagram of a detection device according to Example 1. FIG. 5 is a diagram illustrating a configuration of an information processing system according to Example 2. FIG. 6 is a schematic diagram of a detection device according to Example 2. FIG. 7 is a diagram illustrating a configuration of an information processing system according to Example 3. FIG. 8 is a schematic diagram of a detection device according to Example 3. FIG. 9 is a diagram illustrating a configuration of a contact-type sensor unit according to Example 3. FIG. 10 is a diagram illustrating a configuration of a contact-type sensor unit according to Example 3. FIG. 11 is a diagram illustrating an external force calculation procedure in Example 3. FIG. 12 is a diagram illustrating an external force calculation procedure in Example 3.

[0011] An embodiment of the present invention will be described in detail below. An outline of an information processing system of the present invention is shown in FIG. 1. The information processing system converts the content of a speech into text information or audio information based on biological information, such as muscle activity during speech. In this disclosure, speech includes speech acts that produce zero or almost zero audible sound. For the sake of explanation, speech acts that produce zero or almost zero audible sound will be referred to as silent speech or voiceless speech, and speech acts that produce a normal level of audible sound will be referred to as voiced speech. In other words, in this disclosure, speech includes both silent or voiceless speech and voiced speech.

[0012] (Configuration) The information processing system according to this embodiment is a system that estimates the content of a user's speech (including silent speech) from the user's biometric information. The information processing system according to this embodiment can also be regarded as an information conversion system that converts the user's biometric information into text information or voice information representing the content of the speech.

[0013] The information processing system according to this embodiment mainly includes a detection device 100 that can be worn by a user, and an information processing device 101 that is configured to be able to communicate with the detection device 100 .

[0014] The detection device 100 has a biometric information detection unit 102, a first transmission unit 103, a first receiving unit 104, and a device output unit 105. The biometric information detection unit 102 detects biometric information from one or more locations on the user. The first transmission unit 103 transmits the biometric information detected by the biometric information detection unit 102 to the information processing device 101. The first receiving unit 104 receives a signal transmitted from the information processing device 101. The device output unit 105 provides information to the user based on the signal received by the first receiving unit 104.

[0015] 1 shows a configuration in which the detection device 100 has one biometric information detection unit, but the detection device 100 may have two or more biometric information detection units. The detection device 100 may have a mechanism for acquiring audio information in addition to biometric information. The detection device 100 can also be referred to as an acquisition unit that acquires biometric information.

[0016] The biometric information detection unit 102 is composed of sensors that detect biometric information related to the user's muscle movements, skin and tongue movements, etc. The biometric information detected by the biometric information detection unit 102 is, for example, biometric information related to the movements of the user's vocal organs. The biometric information detection unit 102 is, for example, an acceleration sensor and an angular velocity sensor that detect the movements of the user's mouth and tongue. Alternatively, the biometric information detection unit 102 may be an acceleration sensor, an angular velocity sensor, or a myoelectric potential sensor that detects the movements of the user's mouth and tongue. Alternatively, the biometric information detection unit 102 may be an acceleration sensor, an angular velocity sensor, or a tactile sensor that detects the movements of the user's mouth and tongue.

[0017] The acceleration sensor is a sensor that detects acceleration and outputs data or a signal corresponding to the detected acceleration. The angular velocity sensor (gyro sensor) is a sensor that detects angular velocity and outputs data or a signal corresponding to the detected angular velocity. The tactile sensor is a sensor that outputs data or a signal corresponding to a force or the direction of the force transmitted to the sensor. The acceleration sensor, angular velocity sensor, and tactile sensor in the biometric information detection unit 102 may be integrated.

[0018] The biometric information detection unit 102 is installed in a position where it can detect the movement of the user's mouth and tongue. For example, the biometric information detection unit 102 is installed around the user's mouth. Specifically, the biometric information detection unit 102 can be installed in at least one of the user's lower jaw, cheek, around the mouth, throat, under the ear, neck, and temple. The biometric information detection unit 102 may be installed in a position other than the above-mentioned positions as long as it is a position where it can detect biometric information related to the movement of the mouth and tongue.

[0019] The bioinformation detection unit 102 may include a myoelectric potential sensor, an ultrasonic sensor, an optical sensor, a pressure sensor, etc. Specifically, the bioinformation detection unit 102 may be an acceleration sensor and an angular velocity sensor with a myoelectric potential sensor or a pressure sensor incorporated therein. The bioinformation detection unit 102 may also be one with a geomagnetic sensor incorporated therein. The bioinformation detection unit 102 can also acquire information related to skin movement and muscle movement at the same position.

[0020] The acceleration sensor and angular velocity sensor in the biometric information detection unit 102 can detect movements other than the movement of the mouth and tongue, such as the user's blinking and head movement.

[0021] The biometric information detected by the biometric information detection unit 102 is used both for the estimation process of corresponding text information or voice information and for the determination process of whether or not it corresponds to a specific movement for system control. In other words, the biometric information to be converted into voice information or text information and the biometric information to be determined to correspond to a specific movement for system control are detected by the same sensor.

[0022] The first receiving unit 104 can receive a signal from the information processing device 101. The device output unit 105 outputs to the user any of sound, light, or vibration in accordance with the signal received by the first receiving unit 104. The device output unit 105 may include earphones, a light-emitting diode, a vibration element, etc. If the device output unit 105 includes earphones, it outputs sound to the user in accordance with the signal received by the first receiving unit 104. Similarly, if the device output unit includes a light-emitting diode or a vibration element, it outputs light or vibration to the user in accordance with the signal received by the first receiving unit 104.

[0023] The information processing device 101 is a computer including a processor, a memory, an input / output device, and a communication device. The processor executes a program stored in the memory, and the information processing device 101 functions as a second receiving unit 106, a determination unit 107, a second transmitting unit 108, an estimation unit 109, and a system control unit 112. The second receiving unit 106 receives biometric information transmitted from the detection device 100. The determination unit 107 determines whether the biometric information received by the second receiving unit 106 corresponds to a preset movement. The second transmitting unit 108 transmits information based on the determination result of the determination unit 107 to the detection device 100. The estimation unit 109 estimates text information or audio information corresponding to the biometric information based on the received biometric information. The system control unit 112 controls the entire system based on the determination result by the determination unit 107.

[0024] The information processing device 101 may be, but is not limited to, a smartphone, a personal computer (PC), a tablet PC, or the like. The information processing device 101 may also be mounted in some kind of device, such as a camera. If the information processing device 101 is, for example, a personal computer, the character information estimated by the estimation unit 109 is transmitted to a display unit 110, such as a display.

[0025] The display unit 110 displays text information. The information processing device 101 may have a display control unit (not shown) that controls the display mode on the display unit 110.

[0026] The estimation unit 109 uses a predetermined estimation algorithm (conversion algorithm) to estimate corresponding text information or audio information from the biometric information detected by the biometric information detection unit 102. The estimation unit 109 can also be expressed as a conversion unit that converts the received biometric information into text information or audio information. The estimation unit 109 may estimate audio information corresponding to the text information based on the estimated text information. Alternatively, the estimation unit 109 may directly estimate audio information based on the received biometric information.

[0027] The estimation unit 109 uses a predetermined estimation method when estimating text information or audio information from the biometric information detected by the biometric information detection unit 102. The estimation method uses a trained model (first trained model) based on an architecture configured with a neural network. The information processing device 101 has a memory unit (not shown) that stores the trained model. The estimation unit 109 has a function of performing inference using the trained model. The trained model is a model generated using a deep learning convolutional neural network (CNN), a recurrent neural network (RNN), or a transformer. The trained model may be a model derived from a CNN, an RNN, or a transformer, or may be a model based on other machine learning techniques such as a support vector machine, a logistic regression, a random forest, or a hidden Markov model. Furthermore, the estimation unit 109 may perform estimation using a rule-based method instead of or in addition to using a trained model.

[0028] 1, the estimation unit 109 is configured in the information processing device 101. Note that the estimation unit 109 may also be configured on the cloud.

[0029] The estimation unit 109 can estimate text information or audio information of the speech content from biometric information acquired during speech, i.e., during voiced speech and silent speech. Silent speech is a period during which the user is performing a speech act in which no or almost no audible sound is produced. Voiced speech is a period during which the user is performing a speech act in which audible sound is produced.

[0030] The voice information estimated by the estimation unit 109 is transmitted to the voice information output unit 111. The voice information output unit 111 is a speaker and can reproduce the voice information. The character information and voice information estimated by the estimation unit 109 can also be displayed and reproduced at the transfer destination by transferring the information to another recipient via a network or the like.

[0031] The determination unit 107 determines whether the biometric information detected by the biometric information detection unit 102 corresponds to one or more specific movements for controlling a pre-set system. The determination unit 107 makes the above determination using a determination algorithm that uses, for example, a trained model (second trained model) with an architecture configured by a neural network. The information processing device 101 has a memory unit (not shown) that stores the trained model. The determination unit 107 has a function of making an estimation using the trained model. The trained model is a model generated using a deep learning convolutional neural network (CNN), a recurrent neural network (RNN), or a transformer. The trained model may be a model derived from a CNN, an RNN, or a transformer, or may be a model based on other machine learning techniques such as a support vector machine, a logistic regression, a random forest, or a hidden Markov model. Furthermore, the estimation unit 109 may perform estimation using a rule-based method instead of or in addition to using a trained model.

[0032] 1, the determination unit 107 is configured within the information processing device 101, but it may also be configured within the detection device 100. In this case, the determination result of the determination unit 107 can be directly output to the user from the device output unit 105. Furthermore, the determination unit 107 and the estimation unit 109 may be integrated to perform determination and estimation via a single trained model. In other words, the trained model used by the determination unit 107 (first trained model) and the trained model used by the estimation unit 109 (second trained model) may be the same trained model.

[0033] The system control unit 112 controls the system based on the determination result of the determination unit 107. More specifically, when the determination unit 107 determines that the biometric information detected from one or more parts of the user corresponds to any of the specific movements for system control, the system control unit 112 performs system control according to the movement.

[0034] The "specific movement for system control" (hereinafter also referred to as "specific movement") may be any movement that can be acquired by the detection device 100, and may be, for example, a movement different from the movement made when speaking. An example of the "specific movement" is any one of mouth movement, tongue movement, cheek movement, eye or eyelid movement, and facial movement, or a combination of these. The "specific movement" may also be a movement made when speaking a predetermined keyword.

[0035] "System control" may be any control related to the detection device 100 or the information processing device 101. Examples of "system control" include starting or stopping the conversion of detected biometric information into text information or audio information, and specifying an operating mode when converting detected biometric information into text information or audio information. For example, when the estimation process is not being performed, if it is determined that the biometric information corresponds to a first specific movement, the estimation unit 109 starts estimation. Also, when the estimation process is being performed, if it is determined that the biometric information corresponds to a second specific movement, the estimation unit 109 ends estimation. That is, when it is determined that the biometric information corresponds to a first specific movement, the system control unit 112 starts estimation by the estimation unit 109, and when it is determined that the biometric information corresponds to a second specific movement, the system control unit 112 stops estimation by the estimation unit 109. Thus, the first specific movement signals the start of estimation by the estimation unit 109, and the second specific movement signals the end of estimation by the estimation unit 109. The user can control estimation by the estimation unit 109 by performing either the first specific movement or the second specific movement.

[0036] Alternatively, the system is controlled to switch to an operating mode corresponding to the movement or to launch an application corresponding to the movement depending on which of a plurality of predetermined specific movements the biometric information corresponds to.

[0037] For example, the system control unit 112 may start displaying text information on the display unit 110 when it is determined that the biometric information corresponds to a third specific movement, and may stop displaying the text information on the display unit 110 when it is determined that the biometric information corresponds to a fourth specific movement. Furthermore, the system control unit 112 may start outputting audio information on the sound information output unit 111 when it is determined that the biometric information corresponds to a fifth specific movement. Furthermore, the system control unit 112 may stop outputting audio information on the sound information output unit 111 when it is determined that the biometric information corresponds to a sixth specific movement. In other words, the system control unit 112 can control components such as the estimation unit 109, the display unit 110, and the sound information output unit 111, i.e., the system, depending on the type of movement based on the biometric information.

[0038] The information processing device 101 may also be configured on a cloud. In this case, the first transmission unit 103 in the detection device 100 transmits the biometric information detected by the biometric information detection unit 102 to the cloud. The cloud converts the biometric information into text information or audio information, and transmits the converted text information to the display unit 110. The display unit 110 displays the text information. The converted text information may also be further converted into audio information and transmitted to the sound information output unit 111. The biometric information may also be directly converted into audio information and transmitted to the sound information output unit 111.

[0039] Communication between the detection device 100 and the information processing device 101 may be wired or wireless. In the case of wired communication, the first transmitting unit 103 and the first receiving unit 104 in the detection device 100 and the second receiving unit 106 and the second transmitting unit 108 in the information processing device 101 are connected by a wire such as a USB cable. In the case of wireless communication, the above-mentioned units are connected wirelessly by wireless LAN communication such as Wi-Fi or short-range wireless communication such as Bluetooth (registered trademark).

[0040] If the information processing device 101 is a smartphone or a tablet PC, the display unit 110 is a display. The sound information output unit 111 is a speaker installed in the smartphone or tablet PC, or an earphone connected to the smartphone or tablet PC.

[0041] The information processing device 101 can be regarded as an information conversion device that converts biometric information detected by the biometric information detection unit 102 into text information or audio information. The information processing device 101 can also be regarded as an information estimation device that estimates text information or audio information from the biometric information detected by the biometric information detection unit 102. Furthermore, the information processing device 101 can also include a processing unit that evaluates the biometric information detected by the biometric information detection unit 102 based on predetermined evaluation criteria and deletes text information or audio information corresponding to biometric information that does not satisfy the predetermined evaluation criteria.

[0042] (Processing) FIG. 2 is a flowchart showing the flow of an information processing method executed by the information processing system according to this embodiment.

[0043] Step S100: The user wears the detection device 100 in a position where the biometric information can be detected by the biometric information detection unit 102. The user wears the detection device 100 on the head, neck, or around the head or neck. The biometric information detection unit 102 detects biometric information including acceleration information and angular velocity information, or acceleration information and angular velocity information plus at least one of a myoelectric signal, tactile information, etc.

[0044] Step S101: The first transmission unit 103 transmits biometric information to the information processing device 101. The biometric information is provided with additional information related to time information (time information). The first transmission unit 103 can transmit the biometric information and the time information to the information processing device 101.

[0045] Step S102: The biological information detected by the detection device 100 is received by the second receiving unit 106 of the information processing device 101. The second receiving unit 106 transmits the received biological information to the determination unit 107.

[0046] Step S103: The determination unit 107 determines whether the biometric information corresponds to a specific movement (including not moving the mouth for a certain period of time) using a predetermined determination algorithm. If it is determined that the biometric information corresponds to a movement corresponding to the start of conversion, the system control unit 112 sends a signal via the second transmission unit 108 to the device output unit 105 via the first receiving unit 104 of the detection device 100, and the device output unit 105 notifies the user of the start of conversion by sound. The system control unit 112 also controls the estimation unit 109 to start estimation of the biometric information acquired thereafter. Furthermore, if the determination unit 107 determines that the biometric information indicates that the mouth has not moved for a certain period of time after the start of estimation, the system control unit 112 controls the estimation unit 109 to end estimation.

[0047] Step S104: The estimation unit 109 estimates character information, voice information, or command information from the biometric information using a predetermined estimation algorithm. The estimation unit 109 may also estimate voice information from the biometric information, or may estimate character information and then convert it into voice information by performing voice synthesis. The estimation unit 109 outputs the information estimated from the biometric information.

[0048] Step S105: The display unit 110 displays the character information estimated by the estimation unit 109. The sound information output unit 111 outputs sound information obtained by further converting the character information estimated by the estimation unit 109.

[0049] Furthermore, the character information estimated by the estimation unit 109 can be stored in a storage unit of the information processing device 101. The character information can also be transferred via a network and displayed on an external terminal.

[0050] Furthermore, by converting text information into audio information, it is possible to play and record it on an external device, and the user can listen to the played audio through earphones. This allows the user to confirm whether the estimation is correct. Furthermore, the audio information can be transferred via a network and played back and recorded on another external device. By repeating this series of steps, continuous communication using biometric information becomes possible.

[0051] The information processing system according to this embodiment allows the user to control the system hands-free by moving their mouth, face, cheek, etc., which is highly convenient for the user. Furthermore, the information processing system according to this embodiment uses biometric information used for voice recognition to determine specific movements for system control, eliminating the need for a separate biometric information detection device for system control. This eliminates the need for the user to use an additional device, improving convenience and reducing product costs.

[0052] (Example 1) Next, a description will be given of an information processing system according to Example 1. The information processing system according to this example converts sensing data (biometric information) obtained by an acceleration sensor and an angular velocity sensor attached to a user's cheek and below the ear into text information or voice information.

[0053] Fig. 3 shows an outline of an information processing system according to this embodiment, and Fig. 4 shows a schematic diagram of a detection device 100 according to this embodiment. Here, an information processing device of the information processing system is a smartphone 300. The detection device 100 is powered by a battery (not shown).

[0054] As shown in Fig. 4, the user wears the detection device 100 around the neck. The detection device 100 is a neckband type that is worn by pinching the user's neck. The detection device 100 is composed of a first biological information detection unit 401, a second biological information detection unit 402, a device transmission unit 403, a device reception unit 404, and an output unit 405. The first biological information detection unit 401 and the second biological information detection unit 402 are connected to the device transmission unit 403 by wire. The device transmission unit 403 can transmit the biological information detected by the first biological information detection unit 401 and the second biological information detection unit 402 to an external device.

[0055] Both the first biological information detection unit 401 and the second biological information detection unit 402 are six-axis acceleration / angular velocity sensors that can measure, for example, three-axis translational acceleration and three-axis angular acceleration.

[0056] The first biometric information detection unit 401 and the second biometric information detection unit 402 are adhered to the user's cheek and under the ear, respectively, using, for example, a self-adhesive gel. This allows the first biometric information detection unit 401 and the second biometric information detection unit 402 to detect biometric information (acceleration, angular velocity, etc.). This allows the first biometric information detection unit 401 and the second biometric information detection unit 402 to detect information related to the movement of the user's body parts.

[0057] The biological information detected by the first biological information detection unit 401 and the second biological information detection unit 402 is transmitted to, for example, a Wi-Fi-connected smartphone 300 by a device transmission unit 403. A receiving unit 301 in an application of the smartphone 300 receives the data.

[0058] The determination unit 302 determines, using a trained model, whether the data received by the receiving unit 301 is biometric information corresponding to a specific movement or the absence of mouth movement for one second. An example of the specific movement is closing the eyes continuously for 0.5 seconds or more. If the determination unit 302 determines that the received data is biometric information corresponding to the movement of closing the eyes for 0.5 seconds or more, the determination unit 302 sends the information to the device receiving unit 404 via the transmission unit 304. The output unit 405 signals to the user by sound that the subsequent biometric information will be converted. After hearing the sound, the user can estimate the biometric information by performing a silent movement corresponding to the content they want to convert. Furthermore, if the determination unit 302 determines that the received data shows no mouth movement for one second or more after starting estimation, it determines that data acquisition for estimation is complete and does not transmit the biometric information to the estimation unit 303 until the next estimation start determination is made.

[0059] The determination process in the determination unit 302 is based on a trained model. To generate the trained model in this embodiment, biometric information for 20 pre-set movements, such as movements with eyes closed, was acquired from five users for 0.5 seconds or more, 30 times for each movement. Here, 80% of the total data was used as training data, and the remaining 20% ​​was used as evaluation data. For training, a neural network was trained for 500 epochs using the training data to assign serial numbers to each movement as correct labels, and a trained model was created.

[0060] The estimation unit 303 inputs the biometric information received by the receiving unit 301 into a trained model and estimates corresponding character information or voice information. Specifically, the character information or voice information is estimated by using sensor information from a total of 12 axes, consisting of 6-axis x 2 acceleration / angular velocity sensors, as input data to the trained model.

[0061] When the determination unit 302 determines that the biometric information detected from one or more parts of the user corresponds to one of the specific movements for system control, the system control unit 307 performs system control according to the movement.

[0062] To generate the trained model in this embodiment, biometric information was obtained from 5,000 sentences from five users during silent movements. Here, 80% of the total data was used as training data, and the remaining 20% ​​was used as evaluation data. The training data was used to convert sentences into phonemes, which were then used as correct labels. The neural network was trained for 2,000 epochs to create the trained model. The estimation unit 303 has a function to estimate character information, including a mixture of kanji and hiragana, from the estimated phoneme sequence. It also has a function to estimate speech information from the phoneme sequence.

[0063] The character information estimated (converted) by the estimation unit 303 is displayed on a display, which is the smartphone's display unit 305. The sound information output unit 306 reproduces the sound information estimated (converted) by the estimation unit 303.

[0064] (Example 2) Next, a description will be given of an information processing system according to Example 2. The information processing system according to this example converts sensing data (biometric information) obtained by an acceleration sensor, an angular velocity sensor, and an electromyography sensor attached to the cheek and lower jaw of a user into text information or audio information.

[0065] Fig. 5 shows an overview of an information processing system according to this embodiment, and Fig. 6 shows a schematic diagram of a detection device 100 according to this embodiment. Here, an embodiment is shown in which the information processing device of the information processing system is a smartphone 500. The detection device is powered by a battery (not shown).

[0066] As shown in FIG. 6 , the user wears the detection device 100 around the neck and ears. The detection device 100 is composed of a neckband-type portion worn by pinching the user's neck and an earphone-type portion. The neckband-type portion and the earphone-type portion are connected via a wire. The detection device 100 may be provided with a connecting portion that connects both ends of an elastic member so that it can be worn around the user's neck. The detection device 100 is composed of a first biometric information detection unit 601, a second biometric information detection unit 602, a transceiver unit 603, and a sound output unit 604. The first biometric information detection unit 601, the second biometric information detection unit 602, and the sound output unit 604 are connected to the transceiver unit 603. The transceiver unit 603 can transmit the biometric information detected by the first biometric information detection unit 601 and the second biometric information detection unit 602 to an external device. Furthermore, the transceiver unit 603 can receive information determined by the determination unit 502 and information converted by the estimation unit 506 via the transmitter unit 505.

[0067] The first biological information detection unit 601 and the second biological information detection unit 602 each integrate a six-axis acceleration sensor, an angular velocity sensor, and a myoelectric potential sensor. The six-axis acceleration sensor and angular velocity sensor are sensors that can measure, for example, three-axis translational acceleration and three-axis angular acceleration. Tripolar Ag electrodes are used as the electrodes of the myoelectric potential sensor. In other words, the first biological information detection unit 601 and the second biological information detection unit 602 each are composed of a plurality of different sensors.

[0068] The first biometric information detection unit 601 and the second biometric information detection unit 602 are pressed against the user's cheek and chin, respectively, by support members extending from the sound output unit 604. To further improve wearability, the first biometric information detection unit 601 and the second biometric information detection unit 602 may be adhered with a self-adhesive gel, for example. In this way, the first biometric information detection unit 601 and the second biometric information detection unit 602 can detect biometric information (various types of acceleration and myoelectric potential). This allows the first biometric information detection unit 601 and the second biometric information detection unit 602 to detect information related to the movement of the user's body parts.

[0069] The biological information detected by the biological information detection units 601 and 602 is transmitted to, for example, the Wi-Fi connected smartphone 500 by the transmission / reception unit 603. The reception unit 501 in the application of the smartphone 500 receives the data.

[0070] The determination unit 502 determines, using a trained model, whether the biometric information received by the receiving unit 501 corresponds to a specific movement or to the absence of mouth movement for one second. An example of a specific movement is a nodding movement, i.e., a movement in which the user nods their head, or a movement in which the user closes their eyes for 0.5 seconds or more. If the determination unit 502 determines that the user has nodded, the smartphone 500 transmits the determination result from the transmission unit 505 to the transceiver unit 603, and the detection device 100 uses the sound output unit 604 to notify the user of the start of command input mode by sound. If the determination unit 502 determines that the user has closed their eyes for 0.5 seconds or more, the smartphone 500 transmits the determination result from the transmission unit 505 to the transceiver unit 603, and the detection device 100 uses the sound output unit 604 to notify the user of the start of direct conversion mode by sound. The movements related to these mode determinations are pre-set movements, but other movements may also be used. The determination unit 502 determines the subsequent conversion mode based on the determination results. Furthermore, if the determination unit 502 determines that the mouth has not moved for one second or more in the received data after the start of conversion, it determines that data acquisition for conversion has been completed, and does not transmit the biometric information to the estimation unit 506 until a determination is made to start the next conversion.

[0071] The determination process in the determination unit 502 is based on a trained model. To generate the trained model in this embodiment, biometric information for 20 pre-defined movements, such as nodding, was acquired 30 times for each movement from five users. Here, 80% of the total data was used as training data, and the remaining 20% ​​was used as evaluation data. For training, a neural network was trained for 500 epochs using the training data to assign serial numbers to each movement as correct labels, and a trained model was created.

[0072] When the determination unit 502 determines that the biometric information detected from one or more parts of the user corresponds to one of the specific movements for system control, the system control unit 507 performs system control according to the movement.

[0073] The estimation unit 506 inputs the biometric information received by the receiving unit 501 into the trained model and estimates corresponding character information or voice information. For example, when the operation mode is the direct conversion mode, the estimation unit 506 estimates character information or voice information corresponding to the content of the user's utterance from the received biometric information.

[0074] To generate the trained model in this example, biometric information was collected from five users during silent movements for 5,000 sentences. 80% of the data was used as training data, and the remaining 20% ​​was used as evaluation data. Training was performed on the neural network using the training data, converting sentences into phonemes and using these as correct labels, for 2,000 epochs to create the trained model.

[0075] In the direct conversion mode, the content estimated by the estimation unit 506 is converted from phoneme information to character information and displayed on the display, which is the display unit 503 of the smartphone. The sound information output unit 504 reproduces voice information from the phoneme information estimated by the estimation unit 506. It is also possible for the voice information estimated by the estimation unit 506 to be transmitted to the device by the transmission unit 505, received by the device's transmission / reception unit 603, and reproduced by the sound output unit 604. On the other hand, in the command input mode, the estimated content becomes a control command for the smartphone. For example, if the estimated content is "display email," the content of a new email stored in the smartphone is displayed on the display unit 503. If the estimated content is "read email," the content of a new email stored in the smartphone is synthesized into voice and output from the sound information output unit 504 or the sound output unit 604.

[0076] Next, an information processing system according to Example 3 will be described. The information processing system according to Example 3 converts sensing data (biometric information) obtained by an acceleration sensor, an angular velocity sensor, and a contact sensor attached to the cheek and lower jaw of a user into text information or audio information.

[0077] FIG. 7 shows an outline of an information processing system according to this embodiment, and FIG. 8 shows a schematic diagram of a detection device 100. As shown in FIG.

[0078] The detection device 100 is a headset type device worn on the user's ears. The detection device 100 is composed of a first biological information detection unit 701, a second biological information detection unit 702, a transmission / reception unit 703, and a sound information output unit 704. The first biological information detection unit 701 and the second biological information detection unit 702 are each composed of a contact-type sensor unit.

[0079] FIG. 9 is a diagram showing the configuration of a contact-type sensor unit 900 used in this embodiment.

[0080] The sensor unit 900 is composed of a circuit board 903 on which a magnetic sensor IC 901 and an inertial sensor IC 902 are mounted, an elastic member 904, a permanent magnet 905, and a contact member 906. The inertial sensor IC 902 has a built-in detection unit for detecting three-axis acceleration and three-axis angular velocity. The contact member 906 is a member that comes into contact with the skin.

[0081] An elastic member 904 is disposed between the circuit board 903 and the contact member 906. The elastic member 904 is deformable by an external force applied between the contact member 906 and the circuit board 907. The elastic member 904 may be made of a flexible material such as silicon gel, synthetic rubber, or sponge, or may be a metal or resin spiral spring.

[0082] A permanent magnet 905 is fixed to the contact member 906 side of the elastic member 904. The magnetic sensor IC 901 detects the relative displacement between the permanent magnet 905 and the magnetic sensor IC 901. The magnetic sensor IC 901 is an element having four magnetic field detection points inside. Alternatively, the magnetic sensor IC 901 may be a three-axis geomagnetic sensor capable of detecting the direction of a magnetic field vector. The sensor unit 900 uses the output from the magnetic sensor IC 901 to detect three-dimensional displacement of the magnet using a downstream signal processing circuit (not shown), and calculates the external force based on the detected amount of displacement.

[0083] 10 is a diagram showing the configuration of another sensor unit 1000 used in this embodiment. The sensor unit 1000 is an optical tactile sensor.

[0084] The sensor unit 1000 comprises a circuit board 1010 on which four-segment photodiodes 1007a and 1007b and an inertial sensor IC 1002 are mounted, an elastic member 1004, a light-emitting element 1009, and a contact member 1006. The inertial sensor IC 1002 incorporates a detection unit for detecting three-axis acceleration and three-axis angular velocity. The contact member 1006 is a member that comes into contact with the skin.

[0085] An opening member 1011 and an elastic member 1004 are disposed between the circuit board 1010 and the contact member 1006. The elastic member 1004 is deformable by an external force applied between the contact member 1006 and the circuit board 1010.

[0086] A light-emitting element 1009 consisting of an LED is fixed to the contact member 1006 side of the elastic member 1004. The four-segment photodiodes 1007a and 1007b are light-receiving elements each having a light-receiving surface divided into four parts vertically and horizontally. The aperture member 1011 has openings 1030a and 1030b in the optical path between the light-emitting element 1009 and the four-segment photodiodes 1007a and 1007b. The sensor unit 1000 uses the outputs from the four-segment photodiodes 1007a and 1007b to detect three-dimensional displacement of the light-emitting element 1009 using a downstream signal processing circuit (not shown), and calculates external force from the detected displacement amount.

[0087] Referring to FIG. 11 , a procedure for detecting the amount of movement of the light-emitting element 1009 due to an external force and calculating the external force will be described. FIG. 11 is a schematic diagram showing the movement of the light-emitting element 1009 and the movement of the projected spot 1040 when a vertical external force is applied. The distance between the spot pairs formed by one light-emitting element 1009 changes depending on the ratio between the distance between the light-emitting element 1009 and the openings 1030a and 1030b and the distance between the openings 1030a and 1030b and the light-receiving IC surface. When subjected to a force pushing in the surface direction, the light-emitting element 1009 moves downward, reducing the distance between the light-emitting element 1009 and the openings 1030a and 1030b. As a result, the center-to-center distance between the spot pairs increases. Conversely, when subjected to a force pulling in the surface direction, the light-emitting element 1009 moves upward, increasing the distance between the light-emitting element 1009 and the openings 1030a and 1030b. As a result, the center-to-center distance between the spot pairs decreases.

[0088] The sensor unit 1000 can estimate the external force in the vertical direction by calculating the distance between the center position coordinates of the spot pair.

[0089] 12 is a schematic diagram showing the movement of the light-emitting element 1009 and the movement of the projected spot 1040 when an external force in the shear direction is applied. When the elastic member 1004 is subjected to an external force in the shear direction via the abutting member 1006, the light-emitting element 1009 moves in the in-plane direction. As a result, the center of the center position of the spot pair moves in the direction opposite to the external force. The external force in the shear direction can be estimated by calculating the average value of the coordinates of the center position of the spot pair. Because the light-receiving portions of the segmented photodiodes 1007a and 1007b are divided into four, the sensor unit 1000 can estimate two-dimensional external forces in the in-plane direction.

[0090] The first biometric information detection unit 701 and the second biometric information detection unit 702 are pressed against the cheek and under the chin of the user. By bringing the first biometric information detection unit 701 and the second biometric information detection unit 702 into contact with the user's skin, the first biometric information detection unit 701 and the second biometric information detection unit 702 can detect biometric information.

[0091] The transmitting / receiving unit 703 is connected to the first biological information detection unit 701 and the second biological information detection unit 702. The transmitting / receiving unit 703 can transmit the biological information detected by the first biological information detection unit 701 and the second biological information detection unit 702 to the outside. The transmitting / receiving unit 703 can also acquire information from a smartphone 805. The biological information detected by the first biological information detection unit 701 and the second biological information detection unit 702 is transferred wirelessly to the smartphone 805 by the transmitting / receiving unit 703.

[0092] The receiving unit 806 in the application of the smartphone 805 receives the biometric information detected by the first biometric information detection unit 701 and the second biometric information detection unit 702. The determining unit 812 determines, based on the trained model, whether the received biometric information corresponds to a specific action for controlling the smartphone. For example, if the determining unit 812 determines that the biometric information corresponds to a nodding action, the smartphone 805 transmits the determination result to the detection device 100 via the data transferring unit 807 and the transmitting / receiving unit 703. The sound information output unit 704 notifies the user that estimation of biometric information has begun by emitting a sound, and the data transferring unit 807 transfers the biometric information to the cloud 811. The estimating unit 808 on the cloud 811 begins estimating voice information from the biometric information using the trained model. The estimating unit 808 continues the estimation until the determining unit 812 determines that the user's mouth has not moved for one second or more. If the determining unit 812 notifies the estimating unit 808 that the mouth has not moved for one second or more, the estimating unit 808 terminates the estimation process.

[0093] When the determination unit 812 determines that the biometric information detected from one or more parts of the user corresponds to one of the specific movements for system control, the system control unit 813 performs system control according to the movement.

[0094] In this embodiment, the trained model used to determine predetermined movements for system control and to estimate voice information or text information from biometric information is the same. The trained model was generated as follows. First, biometric information was obtained for five users during silent movements for 5,000 sentences and 20 movements. 80% of the total data was used as training data, and the remaining 20% ​​was used as evaluation data. For training, the neural network was trained for 2,000 epochs using the training data to create the trained model.

[0095] The voice information converted by the estimation unit 808 is transferred to the data transfer unit 807. The voice information converted by the estimation unit 808 is transferred to another smartphone 810 and played back. At the same time, the converted voice is transferred to the transmitting / receiving unit via the data transfer unit 807. The transferred voice is played back via the sound information output unit 704, such as an earphone, and the user can check the conversion result.

[0096] Other Embodiments The above embodiment is merely one example of the information processing system according to the present disclosure, and appropriate modifications are possible within the scope of the idea of ​​the present disclosure.

[0097] For example, in the above explanation, examples of detection devices are neckband type, a combination of neckband type and earphone type, and headset type, but the detection device may also be glasses type, mask type, chin mask type, choker type, ring type, etc. In the case of a ring type device, the device is pressed against a part of the body such as the face when inputting information to the system.

[0098] Furthermore, the biometric information used to infer text information or voice information and to determine input for system control is not limited to the specific examples described above. The information processing according to the present disclosure may utilize any biometric information that can be acquired from a part of the user's body. As an example, biometric information related to the movement of the arms, legs, chest, or abdomen may be acquired. Furthermore, biometric information may be detected using a sensor other than a permanent sensor. For example, an image may be captured using a camera, and the movement of a part of the user's body obtained through image analysis may be used as biometric information.

[0099] Furthermore, although an example of an information processing system consisting of a detection device and an information processing device has been given, the information processing system may also be configured to include other devices, or may be an integrated system in which the detection device and the information processing device are included in a single device.

[0100] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0101] The processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, a central processing unit (CPU), a micro processing unit (MPU), and a digital signal processor (DSP). Dedicated processors include, for example, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), and a programmable logic device (PLD). Examples of programmable logic devices include a field programmable gate array (FPGA) and a complex programmable logic device (CPLD).

[0102] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0103] The present invention can also be realized by a process in which a program that realizes one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit that realizes one or more of the functions.

[0104] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to apprise the public of the scope of the present invention.

[0105] This application claims priority based on Japanese Patent Application No. 2023-174025, filed on October 6, 2023, the entire contents of which are incorporated herein by reference.

[0106] 102: Biometric information detection unit 107: Determination unit 109: Estimation unit 112: System control unit

Claims

1. An information processing system comprising: a biometric information detection unit that detects biometric information from one or more locations on a user; an estimation unit that estimates text information or voice information from the biometric information detected by the biometric information detection unit; a determination unit that determines whether the biometric information corresponds to a specific movement for controlling a system; and a control unit that controls the system based on a determination result from the determination unit.

2. The information processing system of claim 1, characterized in that the control unit controls the system so that, when it is determined that the biometric information corresponds to a first specific movement, the estimation unit begins estimating text information or audio information based on the biometric information.

3. The information processing system of claim 1 or 2, characterized in that the control unit controls the system to terminate the estimation of text information or audio information based on the biometric information by the estimation unit when it is determined that the biometric information corresponds to a second specific movement.

4. An information processing system as claimed in any one of claims 1 to 3, characterized in that the control unit controls the system to switch operating modes depending on which of a plurality of specific movements the biometric information corresponds to.

5. An information processing system as claimed in any one of claims 1 to 4, characterized in that the control unit controls the system to launch an application depending on which of a plurality of specific movements the biometric information corresponds to.

6. An information processing system according to any one of claims 1 to 5, characterized in that the specific movement is at least one of the user's facial movement, cheek movement, tongue movement, eyeball or eyelid movement.

7. An information processing system as claimed in any one of claims 1 to 6, characterized in that the biometric information detection unit detects the biometric information for estimating voice information or text information and the biometric information for determining whether or not it corresponds to the specific movement using a same sensor.

8. The information processing system according to any one of claims 1 to 7, characterized in that the biological information detection unit includes at least one of an acceleration sensor or an angular velocity sensor, and a myoelectric potential sensor.

9. The information processing system according to any one of claims 1 to 7, characterized in that the biometric information detection unit includes at least one of an acceleration sensor or an angular velocity sensor, and a tactile sensor.

10. An information processing system according to any one of claims 1 to 9, characterized in that the biometric information detection unit detects biometric information from at least two of the user's lower jaw, cheeks, throat, under the ears, neck, and temples.

11. An information processing system as claimed in any one of claims 1 to 10, characterized in that the control unit starts displaying text information on the display unit when it is determined that the biometric information corresponds to a third specific movement, and ends displaying the text information on the display unit when it is determined that the biometric information corresponds to a fourth specific movement.

12. An information processing system as described in any one of claims 1 to 11, characterized in that the control unit starts outputting audio information by the sound information output unit when it is determined that the biometric information corresponds to a fifth specific movement, and ends outputting audio information by the sound information output unit when it is determined that the biometric information corresponds to a sixth specific movement.

13. An information processing system as described in any one of claims 1 to 12, characterized in that the estimation unit estimates the character information or the voice information from the biometric information using a first learned model, and the determination unit determines whether the biometric information corresponds to the specific movement using a second learned model.

14. The information processing system according to claim 13, wherein the first trained model and the second trained model are the same trained model.

15. An information processing system as described in any one of claims 1 to 14, characterized in that the biometric information detection unit is included in a device wearable by the user, and the estimation unit, the judgment unit, and the control unit are included in an information processing device configured to be able to communicate with the device.

16. The information processing system according to claim 15, characterized in that the device comprises an output unit that outputs the character information or audio information estimated by the estimation unit.

17. An information processing system according to any one of claims 1 to 16, characterized in that the control unit controls the system depending on the type of movement based on the biometric information.

18. An information processing method including: a biometric information detection step of detecting biometric information from one or more locations of a user; an estimation step of estimating text information or voice information from the biometric information detected in the biometric information detection step; a determination step of determining whether the biometric information corresponds to a specific movement for controlling a system; and a control step of controlling the system based on the determination result of the determination step.

19. A program for causing a computer to execute each step of the information processing method according to claim 18.

Images