Information processing device, information processing method, and program

Abstract

Provided are a device and a method with which it is possible to confirm whether a device capable of input processing based on a user operation correctly recognizes the input value based on the user operation. The present invention has a data processing unit that identifies an input value on the basis of a user operation. The data processing unit further generates and displays user operation recognition state information indicating a recognition state of the user operation. The user operation recognition state information represents data that is generated by converting a sensor detection value corresponding to the user operation into three-dimensional or other forms of visualization data, and that includes an input value recognizable region where the recognition probability of an input value based on the user operation is equal to or greater than a prescribed threshold, a recognition probability of the input value, a user operation point, and the like.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program configured to enable confirmation of whether a user's action, such as a finger movement, is correctly recognized when data is input to the information processing device based on the user's action.

[0002] In recent years, many users have been using various information processing devices, such as PCs, smartphones, game consoles, and AR (Augmented Reality) glasses that enable users to view AR images. AR (Augmented Reality) glasses are glasses that enable users to view augmented reality images that combine real images and virtual images.

[0003] When a user inputs data such as text into these information processing devices, the user uses input devices such as a keyboard and a mouse. Recently, voice input has also become popular. Furthermore, devices are also being used that use sensors to detect user movements, such as the movement of the user's fingers, and use this sensor detection information to input data such as text.

[0004] Incidentally, as a conventional technique that discloses a configuration for recognizing a user's gesture and inputting data, for example, Patent Document 1 (JP 2016-042394 A) is known.

[0005] However, when trying to input text data, for example, a number of different texts such as a, b, c, etc., based on the user's finger movements, the user must accurately perform the prescribed motion for each character. Specifically, the user must input text by performing precise motions corresponding to each character, such as inputting the text "a" by bending the thumb and the text "b" by bending the index finger.

[0006] When inputting text using such delicate operations, users may be unsure whether the text has been input as intended, and they must check the input results by looking at the input character string displayed on the display unit of the information processing device.

[0007] If an incorrect character is detected during this verification process, the user is forced to redo the input, but there is a problem in that it is not possible to know exactly how to correct the user's finger movements to ensure accurate input.

[0008] JP 2016-042394 A

[0009] The present disclosure has been made in consideration of, for example, the above-mentioned problems, and provides an information processing device, an information processing method, and a program configured to enable confirmation of the recognition state of a user's action in the information processing device when performing input processing on the information processing device based on user action such as the movement of the user's fingers.

[0010] A first aspect of the present disclosure is an information processing device having a data processing unit that performs an input value identification process based on a user action, wherein the data processing unit generates user action recognition status information indicating a recognition status of the user action and outputs the information to a display unit.

[0011] Furthermore, a second aspect of the present disclosure is an information processing method executed in an information processing device, in which a data processing unit executes a process of identifying an input value based on a user action, and a process of generating user action recognition status information indicating the recognition status of the user action and outputting it to a display unit.

[0012] Furthermore, a third aspect of the present disclosure is a program for causing an information processing device to execute information processing, the program causing a data processing unit to execute a process of identifying an input value based on a user action, and a process of generating user action recognition status information indicating the recognition status of the user action and outputting it to a display unit.

[0013] The program of the present disclosure is, for example, a program that can be provided in a computer-readable format via a storage medium or a communication medium to an information processing device or a computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or the computer system.

[0014] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present disclosure and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are housed in the same housing.

[0015] According to an embodiment of the present disclosure, a device and method are realized that enable a device capable of input processing based on user motion to confirm whether the device correctly recognizes an input value based on the user motion. Specifically, for example, the device includes a data processing unit that identifies an input value based on the user motion. The data processing unit further generates and displays user motion recognition status information indicating a recognition status of the user motion. The user motion recognition status information is data generated by converting sensor detection values ​​corresponding to the user motion into visualized data such as three-dimensional data, and includes an input value recognition region where the recognition probability of the input value based on the user motion is equal to or greater than a specified threshold, the recognition probability of the input value, the user motion point, etc. This configuration realizes a device and method that enables a device capable of input processing based on user motion to confirm whether the device correctly recognizes an input value based on the user motion. Note that the effects described in this specification are merely exemplary and are not limiting, and additional effects may also be present.

[0016] 1 is a diagram illustrating an overview of data input processing into an information processing device based on a user's action. FIG. 1 is a diagram illustrating an example of an information processing device capable of inputting data based on a user's action. FIG. 2 is a diagram illustrating an example of inputting data into an information processing device using sensor detection values ​​using an electromyographic sensor. FIG. 3 is a diagram illustrating an example of processing for detecting finger movement and inputting data into an information processing device. FIG. 4 is a diagram illustrating the configuration and processing of a smartphone as an example of an information processing device of the present disclosure. FIG. 5 is a diagram illustrating an example of "user action recognition status information A (three-dimensional data, etc.)" displayed on a display unit of a smartphone. FIG. 6 is a diagram illustrating details of data constituting "user action recognition status information A (three-dimensional data, etc.)." FIG. 7 is a diagram illustrating details of data constituting "user action recognition status information A (three-dimensional data, etc.)." FIG. 8 is a diagram illustrating an example of "user action recognition status information B (user body part corresponding data) 90b." FIG. 9 is a diagram illustrating that each character, A, B, C, ..., can be input by various actions of five fingers. FIG. 10 is a diagram illustrating details of "user action recognition status information B (user body part corresponding data)." FIG. 11 is a diagram illustrating an example in which "user action recognition status information A (three-dimensional data, etc.)" and "user action recognition status information B (user body part corresponding data)" are displayed side by side. FIG. 12 is a diagram illustrating a specific example of training processing using user action recognition status information. FIG. 1 is a diagram illustrating a specific example of training processing using user action recognition state information. FIG. 2 is a diagram illustrating a specific example of training processing using user action recognition state information. FIG. 3 is a diagram illustrating a specific example of training processing using user action recognition state information. FIG. 4 is a diagram illustrating a specific example of training processing using user action recognition state information. FIG. 5 is a diagram illustrating a specific example of training processing using user action recognition state information. FIG. 6 is a diagram illustrating a specific example of training processing using user action recognition state information. FIG. 7 is a diagram illustrating a flowchart illustrating a processing sequence when a user executes data input processing by user action into an information processing device such as a smartphone. FIG. 8 is a diagram illustrating a detailed sequence of a setup process for registering user action recognition state information A (three-dimensional data, etc.) unique to the user.1 is a diagram illustrating details of a process for calculating an "input value recognizable area" taking into account fluctuations in a user's motion and displaying it on a three-dimensional coordinate system. FIG. 2 is a diagram illustrating details of a process for calculating an "input value recognizable area" taking into account fluctuations in a user's motion and displaying it on a three-dimensional coordinate system. FIG. 3 is a diagram illustrating details of a process for calculating an "input value recognizable area" taking into account fluctuations in a user's motion and displaying it on a three-dimensional coordinate system. FIG. 4 is a diagram illustrating details of a process for calculating an "input value recognizable area" taking into account fluctuations in a user's motion and displaying it on a three-dimensional coordinate system. FIG. 5 is a diagram illustrating an example of redoing the registration process when there are many overlapping areas in the input value recognizable area so as to reduce the overlapping areas. FIG. 6 is a diagram illustrating a display example of user action recognition status information A (three-dimensional data, etc.). FIG. 7 is a diagram illustrating a display example of user action recognition status information A (three-dimensional data, etc.) and a display example of guide information. FIG. 8 is a diagram illustrating an example of display control of user action recognition status information A (three-dimensional data, etc.). FIG. 9 is a diagram illustrating an example of display control of user action recognition status information A (three-dimensional data, etc.). 1 is a diagram illustrating an example of a display of user action recognition status information when data is input to an information processing device based on a user's hand or arm movement, for example, a movement of slightly raising a hand; FIG. 2 is a diagram illustrating an example of a user action when data is input to an information processing device based on a user's hand or arm movement; FIG. 3 is a diagram illustrating an example of a process using a camera when data is input to an information processing device based on a user's hand or arm movement; FIG. 4 is a diagram illustrating an example of a process using an information processing device such as a smartphone or tablet terminal having a touch panel function; FIG. 5 is a diagram illustrating an example of a process using an information processing device such as a smartphone or tablet terminal having a touch panel function; FIG. 6 is a diagram illustrating an example of a process using an information processing device such as a smartphone or tablet terminal having a touch panel function; and FIG. 7 is a diagram illustrating an example of a hardware configuration of an information processing device of the present disclosure.

[0017] The information processing device, information processing method, and program of the present disclosure will be described in detail below with reference to the drawings. The description will be made according to the following items: 1. Overview of data input processing for an information processing device based on user actions 2. Configuration and processing of the information processing device of the present disclosure 3. Training processing for input processing based on user actions 4. Setup processing for generating user-specific user action recognition status information 5. Display example of user action recognition status information when data is input based on user actions after setup 6. Display example of user action recognition status information corresponding to data input processing based on various user actions 7. Example hardware configuration of an information processing device 8. Summary of the configuration of the present disclosure

[0018] [1. Overview of Data Input Processing to Information Processing Device Based on User Action] First, an overview of data input processing to an information processing device based on user action will be described with reference to FIG. 1 and subsequent figures.

[0019] 1 is a diagram illustrating an example in which a user 10 inputs data into a smartphone 20, which is an information processing device, based on a user action. The example shown in FIG. 1 uses the movement of the fingers of the user 10 as the user action.

[0020] For example, as shown in FIG. 1, various texts can be input by moving the fingers of the user 10's hand as follows: By bending the thumb, text [A] can be input into the smartphone 20. By bending the thumb, text [B] can be input into the smartphone 20. By bending the index finger, text [C] can be input into the smartphone 20. By bending the index finger, text [D] can be input into the smartphone 20. Characters other than A to D shown in FIG. 1 can also be input by other finger movements.

[0021] 1 shows a smartphone 20 as an example of an information processing device, but such information processing devices capable of inputting data based on user actions are not limited to smartphones. As shown in FIG. 2, in addition to the smartphone 20, a PC 30, AR glasses 40, and even a tablet terminal can also be used as an information processing device capable of inputting data based on user actions.

[0022] AR (Augmented Reality) glasses are glasses that allow users to observe augmented reality images that combine real images and virtual images, and are classified into transmissive AR glasses and non-transmissive AR glasses.

[0023] When performing data input processing on an information processing device based on the movement of a user's finger, input values ​​from a sensor that can identify user actions are used. The sensor can be of various types, such as a camera, an acceleration sensor, a gyro sensor, or an electromyographic sensor.

[0024] An example of inputting data into an information processing device using the sensor detection values ​​of the electromyographic sensors is shown in Fig. 3. Fig. 3 is a diagram showing an example in which six electromyographic sensors 60 are attached to the arm portion of a user's hand 50.

[0025] The electromyographic sensor detects weak electrical signals generated in the arm muscles in response to finger movements. By attaching six electromyographic sensors 60 to different parts of the arm of the user's hand 50 and analyzing the electrical signals at each part, it is possible to analyze the movement (bending, bending, pinching, etc.) of each finger of the user's hand (thumb, index finger, middle finger, ring finger, little finger).

[0026] 3, the outputs of the six electromyography sensors 60 are input to the smartphone 20, which is the information processing device of the present disclosure, via a sensor detection value input unit 61. The data processing unit inside the smartphone 20 is configured with an electromyography-input value conversion unit 62 as shown in the figure. The electromyography-input value conversion unit 62 analyzes the six sensor detection values ​​of the six electromyography sensors 60, determines the movement of the user's fingers, and executes a process of identifying the input value.

[0027] As shown in the figure, the following user input value determination process is performed based on the analysis results of the six sensor detection values ​​of the six electromyography sensors 60. (a) If it is determined that a thumb bending motion has been performed, it is determined that the user input value = [A]. (b) If it is determined that a thumb bending motion has been performed, it is determined that the user input value = [B]. (c) If it is determined that an index finger bending motion has been performed, it is determined that the user input value = [C]. (d) If it is determined that an index finger bending motion has been performed, it is determined that the user input value = [D]. For characters other than A to D shown in FIG. 3, input value determination is performed based on the analysis results of the motions of the other fingers.

[0028] In this way, when inputting a large number of different characters such as A, B, C, etc., with the movement of the user's finger, the user needs to accurately perform predetermined actions corresponding to each character. However, it may be difficult for the user to accurately perform the predetermined actions corresponding to each character when inputting characters.

[0029] For example, when inputting a long text, if there are multiple input timings for a single character [C], it is difficult to perform the exact same operation for each input timing of the character [C].

[0030] A specific example will be described with reference to Fig. 4. The upper part of Fig. 4 shows a standard motion example (1) corresponding to inputting the character [C]. That is, the standard motion example of bending the index finger is shown.

[0031] The bottom of Figure 4 shows (2) an example of an actual user's input behavior. This example shows a case where, when inputting a long text, multiple input timings for the letter [C] occur, such as t1, t2, t3, t4, etc. It is difficult for a user to perform the exact same behavior for each input timing of the letter [C].

[0032] As a result, for example, as shown in the figure, at time t1, the bending motion of the user's index finger is recognized by the information processing device (e.g., a smartphone) as an input motion for the letter [C], and the letter [C] is successfully input, but at a later time t2, the information processing device (e.g., a smartphone) is unable to recognize the bending motion of the user's index finger as an input motion for the letter [C], resulting in an input error.

[0033] Furthermore, at time t3, the bending motion of the user's index finger is recognized by the information processing device (e.g., a smartphone) as an input motion for the letter [C], and the letter [C] is successfully input, but at a subsequent time t4, the information processing device (e.g., a smartphone) is unable to recognize the bending motion of the user's index finger as an input motion for the letter [C], resulting in an input error.

[0034] When different actions are performed by different users, the six sensor detection values ​​of the six electromyography sensors 60 described above with reference to Figure 3 will be different values, making it difficult for the information processing device (e.g., a smartphone) to determine which character is being input, resulting in an input error.

[0035] When performing input based on such user actions, the user is concerned about whether the text input is as intended. The input result is confirmed by looking at the input character string displayed on the display unit of the information processing device. If an incorrect input character is detected during this confirmation process, the user must redo the input.

[0036] That is, there is a problem that a user cannot confirm whether the text input is as intended by the user while executing a character input process until the input characters are confirmed on the information processing device (e.g., smartphone). The configuration and process of the present disclosure that solve this problem will be described below. For example, while a user is executing text input, the information processing device of the present disclosure executes a process of displaying "user action recognition status information" indicating whether the user action is correctly recognized by the information processing device (e.g., smartphone) on a display unit of the information processing device (e.g., smartphone).

[0037] 2. Configuration and Processing of the Information Processing Apparatus of the Present Disclosure Next, the configuration and processing of the information processing apparatus of the present disclosure will be described.

[0038] 5 shows a smartphone 20 as an example of an information processing device of the present disclosure. As previously described with reference to FIG. 2, the information processing device of the present disclosure is not limited to the smartphone 20, but also includes a PC 30, AR glasses 40, and the like. In other words, various information processing devices capable of inputting data based on user actions are included.

[0039] 5, similar to the example described above with reference to Fig. 3, shows an example in which six myoelectric sensors 60 are attached to the arm portion of a user's hand 50. By attaching the six myoelectric sensors 60 to different parts of the arm of the user's hand 50 and analyzing the electrical signals at each part, the movement (bending, bending, pinching motion, etc.) of each finger of the user's hand (thumb, index finger, middle finger, ring finger, little finger) is analyzed.

[0040] 5, the outputs of the six electromyography sensors 60 are input to the smartphone 20, which is the information processing device of the present disclosure, via a sensor detection value input unit 61. The smartphone 20, which is the information processing device of the present disclosure, includes a electromyography-input value conversion unit 62 as shown in the figure, as well as a user action recognition state information generation unit 63.

[0041] The myoelectric-input value conversion unit 62 analyzes the six sensor detection values ​​of the six myoelectric sensors 60, determines the movement of the user's fingers, and executes a process to identify the input value. The user action recognition state information generation unit 63 executes a process to generate "user action recognition state information" that indicates whether the user action is correctly recognized in the smartphone 20 while the user is executing an input process. In other words, the myoelectric-input value conversion unit 62 of the smartphone 20 correctly recognizes the user action, and generates "user action recognition state information" that indicates whether the user's intended input is being performed.

[0042] The "user action recognition status information" generated by the user action recognition status information generation unit 63 is displayed on the display unit of the smartphone 20, and the user can check whether the user action has been correctly recognized based on this display information.

[0043] As shown in FIG. 5 , the user action recognition state information generation unit 63 includes a user action recognition state visualization data generation unit 71 and a user action recognition probability calculation unit 72. The user action recognition state visualization data generation unit 71 generates user action recognition state visualization data such as three-dimensional data constituting "user action recognition state information" indicating whether a user action has been correctly recognized. In the embodiment described below, an example will be described in which the user action recognition state visualization data generation unit 71 generates user action recognition state visualization data constituted by three-dimensional data. However, the user action recognition state visualization data generated by the user action recognition state visualization data generation unit 71 is not limited to three-dimensional data and may be in other data formats such as two-dimensional data, as long as it is display data that allows the user to check the user action recognition state. The user action recognition probability calculation unit 72 calculates the probability that the user action has been correctly recognized. This probability is also data constituting the "user action recognition state information."

[0044] The user action recognition state visualization data, such as three-dimensional data, generated by the user action recognition state visualization data generation unit 71 of the user action recognition state information generation unit 63 and the probability calculated by the user action recognition probability calculation unit 72 are displayed as "user action recognition state information" on the display unit of the smartphone 20. Based on this display information, the user can check whether the user action has been correctly recognized.

[0045] 6 shows an example of "user action recognition status information A (three-dimensional data, etc.) 90a" displayed on the display unit of the smartphone 20. The user 10 shown in FIG. 6 inputs text to the smartphone 20 using a user action, such as a finger movement. The text input by the user is displayed sequentially in a text input field 91.

[0046] While the user 10 is entering text, user action recognition status information A (e.g., three-dimensional data) 90a is displayed in a partial area of ​​the display unit of the smartphone 20. The data processing unit of the smartphone 20 executes a process of generating user action recognition status information A (e.g., three-dimensional data) 90a and outputting it to the display unit in parallel with the input value identification process based on the user action. By looking at the user action recognition status information A (e.g., three-dimensional data) 90a displayed on the display unit of the smartphone 20, the user 10 can confirm whether the user action is being correctly recognized by the information processing device (smartphone 20).

[0047] The data constituting the "user action recognition state information A (three-dimensional data, etc.) 90a" will be described in detail with reference to Fig. 7. As shown in Fig. 7, the data constituting the user action recognition state information A (three-dimensional data, etc.) 90a includes the following data: (a01) user action point (a02) user action point trajectory data (a11 to a13) input value recognizable area (a21 to a23) input value recognition probability

[0048] The user motion point a01 is three-dimensional position data corresponding to the current state (motion state) of the user's finger. That is, it is a point that represents, as three-dimensional data, the sensor detection value detected by the electromyographic sensor 60 based on the motion of the user 10, i.e., the motion of the finger.

[0049] In the example described above with reference to FIG. 5 , the electromyographic sensor 60 is composed of six sensors, and the sensor detection values ​​are six-dimensional data. For example, the six-dimensional data is composed of the detection values ​​of the six sensors (S1, S2, S3, S4, S5, S6). The user action recognition state visualization data generator 71 shown in FIG. 5 converts this six-dimensional data (S1, S2, S3, S4, S5, S6) into three-dimensional data (X1, Y1, Z1). Note that the dimension reduction process for converting the six-dimensional data into three-dimensional data uses, for example, "PHATE," an existing dimension reduction algorithm.

[0050] In this way, the user motion point a01 is a point displayed on a three-dimensional coordinate system by converting the detection values ​​(S1, S2, S3, S4, S5, S6) of the six sensors into three-dimensional data (X1, Y1, Z1). When the user 10 moves their finger, the user motion point a01 moves on the three-dimensional coordinate system.

[0051] The user motion point trajectory data a02 is data indicating the trajectory of the user motion point a01. As time passes, the user motion point a01 moves, and the user motion point trajectory data a02 is displayed on the path of the movement.

[0052] Input value recognizable areas a11 to a13 indicate areas (three-dimensional areas) in which the smartphone 20, which is the information processing device of the present disclosure, can recognize input values ​​corresponding to user actions (finger movements) with a probability greater than or equal to a specified threshold.

[0053] In the example shown in the figure, three input value recognizable areas are shown: input value recognizable area a11, input value recognizable area a12, and input value recognizable area a13. The input value recognizable area a11 is an area in which the smartphone 20 can recognize the user's thumb bending motion and input the input value = [A] corresponding to the thumb bending motion with a probability equal to or greater than a specified threshold. Note that the probability equal to or greater than the specified threshold is set in advance, for example, to a probability of 70% or greater, or a probability of 80% or greater.

[0054] 7 is an area in which the smartphone 20 can recognize the user's index finger bending motion and input the input value = [C] corresponding to the index finger bending motion with a probability equal to or greater than a specified threshold. Furthermore, the input value recognizable area a13 shown in FIG. 7 is an area in which the smartphone 20 can recognize the user's index finger bending motion and input the input value = [D] corresponding to the index finger bending motion with a probability equal to or greater than a specified threshold.

[0055] The input value recognizable areas a11 to a13 are output in different colors. For example, the input value recognizable area a11 is blue, the input value recognizable area a12 is yellow, and the input value recognizable area a13 is red. In this way, the input value recognizable areas corresponding to each operation and each input value are output in different colors.

[0056] Although Figure 7 shows only three input value recognizable areas, it is also possible to configure the system to display input value recognizable areas corresponding to all actions and input values, or to display only one or more input value recognizable areas located close to the user action point a01 that moves due to the user's action (finger movement).

[0057] The input value recognition probabilities a21 to a23 are data displayed corresponding to each of the input value recognizable areas a11 to a13, and are pie charts showing the probability of recognizing the input value corresponding to each of the input value recognizable areas a11 to a13 based on the user action (finger movement) currently detected by the smartphone 20.

[0058] For example, the input value recognition probability a21 associated with the input value recognizable area a11 is the probability that the smartphone 20 will determine that the input is the input value [A] corresponding to the input value recognizable area a11, based on the user's action (finger movement) currently detected by the smartphone 20. Note that this probability calculation process is executed by the user action recognition probability calculation unit 72 shown in FIG.

[0059] The pie chart shown as the input value recognition probability a21 in Figure 7 indicates approximately 75%, which indicates that based on the current user action (finger movement), the probability that the smartphone 20 will determine that the input is [A] is approximately 75%.

[0060] In addition, the input value recognition probability a22 associated with the input value recognizable area a12 shown in Figure 7 is the probability that the smartphone 20 will determine that the input is the input value [C] corresponding to the input value recognizable area a12 based on the user action (finger movement) currently detected by the smartphone 20, and the pie chart shows a probability of approximately 23%.

[0061] Furthermore, the input value recognition probability a23 associated with the input value recognizable area a13 shown in Figure 7 is the probability that the smartphone 20 will determine that the input value [D] corresponding to the input value recognizable area a13 is an input based on the user action (finger movement) currently detected by the smartphone 20, and the pie chart shows a probability of approximately 18%.

[0062] 7, the input value recognition probabilities a21 to a23 are each shown as a pie chart, but they may also be shown as numerical values, as shown in Fig. 8. The input value recognition probability a31 associated with the input value recognizable area a11 shown in Fig. 8 is the probability that the smartphone 20 will determine that the input is the input value [A] corresponding to the input value recognizable area a11, based on the user's action (finger movement) currently detected by the smartphone 20, and is an example in which the numerical value [75%] is displayed.

[0063] In addition, the input value recognition probability a32 associated with the input value recognizable area a12 shown in Figure 8 is the probability that the smartphone 20 will determine that the input is the input value [C] corresponding to the input value recognizable area a12 based on the user action (finger movement) currently detected by the smartphone 20, and is an example in which the numerical value [23%] is displayed.

[0064] Furthermore, the input value recognition probability a32 associated with the input value recognizable area a12 shown in Figure 8 is the probability that the smartphone 20 will determine that the input is the input value [D] corresponding to the input value recognizable area a13 based on the user action (finger movement) currently detected by the smartphone 20, and is an example in which the numerical value [18%] is displayed.

[0065] While performing a user action (finger movement) to input text into the smartphone 20, the user 10 can view the user action recognition state information A90a consisting of three-dimensional data shown in Figures 7 and 8. The user 10 can observe changes in the user action point a01 and user action point trajectory data a02 that move with the user's finger movement, and can also confirm changes in the input value recognition probability associated with each input value recognizable area that changes with the user's finger movement.

[0066] These confirmation processes enable the user to estimate to some extent whether the input of text or the like by the user's action (finger movement) will be successful or result in an error. Furthermore, by moving the user action point a01 so that it enters one of the input value recognizable areas a11 to a13, the user 10 can recognize that the character associated with that input value recognizable area can be input, thereby improving input efficiency.

[0067] Note that the "user action recognition status information A (three-dimensional data, etc.) 90a" shown in Figures 7 and 8 is an example in which the probability values ​​calculated by the user action recognition probability calculation unit 72 are displayed in correspondence with each of the input value recognizable areas a11 to a13, but the probability values ​​calculated by the user action recognition probability calculation unit 72 may also be configured to be displayed in correspondence with an image of the hand being moved by the user 10.

[0068] A specific example will be described with reference to Fig. 9. Fig. 9 shows an example of "user action recognition status information B (user body part corresponding data) 90b" displayed on the display unit of the smartphone 20. The user 10 shown in Fig. 9 inputs text to the smartphone 20 using a user action, for example, a finger action. The text input by the user is displayed sequentially in a text input field 91.

[0069] While the user 10 is entering text, user action recognition status information B (user body part correspondence data) 90b is displayed in a partial area of ​​the display unit of the smartphone 20. By looking at this user action recognition status information B (user body part correspondence data) 90b, the user 10 can confirm whether the information processing device (smartphone 20) is correctly recognizing the user action.

[0070] The user action recognition status information B (user part correspondence data) 90b is display data showing the pie charts described above with reference to Fig. 7 on each part of the fingers of the user's hand where the user 10 performs a user action for data input processing. The pie chart on each finger indicates the probability of each user action being recognized by the smartphone 20 based on the user action (finger movement) currently detected by the smartphone 20.

[0071] As shown in Fig. 10, the user 10 can input each letter A, B, C, ... by various movements of the five fingers. The example shown in Fig. 10 shows that the letters A, C, E, G, and I can be input by bending the thumb through little finger one by one. Furthermore, the letters B, D, F, H, and J can be input by bending the thumb through little finger one by one. Furthermore, the letters K, L, M, and N can be input by performing a pinching movement with the index finger through little finger.

[0072] The numerous pie charts in the user action recognition status information B (user part correspondence data) 90b shown in Figure 9 are pie charts that show the probability that the smartphone 20 will recognize each of these characters as input based on the user action (finger movement) currently detected by the smartphone 20.

[0073] The details of the "user action recognition state information B (user body part corresponding data) 90b" shown in FIG. 9 will be described with reference to FIG.

[0074] 11 is display data showing pie charts of input value recognition probabilities b01 to b03 for each part of the fingers of the user's hand on which the user 10 performs a user action for data input processing. The pie charts on each finger indicate the probability of each user action being recognized by the smartphone 20 based on the user action (finger movement) currently detected by the smartphone 20.

[0075] The input value recognition probability b01 for the base of each finger is a pie chart showing the input value recognition probability corresponding to the bending motion of each finger. For example, the pie chart for the base of the index finger is a pie chart showing the probability that the current user motion (finger movement) is a bending motion of the index finger, which corresponds to the user motion of input process [C] shown in FIG.

[0076] The input value recognition probability b02 for the tip of each finger is a pie chart showing the input value recognition probability corresponding to the bending motion of each finger. For example, the pie chart for the tip of the index finger is a pie chart showing the probability that the current user motion (finger movement) is a bending motion of the index finger, which corresponds to the input process [D] shown in FIG.

[0077] Furthermore, the input value recognition probability b03 for the middle part of each finger is a pie chart showing the input value recognition probability corresponding to the pinch motion of each finger. For example, the pie chart for the middle part of the index finger is a pie chart showing the probability that the current user motion (finger movement) is a pinch motion of the index finger, which corresponds to the input process [K] shown in FIG.

[0078] In the example shown in Figure 11, the pie chart at the base of the index finger is the one that shows the highest probability (approximately 75%), and it can be confirmed that the smartphone 20 has the highest probability of recognizing the current user action (finger movement) as a bending action of the index finger, i.e., the input process of [C].

[0079] When processing text input into the smartphone 20, the user 10 can check the user actions recognized by the smartphone 20 by looking at the user action recognition status information B (user body part corresponding data) 90b as shown in Figure 11.

[0080] For example, it is possible to check while a user action is being performed whether the smartphone 20 is correctly recognizing the user action corresponding to the input value intended by the user, or whether it is recognizing a different user action, thereby making it possible to prevent input errors from occurring.

[0081] The user action recognition status information A (three-dimensional data, etc.) 90a, which is composed of three-dimensional data as described with reference to Figures 6 to 8, and the user action recognition status information B (user body part corresponding data) 90b, which is described with reference to Figures 9 to 11, may be displayed individually on the display unit of the smartphone 20, or may be displayed side by side as shown in Figure 12.

[0082] By checking the user action recognition status information A (three-dimensional data, etc.) 90a and user action recognition status information B (user body part corresponding data) 90b while inputting text through user action, the user 10 can confirm the user action recognized by the smartphone 20 and the input value, thereby reducing input errors and improving the efficiency of input correction processing.

[0083] 3. Training Processing for Input Processing Based on User Actions Next, training processing for input processing based on user actions will be described.

[0084] In order for the user 10 to perform text input processing based on user actions such as finger movements with high accuracy, it is effective for the user 10 to undergo training.

[0085] The above-mentioned user action recognition state information, i.e., the user action recognition state information A (three-dimensional data, etc.) 90a consisting of three-dimensional data described with reference to Figures 6 to 8, and the user action recognition state information B (user body part corresponding data) 90b described with reference to Figures 9 to 11, can also be effectively used in user training processing.

[0086] A specific example of training processing using user action recognition state information A (three-dimensional data, etc.) 90a and user action recognition state information B (user part corresponding data) 90b will be described with reference to FIG. 13 and subsequent figures.

[0087] FIG. 13 shows an example of training for text input based on user motion (finger motion) by the user 10. A training character display area 92 and a user input character display area 93 are set up on the right side of the display unit of the smartphone 20. On the left side of the display unit of the smartphone 20, user motion recognition status information A (three-dimensional data, etc.) 90a, which is composed of three-dimensional data as previously described with reference to FIGS. 6 to 8, and user motion recognition status information B (user body part corresponding data) 90b, which was previously described with reference to FIGS. 9 to 11, are displayed side by side. Furthermore, a message display area 94 is set up at the bottom of the display unit of the smartphone 20.

[0088] The user 10 performs training in accordance with the message displayed in the message display area 94. First, the following message is displayed in the message display area 94: "Please input A."

[0089] Following the message, the user 10 performs a user action to input A, i.e., a motion of bending the thumb. In response to this user action, the display data of the user action recognition status information A (e.g., three-dimensional data) 90a and the user action recognition status information B (user part corresponding data) 90b are successively updated.

[0090] A user motion point a01 in the user motion recognition state information A (e.g., three-dimensional data) 90a moves within three-dimensional coordinates in response to the user motion, and new user motion point trajectory data a02 is displayed on the path of the movement. Furthermore, pie charts such as input value recognition probability a21 in the user motion recognition state information A (e.g., three-dimensional data) 90a are also sequentially updated in response to the user motion (finger movement).

[0091] Similarly, the pie charts of the input value recognition probability b01 and the like in the user action recognition state information B (user part corresponding data) 90b are also updated sequentially in accordance with the user action (finger movement).

[0092] In the example shown in Figure 13, the user 10 follows the message and performs a user action to input the letter [A], i.e., bending the thumb, while checking the display data of user action recognition status information A (three-dimensional data, etc.) 90a and user action recognition status information B (user body part corresponding data) 90b.

[0093] In this case, the user 10 can recognize that the input value [A] corresponding to the input value recognizable area a11 has been successfully entered by moving the user action point a01 in the user action recognition status information A (three-dimensional data, etc.) 90a so that it is within the area of ​​the input value recognizable area a11.

[0094] In addition, the user 10 can recognize that he or she will be successful in inputting the input value [A] by adjusting the user's action (finger movement) so as to increase the probability value of the pie chart associated with the input value recognizable area a11 in the user action recognition status information A (three-dimensional data, etc.) 90a, i.e., the pie chart shown as the input value recognition probability a21.

[0095] Similarly, it is possible to recognize that the input of input value [A] will be successful by adjusting the user's action (finger movement) to increase the probability value of the pie chart for the base of the thumb in user action recognition status information B (user body part corresponding data) 90b, i.e., the pie chart shown as input value recognition probability b01.

[0096] In this way, the user 10 can learn the correct thumb bending action, which is a user action for inputting the letter [A], while checking the display data of the user action recognition status information A (three-dimensional data, etc.) 90a and the user action recognition status information B (user body part corresponding data) 90b displayed on the display unit of the smartphone 20.

[0097] FIG. 14 shows an example of display data on the smartphone 20 when the user 10 correctly performs the thumb bending action, which is a user action for inputting the letter [A].

[0098] The character [A] input by the user action performed by the user 10, i.e., the bending action of the thumb, is displayed in a user input character display area 93 on the right side of the display unit of the smartphone 20. Furthermore, the message "Good job" is displayed in a message display area 94 at the bottom of the display unit of the smartphone 20.

[0099] Next, training for inputting the next character begins, as shown in Figure 15. The following message is displayed in the message display area 94: "Next, please input C."

[0100] Following the message, the user 10 performs a user action to input C, i.e., a motion of bending the index finger. In response to this user action, the display data of the user action recognition status information A (e.g., three-dimensional data) 90a and the user action recognition status information B (user part corresponding data) 90b are successively updated.

[0101] A user motion point a01 in the user motion recognition state information A (e.g., three-dimensional data) 90a moves within three-dimensional coordinates in response to the user motion, and new user motion point trajectory data a02 is displayed on the path of the movement. Furthermore, pie charts such as input value recognition probability a21 in the user motion recognition state information A (e.g., three-dimensional data) 90a are also sequentially updated in response to the user motion (finger movement).

[0102] Similarly, the pie charts of the input value recognition probability b01 and the like in the user action recognition state information B (user part corresponding data) 90b are also updated sequentially in accordance with the user action (finger movement).

[0103] In the example shown in Figure 15, user 10 follows the message and performs a user action to input the letter [C], i.e., bending the index finger, while checking the display data of user action recognition status information A (three-dimensional data, etc.) 90a and user action recognition status information B (user body part corresponding data) 90b.

[0104] In this case, the user 10 can recognize that the input value [C] corresponding to the input value recognizable area a12 has been successfully entered by moving the user action point a01 in the user action recognition status information A (three-dimensional data, etc.) 90a so that it is within the area of ​​the input value recognizable area a12.

[0105] In addition, the user 10 can recognize that he or she will be able to successfully input the input value [C] by adjusting the user's action (finger movement) so as to increase the probability value of the pie chart associated with the input value recognizable area a12 in the user action recognition status information A (three-dimensional data, etc.) 90a, i.e., the pie chart shown as the input value recognition probability a22.

[0106] Similarly, it is possible to recognize that the input of input value [C] will be successful by adjusting the user's action (finger movement) to increase the probability value of the pie chart for the base of the index finger in user action recognition status information B (user body part corresponding data) 90b, i.e., the pie chart shown as input value recognition probability b02.

[0107] In this way, the user 10 can learn the correct index finger bending action, which is the user action for inputting the letter [C], while checking the display data of the user action recognition status information A (three-dimensional data, etc.) 90a and the user action recognition status information B (user body part corresponding data) 90b displayed on the display unit of the smartphone 20.

[0108] FIG. 16 shows an example of display data of the smartphone 20 when the user 10 fails to correctly perform the index finger bending action, which is a user action for inputting the letter [C].

[0109] Only the previously input character [A] is displayed in a user input character display area 93 on the right side of the display unit of the smartphone 20. Furthermore, nothing is displayed in a message display area 94 at the bottom of the display unit of the smartphone 20.

[0110] 17 shows an example of the data displayed on the smartphone 20 after that. Advice information for the user's action, such as "Please bend your index finger more," is displayed in the message display area 94 at the bottom of the display unit of the smartphone 20. This message is generated by the user action recognition status information generation unit 63 inside the smartphone 20 based on the analysis results of the user's action.

[0111] The user action recognition state information generation unit 63 analyzes the user action point a01 that moves in response to the user action by the user 10 and the user action point trajectory data a02, detects problems in the user action, and generates and displays a message to solve the detected problem.

[0112] If the problem is different, the user action recognition state information generation unit 63 generates and displays a different message to solve the different problem. For example, as shown in Fig. 18, a message such as "Please bend your index finger more slowly" is displayed in the message display area 94 at the bottom of the display unit of the smartphone 20.

[0113] In response to the displayed message, the user 10 again performs a user action to input the letter [C], that is, a motion of bending the index finger.

[0114] While checking the display data of user action recognition status information A (three-dimensional data, etc.) 90a and user action recognition status information B (user part corresponding data) 90b, user 10 performs a user action to input the letter [C], i.e., bending the index finger.

[0115] 19 shows an example of display data on the smartphone 20 when the user 10 again performs a user action to input the character [C]. The user 10 performs a process to move the user action point a01 in the user action recognition state information A (e.g., three-dimensional data) 90a so that it enters the input value recognition area a12.

[0116] As a result, it can be confirmed that the probability value of the pie chart associated with the input value recognizable area a12, i.e., the pie chart shown as the input value recognition probability a22, has increased significantly, and it can be recognized that the input of the input value [C] has been successful.

[0117] Similarly, it can be confirmed that the probability value of the pie chart for the base of the index finger in user action recognition status information B (user body part corresponding data) 90b, i.e., the pie chart shown as input value recognition probability b02, has also increased significantly, and it can be recognized that the input of input value [C] has been successful.

[0118] In this way, even a user who is unfamiliar with input processing using user actions can perform more accurate input processing by entering data based on various user actions while checking the display data of user action recognition status information A (three-dimensional data, etc.) 90a and user action recognition status information B (user body part corresponding data) 90b.

[0119] Furthermore, the data processing unit of the smartphone 20 may be configured to calculate an input accuracy score from the number of incorrect inputs and the number of correct answers in the user input, and display the calculated score. For example, a process may be performed to calculate and display a score such as "Your score is 80 points." Alternatively, a message emphasizing the results of training may be generated and displayed, such as "Through training, your score has increased from 80 points to 90 points."

[0120] 4. Setup Process for Generating User-Specific User Action Recognition State Information Next, a setup process for generating user-specific user action recognition state information will be described.

[0121] Input value recognizable areas a11 to a13 are displayed in the user action recognition state information A (three-dimensional data, etc.) 90a, which was previously described with reference to Figures 6 to 9, etc. As described above, the input value recognizable areas a11 to a13 indicate areas (three-dimensional areas) in which the smartphone 20, which is the information processing device of the present disclosure, can recognize an input value corresponding to a user action (finger movement) with a probability equal to or higher than a specified threshold value.

[0122] 7 is an area in which the smartphone 20 can recognize the user's thumb bending motion and input the input value [A] corresponding to the thumb bending motion with a probability equal to or greater than a predetermined threshold value. The probability equal to or greater than the predetermined threshold value is set in advance, for example, to a probability of 70% or more, or a probability of 80% or more.

[0123] These input value recognizable areas a11 to a13 are areas that are determined according to the user's 10 user actions (finger movements), but since user actions (finger movements) differ from user to user, it is preferable to set a user-specific movement area that is easy for the user to perform as the input value recognizable area.

[0124] Below, we will explain the process of generating user action recognition status information A (three-dimensional data, etc.) 90a, which sets an input value recognizable area according to each user, i.e., the setup process of generating user action recognition status information specific to each user.

[0125] The flowchart shown in FIG. 20 is a flowchart illustrating a processing sequence when a user 10 performs a data input process by user action on an information processing device such as a smartphone 20.

[0126] 20 and subsequent steps are executed in accordance with a program stored in a storage unit of the information processing device (smartphone, etc.) disclosed herein under the control of a data processing unit including a CPU with a program execution function, etc. The processing of each step of the flow shown in FIG. 20 will be described below in order.

[0127] (Step S101) First, in step S101, the user wears the sensor and executes connection processing (either wired or wireless) to input the sensor detection value into an information processing device (smartphone, AR glasses, etc.).

[0128] That is, for example, as described above with reference to Fig. 5, the sensor detection value is connected to an information processing device such as a smartphone so that the sensor detection value is input to the smartphone. Note that the input process of the sensor detection value to the smartphone may be performed in either a wired or wireless manner.

[0129] (Step S102) Next, in step S102, the user performs an action (e.g., waving a hand) to indicate the start of processing. The type of action is detected based on the input value of the sensor. The type of action and the type of processing execution request are registered in advance in the information processing device.

[0130] (Steps S103 to S104) Next, in steps S103 to S104, the information processing device such as a smartphone determines whether the setup process (generation and registration process of user-specific user action recognition status information A (3D data, etc.)) is complete.

[0131] If the setup process (generation and registration of user-specific user action recognition status information A (e.g., three-dimensional data)) has been completed, proceed to step S105. On the other hand, if the setup process (generation and registration of user-specific user action recognition status information A (e.g., three-dimensional data)) has not been completed, proceed to step S106.

[0132] (Step S105) If it is determined in steps S103 and S104 that the setup process (generation and registration of user-specific user action recognition state information A (three-dimensional data, etc.)) is complete, the process proceeds to step S105.

[0133] In this case, data input processing based on user actions is executed using user action recognition state information A (three-dimensional data, etc.) specific to the user that has been generated by the setup processing.

[0134] For example, as described above with reference to FIG. 12, data input processing based on user actions is executed while displaying and checking user-specific user action recognition state information A (three-dimensional data, etc.) 90a, etc.

[0135] (Step S106) On the other hand, if the setup process (the process of registering the user-specific user action recognition state information A (three-dimensional data, etc.)) is not completed, the process proceeds to step S106.

[0136] In this case, in step S106, a setup process, that is, a process of generating and registering user action recognition state information A (three-dimensional data or the like) specific to the user, is executed.

[0137] After the setup process in step S106 is completed, the process returns to step S103, and after confirming the completion of the setup process, the process proceeds to step S105, where data input processing based on user actions is performed while displaying and checking the user-specific user action recognition status information A (three-dimensional data, etc.) 90a, etc., generated in the setup process.

[0138] The detailed sequence of the setup process executed in step S106, i.e., the process of registering user-specific user action recognition status information A (e.g., three-dimensional data), will be described with reference to the flow shown in Fig. 21. The process of each step in the flow shown in Fig. 21 will be described below in order.

[0139] (Step S121) First, in step S121, the data processing unit of the information processing device of the present disclosure, that is, the information processing device such as the smartphone 20, specifies a finger and outputs a message instructing the finger's movement.

[0140] Specifically, a message such as "Please bend your thumb" is displayed on the display unit of the smartphone 20.

[0141] (Step S122) Next, in step S122, the information processing device such as the smartphone 20 acquires sensor detection values ​​(for example, six-dimensional data).

[0142] For example, as described above with reference to Fig. 5, the user 10 wears an electromyographic sensor 60. The electromyographic sensor 60 is composed of six sensors, and the sensor detection values ​​are six-dimensional data. An information processing device such as a smartphone inputs the six-dimensional data composed of the detection values ​​(S1, S2, S3, S4, S5, S6) of the six sensors.

[0143] (Step S123) Next, in step S123, the information processing device such as the smartphone 20 converts the acquired sensor detection values ​​(6-dimensional data) into visualized data such as 3-dimensional data and displays it as a user action point. In this example, the 3-dimensional data is generated and displayed on a 3-dimensional coordinate system.

[0144] This process is executed by the user action recognition state visualization data generator 71, which was previously described with reference to Fig. 5. The user action recognition state visualization data generator 71 converts six-dimensional data (S1, S2, S3, S4, S5, S6) input from the sensors into three-dimensional data (X1, Y1, Z1) and displays it as a user action point on a three-dimensional coordinate system. Note that the dimension reduction process for converting six-dimensional data into three-dimensional data uses, for example, "PHATE," an existing dimension reduction algorithm.

[0145] As shown on the right side of the flow in Fig. 21, a user motion point is displayed on the three-dimensional data. Note that this user motion point is data corresponding to the user motion point a01 displayed in the "user motion recognition state information A (three-dimensional data, etc.) 90a" previously described with reference to Fig. 7, etc.

[0146] (Step S124) Next, in step S124, the information processing device such as the smartphone 20 calculates an "input value recognizable area" that takes into account fluctuations in the user's movements, and displays it on a three-dimensional coordinate system.

[0147] As shown on the right side of the flow in Fig. 21, an "input value recognizable area" is displayed on the three-dimensional data. Note that this "input value recognizable area" is data equivalent to the "input value recognizable areas a11 to a13" displayed in the "user action recognition status information A (three-dimensional data, etc.) 90a" previously described with reference to Fig. 7, etc.

[0148] Details of these two processes, step S123 and step S124, i.e., the process executed in step S123 to convert the sensor detection value (six-dimensional data) into three-dimensional data and display it on three-dimensional coordinates as a user action point, and the process executed in step S124 to calculate an "input value recognizable area" taking into account fluctuations in user action and display it on three-dimensional coordinates, will be described with reference to Figure 22 and subsequent figures.

[0149] 22 shows a partial configuration of a data processing unit configured in the smartphone 20, similar to the configuration described above with reference to FIG. 5. The outputs of the six electromyography sensors 60 are input to the smartphone 20, which is an information processing device of the present disclosure, via a sensor detection value input unit 61. The smartphone 20 has an electromyography-input value conversion unit 62, a user action recognition state information generation unit 63, and a display information generation unit 75.

[0150] The myoelectric-input value conversion unit 62 analyzes the six sensor detection values ​​of the six myoelectric sensors 60, determines the movement of the user's fingers, and identifies the input value. The user action recognition status information generation unit 63 generates "user action recognition status information" that indicates whether the smartphone 20 is correctly recognizing the user action while the user is performing an input process.

[0151] The user action recognition state information generation unit 63 includes a user action recognition state visualization data generation unit 71 and a user action recognition probability calculation unit 72. The user action recognition state visualization data generation unit 71 generates visualization data such as three-dimensional data constituting "user action recognition state information" indicating whether a user action has been correctly recognized. The user action recognition probability calculation unit 72 calculates the probability that a user action has been correctly recognized. This probability is also data constituting the "user action recognition state information." As described above, in the embodiment, an example is described in which the user action recognition state visualization data generation unit 71 generates user action recognition state visualization data constituted by three-dimensional data. However, the user action recognition state visualization data generated by the user action recognition state visualization data generation unit 71 is not limited to three-dimensional data and may be in other data formats such as two-dimensional data, as long as it is display data that allows the user to check the user action recognition state.

[0152] As shown in FIG. 23 , the user action recognition state visualization data generation unit 71 includes a user action point visualization data generation unit 81 and an input value recognizable area visualization data generation unit 82 .

[0153] The user action point visualization data generator 81 executes a process of setting a user action point on a three-dimensional coordinate system, while the input value recognizable area visualization data generator 82 executes a process of setting an input value recognizable area on a three-dimensional coordinate system.

[0154] Referring to Figure 24, the process executed by the user motion point visualization data generator 81, i.e., the process of setting a user motion point on a three-dimensional coordinate system, will be described in detail. As shown in Figure 24, the user motion point visualization data generator 81 inputs six-dimensional data (S1, S2, S3, S4, S5, S6) that are sensor detection values ​​from a sensor, converts them into three-dimensional data (X1, Y1, Z1), and executes a process of displaying the data as a user motion point on a three-dimensional coordinate system. Note that the dimension reduction process for converting six-dimensional data into three-dimensional data uses, for example, "PHATE," an existing dimension reduction algorithm. This process corresponds to the process of step S123 in the flow shown in Figure 21.

[0155] Next, with reference to FIG. 25, the process executed by the input value recognizable area visualization data generating unit 82, that is, the process of setting an input value recognizable area on a three-dimensional coordinate system, will be described in detail.

[0156] As shown in FIG. 25, the input value recognizable area visualization data generation unit 82 receives six-dimensional data (S1, S2, S3, S4, S5, S6) that are sensor detection values ​​from the sensor.

[0157] First, in step S21 shown on the right side of Figure 25, the input value recognizable area visualization data generation unit 82 generates sensor detection value fluctuation range data that takes into account the fluctuations of user movements based on the input six-dimensional data (S1, S2, S3, S4, S5, S6).

[0158] For example, the following six-dimensional sensor detection value fluctuation range data is generated: S1 ±10% = 0.9 x S1 to 1.1 x S1 S2 ±10% = 0.9 x S2 to 1.1 x S2 S3 ±10% = 0.9 x S3 to 1.1 x S3 S4 ±10% = 0.9 x S4 to 1.1 x S4 S5 ±10% = 0.9 x S5 to 1.1 x S5 S6 ±10% = 0.9 x S6 to 1.1 x S6

[0159] In the above process, a range of ±10% is estimated as the fluctuation range of the user's movement for each of the sensor detection values ​​(S1, S2, S3, S4, S5, S6). The fluctuation range of the user's movement may be set to a value other than ±10%, and various processes are possible, such as a setting of ±5%.

[0160] Next, in step S22, the area defined by the six-dimensional sensor detection value fluctuation range data is converted into three-dimensional data and displayed as an "input value identifiable area" on a three-dimensional coordinate system. The dimension reduction process for converting six-dimensional data into three-dimensional data uses the aforementioned "PHATE." This process corresponds to step S124 in the flow shown in FIG. 21.

[0161] Returning to the flow shown in FIG. 21, the processing from step S125 onwards will be described.

[0162] (Step S125) After the process of converting the sensor detection value (six-dimensional data) in step S123 into three-dimensional data and displaying it on three-dimensional coordinates as a user action point, and the process of calculating an "input value recognizable area" taking into account fluctuations in user action and displaying it on three-dimensional coordinates in step S124 are completed, the data processing unit of the information processing device then executes the process of step S125.

[0163] In step S125, it is determined whether there are any unprocessed fingers or movements. If there are any unprocessed fingers or movements, the process returns to step S121, and the processes from step S121 onwards are executed for the unprocessed fingers and movements. On the other hand, if the processes for all fingers and movements have been completed, the process proceeds to step S126.

[0164] (Step S126) When the processing for all fingers and actions has been completed, in step S126, the "input value recognizable areas" corresponding to all user actions are displayed on a three-dimensional coordinate system.

[0165] As shown on the right side of the flow in Fig. 21, the "input value recognizable areas" corresponding to all user actions are displayed on a three-dimensional coordinate system, and the processing ends. Note that data recording the "input value recognizable areas" corresponding to all user actions on a three-dimensional coordinate system is stored in the storage unit of the information processing device (smartphone, etc.).

[0166] This stored data is displayed on the display unit of the information processing device (e.g., smartphone) when performing data input processing based on user actions. That is, it is displayed on the display unit as "user action recognition status information A (e.g., three-dimensional data) 90a" previously described with reference to FIG. 7, etc., and the user can confirm how the user action he or she is performing is recognized by the information processing device. That is, it is possible to confirm whether the user's intended input was successful.

[0167] The three-dimensional data generated in the initial setup process and stored in the storage unit of the information processing device (e.g., smartphone), i.e., "user action recognition status information A (e.g., three-dimensional data) 90a," may be configured to be updated appropriately when the user subsequently inputs text. That is, the information processing device, such as a smartphone, may learn (through machine learning) the correspondence between the user's actions actually performed and the input values, and update the three-dimensional data stored in the storage unit, i.e., "user action recognition status information A (e.g., three-dimensional data) 90a," to reflect the learning results. Performing such an update process makes it possible to reflect the user's habits, etc., and further improve input accuracy.

[0168] For example, if the "input value recognizable areas" corresponding to a large number of user actions displayed on the three-dimensional coordinate system are crowded together or overlap, the accuracy of identifying various finger actions will decrease. Therefore, in such cases, it is preferable to perform processing such as redoing the registration process for user actions corresponding to the crowded or overlapping "input value recognizable areas."

[0169] Ideally, for example, when "(a) the input value recognizable areas are densely packed" as shown in FIG. 26(a), it is preferable to execute processing such as redoing the registration process so that "(b) the input value recognizable areas are spaced apart" as shown in FIG. 26(b).

[0170] In this way, by reducing the overlapping area of ​​the "input value recognizable area" corresponding to the user's actions, it is possible to improve the accuracy of identifying various finger actions.

[0171] When the setup process according to the flow shown in FIG. 21 is completed, i.e., when the generation and registration process of user-specific user action recognition status information A (e.g., three-dimensional data) is completed, the process returns to step S103 of the flow shown in FIG. 20, and after confirming that the setup process has been completed, the process proceeds to step S105.

[0172] In step S105, data input processing based on the user's actions is performed while displaying and checking the user-specific user action recognition status information A (e.g., three-dimensional data) 90a generated in the setup processing. This setup processing makes it possible to use the user-specific user action recognition status information A (e.g., three-dimensional data) 90a, enabling data input processing that reflects the user's unique movements.

[0173] 5. Display Example of User Action Recognition Status Information When Data is Input Based on User Action After Setup Next, a display example of user action recognition status information when data is input based on user action after setup will be described.

[0174] As described above, when the setup process in step S106 of the flowchart shown in FIG. 20 is completed, data recording the "input value recognizable areas" corresponding to all user actions on three-dimensional coordinates is stored in the memory unit of the information processing device (smartphone, etc.).

[0175] This stored data is displayed on the display unit as "user action recognition status information A (three-dimensional data, etc.) 90a" which has been described with reference to FIG. 7 etc. when data input processing based on user action is performed.

[0176] By referring to the "user action recognition status information A (three-dimensional data, etc.) 90a" displayed on the display unit, the user can check how the user action he or she is performing is being recognized by the information processing device. In other words, it is possible to check whether the user's intended input has been successful.

[0177] Referring to Figure 27 and subsequent figures, a display example of "user action recognition status information A (three-dimensional data, etc.) 90a" displayed on the display unit of an information processing device (smartphone, etc.) when data is input by user action will be described.

[0178] 27 shows an example in which a user 10 wears AR glasses (transmissive type) 40 and inputs data by moving their fingers while looking at their own hands. Step S201 shows the state of the display unit of the AR glasses (transmissive type) 40 before input begins, and nothing is displayed.

[0179] Step S202 shows the state of the display unit of the AR glasses (transmissive type) 40 after data input by user action has started. Text based on the user action (finger movement) is displayed sequentially on the display unit of the AR glasses. In addition, "user action recognition status information A (three-dimensional data, etc.) 90a" is displayed along with the text display. The user can check the recognition status of the user action in the information processing device by referring to the "user action recognition status information A (three-dimensional data, etc.) 90a" displayed on the display unit.

[0180] For example, it is possible to check whether the user's intended input was successful, and if an error occurs, the input can be corrected while viewing the "user action recognition status information A (three-dimensional data, etc.) 90a." Specifically, the error can be corrected by, for example, correcting the user action so that the user action point a01 falls within the input value recognition area.

[0181] 28 is a diagram showing an example in which, when an error occurs in an input by a user action, an information processing device generates guide information for the user to correct the error and displays the guide information on the display unit. The example shown in FIG. 28 is an example in which the information processing device (AR glasses 40) cannot determine the input value from the user action. That is, it is an example in which it cannot determine whether the user action is an input action of input value [C] or an input action of input value [D].

[0182] Step S211 shows the state of the display unit of the AR glasses (transmissive type) 40 after the start of data input by the user's action, and "user action recognition state information A (three-dimensional data, etc.) 90a" is displayed. Furthermore, the following message is displayed as a message from the information processing device (AR glasses 40): "Input values ​​C and D cannot be identified."

[0183] This is a message generated by the data processing unit within the information processing device (AR glasses 40), and when it is not possible to determine whether the user action is an input action of input value [C] or an input action of input value [D], the data processing unit of the information processing device generates it and displays it on the child display unit.

[0184] Furthermore, the data processing unit of the information processing device displays specific examples of user actions, i.e., examples of input actions for [C] and [D], on the display unit, as shown in step S212. The user can check the input actions for [C] and [D] based on this display data, and then perform the correct input actions.

[0185] Next, an example of controlling the display position of "user action recognition status information A (three-dimensional data, etc.) 90a" on the display unit of the information processing device will be described with reference to Fig. 29 and subsequent figures. Fig. 29 shows an example in which a data processing unit in the information processing device (AR glasses 40) analyzes the line of sight of the user 10 and controls the display of "user action recognition status information A (three-dimensional data, etc.) 90a" at a position in the line of sight of the display unit.

[0186] The data processing unit within the information processing device (AR glasses 40) displays "user action recognition status information A (three-dimensional data, etc.) 90a" in the line of sight of the user 10 in this manner, making it easier for the user to recognize "user action recognition status information A (three-dimensional data, etc.) 90a."

[0187] In this example, the data processing unit in the information processing device (AR glasses 40) analyzes the gaze direction of the user 10 using analysis information from a gaze direction analyzer attached to the information processing device (AR glasses 40) or images captured by a camera.

[0188] Figure 30 shows an example in which a data processing unit within an information processing device (AR glasses 40) analyzes the head direction of the user 10 and controls the display of "user action recognition status information A (three-dimensional data, etc.) 90a" at the head direction position on the display unit.

[0189] The data processing unit in the information processing device (AR glasses 40) displays the "user action recognition status information A (three-dimensional data, etc.) 90a" in the head direction position of the user 10 in this manner, making it easier for the user to recognize the "user action recognition status information A (three-dimensional data, etc.) 90a."

[0190] In this example, the data processing unit in the information processing device (AR glasses 40) analyzes the head movement of the user 10 using images captured by a camera attached to the information processing device (AR glasses 40). Alternatively, the movement of the user 10 may be analyzed by inputting detection information from a motion sensor (such as an acceleration sensor) attached to the head of the user 10.

[0191] Figure 31 shows an example in which a data processing unit within an information processing device (AR glasses 40) recognizes the action of a user 10, for example, a hand waving action, and controls the display of "user action recognition status information A (three-dimensional data, etc.) 90a" in the position in the direction of the hand wave.

[0192] The data processing unit in the information processing device (AR glasses 40) displays "user action recognition status information A (three-dimensional data, etc.) 90a" in the direction in which the user 10 waves their hand, making it easier for the user to recognize the "user action recognition status information A (three-dimensional data, etc.) 90a."

[0193] In this example, the data processing unit in the information processing device (AR glasses 40) analyzes the movement of the user 10 using images captured by a camera attached to the information processing device (AR glasses 40). Alternatively, the movement of the user 10 may be analyzed by inputting detection information from a motion sensor (such as an acceleration sensor) attached to the arm of the user 10.

[0194] In the examples described with reference to Figures 27 to 31, a display example of "user action recognition status information A (three-dimensional data, etc.) 90a" has been described, but instead of "user action recognition status information A (three-dimensional data, etc.) 90a", it is also possible to display "user action recognition status information B (user body part corresponding data) 90b" in which the input value recognition probability (pie chart) is associated with the user's hand as described with reference to Figure 9 etc.

[0195] Furthermore, as explained with reference to Figure 12, "user action recognition status information A (three-dimensional data, etc.) 90a" and "user action recognition status information B (user body part corresponding data) 90b" may be displayed side by side.

[0196] 6. Display Examples of User Action Recognition Status Information Corresponding to Data Entry Processes Based on Various User Actions Next, display examples of user action recognition status information corresponding to data entry processes based on various user actions will be described.

[0197] In the above-described embodiment, an example has been described in which the movement of a user's finger is used as a user action when inputting data into an information processing device (such as AR glasses). The user action when inputting data into an information processing device is not limited to the movement of a user's finger, and other user actions can also be used. The display process for user action recognition status information of the present disclosure can also be displayed in response to various user actions.

[0198] The example shown in Figure 32 is an example in which the user's action when inputting data into an information processing device (such as AR glasses) is to input "OK" into the information processing device based on the movement of the user's 10 hand or arm, for example, the movement of slightly raising the hand.

[0199] A motion sensor (such as an acceleration sensor or gyro) 100 is attached to the user's arm, and an information processing device (such as AR glasses) inputs the detection values ​​of the motion sensor 100 to analyze the motion of the user's hand and arm.

[0200] Furthermore, the information processing device (AR glasses, etc.) displays "user action recognition status information A (three-dimensional data, etc.) 90a" as shown in Fig. 32. The "user action recognition status information A (three-dimensional data, etc.) 90a" displays the following data: user action point a01, user action point trajectory data a02, input value identifiable area (recognizable area for slightly raising a hand (OK input)) a15, input value identifiable area (recognizable area for waving a hand (NG input)) a16,

[0201] The "user motion point a01" moves in accordance with the user's hand motion, that is, each motion such as raising a hand, lowering a hand, or waving a hand.

[0202] The "input value identifiable area (area where raising the hand slightly (OK input) can be recognized) a15" is an area where the information processing device can recognize the "user action = raising the hand slightly" action corresponding to the OK input with a predetermined probability (e.g., 70%) or more. The "input value identifiable area (area where waving the hand (NG input) can be recognized) a16 is an area where the information processing device can recognize the "user action = waving the hand" action corresponding to the NG input with a predetermined probability (e.g., 70%) or more. Note that these areas are stored in the memory unit of the information processing device by previously executing a process similar to the setup process described above with reference to FIG. 21 etc.

[0203] The user can check whether the user's intended input (OK) has been recognized by the information processing device by referring to the "User action recognition status information A (three-dimensional data, etc.) 90a" displayed on the display unit as shown in Figure 32.

[0204] The example shown in Figure 32 is an example of inputting "OK" into an information processing device (AR glasses, etc.), that is, an example of inputting "OK" into an information processing device based on the action of user 10 slightly raising his / her hand, but similar processing is possible for other user actions.

[0205] For example, various processes are possible as input examples based on user actions, as shown in Fig. 33. (1) Input example 1 based on user action (user action = slightly raising hand (input = OK)) (2) Input example 2 based on user action (user action = putting hand forward (input = next)) (3) Input example 3 based on user action (user action = waving hand up (input = opening menu)) (4) Input example 4 based on user action (user action = waving hand away (input = closing menu))

[0206] The information processing device (AR glasses, etc.) generates and displays "user action recognition status information A (three-dimensional data, etc.) 90a" which includes input value identifiable areas corresponding to each of these actions, allowing the user to check whether the action they are performing is being recognized by the information processing device (AR glasses, etc.).

[0207] In the examples shown in Figures 32 and 33, a motion sensor (such as an acceleration sensor or gyro) 100 is attached to the user's hand or arm, and an information processing device (such as AR glasses) inputs the detection value of the motion sensor 100 to analyze the movement of the user's hand or arm. However, a configuration in which the movement of the user's hand or arm is analyzed using, for example, an image captured by a camera may also be used.

[0208] 34, a camera 101 that captures an image in front of the AR glasses 40, which are an information processing device, is used to capture an image of the hand of the user 10. The data processing unit of the AR glasses 40 uses the image captured by the camera 101 to analyze the movement of the user's hand and arm.

[0209] The data processing unit of the AR glasses 40 generates user motion points a01 and user motion point trajectory data a02 based on the analysis results, and further generates and displays "user motion recognition status information A (three-dimensional data, etc.) 90a" including input value identifiable areas corresponding to each motion. By referring to the "user motion recognition status information A (three-dimensional data, etc.) 90a," the user can confirm whether the motion they are performing is being recognized by the information processing device (AR glasses, etc.).

[0210] Next, with reference to FIG. 35 and subsequent figures, a processing example using an information processing device such as a smartphone or tablet terminal having a touch panel function will be described.

[0211] 35 shows a smartphone 20. The smartphone 20 has a touch panel function, and the user 10 can input various data to the smartphone 20 by touching the display panel of the smartphone 20 with a finger and sliding it along a predetermined trajectory. The touch panel has a function of detecting the touch position of the user's finger using a touch sensor.

[0212] For example, when the user 10 moves his / her finger in a circular motion on the touch panel, the data processing unit of the smartphone 20 recognizes that the user's finger movement is a circular motion and determines the input data by the user based on this recognition result. For example, it determines that the user's input data is [OK]. The correspondence between the user's finger movement and the input data is registered in advance in the memory unit of the smartphone 20.

[0213] However, for example, even if the user intends to draw a circle, the actual movement of the finger may be a square rather than a circle. In such a case, the user intends to input [OK], but the smartphone 20 processes this as input of data registered corresponding to a "square."

[0214] The data processing unit of the smartphone 20 displays "user action recognition status information A (three-dimensional data, etc.) 90a" in response to a user's operation on the touch panel, as shown in Fig. 35. As shown in Fig. 35, the "user action recognition status information A (three-dimensional data, etc.) 90a" displays the following data: user action point a01, user action point trajectory data a02, input value identifiable area (circle drawing (OK input) recognizable area) a17,

[0215] The "user action point a01" moves in accordance with the user's actions, that is, each action such as a slide operation on the touch panel of the smartphone 20.

[0216] The "input value identifiable area (area where drawing a circle (OK input) can be recognized) a17" is an area where the information processing device can recognize "user action = action of drawing a circle on the touch panel" with a predetermined probability (e.g., 70%) or more. Note that these areas are stored in the storage unit of the information processing device by previously executing a process similar to the setup process described above with reference to FIG. 21 etc.

[0217] The user can check whether the user's intended input (OK) has been recognized by the information processing device by referring to the "User action recognition status information A (three-dimensional data, etc.) 90a" displayed on the display unit as shown in Figure 35.

[0218] 36 shows an example in which the user 10 draws a circle on the display panel of the smartphone 20 and then draws an L-shape. The data processing unit of the smartphone 20 recognizes the circle drawing action as an input of [OK] and the L-shape drawing action as an input process of [Next]. The correspondence between these user finger movements and input data is registered in advance in the memory unit of the smartphone 20.

[0219] The data processing unit of the smartphone 20 displays "user action recognition status information A (three-dimensional data, etc.) 90a" in response to a user's operation on the touch panel, as shown in Fig. 36. As shown in Fig. 36, the "user action recognition status information A (three-dimensional data, etc.) 90a" displays the following data: user action point a01, user action point trajectory data a02, input value identifiable area (area where a circle can be drawn (OK input) can be recognized) a17, input value identifiable area (area where an L can be drawn (next input) can be recognized) a18,

[0220] The "user action point a01" moves in accordance with the user's actions, that is, each action such as a slide operation on the touch panel of the smartphone 20.

[0221] The "input value identifiable area (area where drawing a circle (OK input) can be recognized) a17" is an area where the information processing device can recognize "user action = action of drawing a circle on the touch panel" with a predetermined probability (e.g., 70%) or more. The "input value identifiable area (area where drawing an L (input next) can be recognized) a18" is an area where the information processing device can recognize "user action = action of drawing an L letter on the touch panel" with a predetermined probability (e.g., 70%) or more. Note that these areas are stored in the storage unit of the information processing device by previously executing a process similar to the setup process described above with reference to FIG. 21 etc.

[0222] By referring to the "User Action Recognition Status Information A (3D data, etc.) 90a" displayed on the display unit as shown in Figure 36, the user can check whether the user's intended input (OK) or (Next) has been recognized by the information processing device.

[0223] 37 shows an example in which the user 10 simultaneously draws a circle with two fingers, the index finger and the middle finger, on the display panel of the smartphone 20. Note that the action of drawing a circle with two fingers is recognized by the data processing unit of the smartphone 20 as an input process for "end." Note that the correspondence between this user's finger movement and input data is registered in advance in the memory unit of the smartphone 20.

[0224] The data processing unit of the smartphone 20 displays "user action recognition status information A (three-dimensional data, etc.) 90a" in response to a user's operation of the touch panel, as shown in Fig. 37. As shown in Fig. 36, the "user action recognition status information A (three-dimensional data, etc.) 90a" displays the following data: user action point a01-1, user action point a01-2, user action point trajectory data a02-1, user action point trajectory data a02-2, input value identifiable area (area where double circle drawing (end input) is recognizable) a19-1, input value identifiable area (area where double circle drawing (end input) is recognizable) a19-2,

[0225] "User action point a01-1" and "user action point a01-2" move in accordance with the user's action, i.e., the sliding action of the index finger and middle finger on the touch panel of the smartphone 20. If the smartphone 20 is equipped with a device that can perform fingerprint authentication no matter where on the smartphone screen the user touches, such as a full-screen fingerprint authentication sensor device, the sliding action of the index finger and the sliding action of the middle finger can be distinguished and identified by comparing the fingerprint data of the user's index finger and middle finger that have been registered in advance on the smartphone 20. Therefore, the displayed "user action recognition status information A (three-dimensional data, etc.) 90a" can also display the information of the index finger and middle finger as separate and independent information.

[0226] The "input value identifiable area (area where a double circle is drawn (end input) can be recognized) a19-1" and the "input value identifiable area (area where a double circle is drawn (end input) can be recognized) a19-2" are areas where the information processing device can recognize "user action = action of drawing a double circle on the touch panel" with a predetermined probability (e.g., 70%) or higher. Note that these areas are stored in the storage unit of the information processing device by previously executing a process similar to the setup process described above with reference to FIG. 21 etc.

[0227] By referring to the "user action recognition status information A (three-dimensional data, etc.) 90a" displayed on the display unit as shown in FIG. 37, the user can confirm whether the information processing device has recognized the user's intended input (end). The "user action recognition status information A (three-dimensional data, etc.) 90a" may be controlled to be displayed in a position on the display unit of the smartphone 20 that is easily visible to the user. In the situations shown in FIGS. 35 to 37, the user is performing a touch operation on the touch panel in the right half of the smartphone 20 screen, and the touch panel can determine which position the user is touching. Therefore, in this case, for example, the "user action recognition status information A (three-dimensional data, etc.) 90a" is displayed in the left half of the display screen of the smartphone 20. This prevents the user from obscuring the "user action recognition status information A (three-dimensional data, etc.) 90a" due to the finger that performed the touch operation.

[0228] 7. Hardware Configuration Example of Information Processing Device Next, a hardware configuration example of an information processing device that constitutes a smartphone, AR glasses, tablet terminal, PC, etc. that executes processing according to the above-described embodiment will be described with reference to Fig. 39. The hardware shown in Fig. 38 is an example of the hardware configuration of the information processing device of the present disclosure. The hardware configuration shown in Fig. 38 will be described.

[0229] The CPU (Central Processing Unit) 301 functions as a data processing unit that executes various processes according to programs stored in the ROM (Read Only Memory) 302 or the storage unit 308. For example, it executes processes according to the sequences described in the above-mentioned embodiments. The RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are interconnected by a bus 304.

[0230] The CPU 301 is connected to an input / output interface 305 via a bus 304, and the input / output interface 305 is connected to an input unit 306 including various sensors, cameras, switches, microphones, etc., and an output unit 307 including a display, speaker, etc.

[0231] The storage unit 308 connected to the input / output interface 305 is formed of, for example, a hard disk, and stores various data and programs executed by the CPU 301. The communication unit 309 functions as a transmitter / receiver for data communication via a network such as the Internet or a local area network, and communicates with external devices.

[0232] A drive 310 connected to the input / output interface 305 drives removable media 311 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory such as a memory card, and executes recording or reading of data.

[0233] [8. Summary of the Configuration of the Present Disclosure] The embodiments of the present disclosure have been described above in detail with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0234] The technology disclosed in this specification can be configured as follows: (1) An information processing device having a data processing unit that executes an input value identification process based on a user action, wherein the data processing unit generates user action recognition state information indicating a recognition state of the user action and outputs the information to a display unit.

[0235] (2) The information processing device according to (1), wherein the user action recognition status information is data generated by converting sensor detection values ​​corresponding to user action into visualized data, and the data includes an input value recognizable area indicating an area where the recognition probability of an input value based on user action is equal to or greater than a specified threshold value.

[0236] (3) The information processing device according to (2), wherein the user action recognition state information is data including at least one of a user action point that changes in response to a user action and a trajectory of the user action point that changes in response to a user action.

[0237] (4) The information processing device according to (2) or (3), wherein the user action recognition status information is data generated by converting sensor detection values ​​corresponding to user action into three-dimensional data, and the data includes an input value recognizable area indicating an area where the recognition probability of an input value based on user action is equal to or greater than a specified threshold value.

[0238] (5) The information processing device according to any one of (2) to (4), wherein the user action recognition state information is data including a recognition probability of an input value based on a user action.

[0239] (6) The information processing device according to (5), wherein the recognition probability of the input value based on the user's action is data displayed as a pie chart or numerical values.

[0240] (7) The information processing device according to any one of (2) to (6), wherein the user action recognition state information is data that displays a recognition probability of an input value in association with a part of the user performing the user action.

[0241] (8) The information processing device according to (7), wherein the recognition probability of the input value based on the user's action is data displayed as a pie chart or numerical values.

[0242] (9) The information processing device according to any one of (1) to (8), wherein the data processing unit executes a process of identifying an input value based on the movement of a user's finger, and the user action recognition status information is information indicating a recognition status of the movement of the user's finger.

[0243] (10) The information processing device according to any one of (1) to (9), wherein the user action recognition status information is input from at least one of a myoelectric sensor or a motion sensor attached to the user, an image captured by a camera capturing an image of the user, or a touch sensor, based on a sensor detection value corresponding to the user action.

[0244] (11) The information processing device according to any one of (1) to (10), wherein the data processing unit generates the user action recognition state information and outputs it to a display unit when executing a training process for input processing based on a user action.

[0245] (12) The information processing device according to (11), wherein the data processing unit outputs a message indicating input text when executing the training process, and outputs a message indicating whether input according to the message was successful.

[0246] (13) The information processing device according to (12), wherein the data processing unit generates and outputs explanatory information regarding correct user actions for making input in accordance with the message if the user fails to make input in accordance with the message when executing the training process.

[0247] (14) The information processing device according to any one of (1) to (13), wherein the data processing unit executes a setup process for generating user action recognition state information unique to a user.

[0248] (15) The information processing device according to (14), wherein the data processing unit, in the setup process, sequentially causes the user to perform user actions corresponding to input data, and sequentially determines input value recognizable areas indicating areas where the recognition probability of the input value based on the user actions is equal to or greater than a specified threshold value.

[0249] (16) The information processing device according to (14) or (15), wherein the data processing unit stores user-specific user action recognition state information generated in the setup process in a storage unit, and executes processing to display the information on a display unit when executing input processing based on user action by the user after the setup process.

[0250] (17) The information processing device according to any one of (1) to (16), wherein the data processing unit executes control to change the display position of the user action recognition status information in accordance with the user's line of sight, the user's head direction, or a user action.

[0251] (18) The information processing device according to any one of (1) to (17), wherein the data processing unit executes the process of generating the user action recognition state information and outputting it to the display unit in parallel with the process of identifying an input value based on a user action.

[0252] (19) An information processing method executed in an information processing device, in which a data processing unit executes a process of identifying an input value based on a user action, and a process of generating user action recognition status information indicating a recognition status of the user action and outputting the information to a display unit.

[0253] (20) A program for executing information processing in an information processing device, the program causing a data processing unit to execute a process of identifying an input value based on a user action, and a process of generating user action recognition status information indicating the recognition status of the user action and outputting it to a display unit.

[0254] The series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program on a computer from a recording medium, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.

[0255] The various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are all located in the same housing.

[0256] As described above, according to a configuration of an embodiment of the present disclosure, a device and a method are realized that enable a device capable of input processing based on user motion to check whether the device correctly recognizes an input value based on the user motion. Specifically, for example, the device includes a data processing unit that identifies an input value based on the user motion. The data processing unit further generates and displays user motion recognition status information indicating a recognition status of the user motion. The user motion recognition status information is data generated by converting sensor detection values ​​corresponding to the user motion into visualized data such as three-dimensional data, and includes an input value recognition region where the recognition probability of the input value based on the user motion is equal to or greater than a specified threshold, the recognition probability of the input value, the user motion point, etc. This configuration realizes a device and a method that enable a device capable of input processing based on user motion to check whether the device correctly recognizes an input value based on the user motion.

[0257] 10 User 20 Smartphone 30 PC 40 AR Glasses 50 Hand 60 Myoelectric Sensor 61 Sensor Detection Value Input Unit 62 Myoelectric-Input Value Conversion Unit 63 User Action Recognition State Information Generation Unit 71 User Action Recognition State Visualization Data Generation Unit 72 User Action Recognition Probability Calculation Unit 75 Display Information Generation Unit 81 User Action Point Visualization Data Generation Unit 82 Input Value Recognizable Area Visualization Data Generation Unit 90 User Action Recognition State Information 91 Text Input Field 92 Training Character Display Area 93 User Input Character Display Area 94 Message Display Area 100 Motion Sensor 101 Camera 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / Output Interface 306 Input Unit 307 Output Unit 308 Storage Unit 309 Communication Unit 310 Drive 311 Removable Media

Claims

1. An information processing device having a data processing unit that executes an input value identification process based on a user action, wherein the data processing unit generates user action recognition status information indicating a recognition status of the user action and outputs the information to a display unit.

2. The information processing device according to claim 1, wherein the user action recognition status information is data generated by converting sensor detection values ​​corresponding to user action into visualized data, and the data includes an input value recognizable area indicating an area where the recognition probability of an input value based on user action is equal to or greater than a specified threshold value.

3. The information processing device according to claim 2, wherein the user action recognition state information is data including at least one of a user action point that changes in response to a user action, or a trajectory of a user action point that changes in response to a user action.

4. The information processing device according to claim 2, wherein the user action recognition status information is data generated by converting sensor detection values ​​corresponding to user action into three-dimensional data, and the data includes an input value recognizable area indicating an area where the recognition probability of an input value based on user action is equal to or greater than a specified threshold value.

5. The information processing device according to claim 2, wherein the user action recognition state information is data further including a recognition probability of an input value based on the user action.

6. The information processing device according to claim 5, wherein the recognition probability of the input value based on the user's action is data displayed as a pie chart or numerical values.

7. The information processing device according to claim 2, wherein the user action recognition state information is data that displays a recognition probability of an input value in association with a part of the user performing the user action.

8. The information processing device according to claim 7, wherein the recognition probability of the input value based on the user's action is data displayed as a pie chart or numerical values.

9. The information processing device according to claim 1, wherein the data processing unit executes a process for identifying an input value based on the movement of a user's finger, and the user action recognition status information is information indicating the recognition status of the movement of the user's finger.

10. The information processing device according to claim 1, wherein the user action recognition status information is input from at least one of a myoelectric sensor worn by the user, a motion sensor, an image captured by a camera capturing the user, or a touch sensor, based on sensor detection values ​​corresponding to the user action.

11. The information processing device according to claim 1, wherein the data processing unit generates the user action recognition state information and outputs it to a display unit when executing a training process for input processing based on user action.

12. An information processing device according to claim 11, wherein the data processing unit outputs a message indicating input text when executing the training process, and outputs a message indicating whether input according to the message was successful.

13. The information processing device according to claim 12, wherein the data processing unit generates and outputs explanatory information regarding the correct user actions to make input in accordance with the message if the user fails to make input in accordance with the message when executing the training process.

14. The information processing device according to claim 1, wherein the data processing unit executes a setup process for generating user action recognition state information specific to a user.

15. An information processing device as described in claim 14, wherein the data processing unit, during the setup process, performs a process of having the user sequentially perform user actions corresponding to input data, and sequentially determining input value recognizable areas indicating areas where the recognition probability of the input value based on the user actions is equal to or greater than a specified threshold value.

16. The information processing device according to claim 14, wherein the data processing unit stores user-specific user action recognition status information generated in the setup process in a memory unit, and executes processing to display the information on a display unit when an input process based on user action by the user is executed after the setup process.

17. The information processing device according to claim 1, wherein the data processing unit executes control to change the display position of the user action recognition status information in accordance with the user's line of sight, the user's head direction, or a user action.

18. The information processing device according to claim 1, wherein the data processing unit executes the process of generating the user action recognition status information and outputting it to the display unit in parallel with the process of identifying an input value based on a user action.

19. An information processing method executed in an information processing device, in which a data processing unit executes a process of identifying an input value based on a user action, and a process of generating user action recognition status information indicating the recognition status of the user action and outputting it to a display unit.

20. A program for executing information processing in an information processing device, which causes a data processing unit to execute a process for identifying input values ​​based on user actions, and a process for generating user action recognition status information indicating the recognition status of the user action and outputting it to a display unit.

Images