Processing program, processing device, and robot operation system

Abstract

This processing program causes a computer to function as: a storage device for storing feature data including a plurality of coordinate points indicating features of a plurality of electromyographic signals acquired as a first motion of a user (R) giving an instruction to a robot (H), and a plurality of coordinate points indicating features of a plurality of electromyographic signals acquired as a second motion of the user (R) giving an instruction to the robot (H); and an instruction unit (132) for giving, upon an electromyographic signal of a new motion of the user (R) being input, an instruction to the robot (R) on the basis of whether or not coordinate points indicating features of the electromyographic signal of the new motion are included in a region including the coordinate points of the first motion and the coordinate points of the second motion.

Description

Processing program, processing device and robot operation system

[0001] The present disclosure relates to a processing program, a processing device, and a robot operation system.

[0002] There is a myoelectric prosthetic arm that receives a user's myoelectric potential as input and outputs a motion signal for the electrically powered prosthetic arm (Patent Document 1). In Patent Document 1, the robot learns the correspondence between biosignals and robot movements without using training data.

[0003] In Patent Document 1, an action identifier is associated with a class that classifies an action signal. The robot is driven according to the action identifier associated with the action signal. If this association is easy for the user to use, the association is maintained, but if it is difficult to use, the association is eliminated.

[0004] Biofeedback is a technique that detects phenomena caused by involuntary muscles, converts them into a form that people can recognize, and makes them aware of them, allowing them to consciously control them.

[0005] One example of the use of biofeedback is a method of displaying the amplitude of signals acquired from an electromyographic sensor during training to operate a myoelectric prosthetic hand (Non-Patent Document 1). According to the method described in Non-Patent Document 1, a person can visually grasp which muscles are contracting and to what extent, and can consciously practice isolating muscle contractions.

[0006] International Publication No. 2021 / 141127

[0007] Cosima Prahm and two others, "MyoBeatz: Using music and rhythm to improve prosthetic control in a mobile game for health," [online], August 2019, ResearchGate, [Retrieved July 28, 2023], Internet <URL: https: / / www.researchgate.net / profile / Fares-Kayali / publication / 336875833_MyoBeatz_Using_music_and_rhythm_to_improve_prosthetic_control_in_a_mobile_game_for_health / links / 5e42983e92851c7f7f2f4a0c / MyoBeatz-Using-music-and-rhythm-to-improve-prosthetic-control-in-a-mobile-game-for-health.pdf?origin=publication_detail>

[0008] Patent Document 1 discloses randomly associating action identifiers with classes that classify action signals.

[0009] The present disclosure has been made in consideration of the above circumstances, and a purpose of the present disclosure relates to a technology for driving a robot by associating an appropriate action identifier with a user's myoelectric potential.

[0010] A processing program according to one aspect of the present disclosure causes a computer to function as a memory unit that stores, in a storage device, feature data including a plurality of coordinate points indicating the features of a plurality of electromyography signals acquired as a first action of a user instructing a robot, and a plurality of coordinate points indicating the features of a plurality of electromyography signals acquired as a second action of the user instructing the robot, and an instruction unit that, when an electromyography signal of a new action of a user is input, instructs the robot based on whether or not a region including the coordinate points of the first action and the coordinate points of the second action includes coordinate points indicating the features of the electromyography signals of the new action.

[0011] A processing device according to one aspect of the present disclosure includes a storage device that stores feature data including a plurality of coordinate points indicating features of a plurality of electromyography signals acquired as a first action of a user to instruct a robot, and a plurality of coordinate points indicating features of a plurality of electromyography signals acquired as a second action of the user to instruct the robot, and an instruction unit that, when an electromyography signal of a new action of the user is input, instructs the robot based on whether or not a region including the coordinate points of the first action and the coordinate points of the second action includes coordinate points indicating features of the electromyography signals of the new action.

[0012] A robot operation system according to one aspect of the present disclosure includes a robot and a processing device electrically connected to the robot and configured to drive the robot based on a user's electromyography signals, the processing device including a memory device configured to store feature data including a plurality of coordinate points indicating features of a plurality of electromyography signals acquired as a first action of the user to instruct the robot, and a plurality of coordinate points indicating features of a plurality of electromyography signals acquired as a second action of the user to instruct the robot, and an instruction unit configured to, when an electromyography signal of a new action of the user is input, instruct the robot based on whether or not a region including the coordinate points of the first action and the coordinate points of the second action includes coordinate points indicating features of the electromyography signals of the new action.

[0013] According to the present disclosure, it is possible to provide a technology for driving a robot by associating an appropriate action identifier with a user's myoelectric potential.

[0014] FIG. 1 is a schematic diagram of a robot operation system according to the present disclosure. FIG. 2 is a diagram illustrating functional blocks of a processing device according to a first embodiment. FIG. 3 is a diagram illustrating an example of the data structure and data of score data. FIG. 4 is a diagram illustrating an example of the data structure and data of area data according to the first embodiment. FIG. 5 is a diagram illustrating processing by a learning unit according to the first embodiment. FIG. 6 is a flowchart illustrating learning processing by the learning unit according to the first embodiment. FIG. 7 is a flowchart illustrating determination processing by a determination unit according to the first embodiment. FIG. 8 is a diagram illustrating functional blocks of a processing device according to a second embodiment. FIG. 9 is a diagram illustrating an example of the data structure and data of area data according to the second embodiment. FIG. 10 is a diagram illustrating processing by a learning unit according to the second embodiment. FIG. 11 is a flowchart illustrating learning processing by the learning unit according to the second embodiment. FIG. 12 is a flowchart illustrating determination processing by the determination unit according to the second embodiment. FIG. 13 is a diagram illustrating functional blocks of a processing device according to a third embodiment. FIG. 14 is a diagram illustrating an example of the data structure and data of signal data according to the third embodiment. FIG. 15 is a diagram illustrating an example of the data structure and data of score data according to the third embodiment. FIG. 16 is a diagram illustrating an example of the data structure and data of class data according to the third embodiment. FIG. 17 is an example of a scatter diagram displaying scores for each action type according to the third embodiment. FIG. 18 is an example of a scatter diagram displaying scores input in real time according to the third embodiment. FIG. 19 is an example of a scatter diagram displaying evaluation values ​​of scores input in real time according to the third embodiment. FIG. 20 is a diagram illustrating processing for referencing evaluation values ​​using cosine similarity according to the third embodiment. FIG. 21 is a diagram illustrating processing for referencing evaluation values ​​using Euclidean distance according to the third embodiment. FIG. 22 is a flowchart illustrating processing according to the third embodiment. FIG. 23 is a diagram illustrating the hardware configuration of a computer used in the processing device.

[0015] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

[0016] (Robot Operation System) A processing device 100 according to an embodiment of the present invention converts biosignals generated in association with the activity of a user R into operation signals that instruct the operation of a robot. The processing device 100 is used, for example, in a robot operation system 5 shown in Fig. 1. In the embodiment of the present invention, the biosignals are myoelectric potentials, and the robot is a myoelectric prosthetic arm H.

[0017] As shown in Figure 1, the will of user R is transmitted as a nerve signal to the desired muscle cell of user R, causing the muscle cell to contract. The contraction of the muscle cell is detected as myoelectric potential using an electromyographic sensor or the like. The processing device 100 detects the transition of the myoelectric potential in the muscle and converts the change in the myoelectric potential into an operation signal for the robot to drive the myoelectric prosthetic arm H.

[0018] Before the user R operates the robot, the processing device 100 learns the correspondence between changes in myoelectric potential and movement signals for the myoelectric prosthetic hand H. The user R inputs the movement type of the myoelectric prosthetic hand H and performs the muscle movement that is to be associated with that movement type. When the robot is a myoelectric prosthetic hand H, the movement types of the myoelectric prosthetic hand H include rest, open palm, gripping strength grasp, precision grasp, lateral grasp, wrist palmar flexion, wrist dorsiflexion, wrist pronation, and wrist supination. The movement types differ depending on the robot.

[0019] The processing device 100 associates and stores the movement types input by the user R with the myoelectric potential signals measured from the user R. The processing device 100 learns the relationship between the movement types and the myoelectric potential signals.

[0020] When learning is complete, the processing device 100 instructs the myoelectric prosthetic hand H to perform an action in accordance with the correspondence between the learned action types and the myoelectric signal. The user R performs muscle movements corresponding to the action type that the user R wants the myoelectric prosthetic hand H to perform. More specifically, the user R reproduces the muscle movements that were performed when the user R input the action type that the user R wants the myoelectric prosthetic hand H to perform during learning. The processing device 100 identifies the action type that corresponds to the myoelectric signal from the user R based on the relationship between the learned action types and the myoelectric signal. The processing device 100 instructs the myoelectric prosthetic hand H of the identified action type. This allows the myoelectric prosthetic hand H to be driven as the user R wishes.

[0021] The processing device 100 according to the present disclosure learns the correspondence between the movement of muscle cells by the user R and the robot's operation intended by the user R. The processing device 100 outputs an operation signal for driving the robot in accordance with the learned correspondence, and the processing device 100 can improve the operability of the myoelectric prosthetic hand H by the user R.

[0022] In the embodiment of the present invention, an example of a robot will be described as a myoelectric prosthetic arm H, but the robot is not limited to this. For example, the robot can be converted into various robots, such as a prosthetic arm or leg that compensates for a missing part of a human body, an industrial robot that performs a predetermined operation, a nursing robot, an agricultural robot, or the like.

[0023] In the embodiment of the present invention, the processing device 100 is described as being located in the same space as the user R and the robot, but this is not limiting. For example, when the user R operates a robot located in a remote location, the processing device 100 may be located in a different space from the user R or the robot, or from the user R and the robot. In addition, the processing device 100 is described as converting biosignals into operation signals for a robot so that one user R can operate one robot, but the processing device 100 may convert biosignals into operation signals for a robot for multiple pairs of user R and robot.

[0024] The processing device 100 according to the first embodiment and the processing device 100a according to the second embodiment described in this disclosure will explain a process of associating the motion of the robot with a myoelectric potential signal input to the robot by operation of the user R. In this disclosure, the robot corresponds to the myoelectric prosthetic hand H.

[0025] The processing device 100 and the processing device 100a store in the storage device feature data indicating features of the myoelectric potential signals of the user R. For example, the feature data includes a plurality of coordinate points indicating features of a plurality of myoelectric potential signals acquired as a first movement of the user R instructing the robot, and a plurality of coordinate points indicating features of a plurality of myoelectric potential signals acquired as a second movement of the user instructing the robot.

[0026] In the present disclosure, feature data is shown as score data 111 of the processing device 100 shown in FIG. 2 or score data 111 of the processing device 100a shown in FIG. 8, but is not limited to this. Furthermore, in the present disclosure, coordinate points may be referred to as scores. In the present disclosure, coordinate points or scores are examples of names of data converted from myoelectric potential signals input to a robot by operation by user R for use in processing related to the present disclosure. Coordinate points have any dimension. In the present disclosure, scores are expressed in two dimensions, but may be expressed in three or more dimensions as long as the number of dimensions is recognizable by user R.

[0027] The processing device 100 and the processing device 100a each have an instruction unit that, when a myoelectric potential signal of a new movement of the user R is input, gives instructions to the robot based on whether or not a coordinate point indicating the characteristics of the myoelectric potential signal of the new movement is included in an area including the coordinate points of the first movement and the coordinate points of the second movement.

[0028] The instruction section corresponds to the instruction section 132 of the processing device 100 shown in FIG. 2 or the instruction section 132a of the processing device 100a shown in FIG.

[0029] First Embodiment A processing device 100 according to a first embodiment will be described with reference to Fig. 2. In the first embodiment, the processing device 100 generates a convex hull of an EMG signal for each motion type of the robot and learns the correspondence between the motion type and the convex hull area. The processing device 100 identifies the motion of the robot based on which motion type's convex hull the input EMG signal is contained in.

[0030] The processing device 100 includes score data (feature data) 111 and area data 112, and the functions of a learning unit 120 and a determination unit 130. Each piece of data is stored in a storage device such as a memory 902 or a storage 903. Each function is implemented in a CPU 901.

[0031] The score data 111 includes a plurality of two-dimensional scores converted from a plurality of myoelectric potential signals. The score data 111 includes at least a plurality of myoelectric potential signals acquired as a first movement instructed to the robot and a plurality of myoelectric potential signals acquired as a second movement instructed to the robot.

[0032] The score data (feature data) 111 includes a plurality of coordinate points indicating the features of a plurality of myoelectric potential signals acquired as a first action of the user instructing the robot, and a plurality of coordinate points indicating the features of a plurality of myoelectric potential signals acquired as a second action of the user instructing the robot.

[0033] Specifically, as shown in Fig. 3, the score data 111 is data that associates two-dimensional scores (coordinate points) converted from the myoelectric potential signals input by the user R with the types of movements specified by the user R when inputting the myoelectric potential signals and the times of acquisition. The two-dimensional score includes a first component and a second component. The score data 111 has a plurality of data sets including the score, the type of movement, and the time of acquisition. The score data 111 may associate an acquisition order, which specifies the order in which the myoelectric potential signals were acquired, instead of the time of acquisition.

[0034] The region data 112 is data that identifies the convex hull of the scores for each action type and the tabu region where the two convex hulls overlap in two-dimensional coordinates that expand the two-dimensional scores. The region data 112 includes, for example, information that identifies the first convex hull, the second convex hull, the third convex hull, as well as the first tabu region and the second tabu region, as shown in FIG. 4 . In the present disclosure, a convex hull is the smallest convex polygon that encompasses all of the specified points. A tabu region is the overlapping region of two convex hulls.

[0035] In Figure 4, the region "first convex hull" specifies a convex hull that includes each two-dimensional score corresponding to the first action type in two-dimensional coordinates. The region "second convex hull" specifies a convex hull that includes each two-dimensional score corresponding to the second action type in two-dimensional coordinates. The region "first tabu region" is an overlapping region of the convex hull corresponding to the first action type and the convex hull corresponding to the second action type in two-dimensional coordinates.

[0036] The learning unit 120 learns the association between the myoelectric potential signal input by the user R and the action type. The action type is an action instructed to the robot. The learning unit 120 includes a first acquiring unit 121 and a setting unit 122.

[0037] The first acquisition unit 121 acquires a myoelectric potential signal and a type of movement corresponding to the myoelectric potential signal through an operation by the user R. The myoelectric potential signal is treated as a feature vector identified from the myoelectric potential acquired at a predetermined time. The feature vector is a vector converted from a time-series signal using FFT (First Fourier Transformation) or the like. The myoelectric potential signal may be detected from one location or from multiple locations.

[0038] The first acquiring unit 121 converts the acquired myoelectric potential signal into a two-dimensional score. The first acquiring unit 121 inputs the converted two-dimensional score and the movement type corresponding to the score to the setting unit 122.

[0039] The first acquiring unit 121 may calculate the two-dimensional score by any method. In the present disclosure, the two-dimensional score converted by the first acquiring unit 121 is, for example, a score obtained by normalizing the values ​​of a first principal component and a second principal component obtained by principal component analysis of a multidimensional myoelectric potential signal, but is not limited to this.

[0040] Here, an example of the process of converting a myoelectric potential signal into a two-dimensional score will be described with reference to equations (1) to (10).

[0041] Equation (1) shows the definition of the D-dimensional feature quantity obtained from the myoelectric potential signal acquired by the first acquisition unit 121.

[0042]

[0043] When feature amounts are acquired and stored n times for one action type, the set of feature amounts for the first action type is shown in formula (2). The first acquisition unit 121 acquires feature amounts n times for each of the action types to be instructed to the robot.

[0044]

[0045] Equation (3) defines a set of all feature amounts acquired by the first acquisition unit 121.

[0046]

[0047] Next, the first acquiring unit 121 calculates the average value of the matrix Z defined by the formula (3) using the formula (4) and calculates the standard deviation using the formula (5).

[0048]

[0049]

[0050] The first acquiring unit 121 calculates a matrix Z′ by normalizing the matrix Z using the average value calculated by equation (4) and the standard deviation calculated by equation (5). The elements of the matrix Z′ are normalized feature quantities.

[0051]

[0052] The first acquisition unit 121 applies principal component analysis (PCA) to the matrix Z′ of the normalized feature quantities calculated by equation (6), where equation (7) calculates the variance matrix ss.

[0053]

[0054] The first obtaining unit 121 calculates the correlation matrix R from the variance matrix of equation (7) using equation (10).

[0055]

[0056] The first obtaining unit 121 performs eigenvalue decomposition on the correlation matrix R using equation (12).

[0057]

[0058] The first obtaining unit 121 calculates a matrix Zf of scores after principal component analysis and normalization for each feature using equation (10). Note that the matrix Zf is a D×n matrix, similar to Z in equation (3) and Z′ in equation (6).

[0059]

[0060] The first acquisition unit 121 converts each element of the matrix Zf into a two-dimensional score including the value of the first principal component and the value of the second principal component using Equation (10). Specifically, the first acquisition unit 121 sets the scores corresponding to the eigenvectors v1 and v2 of each element of the matrix Zf as the value of the first principal component and the value of the second principal component, respectively. The first acquisition unit 121 inputs the two-dimensional score of each element to the setting unit 122 in association with the movement type. The first acquisition unit 121 may also input the time or order of acquisition of the myoelectric potential signals used to calculate each score to the setting unit 122 in association with the scores.

[0061] The first acquisition unit 121 inputs two-dimensional scores for n×k myoelectric potential signals to the setting unit 122. Here, n is the number of feature amounts acquired for one movement type, and k is the number of movement types instructed to the robot.

[0062] The first acquisition unit 121 may further acquire new myoelectric potential signals and movement types and calculate new scores. For example, the first acquisition unit 121 converts the newly acquired myoelectric potential feature amount into a two-dimensional score using equation (11). The first acquisition unit 121 inputs the converted score, movement type, acquisition time, etc. to the setting unit 122. At this time, the first acquisition unit 121 uses the average value and standard deviation when calculating the feature amounts for n×k myoelectric potential signals. The average value is calculated in advance using equation (4). The standard deviation is calculated in advance using equation (5).

[0063]

[0064] The first acquisition unit 121 normalizes the newly acquired feature quantity using equation (11) and then generates a score including the values ​​of the first principal component and the second principal component. The score calculated by the first acquisition unit 121 is processed by the setting unit 122.

[0065] The setting unit 122 sets the overlapping region of a first convex hull set for multiple coordinate points of a first motion and a second convex hull set for multiple coordinate points of a second motion as a tabu region.

[0066] Specifically, the setting unit 122 sets the overlapping area of ​​a first convex hull set for multiple scores (coordinate points) of a first action and a second convex hull set for multiple scores of a second action in two-dimensional coordinates as a taboo area.

[0067] The processing of the setting unit 122 will be described with reference to FIG.

[0068] The setting unit 122 associates the score, the action type, and the time of acquisition acquired from the first acquisition unit 121 and inserts them into the score data 111 .

[0069] As shown in Fig. 5(a), the setting unit 122 arranges each two-dimensional score for the myoelectric prosthetic hand H, which is a robot, in two-dimensional coordinates. The x-axis of the two-dimensional coordinates is the first component of the two-dimensional score, which in the present disclosure is the first principal component of the principal component analysis. The y-axis of the two-dimensional coordinates is the second component of the two-dimensional score, which in the present disclosure is the second principal component of the principal component analysis.

[0070] 5(b), the setting unit 122 sets a convex hull of each two-dimensional score for each movement type of the myoelectric prosthetic hand H. The convex hull for each movement type is shown in Equation (11).

[0071]

[0072] FIG. 5B includes a first convex hull P1 containing the scores for the first action type and a second convex hull P2 containing the scores for the second action type.

[0073] The setting unit 122 sets a taboo region where the convex hulls for the two action types overlap, as shown in Fig. 5(c). Fig. 5(b) generates a taboo region P12 where the first convex hull P1 and the second convex hull P2 overlap. The taboo region where the set of scores for action type a and the set of scores for action type b overlap is shown in equation (13). The taboo region where the scores for action type a and the scores for action type b overlap is specified by the logical product of the set of scores for action type a and the set of scores for action type b.

[0074]

[0075] The setting unit 122 associates a region identifier with information that identifies the region for each of the set convex hulls and tabu regions, and generates the region data 112 shown in FIG.

[0076] When a new score is input from the first acquisition unit 121, the setting unit 122 determines whether to insert the new score into the score data 111. If the new score is contained in the first convex hull P1 for the first action, the setting unit 122 discards the new score. Even if the new score is placed in the two-dimensional coordinates, no change occurs to the already generated first convex hull P1.

[0077] On the other hand, if the new score for the first movement is not contained in the first convex hull P1, the setting unit 122 discards the oldest score from the score data 111 when an EMG signal was acquired, and updates the score data 111 by inserting the new score into the score data 111. The setting unit 122 refers to the updated score data 111 and resets the first convex hull P1 and tabu region P12.

[0078] Figure 5(c) shows the two-dimensional coordinates after the new score T is set. Since the new score T is outside the first convex hull P1 shown in Figure 5(b), the setting unit 122 inserts the new score T into the score data 111 to update it. The setting unit 122 refers to the updated score data 111 and updates it to the first convex hull P1' and tabu region P12'.

[0079] The learning process according to the first embodiment will be described with reference to FIG.

[0080] First, in step S101, the setting unit 122 expands each score input from the first acquisition unit 121 into a two-dimensional coordinate system. In the two-dimensional coordinate system, the first component and the second component of each score are set as the horizontal axis and the vertical axis, respectively.

[0081] In step S102, the setting unit 122 sets a first convex hull P1 that encloses the score for the first action type. In step S103, the setting unit 122 sets a second convex hull P2 that encloses the score for the second action type. In step S104, if the first convex hull P1 and the second convex hull P2 overlap, the setting unit 122 sets the overlapping region as a tabu region P12. The setting unit 122 stores each region set in the processing of steps S102 to S104 in the region data 112.

[0082] In step S105, when a new score for the first action type is input, the setting unit 122 determines whether or not the new score is included in the first convex hull P1 set in step S102.

[0083] If the new coordinate point of the first action is not contained in the first convex hull, in step S106 the setting unit 122 discards the oldest coordinate point from the feature data (score data) that was used to acquire the user's action, adds the new coordinate point of the first action to the feature data, and resets the first convex hull and tabu region by referring to the feature data.

[0084] Specifically, if the first convex hull P1 does not include the first action type, in step S106 the setting unit 122 discards the oldest score for the first action type in the score data 111 and inserts the new score into the score data 111, thereby updating the score data 111. The setting unit 122 expands each score in the updated score data 111 into two-dimensional coordinates. The setting unit 122 returns to step S102, resets the first convex hull, tabu region, etc., and updates the region data 112.

[0085] In step S105, if the point is included in the first convex hull P1, in step S107 the setting unit 122 discards the new score and waits for input of a new score.

[0086] In the example shown in FIG. 6, the process when a new score is input for the first action will be described, but the process is similar when a new score is input for the second action.

[0087] In this way, the setting unit 122 first sets a convex hull for each action type, and if two convex hulls overlap, sets a tabu region. When a new score for a predetermined action type is input, if the new score is not contained in a convex hull created in advance for that action type, the setting unit 122 discards the oldest score in the convex hull created in advance and instead generates a new convex hull taking the new score into consideration.

[0088] In the first embodiment, the learning unit 120 specifies a range of myoelectric potentials corresponding to each movement type, reflecting the latest myoelectric potential signals input by the user R. When learning by the learning unit 120 is completed, a determination is made by the determination unit 130. Note that the learning unit 120 may construct a learning model using the movement types and myoelectric potential signal data sets input by the user R as training data, or may tune a learning model that has been trained in advance using predetermined training data, based on the movement types and myoelectric potential signal data sets input by the user R.

[0089] The determination unit 130 refers to the area data 112 learned by the learning unit 120 and identifies the type of movement from the myoelectric potential signal input by the user R. The determination unit 130 instructs the robot to perform the identified type of movement, thereby driving the robot. The determination unit 130 includes a second acquisition unit 131 and an instruction unit 132.

[0090] The second acquisition unit 131 acquires an EMG signal input as an instruction to the robot and converts the acquired EMG signal into a two-dimensional score. The second acquisition unit 131 converts the feature quantities of the multidimensional EMG signal into a standardized multidimensional score using equation (11). The second acquisition unit 131 further converts the standardized multidimensional score into a two-dimensional score using the values ​​of the first principal component and the second principal component as elements. The second acquisition unit 131 inputs the two-dimensional score to the instruction unit 132.

[0091] When the EMG signal of a new movement of user R is input, the instruction unit 132 instructs the robot based on whether or not the area including the coordinate points of the first movement and the coordinate points of the second movement includes coordinate points that indicate the characteristics of the EMG signal of the new movement.

[0092] Specifically, the instruction unit 132 refers to the area data 112 and determines the type of robot movement based on the two-dimensional score input from the second acquisition unit 131, and converts the determined type of movement into a movement signal for the robot.

[0093] The instruction unit 132 instructs the robot to perform a first action if the coordinate point of the new action is contained in the first convex hull but not contained in the taboo region, instructs the robot to perform a second action if the coordinate point is contained in the second convex hull but not contained in the taboo region, and determines that the action is impossible if it is contained in the taboo region.

[0094] Specifically, the instruction unit 132 instructs the robot to perform an action of a first action type when a two-dimensional score converted from an EMG signal input as an instruction to the robot is contained in the first convex hull P1 but not contained in the taboo region P12. The determination unit 130 instructs the robot to perform an action of a second action type when the two-dimensional score is contained in the second convex hull P2 but not contained in the taboo region P12. The determination unit 130 determines that the action is impossible when the two-dimensional score is contained in the taboo region P12.

[0095] If the score input from the second acquisition unit 131 is contained in the convex hull corresponding to a predetermined action type and is not contained in any taboo region, the determination unit 130 determines that the instruction from the user R is an instruction for that predetermined action type. The determination unit 130 generates an action signal corresponding to that predetermined action type and inputs it to the robot.

[0096] If the score input from the second acquisition unit 131 is included in any of the taboo areas, the determination unit 130 determines that the myoelectric potential signal input from the user R cannot be determined and that the determination is impossible. The determination unit 130 prompts the user R to re-input or to re-learn. At this time, the determination unit 130 may re-learn so as to reduce the taboo areas.

[0097] The determination process according to the first embodiment will be described with reference to FIG.

[0098] In step S151, the determination unit 130 acquires a new score and determines the action type of the new score.

[0099] In step S152, it is determined whether the new score is included in the first convex hull P1 but not included in the taboo region P12. If the new score is included in the first convex hull P1 but not included in the taboo region P12, in step S153 the determination unit 130 instructs the robot to perform an action of the first action type.

[0100] In step S154, it is determined whether the new score is included in the second convex hull P2 but not included in the taboo region P12. If the new score is included in the second convex hull P2 but not included in the taboo region P12, in step S155 the determination unit 130 instructs the robot to perform an action of the second action type.

[0101] If the new score is not included in either the first convex hull P1 or the second convex hull, or if it is included in either taboo region P12, the determination unit 130 determines that the operation is impossible in step S156. The determination unit 130 may instruct a notification unit (not shown) to notify the user R that the operation is impossible. The notification unit is, for example, a speaker, a display, or the like.

[0102] The processing device 100 according to the first embodiment can reflect the latest myoelectric potential signal input by the user R, learn the range of myoelectric potential corresponding to each type of movement, associate the identifier of an appropriate movement with the myoelectric potential of the user R, and drive the robot.

[0103] Second Embodiment A processing device 100a according to a second embodiment will be described with reference to Fig. 8. In the first embodiment, the processing device 100a acquires myoelectric potential signals for each type of movement of the robot, arranges the myoelectric potential signals in space, and learns the correspondence between the most frequent type of movement in each region and the region. The processing device 100a identifies the robot's movement based on the most frequent type of movement for the myoelectric potential signals in the region to which the input myoelectric potential signals belong.

[0104] The processing device 100a shown in Fig. 8 has the same configuration as the processing device 100 according to the first embodiment shown in Fig. 2, but differs in the region data 112a, the setting unit 122a, and the instruction unit 132a. The other data and processing units are as described in the first embodiment.

[0105] Region data 112a according to the second embodiment will be described with reference to Fig. 9. In the second embodiment, the two-dimensional coordinates are divided into a plurality of regions specified by a region of the first component and a region of the second component, as shown by the dashed lines in Fig. 10. The region data 112a specifies, for each region, the region of the first component that serves as the horizontal axis and the region of the second component that serves as the vertical axis.

[0106] In the present disclosure, a case where two-dimensional coordinates are divided into rectangular regions will be described, but this is not limiting. The regions may be formed in any manner as long as it is possible to combine nearby scores into one region.

[0107] The setting unit 122a of the learning unit 120a divides the coordinates including the plurality of coordinate points of the first movement and the plurality of coordinate points of the second movement into a plurality of regions.

[0108] Specifically, the setting unit 122a divides the two-dimensional coordinates whose axes are the first and second components of the two-dimensional score input from the first acquisition unit 121 into a plurality of regions specified by each section of the first and second components. The setting unit 122a generates region data 112a by associating the identifiers of each region with the sections of the first and second components that specify each region.

[0109] In this case, the setting unit 122a sets the maximum and minimum values ​​of the first component of each score included in the score data 111 as the maximum and minimum values ​​of the x-axis of the two-dimensional coordinate system, and sets the maximum and minimum values ​​of the second component as the maximum and minimum values ​​of the y-axis of the two-dimensional coordinate system. By dividing the x-axis direction into a plurality of zones and the y-axis direction into a plurality of zones, the setting unit 122a can divide the two-dimensional coordinate system into a plurality of regions as shown in FIG. 10 .

[0110] When the x-axis and y-axis directions are divided into N regions, the two-dimensional coordinates are divided into N squared regions. Each region is defined by four vertices shown in equation (14).

[0111]

[0112] The setting unit 122a generates multiple regions in two-dimensional coordinates based on the scores acquired from the first acquisition unit 121, and then acquires new scores from the first acquisition unit 121. If the number of scores included in the score data 111 is the upper limit value, the setting unit 122a deletes the oldest acquired score from the score data 111, inserts the new score, and updates the score data 111. If the number of scores included in the score data 111 is not the upper limit value, the setting unit 122a inserts the new score into the score data 111 and updates the score data 111.

[0113] The setting unit 122a generates new two-dimensional coordinates and divides the area into a plurality of new areas by referring to the updated score data 111. The setting unit 122a updates the area data 112a for the new plurality of areas.

[0114] The scores belonging to each region in such two-dimensional coordinates are considered to have similar characteristics, and it is presumed that these scores correspond to myoelectric potential signals for a common type of movement.

[0115] The learning process according to the second embodiment will be described with reference to FIG.

[0116] First, in step S201, the setting unit 122a generates two-dimensional coordinates in which the range of values ​​of the first component of each score in the score data 111 is set on the horizontal axis and the range of values ​​of the second component is set on the vertical axis.

[0117] In step S202, the setting unit 122a divides the two-dimensional coordinates generated in step S201 into a plurality of regions, and generates region data 112a for identifying each region.

[0118] In step S203, the setting unit 122a obtains a new score. In step S204, the setting unit 122a determines whether the number of scores in the score data 111 is the upper limit value.

[0119] If it is the upper limit value, in step S205, the setting unit 122a discards the oldest score in the score data 111, inserts the new score, and updates the score data 111. If it is not the upper limit value, in step S206, the setting unit 122a inserts the new score and updates the score data 111.

[0120] When the score data 111 is updated, in step S201, the setting unit 122a references the updated score data 111 and generates two-dimensional coordinates again.

[0121] In the second embodiment, the learning unit 120a identifies, as an area, a range of feature quantities having similar feature quantities, reflecting the latest myoelectric potential signal input by the user R. When learning by the learning unit 120a is completed, a determination is made by the determination unit 130a. Note that the learning unit 120 may construct a learning model using the data set of movement types and myoelectric potential signals input by the user R as training data, or may tune a learning model that has been trained in advance using predetermined training data, based on the data set of movement types and myoelectric potential signals input by the user R.

[0122] When a myoelectric potential signal of a new movement of the user R is input, the coordinate points of the new movement converted from the myoelectric potential signal are input to the instructing unit 132a of the determining unit 130a from the second acquiring unit 131. The instructing unit 132a identifies the region to which the coordinate points of the new movement belong from among the multiple regions, and instructs the robot based on the number of coordinate points of the first movement and the number of coordinate points of the second movement that belong to the identified region.

[0123] Specifically, the instructing unit 132a of the determining unit 130a identifies an area to which two-dimensional scores including first and second components converted from an EMG signal input as an instruction to the robot belong. For each score in the score data 111 belonging to the identified area, if the score corresponding to the first action type is high, the instructing unit 132a instructs the robot to perform an action corresponding to the first action type. If the score corresponding to the second action type is high, the instructing unit 132a instructs the robot to perform an action corresponding to the second action type.

[0124] When the number of scores corresponding to the first action type and the number of scores corresponding to the second action type are the same for each score belonging to the identified area, the instructing unit 132a may determine the action type by taking into consideration the history of score acquisition in the score data 111. For example, the instructing unit 132a instructs the robot to perform the action type corresponding to the most recently acquired score, or the action type with the most recently acquired score out of multiple scores.

[0125] The determination process according to the second embodiment will be described with reference to FIG.

[0126] In step S251, the determination unit 130a acquires a new score and determines the action type of the new score.

[0127] In step S252, the determination unit 130a identifies the region defined by the region data 112a to which the new score acquired in step S251 belongs, and determines which actions the other scores belonging to the identified region relate to.

[0128] If the score for the first action is high, the process proceeds to step S253. In step S253, the determination unit 130a instructs the robot to perform the first action. If the score for the second action is high, the process proceeds to step S254. In step S254, the determination unit 130a instructs the robot to perform the second action.

[0129] The processing device 100a according to the second embodiment can reflect the latest myoelectric potential signal input by the user R, learn a range of similar myoelectric potentials, associate the user R's myoelectric potential with an appropriate action identifier, and drive the robot.

[0130] (Third embodiment) A processing device 100b according to a third embodiment will be described with reference to Fig. 13. In the third embodiment, a training mode for training the user R in operating the myoelectric prosthetic hand H will be described.

[0131] Non-Patent Document 1 only displays the amplitude of the signal. Non-Patent Document 1 cannot display multiple indices of the myoelectric potential signal. Therefore, the processing device 100b according to the third embodiment displays multiple indices of the myoelectric potential signal in a manner that is easy for the user to recognize.

[0132] Note that each unit of the processing device 100b shown in Fig. 13 may be additionally implemented in at least one or both of the processing device 100 shown in Fig. 2 and the processing device 100a shown in Fig. 8, or may be implemented in a device different from these. Furthermore, among the units of the processing device 100b shown in Fig. 13, those that are similar or identical to the units of the processing device 100 shown in Fig. 2 or the processing device 100a shown in Fig. 8 may be similar or identical to the units of the processing device 100 shown in Fig. 2 or the processing device 100a shown in Fig. 8.

[0133] The processing device 100 b includes a learning phase unit 610 , a real-time input unit 620 , and a display unit 630 .

[0134] In the learning phase in which learned class data 616 is formed, the learning phase unit 610 learns the scores and movement types corresponding to the myoelectric potential signals presented by the user R, and identifies the scores corresponding to each movement type.

[0135] After the learning is completed, the real-time input unit 620 converts the myoelectric potential signal 622 presented by the user R into a score 624. The real-time input unit 620 may be used when learning a new class in the learning phase. Alternatively, the real-time input unit 620 may be used when the user R trains the muscle movements corresponding to a desired movement type in the estimation phase in which an movement type is estimated from the learned class data 616.

[0136] When an EMG signal of a new movement of user R is input in training mode, the display unit 630 displays a plurality of coordinate points corresponding to the first movement, a plurality of coordinate points corresponding to the second movement, and coordinate points indicating characteristics of the EMG signal of the new movement. The plurality of coordinate points corresponding to the first movement and the plurality of coordinate points corresponding to the second movement are the results of learning in the first embodiment or the second embodiment. In the third embodiment, the plurality of coordinate points corresponding to the first movement and the plurality of coordinate points corresponding to the second movement become teacher data for each movement type specified by the class data 616.

[0137] Specifically, in the learning phase or estimation phase, the display unit 630 displays the teacher data for each movement type identified by the class data 616, and displays the score 624 corresponding to the newly input myoelectric potential signal 622. After learning is completed, the display unit 630 may further calculate an evaluation value for the newly input myoelectric potential signal 622 and display the evaluation value. User R can evaluate the myoelectric potential signal 622 input by user R based on the relationship between the existing teacher data and the newly input score, and train muscle movements.

[0138] (Learning Phase Unit) The learning phase unit 610 is responsible for processing in the learning phase to associate movement types with myoelectric potential signals. The learning phase unit 610 includes various data such as signal data 612, score data 614, and class data 616, as well as various functional units such as a first acquisition unit 611, a first conversion unit 613, and a learning unit 615. Each piece of data is stored in a storage device such as the memory 902 or the storage 903. Each function is implemented in the CPU 901.

[0139] The signal data 612 stores myoelectric potential signals acquired from the muscle cells of the user R in association with the types of movements designated by the user R at the time of acquisition of the myoelectric potential signals. The signal data 612 may be data that associates the myoelectric potential signals, the types of movements, and the times at which the signals were acquired, as shown in Fig. 14. The signal data 612 includes at least a plurality of myoelectric potential signals corresponding to a first movement to be instructed to the robot and a plurality of myoelectric potential signals corresponding to a second movement to be instructed to the robot.

[0140] The signal data 612 may be controlled to hold only a predetermined number of signals for each movement type. When the signal data 612 already holds a predetermined number of signals for a certain movement type and a new myoelectric potential signal for that movement type is input, the processing device 100b may discard the oldest myoelectric potential signal for that movement type from the signal data 612 and insert the new myoelectric potential signal. Each myoelectric potential signal in the signal data 612 becomes a myoelectric potential signal of the teacher data for that movement type. Each myoelectric potential signal in the signal data 612 becomes a myoelectric potential signal of the teacher data for that movement type.

[0141] The score data 614 stores two-dimensional scores (coordinate points) generated from the electromyogram signals in association with the types of movements designated by the user R when the electromyogram signals were acquired. As shown in FIG. 15 , the score data 614 may be data that associates the electromyogram signals, the types of movements, and the times at which the signals were acquired. The score data 614 includes at least a plurality of two-dimensional scores converted from a plurality of electromyogram signals corresponding to first movements to be instructed to the robot, and a plurality of two-dimensional scores converted from a plurality of electromyogram signals corresponding to second movements to be instructed to the robot. Each score in the score data 614 becomes the score of the teacher data for each movement type.

[0142] The class data 616 is data that associates, for each action type, a score range belonging to that action type. As shown in FIG. 16 , the class data 616 may associate an action type identifier, a score range belonging to that action type, and a score that serves as teacher data. The score that serves as teacher data is a score obtained from an EMG signal 622 input by user R and is adopted as teacher data for that action type. The class data 616 may also associate an EMG signal corresponding to the score that serves as teacher data. The score that serves as teacher data is referenced by a display unit 630 (described later) to display the teacher data in two-dimensional coordinates. The EMG signal that serves as teacher data is referenced by an evaluation unit 631 (described later) to calculate an evaluation value.

[0143] Through the operation of the user R, a myoelectric potential signal and a type of movement corresponding to the myoelectric potential signal are acquired. The myoelectric potential signal is treated as a feature vector identified from the myoelectric potential acquired at a predetermined time. The feature vector is a vector converted from a time-series signal using FFT (First Fourier Transformation) or the like. The myoelectric potential signal may be detected from one location or multiple locations. The first acquisition unit 611 associates the acquired myoelectric potential signal and the type of movement at the time of acquisition and stores them in signal data 612.

[0144] The first conversion unit 613 converts the myoelectric potential signal acquired by the first acquisition unit 611 into a two-dimensional score. Because the myoelectric potential signal is a multidimensional signal, the first conversion unit 613 compresses the multidimensional signal into a two-dimensional signal. The first conversion unit 613 associates the converted two-dimensional score and movement type at the time of acquisition and stores them in score data 614.

[0145] In the third embodiment, a case will be described in which a D-dimensional myoelectric potential signal is converted into a two-dimensional score, but this is not limiting, and the first conversion unit 613 may convert into a score with fewer dimensions than D. In the third embodiment, the myoelectric potential signal is formed in D dimensions, where D is 3 or more.

[0146] The first conversion unit 613 may calculate a two-dimensional score from the myoelectric potential signal by any method. In the third embodiment, the two-dimensional score converted by the first conversion unit 613 is, for example, a score obtained by normalizing the values ​​of the first principal component and the second principal component obtained by principal component analysis of the multidimensional myoelectric potential signal, but is not limited to this.

[0147] Equation (15) represents the D-dimensional feature amount of the myoelectric potential signal acquired by the first acquisition unit 611.

[0148]

[0149] Equation (16) shows a matrix of features for an m-th action type when n features are stored and learned for one action type.

[0150]

[0151] Equation (17) shows the matrix Z of all input data when the number of operation types is k. The matrix Z is an n k × D matrix.

[0152]

[0153] Here, equation (18) represents a matrix obtained when the dimensions of the feature quantities of the myoelectric potential signal are compressed by the dimension reduction method adopted by the first transforming unit 613 according to the third embodiment.

[0154]

[0155] Here, the processing of the first conversion unit 613 will be described in detail. The first conversion unit 613 may standardize in advance the multiple myoelectric potential signals input as training data. Standardization allows the influence of the amount of change in each dimension to be evaluated equally.

[0156] Equation (19) shows the process of standardizing each element of matrix Z of all input data. Each element of matrix Z of all input data is defined by equation (17). The matrix of all input data after each element has been standardized is represented as Z'.

[0157]

[0158] The first conversion unit 613 will now be described as reducing the dimensions of the matrix Z′ of the entire input data after standardization. In order to ensure visibility for the user R, the case where the matrix Z′ is compressed to two dimensions will now be described.

[0159] In the third embodiment, a case will be described in which principal component analysis (PCA) is adopted as a dimension reduction technique. PCA makes it possible to efficiently reduce the dimension while maintaining the characteristic information of the original data.

[0160] Here, we show an algorithm for performing PCA on standardized input data Z'. PCA is a method for determining axes that maximize variance for high-dimensional data. PCA can be reduced to an eigenvalue problem of the covariance matrix of the original data. Equation (20) shows the covariance matrix S of the input data Z'.

[0161]

[0162] When the covariance matrix S is subjected to eigenvalue decomposition, equation (21) is obtained.

[0163]

[0164] When the input data Z' is reduced from D dimensions to two dimensions by PCA, the two largest eigenvalues ​​λ １ and λ ２ The eigenvector v corresponding to １ and v ２ become the first component and the second component, respectively.

[0165] From equation (18), the input data after dimension reduction is expressed by equation (22).

[0166]

[0167] The first conversion unit 613 stores the two-dimensional score obtained from equation (22) in score data 614 .

[0168] The learning unit 615 learns the movement types and score ranges by referring to the score data 614. The learning unit 615 executes processing when a predetermined number of training data sets are collected, or at a timing specified by the user R. When learning is complete, the score range for each movement type is stored in the class data 616. The learning unit 615 further stores the training data scores and EMG signals for each movement type in the class data 616. Note that in the third embodiment, a collection of training data belonging to one movement type may be referred to as a class. Note that the learning unit 615 may construct a learning model using a data set of movement types and EMG signals input by the user R as training data, or may tune a learning model that has been trained in advance using predetermined training data based on the data set of movement types and EMG signals input by the user R (trainer data).

[0169] (Real-time input unit) The real-time input unit 620 converts the myoelectric potential signal 622 that is sequentially input into a score 624 that can be compared with a score that is referenced during learning. The real-time input unit 620 includes various data such as the myoelectric potential signal 622 and the score 624, and various functional units such as a second acquisition unit 621 and a second conversion unit 623. Each piece of data is stored in a storage device such as the memory 902 or the storage 903. Each function is implemented in the CPU 901.

[0170] The second acquisition unit 621 acquires a myoelectric potential signal 622 acquired from the muscle cells of the user R. When a new myoelectric potential signal 622 is additionally input to define a new movement type in the learning phase, the second acquisition unit 621 may acquire the new movement type together with the myoelectric potential signal 622. In the estimation phase, the second acquisition unit 621 may acquire a myoelectric potential signal but not a movement type. In this case, the myoelectric potential signal 622 acquired by the second acquisition unit 621 is a signal that reproduces the muscle movement corresponding to the desired movement type of the user R.

[0171] Equation (23) represents the D-dimensional feature amount of the myoelectric potential signal 622 acquired by the second acquisition unit 621 at a certain time t.

[0172]

[0173] The second conversion unit 623 converts the newly input myoelectric potential signal 622 into a new two-dimensional score 624 in the same way as the first conversion unit 613. The second conversion unit 623 uses the conversion matrix V defined in equation (18). Equation (24) shows the score 624 obtained by reducing the dimension of the myoelectric potential signal 622 acquired at a certain time t.

[0174]

[0175] The second conversion section 623 can convert the myoelectric potential signal 622 input by the real-time input section 620 into a score 624 using the same calculation method as the learning phase section 610 .

[0176] (Display Unit) The display unit 630 displays the various data calculated by the learning phase unit 610 and the real-time input unit 620 on a display device and presents it to the user R.

[0177] The display unit 630 displays each score, for example, in a two-dimensional scatter plot, as shown in FIG. 17 . The display unit 630 plots each dimension of each score on the two-dimensional coordinate system, with each dimension corresponding to the x-axis and y-axis. The display unit 630 displays multiple scores corresponding to a first action with a first symbol and multiple scores corresponding to a second action with a second symbol on the two-dimensional coordinate system. FIG. 17 displays scores for three action types on the two-dimensional coordinate system. While FIG. 17 shows a case where the shape of the symbol differs for each action type, other elements of the symbol, such as color, may also be changed.

[0178] When a new myoelectric potential signal 622 is further input, the display unit 630 displays a new score 624 corresponding to the myoelectric potential signal 622 in the two-dimensional coordinate system using a third symbol as shown in Fig. 18. The new score 624 may be converted by processing by the second conversion unit 623, or may be converted by processing in which the first conversion unit 613 converts all of the accumulated signal data 612 at once.

[0179] When the electromyography signal of a new movement input in training mode is acquired as a first movement, the display unit 630 displays the coordinate points of the acquired new movement with attributes common to the multiple coordinate points corresponding to the first movement.

[0180] Specifically, the display unit 630 displays the score 624 corresponding to the newly input EMG signal 622 as a new symbol not shown in the scatter diagram on the scatter diagram displaying the first and second symbols. When the new EMG signal 622 is acquired as the first movement, the third symbol has a predetermined attribute common to the first symbol. Here, the "predetermined attribute" refers to the shape, color, pattern, etc. of the symbol. For example, if the first symbol is a "△" as shown in FIG. 17 , the third symbol is a "▲" symbol with the inside of the first symbol filled in, as shown in FIG. 18 . Note that the display unit 630 may display, as information indicating the class corresponding to the movement, a first convex hull P1 corresponding to the first movement, a second convex hull P2 corresponding to the second movement, or a line indicating a taboo region P12, as shown in FIG. 5B , or lines indicating multiple divided regions, as shown in FIG. 10 . Information indicating the class corresponding to the action is displayed over the screen shown in FIG. 17 or on a separate screen.

[0181] Furthermore, "when a new EMG signal 622 is acquired as the first movement" refers to a case where the user R specifies the first movement type together with the new EMG signal 622 in the learning phase. Alternatively, "when a new EMG signal 622 is acquired as the first movement" refers to a case where it is determined in the estimation phase that the score 624 corresponding to the new EMG signal 622 is included in the range of scores for the first movement type from the class data 616.

[0182] When displaying the new score 624 in the scatter diagram, the display unit 630 displays it using a symbol that shares some of the predetermined attributes with the symbol of the action type corresponding to the score 624. User R can visually recognize the newly displayed score in the scatter diagram and which action type the score relates to.

[0183] For example, in FIG. 18 , the "▲" symbol is a score 624 corresponding to the EMG signal 622 input with the same movement type specified as the "△" symbol. Because the "▲" symbol is located near the outer edge of the cluster of "△" symbols, user R can understand that the newly input EMG signal is slightly off-center compared to the existing training data. In this way, user R can visually understand the relative position of the newly input EMG signal in the relationship between the movement type and EMG signal input in the learning phase. Furthermore, the display unit 630 displays the area shown in FIG. 5( b) or FIG. 10, allowing user R to understand the position of the newly input EMG signal.

[0184] The display unit 630 sequentially displays the score 624 corresponding to the newly input EMG signal 622 in two-dimensional coordinates. This allows user R to evaluate the muscle movement reproduced by user R based on the relative position of the teacher data score and the newly input score 624 and reflect this in training. For example, user R can train the muscle cell movement for each movement type so that a group of scores for each movement type is formed. Alternatively, user R can train the muscle cell movement for a new movement type so that a new group of scores is formed at a position away from the existing group of scores for each movement type. Furthermore, user R can recognize the taboo region P12 shown in FIG. 5(b) and train the muscle cell movement to avoid the taboo region P12. The learning phase unit 610 may re-learn so that coordinate points identified from the EMG signal input by user R are not entered into the taboo region P12.

[0185] The display unit 630 may further include an evaluation unit 631 and a determination unit 632 .

[0186] The evaluation unit 631 calculates an evaluation value for the new myoelectric potential signal 622 based on its relationship with multiple myoelectric potential signals corresponding to a predetermined action type. The display unit 630 displays the calculated evaluation value on a display device. While in the third embodiment, a symbol calculated from the score 624 is displayed in two-dimensional coordinates, the evaluation value is calculated from the myoelectric potential signal 622. The evaluation value is calculated taking into consideration elements of dimensions that are reduced when scoring the myoelectric potential signal 622 out of the multiple dimensions of the myoelectric potential signal 622. The evaluation value can also express features of the myoelectric potential signal 622 that cannot be expressed by the score 624 alone. In other embodiments, the evaluation unit 631 may calculate the evaluation value using the score 624 with reduced dimensions.

[0187] The evaluation unit 631 may calculate an evaluation value for the new myoelectric potential signal 622, for example, from the similarity with a plurality of myoelectric potential signals corresponding to a predetermined type of movement. The predetermined type of movement is the type of movement determined from the class data 616 for the score 624, or the type of movement specified by the user R when inputting the myoelectric potential signal 622 corresponding to the score 624. The user R can check the probability that the newly input myoelectric potential signal 622 will be determined to be the type of movement that the user R intended.

[0188] When displaying the score 624 corresponding to the new myoelectric potential signal 622 on the scatter diagram, the display unit 630 displays it using a symbol corresponding to the evaluation value. For example, the higher the evaluation value, the darker the color, and the lower the evaluation value, the lighter the color. The display unit 630 displays symbols whose predetermined attributes change stepwise depending on the level of the evaluation value.

[0189] The evaluation unit 631 calculates an evaluation value for the myoelectric potential signal of the new movement from the degree of difference between the myoelectric potential signals corresponding to the first movement and the second movement. The myoelectric potential signals corresponding to the first movement and the second movement are the myoelectric potential signals corresponding to the existing movement types.

[0190] Specifically, the evaluation unit 631 may calculate an evaluation value for the new myoelectric potential signal 622 from the degree of difference between the new myoelectric potential signal 622 and a plurality of myoelectric potential signals corresponding to existing action types. The existing action types are action types that have already been defined in the class data 616. When a first action and a second action are defined in the class data 616, the existing action types are the first action and the second action. User R can refer to the evaluation value as an index of whether or not a new action type can be defined for the existing action types, or can train muscle movements for the new action type.

[0191] The determination unit 632 refers to the class data 616 and identifies the movement type to which the score 624 for the newly input myoelectric potential signal 622 belongs. The display unit 630 plots and displays the score 624 in a scatter diagram for the identified movement type.

[0192] The following describes examples of screens displayed by the display unit 630. The display unit 630 displays (1) a screen displaying teacher data in the learned class data 616, (2) a screen displaying new scores 624 for the learned classes, and (3) a screen displaying new scores 624 and evaluation values ​​for the learned classes.

[0193] First, (1) a screen displaying training data for the learned class data 616 will be described. As shown in FIG. 17 , the display unit 630 references the class data 616 and displays the training data scores for each action type in a scatter plot using symbols corresponding to the action type. The display unit 630 may further display the score range for each action type by encircling it with a circle, etc., according to the range of scores for each action type in the class data 616. For example, as information indicating the class corresponding to the action, the display unit 630 may display a first convex hull P1 corresponding to the first action, a second convex hull P2 corresponding to the second action, or a line indicating a tabu region P12, as shown in FIG. 5( b), or may display lines indicating multiple divided regions, as shown in FIG. 10. The information indicating the class corresponding to the action is displayed overlaid on the screen shown in FIG. 17 or on a separate screen.

[0194] Next, (2) a screen displaying a new score 624 for a learned class will be described. Here, a case will be described in which user R is attempting to reproduce an EMG signal 622 similar to that of an existing class. In the learning phase, the display unit 630 displays a screen in which, for each action type, the scores serving as teacher data are displayed in a scatter plot using symbols corresponding to the action type, as shown in FIG. 17 . Thereafter, the display unit 630 displays the score 624 corresponding to the newly input EMG signal 622 using a new symbol, as shown in FIG. 18 . When the EMG signal 622 corresponding to the new score 624 is input together with an existing action type, or when it is determined that the new score 624 corresponds to an existing action type, the display unit 630 displays the new score 624 using a symbol obtained by changing some of the elements of the symbol corresponding to that action type.

[0195] On screen (2), user R can evaluate the similarity with the class corresponding to the existing action type from the position of the score corresponding to the other action type and the position of the newly input score 624. The display unit 630 displays an evaluation value that evaluates the similarity, allowing user R to obtain an objective evaluation of the similarity with the existing class. User R can also train muscle movements to increase the similarity with the existing class.

[0196] Next, (3) a screen displaying a new score 624 and evaluation value for a learned class will be described. Here, a case will be described in which user R is attempting to reproduce an EMG signal 622 that differs from an existing class. The display unit 630 first displays a screen in which, for each movement type, the scores that serve as teacher data are displayed in a scatter diagram with symbols corresponding to the movement type, as shown in FIG. 17. The evaluation unit 631 of the display unit 630 refers to the class data 616 and calculates an evaluation value for the newly input EMG signal 622.

[0197] For example, the display unit 630 may display the evaluation value itself, or may change the symbol of the new score 624 depending on the evaluation value. As shown in FIG. 19 , the display unit 630 displays the score 624 corresponding to the newly input myoelectric signal 622 as a symbol that does not correspond to an existing class and is shaded corresponding to the evaluation value. Alternatively, the display unit 630 may display the evaluation value itself, such as by displaying the evaluation value inside each symbol. Note that in FIG. 19 , the shade of the symbol is represented by the shade of hatching. The symbols corresponding to the new score 624 are darker the farther they are from symbols corresponding to other action types, and lighter the closer they are to symbols corresponding to other action types.

[0198] The user R can recognize from the screens (2) and (3) whether the reproduced muscle movement is the desired motion type. The user R can train the muscle movement so that it will be determined to be the desired motion type. This allows the user R to reproduce muscle movement that will improve the recognition rate of the determination unit 632.

[0199] Next, the evaluation process by the evaluation unit 631 will be described. A case will be described in which the evaluation unit 631 calculates an evaluation value for evaluating the separability from classes corresponding to other movement types. The evaluation value calculated here is an evaluation value for evaluating the difference from the myoelectric potential signal of training data corresponding to existing movement types.

[0200] In the control for identifying patterns of EMG signals, as in the third embodiment, presenting sufficiently separated training data is important for improving the accuracy of the discrimination model. In the third embodiment, in the learning phase for learning a new class, the EMG signals measured in real time are evaluated based on the relative positional relationship between the EMG signals measured in real time and the training data of other classes presented in advance. Specifically, the evaluation unit 631 calculates an evaluation value that represents the difference between the EMG signals measured in real time and the training data of other classes presented in advance. User R can refer to this evaluation value to determine whether the EMG signals input in real time can be associated with a new movement type.

[0201] Equation (25) represents an evaluation value calculated from the D-dimensional feature amount of the myoelectric potential signal 622 acquired by the second acquisition unit 621 at a certain time t.

[0202]

[0203] Equation (25) can feed back the myoelectric potential signal 622 as an evaluation value to the user R. By inputting the myoelectric potential signal 622 to the processing device 100b while checking the evaluation value, the user R is expected to input the myoelectric potential signal 622 with higher separability.

[0204] As specific evaluation methods, a method using cosine similarity and a method using Euclidean distance will be described.

[0205] First, we will explain the method using cosine similarity. The evaluation method using cosine similarity focuses only on the "direction" of the EMG signal, without considering its "magnitude." The evaluation method using cosine similarity is based on the assumption that each movement type has a different vector in the feature space, and only the magnitude of that vector changes depending on the strength of muscle contraction. This method ignores the strength of muscle contraction and performs evaluation by considering only the difference in patterns.

[0206] This method calculates an evaluation value by assuming that data corresponding to a resting movement is located at the origin of the feature space, and that data for each movement, such as gripping and opening, spreads radially, as shown in Fig. 20. Fig. 20 shows that the resting movement is located at the origin, and displays a first class corresponding to a first movement type and a second class corresponding to a second movement type.

[0207] The myoelectric potential signal 622 newly acquired at a certain time t is defined as shown in equation (23). Fig. 20 shows the feature quantities of the myoelectric potential signals for three movement types, including rest. In Fig. 20, the ellipses for the first and second classes indicate the distribution of training data for each class. Equation (26) shows the cosine similarity between the existing class and the newly input myoelectric potential signal 622 in this case.

[0208]

[0209] The evaluation unit 631 calculates the cosine similarity for each existing class i using Equation (26) and calculates the dissimilarity for the class corresponding to the largest cosine similarity among the calculated cosine similarities. This allows the evaluation unit 631 to assign a high score when the user R moves to a position angularly distant from the existing training data. It is also possible that user R repeatedly inputs the EMG signal 622, and the new EMG signal 622 approaches another class. In this case, too, the evaluation unit 631 simply calculates the dissimilarity for the class that is closest at each time. To consistently obtain a high score, user R is required to input an EMG signal that is different from the EMG signals belonging to all other classes. User R can create a new class that is different from any class and associate it with a new movement type.

[0210] Equation (27) shows the evaluation value calculated by the evaluation unit 631. The evaluation unit 631 can adjust the degree of change in the evaluation value by adjusting the sensitivity. Equation (27) quantifies the separability from existing classes in the range of 0 to 100b, with 100b being the perfect score for the evaluation value.

[0211]

[0212] Next, a method using Euclidean distance will be described.

[0213] Unlike cosine similarity, the evaluation method using Euclidean distance simply calculates a high score when the distance in feature space is large. The evaluation method using Euclidean distance is based on the assumption that even if the muscle contraction pattern is the same, differences in movement are expressed by the strength of the contraction. This method evaluates the new myoelectric potential signal 622 from the perspective of both muscle contraction strength and pattern differences.

[0214] This method identifies the movement type with reference to the class data 616 for the score 624 corresponding to the new myoelectric potential signal 622, and calculates the distance of the new myoelectric potential signal 622 to the myoelectric potential signals of the teacher data belonging to the identified movement type. Therefore, a full score is awarded by this method when the estimation probability for all movement types defined by the class data 616 does not exceed a predetermined threshold and is treated as impossible to determine. In other words, if the probability that the new myoelectric potential signal 622 is estimated to belong to each class is low enough to be treated as impossible to determine, the evaluation unit 631 awards a full score.

[0215] A newly acquired myoelectric potential signal 622 at a certain time t is defined as shown in Equation (23). A case will be described where the determination unit 632 determines that the new myoelectric potential signal 622 belongs to the i-th class. Fig. 21 shows the relationship between the new myoelectric potential signal 622 and the myoelectric potential signals of the training data that belong to the i-th class.

[0216] Equation (28) is the evaluation value of the new myoelectric potential signal 622. The evaluation unit 631 can adjust the degree of change in the evaluation value by adjusting the sensitivity. Equation (28) quantifies the separability from existing classes in the range of 0 to 100b, with 100b being the perfect score for the evaluation value.

[0217]

[0218] (Processing Method) A processing method according to the third embodiment will be described with reference to FIG.

[0219] First, in a learning phase, the processing device 100b learns the relationship between myoelectric potential signals and movement types. In step S601, the processing device 100b acquires a plurality of myoelectric potential signals and movement types designated by the user R. In step S602, each of the myoelectric potential signals acquired in step S601 is converted into a two-dimensional score. In step S603, the processing device 100b learns the range of the score converted in step S602 for each of the movement types acquired in step S601, and stores the range in the class data 616.

[0220] In order to evaluate newly input myoelectric potential signals thereafter, the processing device 100b displays the scores for each movement type on a two-dimensional coordinate system in step S604. Here, the display device displays the scores on the two-dimensional coordinate system using different symbols for each movement type.

[0221] In step S605, the processing device 100b waits for input of a new myoelectric potential signal 622. When the new myoelectric potential signal 622 is input, in step S606, the processing device 100b converts the new myoelectric potential signal 622 into a new score 624. In step S607, the processing device 100b calculates an evaluation value from the new myoelectric potential signal 622. In step S608, the processing device 100b displays the score calculated in step S606 and the evaluation value calculated in step S607 on the display device. The processing device 100b displays the new score 624, for example, in two-dimensional coordinates using a symbol corresponding to the evaluation value.

[0222] The processing device 100b according to the present disclosure converts the multidimensional myoelectric potential signals into two-dimensional scores and displays the teacher data scores for each movement type in two-dimensional coordinates, allowing the user R to easily grasp the relationship between the teacher data for each movement type.

[0223] When a new myoelectric potential signal 622 is input, the processing device 100b displays a two-dimensional score 624 corresponding to the new myoelectric potential signal 622 in the two-dimensional coordinate system. The user R can grasp the relationship between the existing movement types and the newly input myoelectric potential signal 622.

[0224] The processing device 100b calculates and displays an evaluation value for the newly input myoelectric potential signal 622. The processing device 100b is suitable for associating a myoelectric potential signal with a new movement type by using the degree of difference from the myoelectric potential signal of training data belonging to an existing movement type as the evaluation value. The user R trains to move his or her muscles so that a myoelectric potential signal that will result in a high evaluation value can be output. The processing device 100b can identify the movement type corresponding to the myoelectric potential signal with high accuracy.

[0225] The processing device 100b may display the new myoelectric potential signal 622 with a symbol corresponding to the evaluation value. The user R can easily grasp the relationship between the scores of the existing movement types and the score of the new myoelectric potential signal 622, as well as the separability from the existing movement types.

[0226] The processing device 100 of the present embodiment described above is, for example, a general-purpose computer system including a CPU (Central Processing Unit, processor) 901, a memory 902, a storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), a communication device 904, an input device 905, and an output device 906. In this computer system, the CPU 901 executes a program such as a processing program loaded on the memory 902, thereby realizing each function of the processing device 100.

[0227] The processing device 100 may be implemented by one computer or by multiple computers, or may be a virtual machine implemented on a computer.

[0228] The processing device 100 of the present embodiment described above is, for example, a general-purpose computer system including a CPU (Central Processing Unit, processor) 901, a memory 902, a storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), a communication device 904, an input device 905, and an output device 906. In this computer system, the CPU 901 executes a program such as a processing program loaded on the memory 902, thereby realizing each function of the processing device 100.

[0229] The processing device 100 may be implemented by one computer or by multiple computers, or may be a virtual machine implemented on a computer.

[0230] The program of the processing device 100 can be stored in a computer-readable recording medium such as a HDD, an SSD, a Universal Serial Bus (USB) memory, a Compact Disc (CD), or a Digital Versatile Disc (DVD), or can be distributed via a network. The computer-readable recording medium is, for example, a non-transitory recording medium.

[0231] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0232] 5 Robot operation system 100 Processing device 111, 614 Score data 112 Area data 120, 615 Learning unit 121, 611 First acquisition unit 122 Setting unit 130, 632 Determination unit 131, 621 Second acquisition unit 610 Learning phase unit 612 Signal data 613 First conversion unit 616 Class data 620 Real-time input unit 622 EMG signal 623 Second conversion unit 624 Score 630 Display unit 631 Evaluation unit 901 CPU 902 Memory 903 Storage 904 Communication device 905 Input device 906 Output device H Myoelectric prosthetic hand R User

Claims

1. A processing program that causes a computer to function as: a memory unit that stores in a storage device feature data including multiple coordinate points indicating features of multiple myoelectric potential signals acquired as a first movement of a user instructing a robot, and multiple coordinate points indicating features of multiple myoelectric potential signals acquired as a second movement of the user instructing the robot; and an instruction unit that, when a myoelectric potential signal of a new movement of a user is input, issues instructions to the robot based on whether or not an area including the coordinate points of the first movement and the coordinate points of the second movement includes coordinate points indicating features of the myoelectric potential signals of the new movement.

2. The processing program according to claim 1, further causing the computer to function as a setting unit that sets an overlapping area of ​​a first convex hull set for a plurality of coordinate points of the first movement and a second convex hull set for a plurality of coordinate points of the second movement to a taboo area, and the instruction unit instructs the robot to perform the first movement if the coordinate points of the new movement are contained in the first convex hull but not contained in the taboo area, instructs the robot to perform the second movement if the coordinate points are contained in the second convex hull but not contained in the taboo area, and determines that the movement is impossible if the coordinate points are contained in the taboo area.

3. The processing program according to claim 2, wherein the setting unit further, if a new coordinate point of the first action is not contained in the first convex hull, discards the oldest coordinate point from the feature data that was used to obtain the user's action, adds the new coordinate point of the first action to the feature data, and resets the first convex hull and tabu region by referring to the feature data.

4. The processing program of claim 1, further causing the computer to function as a setting unit that divides coordinates including multiple coordinate points of the first movement and multiple coordinate points of the second movement into multiple areas, and the instruction unit identifies an area to which the coordinate points of the new movement belong from among the multiple areas, and instructs the robot based on the number of coordinate points of the first movement and the number of coordinate points of the second movement that belong to the identified area.

5. The processing program according to claim 1, further causing the computer to function as a display unit which, when a myoelectric signal of a new movement of the user is input in training mode, displays a plurality of coordinate points corresponding to the first movement, a plurality of coordinate points corresponding to the second movement, and coordinate points showing characteristics of the myoelectric signal of the new movement.

6. The processing program according to claim 5, wherein when the myoelectric signal of the new movement input in the training mode is acquired as the first movement, the display unit displays the coordinate points of the acquired new movement with attributes common to the multiple coordinate points corresponding to the first movement.

7. The processing program according to claim 5, further causing the computer to function as an evaluation unit that calculates an evaluation value for the myoelectric signal of the new movement from the degree of difference between the myoelectric signal of the new movement and a plurality of myoelectric signals corresponding to the first movement and the second movement, and the display unit displays the evaluation value.

8. A processing device comprising: a memory device that stores feature data including multiple coordinate points indicating features of multiple myoelectric potential signals acquired as a first movement of a user to instruct a robot, and multiple coordinate points indicating features of multiple myoelectric potential signals acquired as a second movement of the user to instruct the robot; and an instruction unit that, when a myoelectric potential signal of a new movement of a user is input, instructs the robot based on whether or not an area including the coordinate points of the first movement and the coordinate points of the second movement includes coordinate points indicating features of the myoelectric potential signals of the new movement.

9. The processing device according to claim 8, further comprising a display unit which, when a myoelectric signal of a new user movement is input in training mode, displays a plurality of coordinate points corresponding to the first movement, a plurality of coordinate points corresponding to the second movement, and coordinate points showing characteristics of the myoelectric signal of the new movement.

10. A robot operation system comprising: a robot; and a processing device electrically connected to the robot and driving the robot based on a user's myoelectric potential signal, wherein the processing device comprises: a memory device that stores feature data including a plurality of coordinate points indicating features of a plurality of myoelectric potential signals acquired as a first movement of the user to instruct the robot, and a plurality of coordinate points indicating features of a plurality of myoelectric potential signals acquired as a second movement of the user to instruct the robot; and an instruction unit that, when a myoelectric potential signal of a new movement of the user is input, gives instructions to the robot based on whether or not an area including the coordinate points of the first movement and the coordinate points of the second movement includes coordinate points indicating features of the myoelectric potential signals of the new movement.

Images