Apparatus for detecting gesture or method therefor

Abstract

An apparatus configured to obtain hand gesture information from a sensor signal is disclosed according to an embodiment of the present invention, and the apparatus may comprise: a plurality of electromyography (EMG) sensors arranged in a loop structure; an inertial measurement unit (IMU) sensor including an acceleration sensor and an angular velocity sensor; and a processor that generates a virtual signal for a first EMG sensor from among the plurality of EMG sensors on the basis of signals of the remaining EMG sensors from among the plurality of EMG sensors or a signal of the IMU sensor, and obtains the hand gesture information on the basis of the signals of the remaining EMG sensors from among the plurality of EMG sensors and the virtual signal.

Description

Device for gesture detection or method therefor

[0001] The present invention relates to a device for gesture detection or a method therefor, and more particularly, to a device for detecting gestures or training a gesture inference model using a gesture inference model learned or trained using a learning data set composed of sensor data or a method therefor.

[0002] Gesture recognition technology using electromyography (EMG) sensors measures electrical signals generated by the user's muscle contractions to determine specific movements or gestures. It is being utilized in a variety of fields, including rehabilitation therapy, sports analysis, human-machine interfaces (HMI), and wearable device control. In particular, surface electromyography (sEMG) can noninvasively measure signals through electrodes attached to the skin surface, offering high user convenience and widespread application in wearable sensing devices.

[0003] However, gesture recognition using surface electromyography sensors has several limitations. One of these is the degradation of recognition performance due to changes in the wearing position of wearable sensing devices, such as arm bands. Changes in the wearing position alter the pattern of the EMG signals measured by the sensor, which can lead to different interpretations of the same gesture, significantly reducing the accuracy of gesture recognition. This problem is further complicated by the differences in arm length, muscle structure, and wearing habits across users.

[0004] Therefore, there is a need for robust gesture recognition technology that can maintain high recognition accuracy regardless of the position of a wearable sensing device. The present invention addresses this need by providing technology capable of reliably recognizing gestures without relying on the position of a wearable device, such as an armband.

[0005] The present invention aims to address the above-described problems of the prior art. Specifically, the present invention seeks to provide technology that guarantees gesture recognition performance regardless of the wearer's position.

[0006] In addition, the present invention seeks to provide a technology for highly reliable gesture recognition by using signals from the remaining sensors when some sensors of a wearable device do not operate.

[0007] The problems to be solved by the present invention are not limited to the problems to be solved above, and other problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0008] According to one embodiment of the present invention, a device configured to obtain hand gesture information from a sensor signal is disclosed, the device including a plurality of EMG (Electromyography) sensors arranged in a loop structure; an IMU (Inertial Measurement Unit) sensor including an acceleration and angular velocity sensor; and a processor configured to generate a virtual signal for a first EMG sensor among the plurality of EMG sensors based on a signal of the remaining EMG sensors among the plurality of EMG sensors or a signal of the IMU sensor, and obtain hand gesture information based on the signal of the remaining EMG sensors among the plurality of EMG sensors, the signal of the IMU sensor, and the virtual signal.

[0009] Additionally or alternatively, the processor may be configured to generate the virtual signal based on whether a signal from the first EMG sensor is not received or the reliability of the signal from the first EMG sensor is below a preset value.

[0010] Additionally or alternatively, the remaining EMG sensors may be configured to include EMG sensors adjacent to the first EMG sensor.

[0011] Additionally or alternatively, the device may further comprise a gesture inference model configured to obtain the hand gesture information from the signal of the EMG sensor or the virtual signal.

[0012] Additionally or alternatively, the gesture inference model may include a rotation data augmenter, a feature extractor, a gesture label classifier, a gradient reverse layer, and a domain separator, wherein the rotation data augmenter obtains an additional data set using a rotation transformation for a pre-trained data set, the feature extractor is configured to extract features for inferring hand gesture information from signals of the remaining EMG sensors among the plurality of EMG sensors, signals of the IMU sensor, and the virtual signal, the gesture label classifier is configured to obtain gesture information from the extracted features, the gradient reverse layer is for inducing domain confusion during backward propagation for training the gesture inference model, and the domain separator may be configured to obtain domain information from the extracted features.

[0013] Additionally or alternatively, the processor may be configured to train the gesture inference model based on a global loss function including a gesture loss function associated with weight parameters of the feature extractor and the label classifier and a domain loss function associated with weight parameters of the domain discriminator.

[0014] Additionally or alternatively, the overall loss function can be obtained by subtracting the result of multiplying the domain loss function by a constant for adjusting the strength for domain confusion from the gesture loss function.

[0015] Additionally or alternatively, the rotation data augmenter, the gesture label classifier, the gradient reverse layer and the domain separator may be configured not to be used in the inference phase for obtaining the hand gesture information.

[0016] Additionally or alternatively, the device further includes a signal generation model for generating the virtual signal, and the number of input channels of the signal generation model may be greater than the number of output channels of the signal generation model, or the number of sensor signals input to the signal generation model may be greater than the number of virtual signals output from the signal generation model.

[0017] Additionally or alternatively, the processor may be configured to calculate a reliability of the acquired hand gesture information, output the acquired hand gesture information if the reliability exceeds a preset value, and not output the acquired hand gesture information if the reliability is below the preset value.

[0018] In another embodiment of the present invention, a method is proposed, performed by a device configured to obtain hand gesture information from a sensor signal, the method comprising: generating a virtual signal for a first EMG sensor among a plurality of EMG sensors based on a signal of a remaining EMG sensor among a plurality of EMG sensors or a signal of an IMU sensor; and obtaining hand gesture information based on a signal of a remaining EMG sensor among the plurality of EMG sensors, a signal of the IMU sensor, and the virtual signal.

[0019] Also, according to another embodiment of the present invention, a non-transitory computer-readable storage medium is proposed that stores code configured to be executed by a computer or processor, wherein the code is performed by one or more processors.

[0020] The above problem solving methods are only some of the embodiments of the present invention, and various embodiments reflecting the technical features of the present invention can be derived and understood by a person having ordinary knowledge in the relevant technical field based on the detailed description of the present invention described below.

[0021] The present invention has the following technical effects.

[0022] The present invention can ensure gesture recognition performance regardless of the wearing position of the wearable device.

[0023] In addition, the present invention can provide highly reliable gesture recognition by using signals from the remaining sensors when some sensors of a wearable device do not operate.

[0024] The effects according to the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the detailed description of the invention below.

[0025] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and, together with the detailed description, explain the technical idea of ​​the present invention.

[0026] Figure 1 illustrates a wearable device according to the present invention.

[0027] FIG. 2 illustrates the types of gestures obtained based on sensor signals measured through a wearable device according to the present invention.

[0028] Figure 3 (a) shows the basic wearing position and rotation state (i.e., the basic state) of the wearable device, and Figures 3 (b) and (c) show examples of a state in which the wearable device is rotated from the basic state (Rot) and a state in which the wearable device is rotated and position-shifted from the basic state (Rot+D), respectively.

[0029] FIG. 4 illustrates an inference phase or inference model for obtaining hand gesture information from a sensor signal according to the present invention.

[0030] FIG. 5 illustrates a training or learning phase for inference of obtaining hand gesture information from sensor signals according to the present invention.

[0031] FIG. 6 illustrates a scenario in which signals from some sensors of a wearable device according to the present invention are used to estimate signals from other sensors.

[0032] FIG. 7 illustrates a scenario for estimating a sensor signal according to rotation of a wearable device using signals from some sensors of the wearable device according to the present invention.

[0033] Figure 8 illustrates a scenario in which signals from some sensors of a wearable device according to the present invention are used to estimate signals from other sensors.

[0034] FIG. 9 illustrates the structure of a gesture inference model based on a plurality of sensors and a virtual sensor or virtual sensor signals generated based on signals from a plurality of sensors according to the present invention.

[0035] Figure 10 illustrates a flowchart of a gesture inference method according to the present invention.

[0036] Figure 11 illustrates a system for gesture inference according to the present invention.

[0037] FIG. 12 illustrates a block diagram of a wearable device for training a gesture inference or gesture recognition model according to the present invention.

[0038] FIG. 13 illustrates a block diagram of a device for training a gesture inference or gesture recognition model according to the present invention.

[0039] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0040] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0041] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0042] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0043] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0044]

[0045] Figure 1 illustrates a wearable device according to the present invention.

[0046] The wearable device (100) has a plurality of sensors (1 to 8), and the plurality of sensors (1 to 8) are arranged in a loop structure, whereby the sensors are arranged in a structure adjacent to each other.

[0047] Each of the plurality of sensors may include an electromyography (EMG) sensor for noninvasively measuring muscle electrical activity. An EMG sensor is a type of biosignal sensor that detects electromyographic signals generated when specific muscles in the human body contract or relax, and converts them into electrical signals.

[0048] These EMG sensors can measure minute electrical potentials generated in muscles through electrodes attached to the skin surface. The measured signals are then analyzed through processes such as amplification, filtering, and digital conversion. Incorporating EMG sensors into wearable devices allows users to obtain EMG data in real time during their daily lives without the need for additional, complex equipment. For example, by embedding EMG sensors in smart bands, smartwatches, bracelet-type devices, and clothing-integrated devices, users can continuously monitor their muscle activity patterns.

[0049] EMG sensors can be used in a variety of fields, including rehabilitation training, sports performance analysis, fitness monitoring, prosthetic limb control, and gesture recognition. In particular, their integration with wearable devices offers the advantage of enhancing device portability and convenience, and enabling them to be implemented in a form suitable for long-term wear.

[0050] Each of the plurality of sensors (1 to 8) illustrated in FIG. 1 may be referred to as one channel, and accordingly, the wearable device (100) according to the present invention may be configured as 8 channels.

[0051] The wearable device (100) may include an IMU (Inertial Measurement Unit) sensor. The IMU sensor is a sensor module for measuring the motion state and posture of an object, and generally includes an accelerometer and a gyroscope, and in some implementations, may additionally include a magnetometer.

[0052] Accelerometers detect linear acceleration acting on a device, allowing them to measure the device's direction of movement, changes in its velocity, and the direction of gravity. Gyroscopes detect the device's angular velocity, allowing them to measure its rotational direction and changes in its velocity. Magnetometers detect the Earth's magnetic field, allowing them to measure the device's absolute heading or to compensate for accumulated errors in accelerometers and gyroscopes.

[0053] IMU sensors can estimate the device's position, velocity, and 3D orientation based on data collected from the sensors using sensor fusion algorithms (e.g., Kalman Filter, Madgwick Filter, etc.). This allows wearable devices to track the movements of the user's arms, wrists, fingers, legs, etc. in real time, and perform various functions such as gesture recognition, motion analysis, exercise measurement, and posture correction.

[0054] Meanwhile, although each sensor in FIG. 1 is illustrated as having three electrodes, the present invention is not limited thereto. That is, each of the plurality of sensors of the wearable device (100) according to the present invention may be configured to include one or more electrodes.

[0055]

[0056] FIG. 2 illustrates the types of hand gestures obtained based on sensor signals measured through a wearable device according to the present invention.

[0057] Supination - the movement of rotating the forearm outward,

[0058] Pronation - the movement of rotating the forearm inward,

[0059] Extension - the movement of straightening the fingers,

[0060] Flexion - the movement of bending the fingers inward.

[0061] Rock - Refers to the action of clenching a fist.

[0062] The hand gesture illustrated in FIG. 2 is only an example, and other hand gestures may be configured to be recognized based on sensor signals measured through the wearable device according to the present invention.

[0063]

[0064] Figure 3 (a) shows the basic wearing position and rotation state (i.e., the basic state) of the wearable device, and Figures 3 (b) and (c) show examples of a state in which the wearable device is rotated from the basic state (Rot) and a state in which the wearable device is rotated and position-shifted from the basic state (Rot+D), respectively.

[0065] Referring to (a) of FIG. 3, the “source” data set wearing position is illustrated. The “source” data set wearing position represents the wearing position when the first channel of the wearable device (100) is aligned with the subject’s middle finger.

[0066] Figures 3(b) and (c) illustrate the “target” data set wearing position in contrast to the “source” data set wearing position. In this specification, the “source” data set wearing position may be referred to as the “source domain.” The source domain corresponds to the “wearing position of the wearable device” where the subject wears the wearable device (100) and acquires sensor data after making a hand gesture to be classified, i.e., acquires the learning data set used to train or learn the inference model.

[0067] Additionally, in this specification, the “target” data set wearing position may be referred to as a “target domain”, which corresponds to the “wearing position of the wearable device” that actually causes the subject to perform hand gesture recognition after wearing the wearable device (100), i.e., perform inference through the inference model.

[0068] Referring to (b) of FIG. 3, the default state, the “Rot” data set wearing position, is illustrated. The left side shows a state in which the wearable device (100) is rotated clockwise from the “source” data set wearing position, and the right side shows a state in which the wearable device (100) is rotated counterclockwise from the “source” data set wearing position, and the arrow indicates the direction of rotation.

[0069] Referring to (c) of FIG. 3, the “Rotation+Translation (Rot+D)” data set wearing position is illustrated. The left side shows a state in which the wearable device (100) is rotated counterclockwise from the “source” data set wearing position and moved away from the subject’s torso, and the right side shows a state in which the wearable device (100) is rotated clockwise from the “source” data set wearing position and moved away from the subject’s torso, and the arrows indicate the rotation and movement directions.

[0070] In this way, the wearing position of the wearable device (100) can continuously change depending on the subject or the wearing period. Such changes in the wearing position cause a deterioration in the performance of EMG data (or signal). The deterioration in the performance of the EMG signal reduces the performance of hand gesture recognition based on the EMG signal. Therefore, the present invention proposes a technology that can implement hand gesture recognition with good performance regardless of the wearing position of the wearable device (100). In other words, the present invention proposes a method that can obtain the hand gesture recognition (inference) result in the target domain described above equivalent to the hand gesture recognition (inference) result in the source domain.

[0071]

[0072] FIG. 4 illustrates an inference phase or inference model for obtaining hand gesture information from a sensor signal according to the present invention.

[0073] Sensor signals can be input from the EMG sensor and / or IMU sensor of the wearable device (100).

[0074] Features are acquired through feature extraction (S410) for the input sensor signal, and the acquired features can be input to a label classifier (S420). Feature extraction is a process of extracting features for inferring hand gesture information from the input sensor signal.

[0075] Label classification (S420) is an operation that extracts a class of hand gestures using extracted features as input. The class of hand gestures is, for example, information indicating a predefined hand gesture as illustrated in Fig. 2.

[0076] As a result of the inference phase, class labels of hand gestures are obtained.

[0077] Below, we describe a method for training or learning an inference model.

[0078] The training or learning of the inference model according to the present invention utilizes rotational augmentation that rotates the input sensor signal clockwise or counterclockwise, which enables the inference model to provide robust performance without degradation in various cases of rotation, i.e., changes in the wearing position. In addition, the training or learning of the inference model according to the present invention uses unsupervised domain adaptation to make the inference model robust to changes in PD (i.e., changes in the wearing position moving away from or closer to the subject's torso, as illustrated in (c) of FIG. 3), and to provide performance without any difference or distinction between the source domain and the target domain.

[0079] FIG. 5 illustrates a training or learning phase for inference of obtaining hand gesture information from sensor signals according to the present invention.

[0080] The sensor signal of the wearable device (100) is preprocessed (S510), the preprocessed signal is rotationally augmented (S520), the rotationally augmented signal is feature extracted (S530), and the extracted features are input to gesture label classification (S540) and domain classification (S560), so that a hand gesture class label and a domain label (i.e., a source domain or a target domain, or a location where the wearable device (100) is worn) can be obtained, respectively. The extracted features pass through a GRL (gradient reversal layer) (S550) before being domain distinguished (S560).

[0081] Preprocessing (S510), rotation augmentation (S520), feature extraction (S530), gesture label classification (S540), and domain classification (S560) can correspond to a preprocessor, a rotation data augmentation unit, a feature extractor, a gesture label classifier, and a domain classifier, respectively, and these are logical modules, and can be implemented in hardware configurations with a processor such as a GPU (graphic processing unit), a CPU (central processing unit), an NPU (neural processing unit), or an AP (application processor).

[0082] Preprocessing is a procedure for segmenting sensor signals obtained from a wearable device (100).

[0083] More specifically, this has limitations due to constraints such as the subject's (human) reaction speed and attention span, which correspond to the start cue corresponding to the beginning of a gesture and the end cue corresponding to the end of a gesture during the sensor signal acquisition process. These limitations can lead to unnecessary gesture information being included in the training or learning of the inference model. To extract only valid gesture information, the validly labeled portion can be segmented based on the mean absolute value (MAV). Specifically, as follows:

[0084] 1) When collecting a data set corresponding to each gesture, i.e., during the sensor signal acquisition process, a threshold is set by multiplying the MAV by the weight for each channel.

[0085] 2) Calculate MAV for each time window by sliding the window method for each channel. The earliest time point exceeding the set threshold is extracted as the start point (t_start), and the latest time point is extracted as the end point (t_end).

[0086] 3) For each channel, calculate MAV_ratio using the mathematical formula below.

[0087]

[0088]

[0089] 4) The start and end points of each channel with a MAV_ratio greater than 70% must be determined, which sets the gesture range to the widest time interval that includes channels with high MAV_ratio values.

[0090] 5) If the maximum value of MAV_ratio is less than the threshold, the original time interval is used.

[0091] Accordingly, the preprocessed sensor signal can have data including as many valid gestures of the subject as possible.

[0092] Rotation augmentation is intended to improve the robustness of an inference model against clockwise or counterclockwise rotation of a wearable device (100). It is proposed to generate rotated data to enhance data diversity. Specifically, rotation augmentation rotates previously acquired original EMG signals and original IMU signals by θ to generate or obtain EMG(θ) signals and IMU(θ) signals. Then, the signals generated or obtained through rotation are added or merged into a data set for learning or training. In addition, EMG signals are linearly interpolated, and IMU signals are multiplied by a rotation matrix to reconstruct the signals.

[0093] The rotated EMG signal is expressed by the following mathematical equation, which includes interpolating the signals of two adjacent channels at the target channel location. That is, referring to (c) of Fig. 3, the rotated EMG signal of channel 8 of the target domain can be obtained using channel 1 and channel 8 of the source domain (however, PD movement / change is required separately).

[0094]

[0095]

[0096]

[0097] In the above mathematical formula, ch represents the channel index of the EMG signal, θ represents the rotated angle, and N represents the number of channels of the armband.

[0098] The rotated IMU signal is generated according to the mathematical formula below, which involves multiplying the IMU signal by a rotation matrix R(θ) to obtain a three-dimensional coordinate system along the roll direction.

[0099]

[0100]

[0101] R(θ) in the roll direction can be changed depending on the axis configuration of the IMU sensor of the wearable device (100), and R(θ) in the case of rotation in the x-axis direction and rotation in the y-axis direction can be expressed as follows.

[0102]

[0103]

[0104]

[0105]

[0106] Feature extraction is designed to extract temporal features from sensor signals. For this purpose, a long short-term memory (LSTM) can be utilized. Feature extraction receives sensor signals—EMG and IMU signals (including rotationally augmented signals)—at different sampling rates and processes them individually by passing them through an extraction network using two streams. The data from the two streams is then concatenated.

[0107] The extraction networks corresponding to EMG and IMU signals share the same structure, differing only in the number of input channels. The input sensor signals are configured to pass through a Bi-LSTM layer followed by a batch normalization layer. To concatenate the extracted features, they are passed through a global average pooling layer for dimension matching.

[0108] Model weights for feature extraction are shared regardless of the domain of the sensor signal.

[0109] As previously explained, the present invention proposes unsupervised domain adaptation to address domain differences arising from various factors. The source and target domain data sets differ in terms of the wearer's position.

[0110] The present invention proposes the use of adversarial learning to minimize the distance between the source and target domains. Adversarial learning is widely used in deep learning for domain adaptation. As illustrated in Figure 5, extracted features are input to both hand gesture label classification (S540) and domain classification (S560). Hand gesture label classification infers the class of the gesture, and domain classification determines whether the extracted features belong to the source or target domain.

[0111] During the training phase of the inference model (i.e., the back-propagation step), the gradient reverse layer (GRL) propagates the gradient of the domain discriminator to the parallel term, which causes domain confusion.

[0112] In particular, the overall loss function L is composed of the gesture loss function L_c and the domain loss function L_d, and can be expressed by the following mathematical formula.

[0113]

[0114]

[0115] The gesture loss function includes a cross-entropy loss function for gesture labels. The gesture loss function can be expressed mathematically as follows.

[0116]

[0117]

[0118] The domain loss function includes binary cross-entropy for domain labels. The domain loss function can be expressed mathematically as follows:

[0119]

[0120]

[0121]

[0122] In the above mathematical equations, α is a constant controlling the strength of domain confusion, CEL is the cross-entropy loss function, and p is the predicted probability distribution. x and y represent the input sensor signal and gesture label, respectively, and D_s and D_t represent the data sets of the source domain and the target domain, respectively. θ_f, θ_c, and θ_d represent the weight parameters of the feature extractor, gesture label classifier, and domain classifier, respectively. θ_f and θ_c are learned or trained to minimize the overall loss function, and θ_d is learned or trained to maximize the overall loss function.

[0123] The learning or training of the inference model proposed above ensures that the extracted features possess high discriminative power and high domain similarity for gesture labels. This reduces the distance between the source and target domains and, through the synergy of rotational augmentation, enhances the robustness of the wearable device to changes in its wearing position.

[0124]

[0125] Meanwhile, as illustrated in FIG. 5, the sensor signals input to the preprocessing (S511) in FIG. 5 may include sensor signals of the target domain. That is, in the learning or training phase of the inference model, not only the sensor signals of the source domain but also the sensor signals of the target domain may be included in the learning data set. In conclusion, the learning data set of the inference model may include sensor signals of the source domain (measured using a wearable device), sensor signals of the target domain, and signals obtained by rotationally augmenting the sensor signals of the source / target domains.

[0126] Additionally, although not illustrated in FIG. 5, if there are multiple target domains, i.e., multi-target domains, a domain separator may be added. In this case, the added domain separator may be placed in parallel with the illustrated domain separator (S560).

[0127] The following table shows the accuracy of the inference results according to learning or training of the inference model of the present invention.

[0128] BASE represents the results according to the inference model that does not apply rotation augmentation and domain adaptation, DA represents the results according to the inference model that applies only domain adaptation, and AUG represents the results according to the inference model that applies only rotation augmentation. Rot and Rot+D represent the wearing position / state of the wearable device. Rot represents an attempt at gesture recognition by wearing the device in a rotated state compared to the initial state (i.e., source domain) (see (b) of Fig. 3), and Rot+D represents an attempt at gesture recognition by wearing the device in a rotated and position-shifted state compared to the initial state (i.e., source domain) (see (c) of Fig. 3).

[0129]

[0130] MethodVariationRotRot+DBASE33.97%74.65%DA43.61%74.35%AUG78.06%85.98%Invention82.91%89.57%

[0131]

[0132] FIG. 6 illustrates a scenario in which signals from some sensors of a wearable device according to the present invention are used to estimate signals from other sensors.

[0133] Referring to FIG. 6, the wearable device (100) is configured with 8 channels and thus includes 8 EMG sensors.

[0134] As shown, it is proposed to estimate the sensor signal of channel 1 using the sensor signals of channels 2 and 8. This can be utilized to acquire the sensor signal of the corresponding channel using the sensor signals of other channels when the sensor signal of at least one of the eight channels of the wearable device (100) is not acquired or the reliability of the sensor signal is below a preset value.

[0135]

[0136] FIG. 7 illustrates a scenario for estimating a sensor signal according to rotation of a wearable device using signals from some sensors of the wearable device according to the present invention.

[0137] Referring to FIG. 7, the wearable device (100) is configured with 8 channels and thus includes 8 EMG sensors.

[0138] As shown, it is proposed to estimate the sensor signal of channel 9 using the sensor signals of channels 1 and 2. Channel 9 may include the sensor signal of the “target domain” where channel 0 (clockwise rotation) or channel 1 (counterclockwise rotation) will be located as the wearing position of the wearable device (100) rotates.

[0139] Accordingly, depending on the rotation of the wearable device (100), sensor signals of 8 other rotated channels can be obtained from sensor signals before rotation.

[0140] Alternatively, estimating the sensor signal of channel 9 using the sensor signals of channels 1 and 2 can also be utilized to expand the 8-channel structure of the wearable device (100) to an 8*N (N is an integer greater than or equal to 2) channel structure. That is, even if the wearable device (100) is not rotated, the accuracy of hand gesture inference can be increased by acquiring a virtual sensor signal between two adjacent channels based on the sensor signals therebetween for channel expansion.

[0141]

[0142] Figure 8 illustrates a scenario in which signals from some sensors of a wearable device according to the present invention are used to estimate signals from other sensors.

[0143] Referring to FIG. 8, the wearable device (100) is configured with eight channels and thus includes eight EMG sensors. In addition, although not shown, the wearable device (100) includes an IMU sensor.

[0144] As shown, it is proposed to estimate the sensor signal of channel 1 using the sensor signals of channels 2 to 8. This can be utilized to acquire the sensor signal of the corresponding channel using the sensor signals of other channels when the sensor signal of at least one of the eight channels of the wearable device (100) is not acquired or the reliability of the sensor signal is below a preset value.

[0145] Additionally, by combining the sensor signal of the IMU sensor with the EMG sensor signal, the EMG sensor signal of channel 1 can be obtained.

[0146]

[0147] The sensor signals acquired in FIGS. 6 to 8 can be used in the inference or training / learning phase using the inference model described in FIGS. 4 to 5. In the inference phase, if the sensor signals of some channels of the EMG sensor of the wearable device (100) are not acquired or the reliability is below a preset value, the EMG sensor signals for the corresponding channels can be supplemented.

[0148]

[0149] Below, we will describe a more specific process and structure for acquiring a sensor signal of a specific channel from the sensor signals of some channels.

[0150] FIG. 9 illustrates the structure of a system for gesture inference based on a plurality of sensors and a virtual sensor or virtual sensor signal generated based on signals of a plurality of sensors according to the present invention.

[0151] The system (1) may include a plurality of sensors (11, 12, 13), a gesture inference model (20) and a sensor signal generation model (30).

[0152] One of the plurality of sensors (11, 12, 13) may include a multi-channel EMG sensor, and the others may include IMU sensors. The number of the plurality of sensors does not limit the scope of the present invention.

[0153] Sensor signals from multiple sensors (11, 12, 13) can be input to a sensor signal generation model (30).

[0154] Additionally, sensor signals from multiple sensors (11, 12, 13) and virtual sensor signals obtained from the sensor signal generation model (30) can be input to the gesture inference model (20).

[0155] As previously described, the sensor signal generation model (30) can be configured to acquire virtual sensor signals for sensor signal restoration or sensor channel expansion when sensor signals for some channels are missing or their reliability is below a preset value. Sensor signal restoration provides a means to respond to various cases, such as sensor electrode failure or loss of contact with the subject's skin.

[0156] The gesture inference model (20) is described with reference to FIGS. 4 and 5.

[0157] Meanwhile, virtual sensor signals may also be included in a learning data set for learning or training of a gesture inference model (20).

[0158]

[0159] Fig. 10 illustrates a flowchart of a gesture inference method according to the present invention. The gesture inference method illustrated in Fig. 10 may be performed by a wearable device or a gesture recognition device, each of which will be described below with reference to Figs. 11 to 13. Hereinafter, it will be described that the device (100, 200) performs the procedure of Fig. 10.

[0160] The device (100, 200) may be configured to generate a virtual signal for a first EMG sensor among a plurality of EMG sensors based on signals of the remaining EMG sensors among the plurality of EMG sensors or signals of the IMU sensor (S1010). Each of the plurality of EMG sensors refers to an EMG sensor of each channel of the wearable device (100) described above. The virtual signal may be generated when a signal of the first EMG sensor is not acquired or the reliability of the signal of the first EMG sensor is below a preset value.

[0161] The device (100, 200) may be configured to use a sensor signal generation model to generate a virtual signal.

[0162] The device (100, 200) may be configured to acquire hand gesture information based on signals from the remaining EMG sensors and virtual signals among the plurality of EMG sensors (S1020). The remaining EMG sensors may include EMG sensors adjacent to the first EMG sensor.

[0163]

[0164] Alternatively, the device (100, 200) may be configured to generate a virtual signal for the first EMG sensor based on signals of at least two EMG sensors among a plurality of EMG sensors or signals of an IMU sensor (S1010). Here, the first EMG sensor may include a virtual sensor and may not be an EMG sensor of each channel of the wearable device (100). The at least two EMG sensors may include neighboring EMG sensors.

[0165] The device (100, 200) may be configured to use a sensor signal generation model to generate a virtual signal.

[0166] The device (100, 200) may be configured to acquire hand gesture information based on signals from a plurality of EMG sensors or virtual signals (S1020). That is, the device (100, 200) may be configured to acquire hand gesture information using only virtual signals. Alternatively, the device (100, 200) may be configured to acquire hand gesture information based on signals from a plurality of EMG sensors and virtual signals (S1020).

[0167] The device (100, 200) uses a gesture inference model to obtain hand gesture information, and can input the generated virtual signal into the gesture inference model.

[0168] The gesture inference method according to the present invention, which is not described with reference to FIG. 10, refers to the contents of the present invention described with reference to FIGS. 1 to 9.

[0169]

[0170] Figure 11 illustrates a system for gesture inference according to the present invention.

[0171] Referring to FIG. 11, a wearable device (100) and a gesture recognition device (200) are illustrated as being communicatively connected. According to FIG. 11, the wearable device (100) may be configured to transmit sensor data acquired through a built-in EMG sensor or IMU sensor to the gesture recognition device (200). In addition, the gesture recognition device (200) may infer hand gesture information from the received sensor data. In addition, the gesture recognition device (200) may be configured to transmit the inferred hand gesture information to another device (e.g., the wearable device (100)) or to perform a preset control operation based on the hand gesture information on its own.

[0172] Figure 11 illustrates a system in which a wearable device (100) and a gesture recognition device (200) operate with separate functions, and is merely an example. Accordingly, the wearable device (100) and the gesture recognition device (200) may be integrated.

[0173] In conclusion, the inference of hand gesture information, learning or training of a model for inference of hand gesture information, as described above, can be performed within the system (1).

[0174]

[0175] FIG. 12 illustrates a block diagram of a wearable device for training a gesture inference or gesture recognition model according to the present invention.

[0176] The wearable device (100) of FIG. 12 illustrates a wearable device in a system in which the wearable device (100) and the gesture recognition device (200) operate with separate functions, as illustrated in FIG. 11. However, unlike this, the wearable device (100) of FIG. 12 may be a device that integrates the wearable device (100) and the gesture recognition device (200) illustrated in FIG. 11.

[0177] A wearable device (100) may include a sensor (100), a processor (101), a memory (103), and a transceiver (104).

[0178] The sensor (100) may include an EMG sensor or an IMU sensor.

[0179] The processor (101) can process the sensor acquired by the sensor (100) or perform an operation to control other components within the wearable device.

[0180] The memory (103) may be configured to store acquired EMG sensor signals or IMU sensor signals, or to store codes containing recorded operations for controlling other components within the wearable device.

[0181] The transmitter / receiver (104) may be configured to transmit the acquired EMG sensor signal or IMU sensor signal to the gesture recognition device (200), or to receive a signal or data from the gesture recognition device (200).

[0182] Optionally, the wearable device (100) may include a human-machine interface (HMI), and the HMI may include hardware configurations capable of outputting various modality signals or data, such as a display, speaker, or vibrator.

[0183] The operation of the wearable device (100), which is not described with reference to FIG. 12, may include at least some of those described with reference to FIGS. 1 to 11.

[0184] Above, the operation of the device (100) according to the present invention has been described. Even if not described with reference to FIG. 12, the device (100) of the present invention may perform the operation according to the present invention according to FIGS. 1 to 11 described above.

[0185]

[0186] FIG. 13 illustrates a block diagram of a device for training a gesture inference or gesture recognition model according to the present invention. The device (200) for training a gesture inference or gesture recognition model will be simply referred to as a “gesture recognition device (200).”

[0187] The gesture recognition device (200) illustrates a gesture recognition device in a system in which a wearable device (100) and a gesture recognition device (200) operate with separate functions, as illustrated in FIG. 11. However, unlike this, the gesture recognition device (200) of FIG. 13 may be a device that integrates the wearable device (100) and the gesture recognition device (200) illustrated in FIG. 11.

[0188] The gesture recognition device (200) may include a transmitter / receiver (200), a processor (201), and a memory (203).

[0189] The transmitter / receiver (200) may be configured to receive an EMG sensor signal or an IMU sensor signal from a wearable device (100) or transmit a gesture inference result to the wearable device (100).

[0190] The memory (203) may be configured to store acquired EMG sensor signals or IMU sensor signals, or to store codes in which operations for controlling other components within the gesture recognition device (200) are recorded.

[0191] The processor (201) may be configured to acquire hand gesture information based on an EMG sensor signal or an IMU sensor signal. In addition, the processor (201) may be configured to generate a virtual sensor signal based on the EMG sensor signal or the IMU sensor signal. As an example, the processor (201) may be configured to generate a virtual signal for a first EMG sensor among the plurality of EMG sensors based on a signal of the remaining EMG sensors among the plurality of EMG sensors or a signal of the IMU sensor.

[0192] Additionally, the processor (201) may be configured to acquire hand gesture information based on signals from the remaining EMG sensors among the plurality of EMG sensors and the generated virtual signals. The remaining EMG sensors may include EMG sensors adjacent to the first EMG sensor.

[0193] The processor (201) may be configured to generate a virtual signal based on whether a signal from the first EMG sensor is not acquired or the reliability of the signal from the first EMG sensor is below a preset value.

[0194] Meanwhile, the device (200) may include a gesture inference model configured to obtain hand gesture information from a signal of an EMG sensor or a generated virtual signal, which may be stored in the memory (203). The gesture inference model may include a rotation data augmenter, a feature extractor, a gesture label classifier, a gradient reverse layer, and a domain discriminator. The rotation data augmenter, the gesture label classifier, the gradient reverse layer, and the domain discriminator are not used in the inference phase for obtaining the hand gesture information. That is, they are used only in the training or learning phase.

[0195] The device (200) may include a signal generation model for generating a virtual signal, which may be stored in the memory (203). The number of input channels of the signal generation model may be greater than the number of output channels of the signal generation model, or the number of sensor signals input to the signal generation model may be greater than the number of virtual signals output from the signal generation model.

[0196] The rotation data augmenter may be configured to acquire an additional data set using a rotation transformation for a planned or pre-trained data set. The feature extractor may be configured to extract features for inferring hand gesture information from signals of the remaining EMG sensors among the plurality of EMG sensors, signals of the IMU sensor, and the virtual signal. The gesture label classifier may be configured to acquire gesture information from the extracted features. The gradient reverse layer may be configured to induce domain confusion during backward propagation for training the gesture inference model. The domain discriminator may be configured to acquire domain information from the extracted features.

[0197] The processor (201) may be configured to train a gesture inference model based on a full loss function including a gesture loss function associated with weight parameters of a feature extractor and a label classifier and a domain loss function associated with weight parameters of a domain discriminator. The full loss function may be obtained by subtracting the result of multiplying the domain loss function by a constant for adjusting the strength for domain confusion from the gesture loss function.

[0198] The processor (201) may be configured to calculate the reliability of the acquired hand gesture information, output the acquired hand gesture information if the calculated reliability exceeds a preset value, and not output the acquired hand gesture information if the calculated reliability is lower than the preset value. The hand gesture information may be output through a human-machine interface (HMI).

[0199] The operation of the gesture recognition device (200), which is not described with reference to FIG. 13, may include at least some of those described with reference to FIGS. 1 to 11.

[0200] Above, the operation of the device (200) according to the present invention has been described. Even if not described with reference to FIG. 13, the device (200) of the present invention may perform the operation according to the present invention according to FIGS. 1 to 11 described above.

[0201]

[0202] In addition, as another aspect of the present invention, the operation of the proposal or invention described above may be implemented, performed or executed by a “computer” (a comprehensive concept including a system on chip (SoC) or a (micro) processor, etc.), or may be provided as a code or a computer-readable storage medium storing or including the code or a computer program product, and the scope of the present invention may be extended to the code or the computer-readable storage medium storing or including the code or the computer program product.

[0203] Furthermore, the functions of the elements disclosed herein may be implemented using circuits or processing circuits configured or programmed to perform the functions disclosed, including general purpose processors, special purpose processors, integrated circuits, Application Specific Integrated Circuits (ASICs), conventional circuits, and / or combinations thereof. A processor is considered a processing circuit or circuit because it includes transistors and other circuits therein. In this specification, circuitry, unit, or means refers to hardware that performs or is programmed to perform the functions described. The hardware may be any hardware disclosed herein or hardware known to those skilled in the art, and may be programmed or configured to perform the functions described. If the hardware is a processor, it may be considered a type of circuit, and in this case, the circuitry, unit, or unit is a combination of hardware and software, wherein the software is used to configure the hardware and / or processor.

[0204] The detailed description of the preferred embodiments of the present invention disclosed above has been provided to enable those skilled in the art to implement and practice the present invention. While the above description has been made with reference to preferred embodiments of the present invention, those skilled in the art will appreciate that various modifications and variations of the present invention, as defined by the following claims, are possible. Accordingly, the present invention is not intended to be limited to the embodiments disclosed herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. Multiple EMG (Electromyography) sensors arranged in a loop structure; An IMU (Inertial Measurement Unit) sensor including an acceleration and angular velocity sensor; and Generating a virtual signal for a first EMG sensor among the plurality of EMG sensors based on a signal of the remaining EMG sensors among the plurality of EMG sensors or a signal of the IMU sensor, A processor that obtains hand gesture information based on signals of the remaining EMG sensors among the plurality of EMG sensors, signals of the IMU sensor, and the virtual signal. , device.

2. In paragraph 1, The processor is configured to generate the virtual signal based on whether a signal from the first EMG sensor is not acquired or the reliability of the signal from the first EMG sensor is below a preset value. , device.

3. In paragraph 1, The remaining EMG sensors include EMG sensors adjacent to the first EMG sensor. , device.

4. In paragraph 1, Further comprising a gesture inference model configured to obtain the hand gesture information from the signal of the EMG sensor or the virtual signal. , device.

5. In the fourth paragraph, the gesture inference model It includes a rotation data augmenter, a feature extractor, a gesture label classifier, a gradient reverse layer, and a domain discriminator. The above rotation data augmenter obtains an additional data set using rotation transformation for the previously trained data set, The above feature extractor is configured to extract features for inferring hand gesture information from signals of the remaining EMG sensors among the plurality of EMG sensors, signals of the IMU sensor, and the virtual signal, The gesture label classifier is configured to obtain gesture information from the extracted features, The above gradient reverse layer is configured to induce domain confusion during backward propagation for training the gesture inference model, The above domain classifier is configured to obtain domain information from the extracted features. , device.

6. In the fifth paragraph, the processor configured to train the gesture inference model based on a full loss function including a gesture loss function related to weight parameters of the feature extractor and the label classifier and a domain loss function related to weight parameters of the domain classifier. , device.

7. In paragraph 6, The above overall loss function is obtained by subtracting the result of multiplying the domain loss function by a constant for adjusting the strength for domain confusion from the gesture loss function. , device.

8. In paragraph 5, The rotation data augmenter, the gesture label classifier, the gradient reverse layer, and the domain separator are not used in the inference phase for obtaining the hand gesture information. , device.

9. In paragraph 1, Further comprising a signal generation model for generating the above virtual signal, The number of input channels of the signal generation model is greater than the number of output channels of the signal generation model, or The number of sensor signals input to the above signal generation model is greater than the number of virtual signals output from the above signal generation model. , device.

10. In the first paragraph, the processor Calculate the reliability of the hand gesture information obtained above, If the reliability exceeds a preset value, the acquired hand gesture information is output; if the reliability is less than a preset value, the acquired hand gesture information is configured not to be output. , device.

11. A method performed by a device configured to obtain hand gesture information from a sensor signal, A step of generating a virtual signal for a first EMG sensor among the plurality of EMG sensors based on a signal of the remaining EMG sensors among the plurality of EMG sensors or a signal of the IMU sensor; and A step of obtaining hand gesture information based on the signal of the remaining EMG sensor among the plurality of EMG sensors, the signal of the IMU sensor, and the virtual signal. , method.

12. In paragraph 11, A step of generating the virtual signal based on whether the signal of the first EMG sensor is not acquired or the reliability of the signal of the first EMG sensor is below a preset value. , method.

13. In paragraph 11, The remaining EMG sensors include EMG sensors adjacent to the first EMG sensor. , method.

14. In paragraph 11, A step of obtaining the hand gesture information from the signal of the EMG sensor or the virtual signal based on a gesture inference model configured to obtain the hand gesture information. , method.

15. In paragraph 14, the gesture inference model: It includes a rotation data augmenter, a feature extractor, a gesture label classifier, a gradient reverse layer, and a domain discriminator. The above rotation data augmenter obtains an additional data set using rotation transformation for the previously trained data set, The above feature extractor is configured to extract features for inferring hand gesture information from signals of the remaining EMG sensors among the plurality of EMG sensors, signals of the IMU sensor, and the virtual signal, The gesture label classifier is configured to obtain gesture information from the extracted features, The above gradient reverse layer is configured to cause domain confusion during backward propagation for training the above inference model, The above domain classifier is configured to obtain domain information from the extracted features. , method.

16. In paragraph 15, A step of training the gesture inference model based on an overall loss function including a gesture loss function related to weight parameters of the feature extractor and the label classifier and a domain loss function related to weight parameters of the domain classifier. , method.

17. In paragraph 16, The above overall loss function is obtained by subtracting the result of multiplying the domain loss function by a constant for adjusting the strength for domain confusion from the gesture loss function. , method.

18. In paragraph 15, The rotation data augmenter, the gesture label classifier, the gradient reverse layer, and the domain separator are not used in the inference phase for obtaining the hand gesture information. , method.

19. In paragraph 11, A step of calculating the reliability of the acquired hand gesture information; and A step of outputting the acquired hand gesture information when the reliability exceeds a preset value; and a step of not outputting the acquired hand gesture information when the reliability is less than or equal to a preset value. , method.

20. A non-transitory computer-readable storage medium storing code configured to be executed by a computer or processor according to any one of claims 11 to 19.

Images