Touchless operating system for predicting operations using brain and body activities

Abstract

A touchless operating system for predicting operations using brain and body activities includes a body activity determination module, a brain activity determination module, an integration module and a computing device. A processing unit of the integration module receives a first prediction signal from the body activity determination module and a second prediction signal from the brain activity determination module to produce an adjacency matrix signal. An adaptive pre-trained model of the integration module receives the adjacency matrix signal and produces an operation signal. The computing device receives the operation signal and executes an operation corresponding to the operation signal. Thus, signal delay or determination error can be prevented and computing speed and use fluency can be enhanced in the touchless operating system.

Description

[Technical Field]

[0001] This invention relates to a touchless operating system, and more particularly to a touchless operating system with brain activity and body activity prediction operation function. The touchless operating system can be applied to touchless operation devices such as virtual reality computing devices, holographic projection computing devices, vehicle computing devices and robot operation. [Previous Technology]

[0002] A conventional touchless operating system determines the desired operation function (such as a single action or continuous action such as grabbing, holding, or clicking) by detecting gestures. However, when detecting gestures, signal delay or incorrect judgment may occur due to the switching of the detected gestures, causing the touchless operating system to fail to react immediately or to produce incorrect operation functions. [Summary of the Invention]

[0003] The main objective of this invention is to increase the computational and operational smoothness of a touchless operating system by using a brain activity judgment module, a body activity judgment module and an integrated judgment module, so as to avoid signal delay or misjudgment problems.

[0004] A touchless operating system with brain activity and body activity prediction operation functions according to the present invention includes a body activity judgment module, a brain activity judgment module, an integrated judgment module, and a computing device. The body activity judgment module includes an image capture device, a body activity signal processing unit, and a body image pre-training model. The image capture device is used to capture a body image and form an image signal. The body activity signal processing unit is used to receive the image signal and form an adjacency matrix signal of body node features. The body image pre-training model is used to receive the adjacency matrix signal of body node features and form a first prediction operation signal. The brain activity judgment module includes a brain activity capture device and a brain activity signal processing unit. The system includes a brain activity pre-training model, a brain activity capture device for capturing brain activity and forming a brain activity signal, a brain activity signal processing unit for receiving the brain activity signal and forming a current-time connection matrix signal, and a brain activity pre-training model for receiving the current-time connection matrix signal and forming a second predicted operation signal. The integrated judgment module includes an operation signal processing unit and an adaptive operation pre-training model. The operation signal processing unit is used to receive the first predicted operation signal and the second predicted operation signal and form an adjacency matrix signal. The adaptive operation pre-training model is used to receive the adjacency matrix signal and form an operation signal. The computing device is used to receive the operation signal and execute an operation function corresponding to the operation signal.

[0005] The present invention uses the integrated judgment module to receive the first predictive operation signal of the body activity judgment module and the second predictive operation signal of the brain activity judgment module to form the operation signal, which can avoid the problem of signal delay or misjudgment and increase the smoothness of operation and use of the touchless operating system.

Implementation Method

[0006] Please refer to Figure 1, which shows the touchless operating system 100 of the present invention with brain activity and body activity prediction operation function. It includes a body activity judgment module 110, a brain activity judgment module 120, an integrated judgment module 130 and a computing device 140. The body activity judgment module 110 is used to generate a first predicted operation signal S13, the brain activity judgment module 120 is used to generate a second predicted operation signal S23, the integrated judgment module 130 is used to receive the first predicted operation signal S13 and the second predicted operation signal S23 and generate an operation signal S32, and the computing device 140 is used to receive the operation signal S32 and execute an operation function corresponding to the operation signal S32. The operation function can be selected from an action (such as a click) or a continuous action (such as sliding, dragging, grabbing, releasing, hugging, jumping, moving, etc.) that is close to or the same as the body activity.

[0007] Please refer to Figures 1, 2A and 2B. The body activity judgment module 110 includes an image capture device 111, a body activity signal processing unit 112 and a body image pre-training model 113. The image capture device 111 is used to capture a body image O and form an image signal S11. The body image O is a limb image O1 and / or the torso image O2. The limb image O1 is selected from one or a combination of the arm, hand, leg and foot. The torso image O2 is selected from one or a combination of the chest and abdomen. The image capture device 111 can be selected from an image capture device such as a camera. The image signal S11 is a digital signal after image processing and digitization.

[0008] Please refer to Figure 1. The body activity signal processing unit 112 is used to receive the image signal S11 and form an adjacency matrix signal S12 of body node features. The body activity signal processing unit 112 can be selected from the limb node data extraction method (such as the limb node data extraction method disclosed at https: / / developers.google.com / mediapipe / solutions / vision / hand_landmarker), the trunk node data extraction method (such as the trunk node data extraction method disclosed at https: / / developers.google.com / ml-kit / vision / pose-detection?hl=zh-tw), and / or the body (limb and trunk) node data integration algorithm (such as the adjacency matrix algorithm and graph representation algorithm disclosed at https: / / huggingface.co / blog / intro-graphml). The body image pre-training model 113 is used to receive the adjacency matrix signal S12 of body node features and form the first prediction operation signal S13.

[0009] Please refer to Figures 1 and 2B. In the first embodiment, the limb image O1 is illustrated as a hand, but this is not a limitation. The image capturing device 111 captures the limb image O1. The image signal S11 is a limb image signal S11a containing a plurality of limb nodes P1. The image capturing device 111 is used to capture the limb image O1 and form the limb image signal S11a. The body activity signal processing unit 112 includes a limb activity signal processing unit 112A. In this embodiment, the limb activity signal processing unit 112A is selected from the limb node data extraction method (such as the limb node data extraction method disclosed at https: / / developers.google.com / mediapipe / solutions / vision / hand_landmarker, which can be achieved through a key point detection model (Landmark Detection). (Model) extracts limb node data, but is not limited thereto. The adjacency matrix signal S12 of the body node features includes an adjacency matrix signal S12a of limb node features. The limb activity signal processing unit 112A is used to receive the limb image signal S11a and extract the coordinate data of the limb nodes P1 of the limb image signal S11a to form the adjacency matrix signal S12a of the limb node features.

[0010] Please refer to Figures 1 and 4A. In this embodiment, the body image pre-training model 113 includes a pre-trained matrix signal data set D5 of limb node features. The pre-trained limb node feature matrix signal data set D5 includes a plurality of pre-trained limb node feature matrix data D51. These pre-trained limb node feature matrix data D51 can be limb node data, adjacency matrix data, and represented operation functions (such as click, drag, etc.) constructed by one or more experimenters at different time units (such as 0-1 second, 1-2 seconds, etc.). The body image pre-training model 113 is used to receive the adjacency matrix signal S12a of the limb node features and compare it with these pre-trained limb node feature matrix data D51 to form the first predicted operation signal S13 corresponding to one of the pre-trained limb node feature matrix data D51.

[0011] Please refer to Figures 1 and 2A. In the second embodiment, the image capturing device 111 captures the torso image O2. In this embodiment, the image signal S11 is a torso image signal S11b containing a plurality of torso nodes P2. The image capturing device 111 is used to capture the torso image O2 and form the torso image signal S11b. The body activity signal processing unit 112 includes a torso activity signal processing unit 112B. In this embodiment, the torso activity signal processing unit 112B is selected from the method for capturing torso node data (such as the method for capturing torso node data disclosed at https: / / developers.google.com / ml-kit / vision / pose-detection?hl=zh-tw, which can be achieved through a pose estimation model). (Model) extracts torso node data, but is not limited thereto. The adjacency matrix signal S12 of the body node features includes an adjacency matrix signal S12b of torso node features. The torso activity signal processing unit 112B is used to receive the torso image signal S11b and extract the coordinate data of the torso nodes P2 of the torso image signal S11b to form the adjacency matrix signal S12b of the torso node features.

[0012] Please refer to Figures 1 and 4B. In this embodiment, the body image pre-training model 113 includes a pre-trained matrix signal data set D6 of torso node features. The pre-trained matrix signal data set D6 includes a plurality of pre-trained torso node feature matrix data D61. These pre-trained torso node feature matrix data D61 can be torso node data, adjacency matrix data, and represented operation functions (such as jump, turn, etc.) constructed by one or more experimenters at different time units (such as 0-1 second, 1-2 seconds, etc.). The body image pre-training model 113 is used to receive the adjacency matrix signal S12b of the torso node features and compare it with these pre-trained torso node feature matrix data D61 to form the first predicted operation signal S13 corresponding to one of the pre-trained torso node feature matrix data D61.

[0013] Please refer to Figures 1, 2A, and 2B. In the third embodiment, the image cropping device 111 crops the limb image O1 and the torso image O2. In this embodiment, the image signal S11 includes the limb image signal S11a and the torso image signal S11b. The image cropping device 111 is used to crop the limb image O1 and the torso image O2 and respectively form the limb image signal S11a and the torso image signal S11b. The body activity signal processing unit 112 includes the limb activity signal processing unit 112A, the torso activity signal processing unit 112B, and an activity signal integration processing unit 112C. The activity signal integration processing unit 112C is selected from body (limb and torso) node data integration algorithms (such as the adjacency matrix algorithm and graph representation algorithm disclosed at https: / / huggingface.co / blog / intro-graphml). (algorithm), but not limited thereto, the adjacency matrix signal S12 of the body node features includes an adjacency matrix signal S12c that integrates the features of limb nodes and trunk nodes. The limb activity signal processing unit 112A is used to receive the limb image signal S11a and extract the coordinate data of the limb nodes P1 of the limb image signal S11a. The trunk activity signal processing unit 112B is used to receive the trunk image signal S11b and extract the coordinate data of the trunk nodes P2 of the trunk image signal S11b. The activity signal integration processing unit 112C is used to receive the coordinate data of the limb nodes P1 and the coordinate data of the trunk nodes P2 to form the adjacency matrix signal S12c that integrates the features of limb nodes and trunk nodes.

[0014] Please refer to Figures 1 and 4C. In this embodiment, the body image pre-training model 113 includes a pre-trained matrix signal data set D7 integrating limb node features and trunk node features. The pre-trained matrix signal data set D7 includes a plurality of pre-trained matrix data D71 integrating trunk node and limb node features. These pre-trained matrix data D71 integrating trunk node and limb node features can be generated by one or more experimenters at different time units (such as 0-1 second, 1-2 seconds, etc.). The body image pre-training model 113 receives the adjacency matrix signal S12c of the integrated limb node features and torso node features constructed by the unit, and compares it with the pre-trained integrated pre-trained torso node and limb node feature matrix data D71 to form the first predicted operation signal S13 corresponding to one of the pre-trained integrated pre-trained torso node and limb node feature matrix data D71.

[0015] Please refer to Figure 1. The brain activity judgment module 120 includes a brain activity capture device 121, a brain activity signal processing unit 122, and a brain activity pre-training model 123. The brain activity capture device 121 is selected from functional magnetic resonance imaging (fMRI) or electroencephalography (EEG) and other brain activity capture devices. The brain activity capture device 121 is used to capture a brain activity and form a brain activity signal S21 (such as a magnetic resonance signal or an electrical signal). The brain activity signal processing unit 122 is used to receive the brain activity signal S21 and form a current-time connectivity matrix signal S22. The brain activity signal processing unit 122 is selected from connectivity analysis methods (such as the connectivity analysis methods disclosed at https: / / www.humanconnectome.org, Dynamic Functional Connectivity). Connectivity), time-frequency analysis, graph theory analysis, and Hidden Markov Models (HMM) for State Detection, but not limited thereto, the brain activity pre-trained model 123 is used to receive the current time connection matrix signal S22 and form the second prediction operation signal S23.

[0016] Please refer to Figure 1. In this embodiment, the brain activity pre-training model 123 includes a first pre-training model 123A and a second pre-training model 123B. The first pre-training model 123A is used to receive the current time connection matrix signal S22 and form the next time connection matrix signal S22a. The second pre-training model 123B is used to receive the next time connection matrix signal S22a and form the second prediction operation signal S23.

[0017] Please refer to Figures 1 and 3A. In this embodiment, the first pre-trained model 123A has a pre-trained current time unit link matrix signal data set D1 and a pre-trained next time unit link matrix signal data set D2. The pre-trained current time unit link matrix signal data set D1 contains a plurality of pre-trained current time unit link matrix data D11. These pre-trained current time unit link matrix data D11 can be current time matrix data constructed by one or more experimenters in different current time units (such as 0~1 second, 1~2 seconds, etc.). The pre-trained next time unit matrix signal data set D2 contains a plurality of pre-trained next time units (such as 1~2 seconds, 2~3 seconds, etc.). ...time unit) link matrix data D21, these pre-trained next time unit link matrix data D21 can be the next time matrix data constructed by the same experimenter who constructed the current time matrix data in the next time unit, each of the pre-trained current time unit link matrix data D11 is followed by each of the pre-trained next time unit link matrix data D21, the first pre-trained model 123A is used to receive the current time link matrix signal S22 and compare the pre-trained time unit link matrix data D11 and the pre-trained next time unit link matrix data D21 to form the next time link matrix signal S22a corresponding to one of the pre-trained next time unit link matrix data D21.

[0018] Please refer to Figures 1 and 3B. In this embodiment, the second pre-trained model 123B includes a pre-trained connection matrix dataset D3 and a pre-trained operational function dataset D4. The pre-trained connection matrix dataset D3 includes a plurality of pre-trained connection matrix datasets D31. These pre-trained connection matrix datasets D31 can be connection matrix signals constructed by one or more experimenters at different time units (such as 0-1 second, 1-2 seconds, etc.). The pre-trained operational function datasets D4... Set D4 contains a plurality of pre-trained operation function data D41 (such as click, drag, etc.), and each of the pre-trained connection matrix data D31 corresponds to each of the pre-trained operation function data D41. The second pre-trained model 123B is used to receive the connection matrix signal S22a at the next time and compare the pre-trained connection matrix data D31 and the pre-trained operation function data D41 to form a second predicted operation signal S23 corresponding to one of the pre-trained operation function data D41.

[0019] Please refer to Figure 1. The integrated judgment module 130 includes an operation signal processing unit 131 and an adaptive operation pre-training model 132. The operation signal processing unit 131 is used to receive the first predicted operation signal S13 and the second predicted operation signal S23 and form an adjacency matrix signal S31. The operation signal processing unit 131 can be selected from adjacency matrix algorithms (such as the adjacency matrix algorithm and graph representation algorithm disclosed at https: / / huggingface.co / blog / intro-graphml). The adaptive operation pre-training model 132 is used to receive the adjacency matrix signal S31 and form the operation signal S32.

[0020] Please refer to Figures 1 and 5. In this embodiment, the adaptive operation pre-training model 132 includes a pre-trained adaptive operation function adjacency matrix signal data set D8 and a pre-trained adaptive operation function data set D9. The pre-trained adaptive operation function adjacency matrix signal data set D8 includes a plurality of pre-trained adaptive operation function adjacency matrix signal data D81. These pre-trained adaptive operation function adjacency matrix signal data D81 can be adjacency matrix signal data constructed by one or more experimenters at different time units (such as 0~1 second, 1~2 seconds, etc.). The training adaptive operation function dataset D9 contains a plurality of pre-trained adaptive operation function data D91 (such as click, drag, etc.). Each pre-trained adaptive operation function adjacency matrix signal data D81 corresponds to each pre-trained adaptive operation function data D91. The adaptive operation pre-training model 132 is used to receive the adjacency matrix signal S31 and compare the pre-trained adaptive operation function adjacency matrix signal data D81 and the pre-trained adaptive operation function data D91 to form the operation signal S32 corresponding to one of the pre-trained adaptive operation function data D91.

[0021] Please refer to Figure 1. The computing device 140 is used to receive the operation signal S32 and execute the operation function corresponding to the operation signal S32 of the pre-trained adaptive operation function data D91, that is, to execute at least one of the operation functions (such as click, drag, etc.) corresponding to the pre-trained adaptive operation function data D91.

[0022] Please refer to Figures 1, 3A-3B, 4A-4C and 5. The body image pre-training model 113, the first pre-training model 123A, the second pre-training model 123B and the adaptive operation pre-training model 132 are pre-trained models that collect various operation functions (such as click, drag, etc.) performed by the same or different experimenters in a time unit (such as 0~1 second, 1~2 seconds, etc.) and respectively obtain node data, matrix data, connection matrix data, adjacency matrix data and operation functions through the limb activity signal processing unit 112A, the trunk activity signal processing unit 112B, the activity signal integration processing unit 112C, the brain activity signal processing unit 122 and / or the operation signal processing unit 131. They are then pre-trained models that are supervised learning, unsupervised learning, machine learning based and / or deep learning based.

[0023] The integrated judgment module 130 of the present invention forms the operation signal S32 by the first predicted operation signal S13 formed by the body activity judgment module 110 and the second predicted operation signal S23 formed by the brain activity judgment module 120, which can increase the operation and smoothness of the touchless operating system 100 and avoid the problems of signal delay or misjudgment.

[0024] The scope of protection of this invention shall be determined by the appended claims. Any changes and modifications made by those skilled in the art without departing from the spirit and scope of this invention shall fall within the scope of protection of this invention. [Simplified Explanation of the Diagram]

[0025] Figure 1: Block diagram of the touchless operating system of the present invention. Figure 2A: Schematic diagram of body nodes. Figure 2B: Schematic diagram of gesture nodes. Figure 3A: Schematic diagram of the first pre-trained model of the present invention. Figure 3B: Schematic diagram of the second pre-trained model of the present invention. Figures 4A-4C: Schematic diagram of the body image pre-trained model of the present invention. Figure 5: Schematic diagram of the adaptive operation pre-trained model of the present invention.

Claims

1. A touchless operating system with brain activity and body activity prediction operation functions includes: a body activity judgment module, comprising an image capture device, a body activity signal processing unit, and a body image pre-training model, wherein the image capture device is used to capture a body image and form an image signal, the body activity signal processing unit is used to receive the image signal and form an adjacency matrix signal of body node features, and the body image pre-training model is used to receive the adjacency matrix signal of body node features and form a first prediction operation signal; and a brain activity judgment module, comprising a brain activity capture device, a brain activity signal processing unit, and a brain activity pre-training model, wherein the brain activity capture device is used to capture a brain activity and form a brain activity signal, the brain activity signal processing unit is used to receive the brain activity signal and form a current-time connection matrix signal, and the brain activity pre-training model is used to receive the current-time connection matrix signal and form a second prediction operation signal. An integrated judgment module includes an operation signal processing unit and an adaptive operation pre-training model. The operation signal processing unit is used to receive the first predicted operation signal and the second predicted operation signal and form an adjacency matrix signal. The adaptive operation pre-training model is used to receive the adjacency matrix signal and form an operation signal. A computing device is used to receive the operation signal and execute an operation function corresponding to the operation signal.

2. The touchless operating system with brain activity and body activity prediction operation function as described in claim 1, wherein the brain activity pre-training model includes a first pre-training model and a second pre-training model, the first pre-training model being used to receive the current time connection matrix signal and form the next time connection matrix signal, and the second pre-training model being used to receive the next time connection matrix signal and form the second prediction operation signal.

3. As in claim 2, a touchless operating system for predicting operational functions based on brain and body activity, wherein the first pre-trained model has a pre-trained current time unit link matrix signal data set and a pre-trained next time unit link matrix signal data set. The pre-trained current time unit link matrix signal data set contains a plurality of pre-trained current time unit link matrix data sets, and the pre-trained next time unit link matrix signal data set contains a plurality of pre-trained next time unit link matrix data sets. Each pre-trained current time unit link matrix data set follows each pre-trained next time unit link matrix data set. The first pre-trained model is used to receive the current time link matrix signal and compare the pre-trained time unit link matrix data sets and the pre-trained next time unit link matrix data sets to form the next time link matrix signal.

4. As in claim 3, a touchless operating system that predicts operational functions based on brain and body activity, wherein the second pre-trained model includes a pre-trained connection matrix dataset and a pre-trained operational function dataset. The pre-trained connection matrix dataset includes a plurality of pre-trained connection matrix datasets, and the pre-trained operational function dataset includes a plurality of pre-trained operational function datasets. Each pre-trained connection matrix dataset corresponds to each pre-trained operational function dataset. The second pre-trained model is used to receive the connection matrix signal at the next time step and compare the pre-trained connection matrix datasets and the pre-trained operational function datasets to form a second predicted operational signal.

5. A touchless operating system for predicting brain and body activity as described in claim 1, wherein the body image is a limb image, the image signal is a limb image signal containing a plurality of limb nodes, the body activity signal processing unit includes a limb activity signal processing unit, the adjacency matrix signal of the body node features includes an adjacency matrix signal of the limb node features, the body image pre-training model includes a pre-trained matrix signal dataset of limb node features, the pre-trained matrix signal dataset of limb node features includes a plurality of pre-trained matrix data of limb node features, the image capture device is used to capture the limb image and form the limb image signal, the limb activity signal processing unit is used to receive the limb image signal and capture the coordinate data of a plurality of limb nodes of the limb image signal to form the adjacency matrix signal of the limb node features, and the body image pre-training model is used to receive the adjacency matrix signal of the limb node features and compare the pre-trained matrix data of the limb node features to form the first prediction operation signal.

6. A touchless operating system for predicting brain and body activity, as described in claim 1, wherein the body image is a torso image, the image signal is a torso image signal containing a plurality of torso nodes, the body activity signal processing unit includes a torso activity signal processing unit, the adjacency matrix signal of the body node features includes an adjacency matrix signal of the torso node features, the body image pre-training model includes a pre-trained matrix signal dataset of torso node features, and the pre-trained matrix signal dataset of the torso node features... The dataset contains a plurality of pre-trained trunk node feature matrix data. The image capture device is used to capture the trunk image and form the trunk image signal. The trunk activity signal processing unit is used to receive the trunk image signal and capture the coordinate data of a plurality of trunk nodes in the trunk image signal to form an adjacency matrix signal of the trunk node features. The body image pre-training model is used to receive the adjacency matrix signal of the trunk node features and compare these pre-trained trunk node feature matrix data to form the first prediction operation signal.

7. A touchless operating system for predicting brain and body activity, as described in claim 1, wherein the body image includes a limb image and a trunk image, the image signal includes a limb image signal of a plurality of limb nodes and a trunk image signal of a plurality of trunk nodes, the body activity signal processing unit includes a limb activity signal processing unit, a trunk activity signal processing unit and an activity signal integration processing unit, the adjacency matrix signal of the body node features includes an adjacency matrix signal integrating limb node and trunk node features, the body image pre-training model includes a pre-trained matrix signal dataset integrating limb node features and trunk node features, and the pre-trained matrix signal dataset integrating limb node features and trunk node features includes a plurality of pre-trained matrix data integrating pre-trained trunk node and limb node features. The image capture device is used to capture the limb image and the torso image, and respectively form the limb image signal and the torso image signal. The limb activity signal processing unit is used to receive the limb image signal and capture the coordinate data of the limb nodes of the limb image signal. The torso activity signal processing unit is used to receive the torso image signal and capture the coordinate data of the torso nodes of the torso image signal. The activity signal integration processing unit is used to receive the coordinate data of the limb nodes and the coordinate data of the torso nodes to form the adjacency matrix signal integrating the limb node features and the torso node features. The body image pre-training model is used to receive the adjacency matrix signal integrating the limb node features and the torso node features, and compare the pre-trained integrated pre-trained torso node and limb node feature matrix data to form the first prediction operation signal.

8. A touchless operating system for predicting operational functions based on brain and body activity, as described in claim 1, wherein the adaptive operation pre-training model includes a pre-trained adaptive operation function adjacency matrix signal data set and a pre-trained adaptive operation function data set. The pre-trained adaptive operation function adjacency matrix signal data set includes a plurality of pre-trained adaptive operation function adjacency matrix signal data sets and a plurality of pre-trained adaptive operation function data sets. Each pre-trained adaptive operation function adjacency matrix signal data set corresponds to each pre-trained adaptive operation function data set. The adaptive operation pre-training model is used to receive the adjacency matrix signals and compare the pre-trained adaptive operation function adjacency matrix signal data sets and the pre-trained adaptive operation function data sets to form the operation signal.

9. A touchless operating system with brain activity and body activity prediction operation function as described in claim 1, wherein the image capture device is selected from a camera.

10. The touchless operating system for predicting operation functions based on brain and body activity, as described in claim 1, wherein the brain activity capture device is selected from functional magnetic resonance imaging (fMRI) or electroencephalography (EEG).

Images