Painting auxiliary interaction method and system based on in-ear wearable device, device and medium

Abstract

The invention provides a drawing auxiliary interaction method and system based on in-ear wearable equipment, equipment and a medium. The drawing auxiliary interaction method comprises the following steps: acquiring an electroencephalogram signal; the electroencephalogram signals are classified, and an electroencephalogram instruction based on electroencephalogram signal analysis is obtained; acquiring facial muscle and lower jaw movement information near ears, and movement information of the head and the neck; mouth shape information of different pronunciations is analyzed through the facial muscle and lower jaw movement information and the head and neck movement information; and fusing the electroencephalogram instruction with the mouth shape information to decide a final user instruction. The problem that traditional painting switching operation is inconvenient is solved, intention operation of a user is analyzed through the electroencephalogram electrode, the near-infrared sensor and the gyroscope which are arranged on the in-ear wearable device, and smooth switching of painting tool options is directly achieved. According to the scheme, redundant operation of manual step-by-step selection is not needed any more, delay is low, operation is natural, a user is supported to concentrate on creation, and creation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of painting-assisted interaction technology, and in particular to a painting-assisted interaction method and system, device, and medium based on an in-ear wearable device. Background Art

[0002] Compared with traditional painting, tablet painting is popular among users because it is not restricted by location or time, and integrates a rich variety of brush tools and color choices.

[0003] Currently, to select a brush tool and color, you need to touch or click the corresponding toolbar or color palette. Due to the limited screen area, the corresponding tool or color options are often laid out in the form of folds or submenus. So when the user needs to switch, it often takes multiple clicks to select the target. This inconvenient method often interrupts the user's train of thought. Especially when creative inspiration is flowing, frequent tool switching or menu operations can easily increase the user's cognitive load and reduce the user's concentration and fluency; and complex tool options will also increase cognitive costs, which may cause frustration and reduce the joy of creation. Summary of the invention

[0004] In order to achieve the above-mentioned purpose and other advantages of the present invention, the first purpose of the present invention is to provide a painting auxiliary interaction method based on an in-ear wearable device, comprising the following steps:

[0005] Acquire EEG signals;

[0006] Classifying the EEG signals to obtain EEG instructions based on EEG signal analysis;

[0007] Obtain information about facial muscle and jaw movements near the ears, as well as head and neck movements;

[0008] Analyzing the mouth shape information of different pronunciations through the facial muscle and jaw movement information, and the head and neck movement information;

[0009] The EEG command is integrated with the lip shape information to determine the final user command for interactive control.

[0010] Furthermore, the EEG signal is configured as an EEG signal of a temporal lobe region.

[0011] Furthermore, the step of classifying the EEG signals to obtain EEG instructions based on EEG signal analysis includes:

[0012] The EEG signal is classified by an EEG signal classification model.

[0013] Furthermore, the EEG signal classification model training steps are also included:

[0014] Collect the EEG signals of the subjects imagining preset colors and preset tool instructions;

[0015] The collected data is classified and stored according to preset tags, and the data is segmented according to preset timestamps;

[0016] De-noising is performed through pre-processing techniques to obtain an effective training data set;

[0017] The training data set is sent to a deep learning network for training, and finally a pre-trained model based on EEG classification is obtained.

[0018] Furthermore, the step of parsing the lip shape information of different pronunciations through the facial muscle and jaw movement information, and the head and neck movement information comprises:

[0019] The facial muscle and jaw movement information, and the head and neck movement information are classified by a lip shape classification model.

[0020] Furthermore, the method also includes the following steps of training the lip shape classification model:

[0021] Collect information on the subjects' facial muscles and jaw movements when they make different pronunciations, as well as head and neck movements;

[0022] The collected data is classified and stored according to preset tags, and the data is segmented according to preset timestamps;

[0023] De-noising is performed through pre-processing techniques to obtain an effective training data set;

[0024] The training data set is sent to a deep learning network for training, and finally a pre-trained model based on lip shape classification is obtained.

[0025] Furthermore, the step of fusing the EEG instruction with the lip shape information to determine the final user instruction for interactive control includes:

[0026] The EEG instruction is fused with the confidence of the lip shape information, and a decision with a high confidence is selected as the final result.

[0027] The second object of the present invention is to provide a painting auxiliary interactive system based on an in-ear wearable device to implement the above method, including an in-ear wearable device and a main control module. The in-ear wearable device is provided with EEG electrodes, a near-infrared sensor, and a gyroscope. The EEG electrodes are used to collect EEG signals, the near-infrared sensor is used to detect the movement information of facial muscles and jaw, and the gyroscope is used to capture the movement information of the head and neck. The main control module decides the final user instructions based on the EEG signals, the movement information of facial muscles and jaw, and the movement information of the head and neck for interactive control.

[0028] Furthermore, the number of the EEG electrodes is 2, the EEG electrodes are placed on the inner side of the ear canal close to the eardrum, and the EEG electrodes in the left and right in-ear wearable devices are placed symmetrically.

[0029] A third object of the present invention is to provide a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0030] A fourth object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0031] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0032] The present invention overcomes the inconvenience of traditional painting switching operations. It uses the EEG electrodes, near-infrared sensors and gyroscopes configured on the in-ear wearable device to analyze the user's intended operation and directly achieve smooth switching of painting tool options. Compared with the traditional method, the solution provided by the present invention no longer requires redundant operations of manual step-by-step selection, has low latency, and natural operation, which supports users to focus on the creation itself and improves the efficiency of creation.

[0033] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0035] Figure 1 The flowchart of the painting assistance interaction method based on the in-ear wearable device is as follows;

[0036] Figure 2 A schematic diagram of a drawing-assisted interaction method based on an in-ear wearable device;

[0037] Figure 3 Flowchart for training EEG signal classification model;

[0038] Figure 4 Training flow chart for lip shape classification model;

[0039] Figure 5 Schematic diagram of a painting assistance interaction system based on an in-ear wearable device;

[0040] Figure 6 It is a schematic diagram of an in-ear wearable device;

[0041] Figure 7 It is a schematic diagram of computer equipment;

[0042] Figure 8 A schematic diagram of a computer-readable storage medium. DETAILED DESCRIPTION

[0043] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. It should be noted that, under the premise of no conflict, the embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0044] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0045] The figure numbers in this application are only used to distinguish the various steps in the scheme, and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0047] Example 1

[0048] A painting-assisted interaction method based on an in-ear wearable device, such as Figure 1 , Figure 2 As shown, the following steps are included:

[0049] S1, obtain EEG signals;

[0050] This embodiment adopts a non-invasive and portable method for collecting EEG signals. Figure 6 As shown, the in-ear wearable device is provided with EEG electrodes, which are used to collect EEG signals.

[0051] Specifically, two EEG electrodes are built into the in-ear wearable device to collect EEG signals from the temporal lobe area. To improve signal quality, the EEG electrodes can be placed inside the ear canal close to the eardrum, and to eliminate noise, the EEG electrodes in the left and right in-ear wearable devices are placed symmetrically.

[0052] S2, classifying the EEG signals to obtain EEG instructions based on EEG signal analysis;

[0053] In some embodiments, the step of classifying the EEG signal to obtain an EEG instruction based on EEG signal analysis includes:

[0054] The EEG signal is classified by an EEG signal classification model.

[0055] In this embodiment, an EEG signal classification model of preset instructions is pre-trained, and the collected EEG signals are classified by the EEG signal classification model, and finally the instruction control based on the EEG signal analysis is obtained.

[0056] Specifically, Figure 3 As shown, it also includes the EEG signal classification model training steps:

[0057] S21, collecting EEG signals of the subjects imagining preset colors and preset tool instructions;

[0058] In order to balance accuracy and efficiency, it is planned to classify frequently used colors and instructions within a controllable range, such as "red", "yellow", "white", "green", "blue", "black", etc.; commonly used instructions include but are not limited to "brush", "smear", "erase", "quick fill", etc.; in addition, a fine-tuning instruction set is provided, such as "lighter" or "darker" colors.

[0059] Based on the preset instructions that need to be classified, data that can support model training is collected in advance. In order to ensure signal quality, the experiment is generally carried out in a quiet, stable light and electromagnetic interference-free environment. For the color imagination EEG signal collection, different colors are presented on the screen in turn, and the subjects are asked to imagine the colors they see, and each color lasts for several seconds. After each color is presented, the subjects are asked to close their eyes and imagine the color, and the imagination is maintained for a fixed time. Then rest for a preset time and repeat the next color.

[0060] Similarly, for tool instructions, preset semantic instructions were displayed on the screen at a fixed frequency, and the subjects were given enough time to imagine the instructions for data collection.

[0061] S22, classifying and storing the collected data according to preset tags, and segmenting the data according to preset timestamps;

[0062] S23, removing noise through pre-processing techniques such as filtering to obtain an effective training data set;

[0063] S24, sending the training data set to a deep learning network (for example, a convolutional neural network or a long short-term memory network) for training, and finally obtaining a pre-trained model based on EEG classification.

[0064] S3, obtaining facial muscle and jaw movement information near the ears, and head and neck movement information;

[0065] like Figure 6 As shown, the in-ear wearable device is provided with a near-infrared sensor and a gyroscope. The near-infrared sensor is used to detect the movement characteristics of the facial or jaw muscle area and analyze the lip shape information of different pronunciations. The gyroscope is used to capture the head posture EEG electrodes, which are complementary and integrated with the muscle activity information captured by the near-infrared sensor to assist in lip shape analysis.

[0066] S4, analyzing the mouth shape information of different pronunciations through the facial muscle and jaw movement information, and the head and neck movement information;

[0067] In some embodiments, the step of parsing the lip shape information of different pronunciations through the facial muscle and jaw movement information, and the head and neck movement information comprises:

[0068] The facial muscle and jaw movement information, and the head and neck movement information are classified by a lip shape classification model.

[0069] Similar to the EEG signal classification model, it is necessary to build a classification model based on near-infrared sensor data and gyroscope motion data, and the classification label range is consistent with the above EEG category labels. Specifically, Figure 4 As shown, it also includes the lip shape classification model training steps:

[0070] S41. Collect the facial muscle and jaw movement information, as well as the head and neck movement information of the subjects when making different pronunciation mouth shapes;

[0071] S42, classifying and storing the collected data according to preset tags, and segmenting the data according to preset timestamps;

[0072] S43, removing noise through pre-processing techniques such as filtering to obtain an effective training data set;

[0073] S44, sending the training data set to a deep learning network (for example, a convolutional neural network or a long short-term memory network) for training, and finally obtaining a pre-trained model based on lip shape classification.

[0074] S5, fusing the EEG command with the lip shape information to determine the final user command for interactive control. This fusion solution can solve the disadvantage of low accuracy of a single modality.

[0075] It should be noted that the present embodiment adopts "silent control" and lip control in order to solve the problem of voice incompatibility and low voice recognition accuracy in noisy environments in public places, such as libraries.

[0076] This embodiment adopts but is not limited to the decision-level fusion method, which fuses the confidences of two classifier results obtained by analyzing the EEG signal and the lip signal, and selects the decision with the higher confidence as the final result.

[0077] This embodiment provides a painting-assisted interaction method based on an in-ear wearable device. For a painting interaction scenario on an electronic screen, the user's intention information is captured through the electroencephalogram sensor of the in-ear wearable device, and the user's lip shape information is captured through the infrared proximity sensor and gyroscope built into the in-ear wearable device. Finally, the pen tip control during painting is achieved through a multimodal data fusion algorithm.

[0078] Example 2

[0079] Based on the same concept, the present embodiment also provides a painting assistance interaction system based on an in-ear wearable device, and applies the painting assistance interaction method based on the in-ear wearable device provided in Example 1. For a detailed description of the painting assistance interaction method based on the in-ear wearable device provided in Example 1, refer to the corresponding description in Example 1, which will not be repeated here.

[0080] It is understandable that the painting assistance interactive system based on in-ear wearable devices provided in this embodiment includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of the examples disclosed in this embodiment, this embodiment can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of this embodiment.

[0081] A painting assistance interactive system based on in-ear wearable devices, such as Figure 5 , Figure 6As shown, the system 100 includes an in-ear wearable device 110 and a main control module 120. The in-ear wearable device is provided with EEG electrodes, a near-infrared sensor, and a gyroscope. The EEG electrodes are used to collect EEG signals, the near-infrared sensor is used to detect movement information of facial muscles and jaw, and the gyroscope is used to capture movement information of the head and neck. The main control module decides the final user command based on the EEG signals, the movement information of facial muscles and jaw, and the movement information of the head and neck for interactive control.

[0082] In some embodiments, the number of the EEG electrodes is 2 to collect EEG signals in the temporal lobe region. To improve signal quality, the EEG electrodes can be placed inside the ear canal close to the eardrum, and to eliminate noise, the EEG electrodes in the left and right in-ear wearable devices are placed symmetrically.

[0083] The main control module of this embodiment may include the following processes:

[0084] An EEG information acquisition module, used to acquire EEG signals;

[0085] An EEG signal classification module is used to classify the EEG signals and obtain EEG instructions based on EEG signal analysis;

[0086] A motion information acquisition module, used to acquire motion information of facial muscles and jaw near the ears, as well as motion information of the head and neck;

[0087] A lip shape classification module, used for analyzing lip shape information of different pronunciations through the facial muscle and jaw movement information, and the head and neck movement information;

[0088] The user instruction determination module is used to fuse the EEG instruction with the lip shape information to determine the final user instruction for interactive control.

[0089] Based on the technical solution of the above embodiment, optionally, the EEG signal is configured as an EEG signal of the temporal lobe area.

[0090] Based on the technical solution of the above embodiment, optionally, the step of classifying the EEG signal to obtain an EEG instruction based on EEG signal analysis includes:

[0091] The EEG signal is classified by an EEG signal classification model.

[0092] Based on the technical solution of the above embodiment, optionally, the method further includes the following steps:

[0093] Collect the EEG signals of the subjects imagining preset colors and preset tool instructions;

[0094] The collected data is classified and stored according to preset tags, and the data is segmented according to preset timestamps;

[0095] De-noising is performed through pre-processing techniques to obtain an effective training data set;

[0096] The training data set is sent to a deep learning network for training, and finally a pre-trained model based on EEG classification is obtained.

[0097] Based on the technical solution of the above embodiment, optionally, the step of parsing the lip shape information of different pronunciations through the facial muscle and jaw movement information, and the head and neck movement information includes:

[0098] The facial muscle and jaw movement information, and the head and neck movement information are classified by a lip shape classification model.

[0099] Based on the technical solution of the above embodiment, optionally, the method further includes a lip shape classification model training step:

[0100] Collect information on the subjects' facial muscles and jaw movements when they make different pronunciations, as well as head and neck movements;

[0101] The collected data is classified and stored according to preset tags, and the data is segmented according to preset timestamps;

[0102] De-noising is performed through pre-processing techniques to obtain an effective training data set;

[0103] The training data set is sent to a deep learning network for training, and finally a pre-trained model based on lip shape classification is obtained.

[0104] Based on the technical solution of the above embodiment, optionally, the step of fusing the EEG instruction with the lip shape information to determine the final user instruction for interactive control includes:

[0105] The EEG instruction is fused with the confidence of the lip shape information, and a decision with a high confidence is selected as the final result.

[0106] This embodiment provides a painting-assisted interaction method based on an in-ear wearable device. For a painting interaction scenario on an electronic screen, the user's intention information is captured through the electroencephalogram sensor of the in-ear wearable device, and the user's lip shape information is captured through the infrared proximity sensor and gyroscope built into the in-ear wearable device. Finally, the pen tip control during painting is achieved through a multimodal data fusion algorithm.

[0107] Example 3

[0108] A computer device 200, such as Figure 7As shown, it includes a memory 210, a processor 220, and a computer program 230 stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a drawing assistance interaction method based on an in-ear wearable device are implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, which will not be repeated here.

[0109] Example 4

[0110] A computer readable storage medium such as Figure 8 As shown, a computer program is stored thereon, and when the computer program is executed by the processor, the steps of a drawing assistance interaction method based on an in-ear wearable device are implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, and no further description is given here.

[0111] Example 5

[0112] A computer program product, the computer program product comprising a computer program, wherein when the computer program is executed by a processor, the steps of a drawing assistance interaction method based on an in-ear wearable device are implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, which will not be repeated here.

[0113] The number of devices and processing scales described here are used to simplify the description of the present invention. Applications, modifications and variations of the present invention will be obvious to those skilled in the art.

[0114] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and implementation modes. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

[0115] The apparatus, computer device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, computer device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device and non-volatile computer storage medium will not be repeated here.

[0116] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software units for implementing the method and structures within the hardware component.

[0117] The systems, devices or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions in various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or more software and / or hardware.

[0118] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may be in the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of this specification may be in the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0119] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0123] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program units. Generally, program units include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program units may be located in local and remote computer storage media, including storage devices.

[0124] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0125] The above description is only an embodiment of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of one or more embodiments of this specification.

Claims

1. A painting-assisted interaction method based on an in-ear wearable device, characterized in that: The following steps are involved: Acquire EEG signals; Classifying the EEG signals to obtain EEG instructions based on EEG signal analysis; Obtain information about facial muscle and jaw movements near the ears, as well as head and neck movements; Analyzing the mouth shape information of different pronunciations through the facial muscle and jaw movement information, and the head and neck movement information; The EEG command is integrated with the lip shape information to determine the final user command for interactive control.

2. A painting-assisted interaction method based on an in-ear wearable device as claimed in claim 1, characterized in that: The EEG signal is configured as an EEG signal of a temporal lobe area.

3. A painting-assisted interaction method based on an in-ear wearable device as claimed in claim 1, characterized in that: The step of classifying the EEG signals to obtain EEG instructions based on EEG signal analysis includes: The EEG signal is classified by an EEG signal classification model.

4. A painting-assisted interaction method based on an in-ear wearable device as claimed in claim 3, characterized in that: It also includes the steps for training the EEG signal classification model: Collect the EEG signals of the subjects imagining preset colors and preset tool instructions; The collected data is classified and stored according to preset tags, and the data is segmented according to preset timestamps; De-noising is performed through pre-processing techniques to obtain an effective training data set; The training data set is sent to a deep learning network for training, and finally a pre-trained model based on EEG classification is obtained.

5. A painting-assisted interaction method based on an in-ear wearable device as claimed in claim 1, characterized in that: The step of analyzing the lip shape information of different pronunciations through the facial muscle and jaw movement information and the head and neck movement information comprises: The facial muscle and jaw movement information, and the head and neck movement information are classified by a lip shape classification model.

6. A painting-assisted interaction method based on an in-ear wearable device as claimed in claim 5, characterized in that: It also includes the steps for training the lip shape classification model: Collect information on the subjects' facial muscles and jaw movements when they make different pronunciations, as well as head and neck movements; The collected data is classified and stored according to preset tags, and the data is segmented according to preset timestamps; De-noising is performed through pre-processing techniques to obtain an effective training data set; The training data set is sent to a deep learning network for training, and finally a pre-trained model based on lip shape classification is obtained.

7. A painting-assisted interaction method based on an in-ear wearable device as claimed in claim 1, characterized in that: The step of fusing the EEG instruction with the lip shape information to determine the final user instruction for interactive control includes: The EEG instruction is fused with the confidence of the lip shape information, and a decision with a high confidence is selected as the final result.

8. A painting assistance interactive system based on an in-ear wearable device, implementing the method according to any one of claims 1 to 7, characterized in that: The invention comprises an in-ear wearable device and a main control module. The in-ear wearable device is provided with EEG electrodes, a near-infrared sensor and a gyroscope. The EEG electrodes are used to collect EEG signals, the near-infrared sensor is used to detect the movement information of facial muscles and jaw, and the gyroscope is used to capture the movement information of the head and neck. The main control module decides the final user instructions based on the EEG signals, the movement information of facial muscles and jaw, and the movement information of the head and neck for interactive control.

9. A painting-assisted interactive system method based on an in-ear wearable device as claimed in claim 8, characterized in that: The number of the EEG electrodes is 2, and the EEG electrodes are placed on the inner side of the ear canal close to the eardrum, and the EEG electrodes in the left and right in-ear wearable devices are placed symmetrically.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images