Control device and method based on bio-electricity signal and visual perception

Abstract

The invention discloses a control device and method based on bio-electricity signals and visual perception, and the device comprises an interaction interface which is provided with an interaction button which is used for interacting with a target user and obtaining a user instruction sent by the target user; the signal acquisition module is in communication connection with the interaction interface and is used for responding to the user instruction to obtain a bio-electricity signal of a target user; the visual perception module is in communication connection with the interaction interface and is used for responding to the user instruction to obtain the face image information of the target user and the current environment information of the target user; and the instruction generation module is used for receiving a user instruction, the bio-electricity signal and the face image information and generating an interaction control instruction. According to the invention, the bio-electricity signal, the face image information and the current environment information related to the user are determined according to the user instruction, and the interaction control instruction is generated, so that the accuracy of generation of the interaction control instruction can be improved, and the reliability and accuracy of interaction between the control device and the outside are improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent control technology, and more specifically, relates to a control device and method based on bioelectric signals and visual perception. Background Art

[0002] With the development of science and technology, in order to cope with scenarios with inconvenient operations or to improve the operational convenience of various systems, devices, structures, etc., there are various intelligent control methods in the prior art. For example, by capturing the user's body changes, determining the user's control needs, and then performing corresponding control operations on behalf of the user.

[0003] Based on bioelectric signals and visual perception, as a technology that uses human bioelectric signals and visual perception to achieve control, it can be widely used in intelligent control methods in many fields, such as medical, human-computer, virtual reality, etc., so that users can achieve intelligent control of external objects based on bioelectric signals and vision. For example, by using electrodes to capture muscle contraction signals or nerve signals and converting them into control signals, however, this control method does not determine whether the user's current state is suitable for making corresponding operations, so there are great safety risks in many application scenarios. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, the purpose of the present invention is to solve the above-mentioned defects and further propose a control method based on bioelectric signals and visual perception.

[0005] The present invention adopts the following technical solution:

[0006] The first aspect of the present invention discloses a control device based on bioelectric signals and visual perception, the device comprising:

[0007] An interactive interface, wherein the interactive interface is provided with interactive buttons for interacting with a target user and acquiring user instructions issued by the target user;

[0008] A signal acquisition module, connected to the interactive interface for acquiring the bioelectric signal of the target user in response to the user instruction;

[0009] A visual perception module, connected to the interactive interface for responding to the user instruction and for acquiring facial image information of the target user and current environment information of the target user;

[0010] The instruction generation module is used to receive the user instruction, the bioelectric signal and the facial image information, and generate an interactive control instruction.

[0011] In some possible embodiments, the bioelectric signal includes at least one or more of a myoelectric potential signal, current posture information, and posture change information; the signal acquisition module includes: an electromyographic signal acquisition module and a posture information acquisition module;

[0012] The electromyographic signal acquisition module is used to acquire the muscle potential signal; the muscle potential signal is determined based on the potential change of the muscle surface;

[0013] The posture information collection module is used to collect the posture information or the posture change information of the target user.

[0014] In some possible embodiments, the visual perception module includes an image processing module and an environment recognition module;

[0015] The image processing module is used to obtain facial image information of the target user; the facial image information includes facial feature information and target tracking information;

[0016] The environment recognition module is used to obtain the current environment information of the target user; the current environment information includes object recognition information and road detection information.

[0017] A second aspect of the present invention discloses a control method based on bioelectric signals and visual perception, which is applied to the control device based on bioelectric signals and visual perception as described in any one of the above items, and the method comprises:

[0018] Displaying an interactive interface, and receiving a user instruction based on the interactive interface; the user instruction corresponds to a first interactive button or a second interactive button on the interactive interface; the user instruction represents the control demand of the target user for the operating system;

[0019] When the user instruction corresponds to the first interactive button, acquiring a bioelectric signal corresponding to the target user;

[0020] Performing signal analysis on the bioelectric signal to obtain a signal analysis result;

[0021] Based on the signal analysis result, an interactive control instruction matching the control requirement is generated; the interactive control instruction is used to control the operating system to be operated.

[0022] In some possible embodiments, performing signal analysis on the bioelectric signal to obtain a signal analysis result includes:

[0023] When the collected bioelectric signal corresponds to the muscle potential signal, amplifying the muscle potential signal to obtain an amplified muscle potential signal;

[0024] Performing signal conversion processing on the amplified muscle potential signal to obtain a muscle simulation signal;

[0025] A muscle signal waveform corresponding to the muscle simulation signal is output, and a signal analysis is performed based on the muscle signal waveform to determine the signal analysis result, wherein the signal analysis result corresponding to the muscle potential signal includes that the target user's body is in a relaxed state, or that the target user's body is in a tense state.

[0026] In some possible embodiments, performing signal analysis on the bioelectric signal to obtain a signal analysis result includes:

[0027] When the collected bioelectric signal corresponds to the current posture information and / or the posture change information, a posture angle parameter is determined based on the current posture information and / or the posture change information; the posture angle parameter is used to characterize the angle corresponding to any limb part of the target user.

[0028] In some possible embodiments, the method further includes:

[0029] When the user instruction corresponds to the second interactive button, acquiring facial image information of the target user;

[0030] Performing facial recognition on the facial image information to determine the face recognition information of the target user;

[0031] Performing user detection based on the face recognition information to obtain a user detection result; the user detection result indicates that the target user is a known user or the target user is an unknown user;

[0032] Performing face tracking based on the face recognition information and the facial contour recognition information to obtain face tracking information;

[0033] The interactive control instruction is generated based on the face tracking information.

[0034] The third aspect of the present invention discloses an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction and at least one program, and the at least one instruction and the at least one program are loaded and executed by the processor to implement the control method based on bioelectric signals and visual perception as described above.

[0035] The fourth aspect of the present invention discloses a computer storage medium, which stores at least one instruction and at least one program. The at least one instruction and the at least one program are loaded and executed by a processor to implement the control method based on bioelectric signals and visual perception as described above.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] In the present invention, an interactive interface is provided for the control device, which can facilitate the user to intuitively perceive the function of the control device. The user issues instructions based on the interactive interface, thereby realizing information interaction between the user and the control device, thereby improving the convenience of the user in the use of the control device. The control device also includes a signal acquisition module, a visual perception module, and an instruction generation module, so that the control device can respond to the user instructions issued by the user on the interactive interface, obtain the bioelectric signal and image information corresponding to the user instruction, and generate corresponding interactive control instructions according to the bioelectric signal and image information, which can improve the convenience and accuracy of the control operation; because the acquired bioelectric signal and image signal match the current state of the user, it can improve the adaptability between the control method executed by the control device and the actual instruction of the user. As for the control method, after acquiring the bioelectric signal and the image signal, the interactive control instruction is not generated directly. Instead, the bioelectric signal is analyzed to determine whether the user's current state can adapt to the control operation made by the control device, and then the interactive control instruction is generated according to the signal analysis result. When the user's current state can adapt to the control operation made by the control device, the interactive control instruction matching the user instruction is executed. This can reduce the potential risks brought by the direct execution of the interactive control instruction and improve the safety of the user application process. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of the structure of a control device based on bioelectric signals and visual perception provided by an embodiment of the present invention;

[0039] Figure 2 A first schematic diagram of an interactive interface provided by a specific embodiment of the present invention;

[0040] Figure 3 A second schematic diagram of an interactive interface provided by a specific embodiment of the present invention;

[0041] Figure 4 A schematic diagram of an interface corresponding to obtaining posture information provided in a specific embodiment of the present invention;

[0042] Figure 5 A third schematic diagram of an interactive interface provided by a specific embodiment of the present invention;

[0043] Figure 6 is a schematic flow chart corresponding to a control method based on bioelectric signals and visual perception provided by an embodiment of the present invention;

[0044] Figure 7 A schematic diagram of a process flow corresponding to the collection of muscle potential signals provided in an embodiment of the present invention;

[0045] Figure 8 A schematic flow chart corresponding to another control method based on bioelectric signals and visual perception provided by an embodiment of the present invention;

[0046] Fig. 9 A flowchart corresponding to face recognition and face tracking is provided for a specific embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0049] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0050] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0051] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0052] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0053] Figure 1 is a schematic diagram of the structure of the control device based on bioelectric signals and visual perception provided by an embodiment of the present invention; Figure 1 As shown, the structure of the control device based on bioelectric signals and visual perception may include:

[0054] An interactive interface, wherein the interactive interface is provided with interactive buttons for interacting with a target user and acquiring user instructions issued by the target user;

[0055] In a specific embodiment, Figure 2 A first schematic diagram of an interactive interface provided by a specific embodiment of the present invention; Figure 2 As shown, the interactive interface may at least include the system name "Welcome to the bioelectric and visual perception system", the first interactive button "Bioelectric signal acquisition" and the second interactive button "Visual perception"; in addition, it may also include a "Help" button, which can facilitate the target user to understand the operation of the control device by triggering the "Help" button.

[0056] The interactive interface can receive user instructions issued by the user by responding to the user's click operation on the interactive button; it can also obtain user instructions through other methods, such as setting up a voice recognition module to determine the corresponding user instructions by obtaining the user's voice and recognizing the content.

[0057] A signal acquisition module, which is in communication connection with the interactive interface, and is used to obtain the bioelectric signal of the target user in response to the user instruction; wherein the signal acquisition module includes: an electromyographic signal acquisition module and a posture information acquisition module; the bioelectric signal includes at least one or more of a muscle potential signal, current posture information, and posture change information;

[0058] The electromyographic signal acquisition module is used to acquire the muscle potential signal; the muscle potential signal is determined based on the potential change on the muscle surface; the posture information acquisition module is used to acquire the posture information or the posture change information of the target user.

[0059] In a specific embodiment, Figure 3 A second schematic diagram of an interactive interface provided by a specific embodiment of the present invention; Figure 3As shown, when the user instruction corresponds to the first interactive button, the bioelectric signal acquisition interface is entered. In the bioelectric signal acquisition interface, two interactive buttons, "electromyographic signal acquisition" and "posture information acquisition", can also be included, which correspond to the electromyographic signal acquisition module and the posture information acquisition module respectively.

[0060] The electromyographic signal acquisition module is used to collect muscle potential signals. Specifically, it can be the potential signal corresponding to any muscle part of the target user. The specific muscle part to collect the potential signal depends on the specific location setting of the electromyographic signal acquisition module. For example, the muscle near the biceps can be selected to collect muscle potential signals; when neurons transmit information, corresponding potential changes will occur on the muscle surface. By collecting muscle potential signals, the current state of muscle tissue can be analyzed, and corresponding interactions with the outside world can be carried out. For the electromyographic signal acquisition module, the MuscleSensor module can be used. As a high-performance sensor, the Muscle Sensor module can amplify the weak and complex potential changes in the muscle and convert them into simple analog signals. The disposable medical Ag / AgCl (silver / silver chloride) ECG electrodes used in the acquisition process are attached to the designated muscle parts to collect muscle potential signals.

[0061] The posture information acquisition module is used to collect the posture information or posture change information of the target user. The ATK-IMU901 angle sensor module can be selected as the posture information acquisition module. The serial port can directly output the posture angle, which can be used for inclination measurement, posture angle measurement, air pressure measurement, magnetic field measurement, and altitude measurement; the posture information can include posture angle parameters, and the posture change information is used to characterize the posture changes of any limb part of the target user during measurement. Figure 4 The following is a schematic diagram of an interface corresponding to the posture information acquisition provided in a specific embodiment of the present invention, such as Figure 4 As shown, the posture information or posture change information may include acceleration, angular velocity, angle, magnetic field, air pressure and other data. In addition, in addition to the posture information corresponding to the target user, the posture position and other information of the operating system to be operated may also be measured to realize the state detection of the operating system to be operated, so as to facilitate the subsequent judgment of whether the operation performed on the operating system is risky, thereby improving the safety of the control operation process.

[0062] A visual perception module, which is in communication with the interactive interface and is used to respond to the user instruction and obtain the facial image information of the target user and the current environment information of the target user; wherein the visual perception module includes an image processing module and an environment recognition module;

[0063] The image processing module is used to obtain facial image information of the target user; the facial image information is used to obtain face tracking information of the target user; the environment recognition module is used to obtain current environment information of the target user; the current environment information includes object recognition information and road detection information.

[0064] In a specific embodiment, Figure 5 A third schematic diagram of an interactive interface provided by a specific embodiment of the present invention; Figure 5 As shown, when the user instruction corresponds to the second interactive button, the visual perception interface is entered. In the visual perception interface, two interactive buttons, "object recognition" and "person recognition and tracking", can also be included, which correspond to the environment recognition module and the image processing module respectively.

[0065] You can choose a development board equipped with a Kendryte K210 chip as an environmental recognition module, use the yolo target detection algorithm, the Keras-yolo target detection machine learning algorithm, and a certain amount of data sets to train the target detection model. Use the K210 module to implement object recognition and road information detection at the hardware level to obtain real-time data; you can use a camera with the OpenCV computing library as an image processing module, and implement face recognition and tracking by calling the TLD library function. After calling the camera, facial features are detected and related data is collected, and the TLD target tracking algorithm is used to learn the collected information and optimize target tracking.

[0066] The instruction generation module is used to receive the user instruction, the bioelectric signal and the facial image information, and generate an interactive control instruction.

[0067] In a specific embodiment, user instructions, bioelectric signals and facial image information are input into an instruction generation module to obtain interactive control instructions for controlling the operating system to be operated; the output of the interactive instructions can be controlled by a relay. In a specific application scenario, the smart wheelchair is determined to be the operating system to be operated, and the following will take this as an example to describe the generation of interactive control instructions.

[0068] In addition, the control device may further include a communication detection module for detecting the communication status between the signal acquisition module and the visual perception module and the instruction generation module respectively, so as to ensure the reliability of the control device and improve the timeliness of fault detection.

[0069] Figure 6 is a flowchart corresponding to the control method based on bioelectric signals and visual perception provided by an embodiment of the present invention. The execution subject may be a device that can implement the control method based on bioelectric signals and visual perception; please refer to Figure 6In one embodiment, a control method based on bioelectric signals and visual perception includes the following steps:

[0070] Step S601: displaying an interactive interface, and receiving a user instruction based on the interactive interface; the user instruction corresponds to a first interactive button or a second interactive button on the interactive interface; the user instruction represents the control demand of the target user for the operating system;

[0071] In a specific embodiment, an interactive interface is displayed to a target user, and the reception of user instructions is achieved by receiving the triggering operation of the target user on the interactive button; the first interactive button may correspond to bioelectric signal collection, and the second interactive button may correspond to visual perception; based on the user's triggering of the interactive button, the control requirements corresponding to the user instructions are determined.

[0072] Step S602: when the user instruction corresponds to the first interactive button, obtaining a bioelectric signal corresponding to the target user;

[0073] In a specific embodiment, when the user instruction corresponds to the first interactive button, it is necessary to execute the collection of bioelectric signals corresponding to the target user, so it is possible to jump to the page corresponding to the bioelectric signal collection. This page may include two interactive buttons, "electromyographic signal collection" and "posture information acquisition", which correspond to the electromyographic signal collection module and the posture information collection module respectively.

[0074] The bioelectric signal includes at least one or more of a muscle potential signal, current posture information and posture change information; the collection of the bioelectric signal means collecting the muscle potential signal, current posture information and posture change information.

[0075] Step S603: performing signal analysis on the bioelectric signal to obtain a signal analysis result;

[0076] In a specific embodiment, there are different types of bioelectric signals, and there are different signal analysis methods corresponding to different types of bioelectric signals. Specifically, Figure 7 The specific implementation process of step S603 is described.

[0077] Figure 7 A schematic diagram of the process flow corresponding to the muscle potential signal acquisition provided by the embodiment of the present invention; Figure 7 As shown, the signal analysis of the bioelectric signal to obtain the signal analysis result includes:

[0078] Step S701: when the collected bioelectric signal corresponds to the muscle potential signal, amplify the muscle potential signal to obtain an amplified muscle potential signal;

[0079] In a specific embodiment, for muscle potential signals, since there may be weak and complex potential signals, the muscle potential signals need to be amplified to obtain amplified muscle potential signals; the amplified muscle potential signals can amplify the complex and weak potential signals, thereby ensuring the comprehensiveness and reliability of the signal data, and improving the convenience of muscle potential signal analysis.

[0080] Step S702: performing signal conversion processing on the amplified muscle potential signal to obtain a muscle state simulation signal;

[0081] Step S703: Output the muscle state signal waveform corresponding to the muscle state simulation signal, and perform signal analysis based on the muscle state signal waveform to determine the signal analysis result. The signal analysis result corresponding to the muscle potential signal includes that the target user's body is in a relaxed state, or that the target user's body is in a tense state.

[0082] In a specific embodiment, the amplified muscle potential signal is converted into a corresponding analog signal, i.e., a muscle state analog signal, and then a corresponding muscle state signal waveform is obtained based on the muscle state analog signal; a signal analysis is performed based on the muscle state signal waveform to determine whether the target user's body is in a relaxed state or a tense state; for example, when the muscle state signal waveform corresponds to a muscle relaxation state, it can be determined that the target user's body is in a relaxed state, and when the muscle state signal waveform corresponds to a muscle tension state, it can be determined that the target user's body is in a tense state. The tense state and the relaxed state of the target user are determined as the signal analysis results, so as to facilitate the determination of whether the current state of the target user can adapt to the control device to perform corresponding operations on the smart wheelchair.

[0083] In addition, when the bioelectric signal corresponds to posture information or posture change information, the bioelectric signal is subjected to signal analysis to obtain a signal analysis result, including:

[0084] When the collected bioelectric signal corresponds to the current posture information and / or the posture change information, a posture angle parameter is determined based on the current posture information and / or the posture change information; the posture angle parameter is used to characterize the angle corresponding to any limb part of the target user.

[0085] In a specific embodiment, the current posture information is determined by acquiring the posture change of the target user, and the posture change information is determined by acquiring the position change of any limb part of the target user, so as to obtain the posture angle parameters, acceleration, angular velocity, magnetic field pressure and other data; the posture change information can correspond to the change information of a certain part of the target user's body, and obtaining the current posture information and posture change information can determine the current state of the target user, and can also detect the posture position and other information of the wheelchair, so as to facilitate the monitoring of the driving state of the wheelchair; the current posture information and posture change information can be uploaded to the host computer control system to execute the generation of posture angle parameters and other processing.

[0086] In another specific embodiment, the user instruction may also correspond to a second interactive button. Figure 8 A flow chart corresponding to another control method based on bioelectric signals and visual perception provided by an embodiment of the present invention; Figure 8 As shown, the method includes:

[0087] Step S801: when the user instruction corresponds to the second interactive button, acquiring facial image information of the target user;

[0088] In a specific embodiment, when the user instruction corresponds to the second interactive button, it is necessary to perform a visual perception analysis corresponding to the target user, so it is possible to jump to the page corresponding to the visual perception. This page may include two interactive buttons, "Object Recognition" and "Character Recognition and Tracking", which correspond to the environment recognition module and the image processing module respectively.

[0089] Step S802: performing facial recognition on the facial image information to determine the face recognition information of the target user;

[0090] In a specific embodiment, based on the OpenCV computer vision library, a camera can be called to collect facial image information, wherein the facial image information can be current video screen data or current picture screen data. In the present invention, the facial image information is set as video screen data; face detection is performed based on the facial image information to determine the location of the face, and the range of the screen in the video is further segmented based on the location of the face to obtain face recognition information.

[0091] Step S803: Perform user detection based on the face recognition information to obtain a user detection result; the user detection result indicates that the target user is a known user or the target user is an unknown user;

[0092] In a specific embodiment, the cascade classifier provided by OpenCV is used to create an image data set of a specific face for training. The trained model file is used in the prediction program to identify whether the face in the current picture is consistent with the specific face, thereby determining whether the target user is a known user or an unknown user.

[0093] Step S804: performing face tracking based on the face recognition information and the facial contour recognition information to obtain face tracking information;

[0094] In a specific embodiment, in addition to determining whether the target user is a known user, facial tracking can also be performed based on face recognition information to obtain face tracking information; specifically, after calling the cascade classifier to detect the face in the picture, a box is drawn around it for easy identification, and Gaussian filtering and threshold classification are performed on the mouth contour and pupil part to facilitate capturing the corresponding position changes and obtain face contour feature tracking information; the face tracking information can also be converted into a grayscale image for easy viewing.

[0095] Step S805: Generate the interactive control instruction based on the face tracking information and the facial feature tracking information.

[0096] In a specific embodiment, the movement limit is set manually, and the movement range of the mouth and pupil is calibrated, the corresponding position of the line of sight is determined based on the face tracking information and the face feature contour tracking information, and then the control requirements corresponding to the user instructions are judged based on the corresponding position, and interactive control instructions for controlling the operating system (such as a smart wheelchair) are generated.

[0097] Fig. 9 A flowchart corresponding to face recognition and facial feature tracking provided by a specific embodiment of the present invention is as follows: Fig. 9 As shown, after acquiring the facial image information captured by the camera, the facial image information can be range segmented and face detected, and the face detection result can be calibrated with a frame to obtain face recognition information; after obtaining the face recognition information, face verification is performed based on the face recognition information to determine whether the user is an unknown user or a known user, and facial feature detection is performed based on the face recognition information, and the frame is calibrated according to the detection result to obtain facial feature recognition information; and then Gaussian filtering and threshold separation are performed on the recognition information to obtain an output image corresponding to the facial tracking information.

[0098] In another specific embodiment, when the user instruction corresponds to the second interactive button, in addition to obtaining the facial image information of the target user, the current environment information of the target user can also be obtained, and the current environment information may include object recognition information and road detection information.

[0099] Specifically, the Yolo model can be trained, the training results and the corresponding model can be burned into the environment recognition module, and the current environment information can be recognized based on the Yolo model to determine the object recognition information and road detection information in the current environment information. Then, the object recognition information and the road detection information can be combined to determine whether there are obstacles or other dangerous objects that hinder the safe operation (such as movement) of the operating system (i.e., the smart wheelchair) in the current environment, thereby reducing the risks when the operating system is controlled and improving the safety of the control of the operating system.

[0100] Step S604: Based on the signal analysis result, an interactive control instruction matching the control requirement is generated; the interactive control instruction is used to control the operating system to be operated.

[0101] In a specific embodiment, the interactive control may include at least one of a secondary confirmation instruction and a demand control instruction, wherein the secondary confirmation instruction is used to issue a secondary confirmation to the target user to determine whether the target user confirms to execute the control demand in the current state; the demand control instruction may be an instruction that can control the operating system to execute an operation that satisfies the user's control demand. The interactive control instruction may be determined in combination with the signal analysis results, such as whether the state of the target user can adapt to the changes brought about by the control device operating the operating system. For example, if the control demand is to control the intelligent wheelchair to accelerate forward, but the current posture information shows that the target user is not sitting steadily on the intelligent wheelchair, then there is a potential risk of accelerating at this time. The interactive control instruction may again confirm the control demand to the target user to ensure that the operating system is controlled in a safe state, reduce the risks in the control process, and improve safety.

[0102] An embodiment of the present invention also provides an electronic device, comprising: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a control method based on bioelectric signals and visual perception as described in any one of the method embodiments.

[0103] The embodiment of the present invention also provides a storage medium, and the computer-readable storage medium can be a tangible device that keeps and stores the instructions used by the instruction execution device. The computer-readable storage medium can be an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or convex structures in grooves on which instructions are stored, and any suitable combination of the above. The computer-readable storage medium used here is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses through optical fiber cables), or electrical signals transmitted by wires.

[0104] The computer-readable program instructions described in the above content can be downloaded from the computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0105] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0106] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0108] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0109] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the square frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square frames can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square frame in the block diagram and / or flow chart, and the combination of the square frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special-purpose hardware and computer instructions.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific implementation modes of the present invention can still be modified or replaced by equivalents, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A control device based on bioelectric signals and visual perception, characterized in that: The device comprises: An interactive interface, wherein the interactive interface is provided with interactive buttons for interacting with a target user and acquiring user instructions issued by the target user; A signal acquisition module, connected to the interactive interface for acquiring the bioelectric signal of the target user in response to the user instruction; A visual perception module, connected to the interactive interface for responding to the user instruction and for acquiring facial image information of the target user and current environment information of the target user; The instruction generation module is used to receive the user instruction, the bioelectric signal and the facial image information, and generate an interactive control instruction.

2. A control device based on bioelectric signals and visual perception according to claim 1, characterized in that: The bioelectric signal includes at least one or more of a muscle potential signal, current posture information, and posture change information; The signal acquisition module includes: an electromyographic signal acquisition module and a posture information acquisition module; The electromyographic signal acquisition module is used to acquire the muscle potential signal; the muscle potential signal is determined based on the potential change of the muscle surface; The posture information collection module is used to collect the posture information or the posture change information of the target user.

3. A control device based on bioelectric signals and visual perception according to claim 1, characterized in that: The visual perception module includes an image processing module and an environment recognition module; The image processing module is used to obtain facial image information of the target user; the facial image information includes facial feature information and target tracking information; The environment recognition module is used to obtain the current environment information of the target user; the current environment information includes object recognition information and road detection information.

4. A control method based on bioelectric signals and visual perception, applied to the control device based on bioelectric signals and visual perception as claimed in any one of claims 1 to 3, characterized in that: The method comprises: Displaying an interactive interface, and receiving a user instruction based on the interactive interface; the user instruction corresponds to a first interactive button or a second interactive button on the interactive interface; the user instruction represents the control demand of the target user for the operating system; When the user instruction corresponds to the first interactive button, acquiring a bioelectric signal corresponding to the target user; Performing signal analysis on the bioelectric signal to obtain a signal analysis result; Based on the signal analysis result, an interactive control instruction matching the control requirement is generated; the interactive control instruction is used to control the operating system to be operated.

5. The control method based on bioelectric signals and visual perception according to claim 4, characterized in that: The bioelectric signal includes at least one or more of a muscle potential signal, current posture information and posture change information; The performing signal analysis on the bioelectric signal to obtain a signal analysis result includes: When the collected bioelectric signal corresponds to the muscle potential signal, amplifying the muscle potential signal to obtain an amplified muscle potential signal; Performing signal conversion processing on the amplified muscle potential signal to obtain a muscle state simulation signal; Output a muscle state signal waveform corresponding to the muscle state simulation signal, and perform signal analysis based on the muscle state signal waveform to determine the signal analysis result, wherein the signal analysis result corresponding to the muscle potential signal includes that the target user's body is in a relaxed state, or that the target user's body is in a tense state.

6. The control method based on bioelectric signals and visual perception according to claim 5, characterized in that: The performing signal analysis on the bioelectric signal to obtain a signal analysis result includes: When the collected bioelectric signal corresponds to the current posture information and / or the posture change information, a posture angle parameter is determined based on the current posture information and / or the posture change information; the posture angle parameter is used to characterize the angle corresponding to any limb part of the target user.

7. The control method based on bioelectric signals and visual perception according to claim 4, characterized in that: The method further comprises: When the user instruction corresponds to the second interactive button, acquiring facial image information of the target user; Performing facial recognition on the facial image information to determine the face recognition information of the target user; Performing user detection based on the face recognition information to obtain a user detection result; the user detection result indicates that the target user is a known user or the target user is an unknown user; Performing facial tracking based on the face recognition information and the facial contour recognition information to obtain face tracking information; The interactive control instruction is generated based on the face tracking information.

8. An electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction and at least one program, and the at least one instruction and the at least one program are loaded and executed by the processor to implement a control method based on bioelectric signals and visual perception as described in any one of claims 4-7.

9. A computer storage medium, wherein at least one instruction and at least one program are stored in the computer storage medium, wherein the at least one instruction and the at least one program are loaded and executed by a processor to implement a control method based on bioelectric signals and visual perception as described in any one of claims 4 to 7.

Images