Brain-controlled assistive grasping robot system and control method based on visual semantic target stimulation

Abstract

The present invention discloses a brain-controlled grasping robot system and control method for visual semantic target stimulation, including an EEG signal acquisition unit, a visual induced unit, a signal processing unit, and a grasping robot body; the grasping robot body acquires the semantic target image and posture information of each object in the environment; and grasps the object according to the brain-controlled grasping intention; the visual induced unit flashes the visual induced paradigm based on the semantic target stimulation in a preset flashing mode; the EEG signal acquisition unit acquires the EEG signal induced by the visual induced paradigm of the disabled operator; the signal processing unit decodes the EEG signal to obtain the brain-controlled grasping intention, and sends it to the grasping robot body, and also uses the knowledge graph to infer the next step of the brain-controlled grasping intention of the disabled operator, and feeds back to the visual induced unit to generate a visual induced sub-paradigm. The intelligent decision-making and execution capabilities of the brain-controlled grasping robot for assisting the disabled are improved, and the disabled operator is helped to achieve basic self-care.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent control, and relates to a visual semantic target stimulated brain-controlled assistive grasping robot system and a control method. Background Art

[0002] Brain control technology is an emerging technology that enables direct interaction between the brain and external devices. This technology collects the cerebral cortex electrical signals of the disabled operator's brain control intentions, decodes the disabled operator's brain control intentions with the help of machine learning methods, and converts them into control instructions for peripheral devices, such as the control signals of the disabled grasping robot that helps patients with neurological diseases restore basic self-care abilities, thereby achieving direct interaction between the disabled operator and peripheral devices. This technology is currently mainly used to help patients with neurological diseases, such as ALS and stroke patients, restore basic self-care abilities, thereby effectively reducing the burden on families and society.

[0003] However, the current research on brain-computer interaction rehabilitation robot technology is mostly focused on the research of brain control intention decoding algorithms, and there are fewer implementations of brain-computer fusion methods. For example, for brain-controlled grasping robot systems for assisting the disabled, they are generally only induced according to the traditional visual induced paradigm, or can only grasp objects at specific locations according to decoded brain control instructions. This results in the existing brain-controlled grasping robot systems for assisting the disabled having poor human-computer interaction capabilities and insufficient autonomous decision-making capabilities, which in turn leads to the inability to achieve accurate grasping in dynamic unstructured environments, and cannot effectively help disabled operators achieve basic self-care abilities. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art, that is, the brain-controlled assistive grasping robot lacks human-computer interaction ability, insufficient autonomous decision-making ability and simple control strategy in a dynamic unstructured environment, and cannot effectively help disabled operators to achieve basic self-care ability, and to provide a brain-controlled assistive grasping robot system and control method with visual semantic target stimulation.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a visual semantic target stimulation brain-controlled assistive grasping robot system, comprising an electroencephalogram signal acquisition unit, a visual induction unit, a signal processing unit and a grasping robot body; the electroencephalogram signal acquisition unit is sequentially connected to the signal processing unit, the grasping robot body and the visual induction unit;

[0007] The grasping robot body is used to collect scene information in a dynamic unstructured environment, and segment the target image with object information contained in the scene information, and obtain the semantics of each target image; obtain the semantic target image, position information and posture information of each object in the dynamic unstructured environment according to the scene information and the semantics of each target image; and send the semantic target image of each object to the visual induction unit; and receive the brain-controlled grasping intention sent by the signal processing unit, and generate a grasping control signal according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset setting position information, and grasp the object to be grasped according to the grasping control signal and transport it to the set position;

[0008] The visual evoked unit is used to generate a visual evoked paradigm based on semantic target stimulation according to the semantic target image of each object, and flash according to a preset flashing mode;

[0009] The EEG signal acquisition unit is used to acquire EEG signals induced by the disabled operator based on the semantic target stimulation visual induced paradigm, and send them to the signal processing unit;

[0010] The signal processing unit is used to decode the EEG signal to obtain the brain-controlled grasping intention and send it to the grasping robot body; as well as to infer the next brain-controlled grasping intention of the disabled operator and feed it back to the visual induction unit.

[0011] Optionally, the grasping robot body includes a visual perception module, a motion control module and a mechanical arm structure module; the visual perception module is connected to the motion control module and the visual induction unit, and the motion control module is connected to the signal processing unit;

[0012] The visual perception module is used to collect scene information in a dynamic unstructured environment, segment target images with semantic information contained in the scene information, and obtain the semantics of each target image; obtain the semantic target image, position information and posture information of each object in the dynamic unstructured environment according to the scene information and the semantics of each target image; send the semantic image of each object to the visual induction unit, and send the position information and posture information of each object to the motion control module;

[0013] The motion control module is used to receive the brain-controlled grasping intention sent by the signal processing unit, and the position information and posture information of each object sent by the visual perception module; generate a grasping control signal according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset set position information, and send it to the mechanical arm structure module;

[0014] The robot arm structure module is used to grasp the object to be grasped and transport it to the set position according to the grasping control signal.

[0015] Optionally, generating a grasping control signal according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset set position information includes:

[0016] According to the brain-controlled grasping intention, the position information of each object and the posture information of each object, the position information and posture information of the object to be grasped are obtained;

[0017] The Denavit-Hartenberg method is used to perform kinematic modeling on the robot arm structure module, and the position information and posture information of the object to be grasped are combined to obtain the position information and posture information of the robot arm structure module reaching the object to be grasped; and according to the position information and posture information of the robot arm structure module reaching the object to be grasped, the inverse kinematics of the robot joint is solved to obtain the angle at which the robot arm structure module needs to rotate;

[0018] According to the position information of the object to be grasped and the preset set position information, the path planning information of the robot arm structure module is obtained; and according to the angle at which the robot arm structure module needs to rotate and the path planning information of the robot arm structure module, a grasping control signal is generated.

[0019] Optionally, the collecting of scene information in a dynamic unstructured environment, segmenting target images with object information contained in the scene information, and acquiring semantics of each target image includes:

[0020] Collect scene information in a dynamic unstructured environment; call a preset detection and segmentation model to detect and segment the target image with object information contained in the scene image of the scene information, and obtain the semantic information of the target image; wherein the detection and segmentation model adopts a YOLOv5 network model trained with labeled images.

[0021] Optionally, the grasping robot body is also used to obtain thrust feedback information and analyze it to obtain a thrust feedback value; and when the thrust feedback value is greater than a set thrust feedback threshold, a feedback control signal is generated, and according to the feedback control signal, the grasped object to be grasped is transported to the original position.

[0022] Optionally, generating a visual evoked paradigm based on semantic target stimulation according to the semantic target image of each object, and flashing in a preset flashing manner includes:

[0023] After the semantic target images of each object were subjected to image edge contour extraction and grayscale processing, a visual evoked paradigm based on semantic target stimulation was generated, and the visual evoked paradigm based on semantic target stimulation was flashed in a sinusoidal modulation manner.

[0024] Optionally, decoding the EEG signal to obtain the brain control capture intention includes:

[0025] The EEG signals were subjected to 0.5-45Hz Butterworth bandpass filtering and trend removal processing, and then the canonical correlation analysis algorithm based on genetic algorithm and filter bank was used to perform pattern recognition on the target steady-state visual evoked potential to obtain the brain-controlled grasping intention of the disabled operator.

[0026] Optionally, the signal processing unit is further connected to a visual induction unit, wherein the inference of the next brain-controlled grasping intention of the disabled operator includes:

[0027] The signal processing unit is further used to obtain semantic target images of several inference intentions according to the brain-controlled grasping intention through a pre-built dynamic unstructured scene knowledge graph, and send the obtained semantic target images of several inference intentions as the next brain-controlled grasping intention of the disabled operator to the visual induction unit;

[0028] The visual evoked unit is also used to generate a visual evoked sub-paradigm based on semantic target stimulation according to a number of semantic target images of inference intentions, and flash according to a preset flashing mode.

[0029] Optionally, the semantic target images of several reasoning intentions are obtained through a pre-built dynamic unstructured scene knowledge graph according to the brain-controlled grasping intention, including:

[0030] Encode the brain-controlled grasping intention into a feature vector to obtain the intention information vector;

[0031] The intention information vector is matched with the structured vector in the dynamic unstructured scene knowledge graph, and the object information node corresponding to the structured vector with a matching degree higher than a preset value is used as the starting node. According to the path-based search method, the object entity information node representing the further intention of the disabled operator is searched in the dynamic unstructured scene knowledge graph to obtain several entity information nodes of the inferred intention;

[0032] According to several inference intention entity information nodes, the next intention inference result of the disabled operator is obtained, and according to the next intention inference result of the disabled operator, several semantic target images of the inference intention are generated.

[0033] The second aspect of the present invention provides a control method of a visual semantic target stimulation-based brain-controlled handicapped grasping robot system, comprising:

[0034] The robot body is grasped to collect scene information in a dynamic unstructured environment, and the target image with object information contained in the scene information is segmented, and the semantics of each target image is obtained; according to the scene information and the semantics of each target image, the semantic target image, position information and posture information of each object in the dynamic unstructured environment are obtained; and the semantic target image of each object is sent to the visual elicitation unit;

[0035] The visual evoked unit generates a visual evoked paradigm based on semantic target stimulation according to the semantic target image of each object, and flashes according to a preset flashing mode;

[0036] The electroencephalogram (EEG) signal of the disabled operator induced by the visual induced paradigm is collected by the EEG signal collection unit and sent to the signal processing unit;

[0037] The signal processing unit decodes the EEG signal to obtain the brain-controlled grasping intention and sends it to the grasping robot body; and infers the next brain-controlled grasping intention of the disabled operator and feeds it back to the visual evoked unit;

[0038] The grasping robot body receives the brain-controlled grasping intention sent by the signal processing unit, and generates a grasping control signal according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset set position information, and grasps the object to be grasped according to the grasping control signal and transports it to the set position.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The brain-controlled assistive grasping robot system with visual semantic target stimulation of the present invention collects scene information in a dynamic unstructured environment through a grasping robot body, segments target images containing object information, and obtains the semantics of each target image, and then obtains the semantic target image of each object and sends it to a visual induction unit, and generates a visual induction paradigm based on dynamic semantic target stimulation based on the visual induction unit, and the disabled operator gazes at the corresponding semantic target image in the visual induction paradigm to induce his steady-state visual evoked potential to express his brain control intention, and the induced steady-state visual evoked potential signal, i.e., the electroencephalogram signal, is collected by the electroencephalogram signal acquisition unit and sent to the signal processing unit for processing, and the brain control intention of the disabled operator is obtained and the brain-controlled grasping intention is parsed and sent to the grasping robot body, and the grasping robot body generates a grasping control signal according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset set position information, and grasps the object to be grasped according to the grasping control signal and transports it to the set position. At the same time, the signal processing unit will also obtain several semantic target images of reasoning intentions based on the decoded information of the subject's current brain control intentions through the pre-built dynamic unstructured scene knowledge graph, and feed them back to the visual induced unit. It can be seen that the brain-controlled disabled grasping robot system stimulated by visual semantic targets combines the brain-computer interface technology based on biological intelligence with the grasping robot body with autonomous grasping ability, and effectively improves the intelligent decision-making and execution capabilities of the brain-computer fusion grasping robot through human-computer interaction; by extracting the semantic target images of each object in the dynamic unstructured environment, a visual induced brain control paradigm based on the flashing of semantic target images is established, which improves the adaptability of the brain-computer fusion grasping robot in the dynamic unstructured environment, and finally realizes the fast and efficient brain control method to control the grasping robot, effectively helping the disabled operator to achieve basic self-care ability.

[0041] Furthermore, a canonical correlation analysis algorithm based on genetic algorithm and filter group is used to perform pattern recognition on the target steady-state visual evoked potential, and visual evoked potential decoding based on adaptive parameter optimization is realized, thereby improving the decoding accuracy of EEG signals.

[0042] Furthermore, by introducing a knowledge graph intelligent reasoning method based on structured knowledge, the robot's grasping efficiency was improved, the robot's control logic capability was optimized, and the fatigue of disabled operators was effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the structure of the brain-controlled assistive grasping robot system according to an embodiment of the present invention.

[0044] Figure 2 The figure is a schematic diagram of the arrangement of the EEG signal acquisition unit according to an embodiment of the present invention.

[0045] Figure 3 Flow chart of the FB-CCA algorithm in an embodiment of the present invention.

[0046] Figure 4 This is a schematic diagram of the reasoning principle based on the indoor unstructured scene knowledge graph according to an embodiment of the present invention.

[0047] Figure 5 This is a flow chart of the brain-controlled grasping method based on the brain-controlled assistive grasping robot system according to an embodiment of the present invention.

[0048] Figure 6 Schematic diagram of the visual evoked stimulation paradigm interface of an embodiment of the present invention.

[0049] Figure 7 It is a schematic diagram of the physical system and complete scenario of an example of the present invention.

[0050] Among them, 650-graspable objects in dynamic unstructured environment; 660-grasping robot body; 670-signal processing unit; 680-EEG signal acquisition unit; 690-visual evoked unit. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] In response to the problems in the background technology, the present invention combines the visually induced brain control method based on semantic target stimulation with a disabled grasping robot with autonomous grasping function, and designs a brain-machine fusion method of the disabled grasping robot based on the fusion of semantic target stimulation and knowledge graph reasoning method, so as to solve the problems of simple control strategy and poor system robustness of the traditional brain control method of the disabled grasping robot, and thus help the disabled operator to achieve basic self-care ability.

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0055] See also Figure 1 In an embodiment of the present invention, a brain-controlled assistive grasping robot system with visual semantic target stimulation is provided, including an EEG signal acquisition unit, a visual induction unit, a signal processing unit and a grasping robot body; the EEG signal acquisition unit is connected to the signal processing unit, the grasping robot body and the visual induction unit in sequence. Among them, the grasping robot body is used to collect scene information in a dynamic unstructured environment, and segment the target image with object information contained in the scene information, and obtain the semantics of each target image; according to the scene information and the semantics of each target image, the semantic target image, position information and posture information of each object in the dynamic unstructured environment are obtained; and the semantic target image of each object is sent to the visual induction unit; and the brain-controlled grasping intention sent by the signal processing unit is received, and a grasping control signal is generated according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset set position information, and the grasping control signal is generated according to the grasping The control signal is used to grasp the object to be grasped and transport it to the set position; the visual evoked unit is used to generate a visual evoked paradigm based on semantic target stimulation according to the semantic target image of each object, and flash according to a preset flashing mode; the EEG signal acquisition unit is used to collect the EEG signals induced by the visual evoked paradigm of the disabled operator, and send them to the signal processing unit; the signal processing unit is used to decode the EEG signals to obtain the brain-controlled grasping intention, and send it to the grasping robot body, the signal processing unit is used to infer the next brain-controlled grasping intention of the disabled operator and feed it back to the visual evoked unit, and generate a visual evoked sub-paradigm for auxiliary control of the grasping robot.

[0056] The brain-controlled assistive grasping robot system with visual semantic target stimulation of the present invention collects scene information in a dynamic unstructured environment through a grasping robot body, then segments a target image containing object information, and obtains the semantics of each target image, and then obtains the semantic target image of each object and sends it to a visual induction unit, and generates a visual induction paradigm based on dynamic semantic target stimulation based on the visual induction unit, and a disabled operator gazes at the corresponding semantic target image in the visual induction paradigm to induce his steady-state visual evoked potential to express his brain control intention, and the induced steady-state visual evoked potential signal, i.e., the electroencephalogram signal, is collected by an electroencephalogram signal acquisition unit and sent to a signal processing unit for processing, and the brain control intention of the disabled operator is obtained and the brain-controlled grasping intention is parsed and sent to the grasping robot body, and a grasping control signal is generated by the grasping robot body according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset set position information, and the object to be grasped is grasped according to the grasping control signal and transported to the set position. The signal processing unit also obtains semantic target images of several inference intentions based on the brain control grasping intention obtained by decoding the brain control intention through a pre-built dynamic unstructured scene knowledge graph, and sends the obtained semantic target images of several inference intentions to the visual induction unit.

[0057] To sum up, the brain-controlled assistive grasping robot system with visual semantic target stimulation of the present invention combines the brain-computer interface technology based on biological intelligence with the grasping robot body with autonomous grasping ability, and effectively improves the intelligent decision-making and execution capabilities of the brain-computer fusion assistive grasping robot through human-computer interaction; by extracting the semantic target images of each object in the dynamic unstructured environment, a visually induced brain control paradigm based on the flashing of semantic target images is established, thereby improving the adaptability of the brain-computer fusion assistive grasping robot in the dynamic unstructured environment, and finally realizing the fast and efficient control of the assistive grasping robot using brain control, effectively helping disabled operators to achieve basic self-care ability.

[0058] Optionally, the visual evoked unit is connected to the grasping robot body unit by wireless communication, the EEG signal acquisition unit is connected to the signal processing unit by wireless communication, and the signal processing unit is wirelessly connected to the grasping robot body and the visual evoked unit by wireless communication. The grasping robot body can adopt a seven-degree-of-freedom robotic arm, the EEG signal acquisition unit can be a wireless EEG acquisition device under the international standard 10 / 20 channel distribution, and the signal processing unit can be a laptop computer.

[0059] In a possible implementation, the grasping robot body includes a visual perception module, a motion control module and a mechanical arm structure module; the visual perception module is connected to the motion control module and the mechanical arm structure module in sequence, and the motion control module is connected to the signal processing unit.

[0060] The visual perception module is used to collect scene information in a dynamic unstructured environment, and to segment target images with semantic information contained in the scene information, and to obtain the semantics of each target image; based on the scene information and the semantics of each target image, the semantic target image, position information and posture information of each object in the dynamic unstructured environment are obtained; and the semantic image of each object is sent to the visual induction unit, and the position information and posture information of each object is sent to the motion control module; the motion control module is used to receive the brain-controlled grasping intention sent by the signal processing unit, and the position information and posture information of each object sent by the visual perception module; a grasping control signal is generated according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset set position information, and sent to the robotic arm structure module; the robotic arm structure module is used to grasp the object to be grasped and transport it to the set position according to the grasping control signal.

[0061] Among them, the generation of a grasping control signal according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset set position information includes the following steps: according to the brain-controlled grasping intention, the position information of each object and the posture information of each object, the position information and posture information of the object to be grasped are obtained; the Denavit-Hartenberg method is used to perform kinematic modeling on the robotic arm structure module, and the position information and posture information of the object to be grasped are combined to obtain the position information and posture information of the robotic arm structure module reaching the object to be grasped; and according to the position information and posture information of the robotic arm structure module reaching the object to be grasped, the inverse kinematics of the robot joint is solved to obtain the angle at which the robotic arm structure module needs to rotate; according to the position information of the object to be grasped and the preset set position information, the path planning information of the robotic arm structure module is obtained; and according to the angle at which the robotic arm structure module needs to rotate and the path planning information of the robotic arm structure module, a grasping control signal is generated.

[0062] In a possible implementation, a risk avoidance method based on thrust detection is also provided when a grasping error occurs. Specifically, the grasping robot body is also used to obtain thrust feedback information and analyze it to obtain a thrust feedback value; and when the thrust feedback value is greater than a set thrust feedback threshold, a feedback control signal is generated, and according to the feedback control signal, the grasped object to be grasped is transported to the original position of the object to be grasped.

[0063] Specifically, an error avoidance method based on thrust detection is adopted. With the goal of reducing the misoperation of the grasping robot body, a thrust feedback threshold is set in advance for the torque of the grasping robot body. When the disabled operator selects an incorrect object, thrust is applied to the grasping robot body, so that the torque of the grasping robot body exceeds the thrust feedback threshold, and the grasping robot body will automatically put the object back to its original place.

[0064] In a possible implementation, the collecting of scene information in a dynamic unstructured environment, segmenting target images with object information contained in the scene information, and obtaining the semantics of each target image include: collecting scene information in a dynamic unstructured environment; calling a preset detection and segmentation model, detecting and segmenting target images with object information contained in scene images of the scene information, and obtaining semantic information of the target images; wherein the detection and segmentation model is obtained by training a YOLOv5 network model using annotated images. Then, the semantic target image of each object is determined based on the semantics of each target image.

[0065] Specifically, a detection segmentation model is first established, including by collecting and establishing a data set based on pictures of daily necessities; then a YOLOv5 network model framework is established, and the established daily necessities picture data set is sent to the YOLOv5 network model framework for training to obtain a detection segmentation model. The detection segmentation model is used to detect objects, and object recognition is performed based on the pixel unit coordinate points of the detected objects.

[0066] Then, an image of the target area is collected and sent to the detection and segmentation model for target recognition, thereby identifying the semantic target image containing graspable objects in the dynamic unstructured environment, and then the semantic target image identified based on the detection and segmentation model is saved and transmitted to the visual elicitation unit.

[0067] In a possible implementation, the method of generating a visual evoked paradigm based on semantic target stimulation according to the semantic target images of each object and flashing according to a preset flashing method includes: performing image edge contour extraction and grayscale processing on the semantic target images of each object, generating a visual evoked paradigm based on semantic target stimulation, and flashing the visual evoked paradigm based on semantic target stimulation in a sinusoidal modulation manner.

[0068] Among them, different semantic target images are modulated with different frequencies and presented on the screen of the visual evoked unit. The stimulation sequence corresponding to the frequency can be achieved by adjusting the grayscale of the screen by the following formula. The modulation function of the visual evoked unit is as follows:

[0069]

[0070] Among them, R is the refresh rate of the screen of the visual evoked unit, that is, the sampling rate of the sine wave signal, n is the serial number of each frame in the stimulation sequence, and f is the stimulation frequency.

[0071] According to the refresh rate of the screen of the visual evoked unit, the time t in the following formula needs to be discretized, that is:

[0072] t(n)=n / fr,n=1,2,3,

[0073] Finally, the flickering frequency of each semantic target image is determined.

[0074] In one possible implementation, see Figure 2 The EEG signal acquisition unit is placed on the subject's head, and a portable 64-channel wireless EEG acquisition device is preferably used. The EEG signals located at the occipital lobe Oz, O1, O2, POz, PO3, PO4, PO5 and PO6 positions under the international standard 10 / 20 channel are selected.

[0075] In a possible implementation, a method for decoding a semantic target visual evoked EEG signal based on adaptive parameter optimization is provided. Specifically, the decoding of the EEG signal to obtain a brain-controlled grasping intention includes: performing a Butterworth bandpass filter of 0.5 to 45 Hz on the EEG signal and removing trend terms, and then using a GA-FBCCA (Genetic Algorithm-Filter Bank Canonical Correlation Analysis, based on a genetic algorithm and a canonical correlation analysis of a filter bank) algorithm for pattern recognition to obtain a brain-controlled grasping intention. Furthermore, a genetic algorithm is used to optimize the characteristic parameters of the canonical correlation analysis of the filter bank for different disabled operators, and the filter bank canonical correlation analysis method after parameter optimization is used for pattern recognition of the target steady-state visual evoked potential.

[0076] Among them, GA (genetic algorithm) is used to optimize the characteristic parameters of canonical correlation analysis of filter groups of different disabled operators in order to obtain adaptive parameters for different disabled operators, and FBCCA (filter group canonical correlation analysis) is used to decode the target steady-state visual evoked potential to obtain the brain control intention of the disabled operator.

[0077] See also Figure 3 ,The main process of the GA-FBCCA algorithm includes: first, multiple Butterworth bandpass filters of different frequency bands are used to decompose the EEG signals into multiple sub-band signals; then a canonical correlation analysis is performed on each sub-band signal; a genetic algorithm is used to find the optimal parameters of the weights of each sub-band signal, and a weighted summation of each correlation is performed according to the signal response characteristics; finally, the target is judged based on the identified feature set with the maximum correlation coefficient.

[0078] In a possible implementation, the signal processing unit is provided with an EEG signal processing module and a knowledge graph module, wherein the EEG signal processing module is used to decode the EEG signal to obtain the brain-controlled grasping intention, and send the brain-controlled grasping intention to the grasping robot body.

[0079] The knowledge graph module is used to obtain semantic target images of several reasoning intentions according to the brain-controlled grasping intention through a pre-built dynamic unstructured scene knowledge graph, and send the obtained semantic target images of several reasoning intentions to the visual elicitation unit; the visual elicitation unit is also used to generate a visual elicitation sub-paradigm based on semantic target stimulation according to the semantic target images of several reasoning intentions and flash according to a preset flashing method.

[0080] Here, it should be noted that the visual elicitation unit can be considered to perform the same processing on the semantic target image of the inference intention and the semantic target image of each object in the dynamic unstructured scene. Subsequent EEG signal decoding and capture operations will also be performed on the semantic target image of the inference intention.

[0081] Specifically, after the capture is completed, the next intention of the disabled operator is inferred according to the dynamic unstructured scene knowledge graph based on the node information of the disabled operator's living environment and habit information, and the inferred intention result of the disabled operator is transmitted to the visual induction unit in the form of a semantic target image, that is, the semantic target image of the inferred intention, for the next step of EEG signal induction and brain control intention recognition.

[0082] Optional, see Figure 4 , the method of obtaining several semantic target images of inference intentions through a pre-constructed dynamic unstructured scene knowledge graph according to the brain-controlled grasping intention includes: encoding the brain-controlled grasping intention by a feature vector to obtain an intention information vector; calculating the matching degree between the intention information vector and the structured vector in the dynamic unstructured scene knowledge graph, taking the object information node corresponding to the structured vector with a matching degree higher than a preset value as the starting node, searching the dynamic unstructured scene knowledge graph for object entity information nodes representing the further intention of the disabled operator according to a path-based search method, and obtaining several entity information nodes of inference intentions; obtaining the next intention inference result of the disabled operator according to the several inference intention entity information nodes, and generating several semantic target images of inference intentions according to the next intention inference result of the disabled operator.

[0083] Among them, the dynamic unstructured scene knowledge graph is constructed in the following way according to the daily life scenes and living habits of the disabled operator. Among them, the dynamic unstructured scene knowledge graph includes a data layer and a pattern layer. In the data layer, the brain control intention information of the disabled operator is structured and output, and stored in the form of entity-relationship-entity and entity-attribute-attribute value triples; in the pattern layer, a hybrid method of bottom-up and top-down is adopted to construct the pattern layer to form the personal habit information of the disabled operator; secondly, according to the common sense information and experience information of human beings, the common sense relationship and attributes of each item in the pattern layer are expanded to construct a dynamic unstructured scene knowledge graph containing the living environment and habit information of the disabled operator. In this embodiment, a dynamic unstructured scene knowledge graph with a disabled operator as the graph center, including 357 nodes and 310 edges is obtained.

[0084] Among them, the intention reasoning in the pattern layer uses the Knowledge Graph Embedding (KGE) algorithm to pre-process the structured information in the dynamic unstructured indoor scene knowledge graph pattern layer, embed the object entity information of the scene where the disabled operator is located into the reasoning framework, and obtain the structured vector of the object entity information by encoding. Then, the semantic relationship and entity link information in the crawling intention knowledge graph are used to mine the potential association between the disabled operator and each object, so as to realize the further reasoning of the disabled operator's intention.

[0085] It can be seen that the present invention, on the basis of combining the visually induced brain control method based on semantic target stimulation with the disabled assistive grasping robot with autonomous grasping function, further combines the human-computer interaction strategy based on the knowledge graph to further optimize the control strategy of the brain-controlled disabled assistive grasping robot system and improve its system robustness.

[0086] In another embodiment of the present invention, a control method for a brain-controlled assistive grasping robot system based on visual semantic target stimulation is provided, comprising the following steps: collecting scene information in a dynamic unstructured environment through a grasping robot body, and segmenting target images with object information contained in the scene information, and obtaining the semantics of each target image; obtaining the semantic target image, position information and posture information of each object in the dynamic unstructured environment according to the scene information and the semantics of each target image; and sending the semantic target image of each object to a visual induction unit; and obtaining the semantic target image of each object according to the visual induction unit. The method comprises the following steps: a semantic target image of a body is generated, a visual evoked paradigm based on the semantic target stimulation is generated, and the visual evoked paradigm is flashed in a preset flashing manner; the electroencephalogram signal induced by the visual evoked paradigm of the disabled operator is collected by the electroencephalogram signal acquisition unit, and sent to the signal processing unit; the electroencephalogram signal is decoded by the signal processing unit to obtain the brain-controlled grasping intention, and the brain-controlled grasping intention is sent to the grasping robot body; the grasping robot body receives the brain-controlled grasping intention sent by the signal processing unit, and generates a grasping control signal according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset setting position information, and grasps the object to be grasped according to the grasping control signal and transports it to the setting position.

[0087] In a possible implementation, the method for controlling a brain-controlled assisted grasping robot system with visual semantic target stimulation also includes a signal processing unit obtaining, based on the results of EEG signal decoding, a number of semantic target images with reasoning intentions through a pre-constructed dynamic unstructured scene knowledge graph, and sending the obtained semantic target images with reasoning intentions to a visual induction unit to generate a visual induction sub-paradigm to perform auxiliary control of the grasping robot.

[0088] For details, see Figure 5 In this embodiment, the control method of the visual semantic target stimulation brain-controlled assistive grasping robot system comprises the following steps:

[0089] Step 510, collect a large number of pictures of daily necessities commonly used in the ward and annotate the collected daily necessities picture data set, establish a picture data set based on the daily necessities of disabled operators, send the data set to the pre-built YOLOv5 network model framework for training, and obtain a YOLOv5 network model based on semantic target recognition in a dynamic unstructured home environment, that is, a detection and segmentation model.

[0090] Secondly, the scene images in the scene information in the dynamic unstructured environment collected by the grasping robot body are input into the trained detection and segmentation model, the target images with object information contained in the target scene images are detected and segmented, and the semantics of each target image are obtained, so as to obtain the semantic target images of each object in the dynamic unstructured environment, and transmit them to the visual induction unit through wireless communication technology.

[0091] In step 520, the visual evoked unit receives the semantic target image obtained in step 510 through a wireless communication technology, performs edge contour segmentation and grayscale processing on the semantic target image, evenly arranges the image on the visual evoked unit, and flashes the image in a sinusoidal modulation manner with different frequencies. Thus, the disabled operator can express his or her brain control intention by looking at different semantic target images.

[0092] In this embodiment, the visual evoked paradigm is implemented by the PsychToolbox toolbox under the Matlab software to realize the presentation of the visual evoked paradigm of dynamic semantic target flashing, and the obtained brain-controlled visual evoked stimulation paradigm interface based on the dynamic semantic target image is as follows: Figure 6 As shown, the corresponding stimulation frequencies are 7 Hz, 7.5 Hz, 8 Hz, 8.5 Hz, 9 Hz, 9.5 Hz, 10 Hz, 10.5 Hz and 11 Hz respectively.

[0093] Step 530, the visual evoked unit is within the visual range of the disabled operator, and the disabled operator expresses his or her brain control intention by looking at the flashing semantic target image in the visual evoked unit. In this embodiment, the NeuSen-W64 64-lead EEG acquisition device is selected. The CPz of the device is the reference electrode, and GND is the ground electrode. The EEG signals of the 8 electrodes O1, O2, Oz, POz, PO3, PO4, PO5 and PO6 in the occipital lobe area of ​​the disabled operator are collected in real time under the international 10 / 20 standard, and the collected EEG signals are transmitted to the signal processing unit by a wireless communication method.

[0094] Step 540: The signal processing unit pre-processes the collected EEG signal. In this embodiment, the collected EEG signal is subjected to a 0.5-45 Hz fourth-order Butterworth bandpass filter, and then a trend term is removed to obtain a target steady-state visual evoked potential.

[0095] Step 550, pattern recognition is performed on the target steady-state visual evoked potential. In this embodiment, the GA-FBCCA algorithm is used to perform pattern recognition on the target steady-state visual evoked potential, and the obtained brain control intention is transmitted to the knowledge graph submodule through wireless communication. Among them, GA is used to optimize the characteristic parameters of the canonical correlation analysis of the filter group of different disabled operators to obtain adaptive parameters for different disabled operators, and FBCCA is used to decode the target steady-state visual evoked potential to obtain the brain control intention.

[0096] Step 560, the grasping robot body determines the object to be grasped according to the brain control intention and performs motion planning. First, the visual perception module of the grasping robot body obtains the position information and posture information of the object to be grasped through target detection, and then the motion control module of the grasping robot body uses the Denavit-Hartenberg method to perform kinematic modeling, and determines the coordinates and posture of the end effector of the manipulator to reach the object to be grasped according to the position information and posture information of the object to be grasped, and uses the robot joint inverse kinematics to solve the angle that each motor of the manipulator needs to rotate to reach the target position.

[0097] Step 570: The grasping robot body grasps the object to be grasped according to the planning result. First, the mechanical arm structure module of the grasping robot body performs the grasping operation of the object to be grasped, and then sends the grasped object to the disabled operator.

[0098] In step 580, the grasping robot body delivers the object to be grasped to the operator, and the disabled operator feeds back the decoding result of the EEG signal through tactile feedback. When the grasped object to be grasped is confirmed to be what the disabled operator needs, the disabled operator takes the item. When the grasping is wrong, the disabled operator gently pushes the mechanical arm structure module by means of resistance feedback, and the mechanical arm structure module puts the object to be grasped back to its original place. Specifically, the mechanical arm structure module obtains and analyzes the thrust feedback information to obtain the thrust feedback value. When the thrust feedback value is greater than the set thrust feedback threshold, a feedback control signal is generated, and according to the feedback control signal, the grasped object to be grasped is transported to the original position of the object to be grasped in the target area. Among them, the thrust feedback threshold can be adjusted according to different disabled operators and living habits. In this embodiment, the thrust feedback threshold is set to 8N.

[0099] In step 590, the knowledge graph module of the signal processing unit combines the brain control intention to infer the next grasping intention and obtains several semantic target images of the inference intention. In this embodiment, first, the brain control intention information obtained by decoding after the disabled operator is stimulated is encoded with a feature vector, and the similarity between the current disabled operator's intention information vector in the obtained data layer and the structured vector in the pattern layer is calculated to obtain the similarity calculation result. Subsequently, according to the path-based method, the path relationship between entities in the grasping intention knowledge graph is searched in the common sense and experience information, and the next node information of the starting node is matched with the inference entity node information to obtain the semantic target image of the inference object, and it is added to the options displayed to the disabled operator by the next round of visual elicitation unit.

[0100] See also Figure 7 , showing the specific application scenario of the brain-controlled assistive grasping robot system with visual semantic target stimulation of the present invention. The grasping robot body 660 is connected to the visual induction unit 690, and the signal processing unit 670 is connected to both the EEG signal acquisition unit 680 and the visual induction unit 690. The grasping robot body 660 is used to collect information about the graspable object 650 in a dynamic unstructured environment, that is, the scene information in the dynamic unstructured environment. Among them, the grasping robot body 660 and the visual induction unit 690, the signal processing unit 670 and the EEG signal acquisition unit 680, and the signal processing unit 670 and the visual induction unit 690 are all wirelessly connected. The grasping robot body 660 can adopt a disabled grasping robot with a seven-degree-of-freedom mechanical arm, and the visual induction unit 690 can adopt a display screen.

[0101] In summary, the brain-controlled assistive grasping robot system and control method with visual semantic target stimulation of the present invention aims to solve the problems of traditional grasping robots' lack of human-computer interaction ability and insufficient autonomous decision-making ability. It combines the brain-computer interface technology based on biological intelligence with the control method of the grasping robot body with autonomous grasping ability, and effectively improves the intelligent decision-making and execution capabilities of the grasping robot body through human-computer interaction.

[0102] At the same time, through visual perception of dynamic environmental information, graspable targets in dynamic unstructured indoor scenes are extracted, and a visually induced brain control paradigm based on semantic target image flashing is established to improve the adaptability of brain-machine fusion assistive grasping robots in dynamic unstructured indoor scenes and reduce the fatigue of disabled operators.

[0103] In addition, to address the problem of low robustness of traditional visual evoked potential decoding methods, a visual evoked potential decoding method based on adaptive parameter optimization is proposed, which makes full use of the characteristics of EEG signals, optimizes the EEG signal decoding method, and significantly improves the accuracy.

[0104] In addition, an intelligent reasoning method based on knowledge graph is adopted, and the dynamic unstructured scene knowledge graph is used to represent the target entities of the scene where the disabled operator is located, the relationship between the entities and the disabled operator, and the attributes of the target entities in a network knowledge structure. By introducing structured knowledge, the reasoning ability of the disabled operator's intentions can be effectively improved.

[0105] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A visual semantic target stimulation brain-controlled assistive grasping robot system, characterized in that: It includes an electroencephalogram signal acquisition unit, a visual induced unit, a signal processing unit and a grasping robot body; the electroencephalogram signal acquisition unit is sequentially connected to the signal processing unit, the grasping robot body and the visual induced unit; The grasping robot body is used to collect scene information in a dynamic unstructured environment, and segment the target image with object information contained in the scene information, and obtain the semantics of each target image; obtain the semantic target image, position information and posture information of each object in the dynamic unstructured environment according to the scene information and the semantics of each target image; and send the semantic target image of each object to the visual induction unit; and receive the brain-controlled grasping intention sent by the signal processing unit, and generate a grasping control signal according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset setting position information, and grasp the object to be grasped according to the grasping control signal and transport it to the set position; The visual evoked unit is used to generate a visual evoked paradigm based on semantic target stimulation according to the semantic target image of each object, and flash according to a preset flashing mode; The EEG signal acquisition unit is used to acquire EEG signals induced by the disabled operator based on the semantic target stimulation visual induced paradigm, and send them to the signal processing unit; The signal processing unit is used to decode the EEG signal to obtain the brain-controlled grasping intention and send it to the grasping robot body; and to infer the next brain-controlled grasping intention of the disabled operator and feed it back to the visual induced unit; The grabbing robot body includes a visual perception module, a motion control module and a mechanical arm structure module; the visual perception module is connected to the motion control module and the visual induction unit, and the motion control module is connected to the signal processing unit; The visual perception module is used to collect scene information in a dynamic unstructured environment, segment target images with semantic information contained in the scene information, and obtain the semantics of each target image; obtain the semantic target image, position information and posture information of each object in the dynamic unstructured environment according to the scene information and the semantics of each target image; send the semantic image of each object to the visual induction unit, and send the position information and posture information of each object to the motion control module; The motion control module is used to receive the brain-controlled grasping intention sent by the signal processing unit, and the position information and posture information of each object sent by the visual perception module; generate a grasping control signal according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset set position information, and send it to the mechanical arm structure module; The robot arm structure module is used to grasp the object to be grasped and transport it to the set position according to the grasping control signal; The method of generating a grasping control signal according to the brain-controlled grasping intention, the position information of each object, the posture information of each object and the preset set position information includes: According to the brain-controlled grasping intention, the position information of each object and the posture information of each object, the position information and posture information of the object to be grasped are obtained; The Denavit-Hartenberg method is used to perform kinematic modeling on the robot arm structure module, and the position information and posture information of the object to be grasped are combined to obtain the position information and posture information of the robot arm structure module reaching the object to be grasped; and according to the position information and posture information of the robot arm structure module reaching the object to be grasped, the inverse kinematics of the robot joint is solved to obtain the angle at which the robot arm structure module needs to rotate; According to the position information of the object to be grasped and the preset set position information, the path planning information of the robot arm structure module is obtained; and according to the angle at which the robot arm structure module needs to rotate and the path planning information of the robot arm structure module, a grasping control signal is generated; The collecting of scene information in a dynamic unstructured environment, segmenting target images with object information contained in the scene information, and obtaining the semantics of each target image include: Collect scene information in a dynamic unstructured environment; call a preset detection and segmentation model to detect and segment a target image with object information contained in a scene image of the scene information, and obtain semantic information of the target image; wherein the detection and segmentation model is a YOLOv5 network model trained with labeled images; The grasping robot body is also used to obtain thrust feedback information and analyze it to obtain a thrust feedback value; and when the thrust feedback value is greater than a set thrust feedback threshold, a feedback control signal is generated, and according to the feedback control signal, the grasped object to be grasped is transported to the original position; The generating of a visual evoked paradigm based on semantic target stimulation according to the semantic target image of each object and flashing in a preset flashing manner includes: After extracting the edge contour and grayscale processing of the semantic target images of each object, a visual evoked paradigm based on semantic target stimulation is generated, and the visual evoked paradigm based on semantic target stimulation is flashed in a sinusoidal modulation manner; Decoding the EEG signal to obtain the brain control capture intention includes: The EEG signals were subjected to 0.5-45Hz Butterworth bandpass filtering and trend removal processing, and then the canonical correlation analysis algorithm based on genetic algorithm and filter bank was used to perform pattern recognition on the target steady-state visual evoked potential to obtain the brain-controlled grasping intention of the disabled operator. The signal processing unit is also connected to the visual induction unit, wherein the inference of the next brain-controlled grasping intention of the disabled operator includes: The signal processing unit is further used to obtain semantic target images of several inference intentions according to the brain-controlled grasping intention through a pre-built dynamic unstructured scene knowledge graph, and send the obtained semantic target images of several inference intentions as the next brain-controlled grasping intention of the disabled operator to the visual induction unit; The visual evoked unit is further used to generate a visual evoked sub-paradigm based on semantic target stimulation according to a number of semantic target images of inference intentions, and flash according to a preset flashing mode; According to the brain-controlled grasping intention, the semantic target images of several reasoning intentions are obtained through the pre-built dynamic unstructured scene knowledge graph, including: Encode the brain-controlled grasping intention into a feature vector to obtain the intention information vector; The intention information vector is matched with the structured vector in the dynamic unstructured scene knowledge graph, and the object information node corresponding to the structured vector with a matching degree higher than a preset value is used as the starting node. According to the path-based search method, the object entity information node representing the further intention of the disabled operator is searched in the dynamic unstructured scene knowledge graph to obtain several entity information nodes of the inferred intention; According to several inference intention entity information nodes, the next intention inference result of the disabled operator is obtained, and according to the next intention inference result of the disabled operator, several semantic target images of the inference intention are generated.

Images