Multimodal interactive AI wheelchair control method and system

Abstract

The present invention relates to the field of artificial intelligence, and discloses a multimodal interactive AI wheelchair control method, including: defining a spatial filtering algorithm for an EEG signal, filtering the EEG signal using the spatial filtering algorithm to obtain a spatial filtering signal, calculating a forward analysis error and a backward analysis error of the spatial filtering signal, determining a signal timing analysis model for the spatial filtering signal, calculating a signal spectrum of the spatial filtering signal, and extracting EEG signal features of the spatial filtering signal; extracting features from a speech signal and a gesture image to obtain language signal features and gesture image features, and calculating a vertical component and a horizontal component of a target user; extracting environmental features of an external environment, calculating a motion path of a target user using a preset AI path control algorithm, and executing interactive control of the AI ​​wheelchair according to the motion path. The present invention can improve the accuracy of AI wheelchair control.

Description

Technical Field

[0001] The present invention relates to a multi-modal interactive AI wheelchair control method and system, and belongs to the field of artificial intelligence. Background Art

[0002] Multimodal interactive AI wheelchair control refers to an intelligent system that uses multiple interactive methods to control wheelchairs. This system combines artificial intelligence technology to allow users to guide the actions of the wheelchair through different sensory channels and input methods, thereby improving the convenience and adaptability of wheelchair use. Especially for users with limited mobility, this control method can greatly improve their autonomy and quality of life.

[0003] At present, the main method of wheelchair control is to control the movement of the wheelchair through a remote control. This method requires the user to have sound finger movement ability. For patients who have lost the ability to move their hands, there are still movement disorders. Moreover, this control method is relatively simple and cannot meet the needs of users, resulting in poor wheelchair control effect. Summary of the invention

[0004] The present invention provides a multimodal interactive AI wheelchair control method and system, the main purpose of which is to improve the accuracy of AI wheelchair control.

[0005] To achieve the above object, the present invention provides a multimodal interactive AI wheelchair control method, comprising:

[0006] Acquire an AI wheelchair of a target user, analyze the physical state of the target user and the user's wheelchair needs, determine an interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs, configure a sensor network of the AI ​​wheelchair according to the interaction mode, and use the sensor network to collect multimodal signals of the target user, wherein the multimodal signals include: electroencephalogram signals, voice signals, and gesture images;

[0007] A spatial filtering algorithm for the EEG signal is defined, and the EEG signal is filtered using the spatial filtering algorithm to obtain a spatial filtering signal, wherein the spatial filtering algorithm:

[0008]

[0009] Among them, D a (t) represents the spatial filtering signal of the EEG signal at time t after being filtered by the spatial filtering algorithm, d a (t) represents the EEG signal at time t, d i represents the adjacent potential of the i-th reference electrode of the EEG signal corresponding to the target electrode, L a,i represents the distance between the target electrode and the i-th reference electrode;

[0010] Calculating a forward analysis error and a backward analysis error of the spatial filtering signal, determining a signal timing analysis model of the spatial filtering signal based on the forward analysis error and the backward analysis error, calculating a signal spectrum of the spatial filtering signal according to the signal timing analysis model, and extracting an EEG signal feature of the spatial filtering signal according to the signal spectrum;

[0011] Extracting features from the speech signal and the gesture image to obtain language signal features and gesture image features, and calculating the vertical component and the horizontal component of the target user according to the EEG signal features, the language signal features, and the gesture image features;

[0012] The external environment of the target user is collected, environmental features of the external environment are extracted, and based on the environmental features, the vertical component and the horizontal component, a motion path of the target user is calculated using a preset AI path control algorithm, and interactive control of the AI ​​wheelchair is performed according to the motion path.

[0013] Optionally, analyzing the target user's physical condition and the user's wheelchair needs includes:

[0014] Acquire user data of the target user;

[0015] Performing data cleaning on the user data to obtain user cleansing data;

[0016] Classifying the user cleansing data to obtain classified user cleansing data, wherein the classified user cleansing data includes basic information, living environment data, user activity data, user function data, nervous system data, sensory ability data, and cognitive ability data;

[0017] Analyze the daily activity needs of the target user based on the basic information, living environment data and user activity data;

[0018] Analyzing the user activity capability of the target user according to the user function data;

[0019] Determining the target user's neurological capacity through the neurological data;

[0020] Analyzing the sensory ability of the target user according to the sensory ability data;

[0021] Analyzing the cognitive ability of the target user based on the cognitive ability data;

[0022] Determine the physical state of the target user by combining the user's activity ability, the nervous system ability, the sensory ability, and the cognitive ability;

[0023] A user wheelchair requirement of the target user is determined according to the physical state and the daily activity requirement.

[0024] Optionally, determining the interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair requirements includes:

[0025] Based on the user's wheelchair requirements, analyzing the interactive modes of the AI ​​wheelchair;

[0026] analyzing interaction characteristics of the interactive mode;

[0027] Analyzing the user load and interaction convenience coefficient of the interactive mode on the physical state according to the interactive characteristics;

[0028] Calculating the interaction mode score of the interactive mode according to the user load and the interaction convenience coefficient;

[0029] Based on the interaction mode score, an interaction mode of the AI ​​wheelchair is determined.

[0030] Optionally, calculating the forward analysis error and the backward analysis error of the spatial filtering signal includes:

[0031] Determining a reference signal of the spatial filtering signal;

[0032] Based on the reference signal, the forward analysis error of the spatial filtering signal is calculated using the following formula:

[0033] E fwd (t) = D ref (t)-D a (t)

[0034] Among them, E fwd (t) represents the forward analysis error of the spatial filtering signal at time t, D ref (t) represents the reference signal of the spatial filtering signal at time t, D a (t) represents the spatial filtered signal of the EEG signal at time t after being filtered by the spatial filtering algorithm;

[0035] Performing inverse filtering on the spatial filtering signal to obtain an inverse filtering signal;

[0036] Based on the inverse filtered signal, the backward analysis error of the spatial filtered signal is calculated using the following formula:

[0037] E bwd (t) = D a (t)-D rev (t)

[0038] Among them, E bwd(t) represents the backward analysis error of the spatial filtering signal at time t, D a (t) represents the spatial filtering signal of the EEG signal at time t after being filtered by the spatial filtering algorithm, D ref (t) represents the inverse filtered signal of the spatial filtered signal at time t.

[0039] Optionally, calculating the signal spectrum of the spatial filtering signal according to the signal timing analysis model includes:

[0040] Normalizing the spatial filtering signal to obtain a normalized filtering signal;

[0041] identifying a stationary coefficient of the normalized filtered signal;

[0042] Based on the stationary coefficient, the normalized filtered signal is differentiated to obtain a differential filtered signal;

[0043] marking potential signals of the differential filtering signal using the signal timing analysis model;

[0044] combining the potential signal and the differential filter signal to obtain a target EEG signal;

[0045] Dividing the target EEG signal into data segments;

[0046] A preset FFT algorithm is applied to the data segment to obtain the signal spectrum.

[0047] Optionally, the extracting features of the speech signal and the gesture image to obtain language signal features and gesture image features includes:

[0048] Denoising the speech signal to obtain a denoised speech signal;

[0049] Pre-emphasize the denoised speech signal to obtain an emphasized speech signal;

[0050] Splitting the emphasized speech signal into short-time frame signals;

[0051] Marking a signal starting point and a signal ending point of the short-time frame signal;

[0052] Based on the signal starting point and the signal ending point, calculating the Mel frequency cepstrum coefficients of the short time frame signal;

[0053] Analyzing the language signal characteristics of the speech signal through the Mel-frequency cepstral coefficients;

[0054] extracting a gesture area of ​​the gesture image;

[0055] Extracting gesture shape features, gesture movement features, and gesture surface features of the gesture area;

[0056] Based on the gesture shape features, gesture movement features, and gesture surface features, gesture image features of the gesture image are determined.

[0057] Optionally, the extracting of gesture shape features, gesture movement features, and gesture surface features of the gesture area includes:

[0058] identifying a gesture contour of the gesture area;

[0059] Analyzing gesture shape features of the gesture area according to the gesture contour;

[0060] Marking the gesture centroid of the gesture area;

[0061] analyzing a movement pattern of the center of mass of the gesture;

[0062] Defining gesture motion features of the gesture area according to the motion pattern;

[0063] Analyzing gesture color distribution in the gesture area;

[0064] Based on the gesture color distribution, a gesture surface feature of the gesture area is determined.

[0065] Optionally, calculating the vertical component and the horizontal component of the target user according to the EEG signal feature, the language signal feature and the gesture image feature includes:

[0066] Extracting left brain signal features and right brain signal features of the EEG signal features;

[0067] Determine a left brain vertical signal amplitude and a left brain horizontal signal amplitude of the left brain signal characteristic, and determine a right brain vertical signal amplitude and a right brain horizontal signal amplitude of the right brain signal characteristic;

[0068] Analyzing the left brain signal weight of the left brain signal feature;

[0069] Analyzing the right brain signal weight of the right brain signal feature;

[0070] According to the target user's corresponding physical state, respectively calculating the EEG signal weight, the language signal weight and the gesture image weight of the EEG signal feature, the language signal feature and the gesture image feature;

[0071] According to the left brain vertical signal amplitude, the left brain horizontal signal amplitude, the right brain vertical signal amplitude, the right brain horizontal signal amplitude, the left brain signal weight, the right brain signal weight, the EEG signal weight, the language signal weight and the gesture image weight, the vertical component and the horizontal component of the target user are calculated using the following formula:

[0072] Y=W α (W r R y +W z Z y )+W β T y +W γ S y +C

[0073] X=W α (W r R x +W z Z x )+W β T x +W γ S x +C

[0074] Where Y represents the vertical component, W α represents the EEG signal weight, W β represents the language signal weight, W γ represents the gesture image weight, W r represents the right brain signal weight, W z represents the left brain signal weight, R y Represents the vertical signal amplitude of the right brain, R x represents the amplitude of the right brain horizontal signal, Z y Represents the vertical signal amplitude of the left brain, Z x represents the horizontal signal amplitude of the left brain, T y Indicates the language vertical signal feature corresponding to the language signal feature, T x Indicates the language level signal feature corresponding to the language signal feature, S y Indicates the vertical signal feature of the gesture corresponding to the gesture image feature, S x represents the gesture level signal feature corresponding to the gesture image feature, and C represents a constant.

[0075] Optionally, the calculating the motion path of the target user by using a preset AI path control algorithm based on the environmental feature, the vertical component, and the horizontal component includes:

[0076] Determining an initial motion path of the target user according to the vertical component and the horizontal component;

[0077] According to the environmental characteristics, identifying obstacles in the initial motion path, and analyzing physical characteristics and position characteristics of the obstacles;

[0078] According to the physical features and the position features, a preset AI path control algorithm is used to respectively calculate a vertical adjustment parameter of the vertical component and a horizontal adjustment parameter of the horizontal component;

[0079] According to the vertical adjustment parameter and the horizontal adjustment parameter, the vertical component and the horizontal component are adjusted to obtain an optimized vertical component and an optimized horizontal component;

[0080] A motion path of the target user is constructed according to the optimized vertical component and the optimized horizontal component.

[0081] In order to solve the above problems, the present invention also provides a multimodal interactive AI wheelchair control system, the system comprising:

[0082] a multimodal signal acquisition module, used to acquire the AI ​​wheelchair of the target user, analyze the physical state of the target user and the user's wheelchair needs, determine the interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs, configure the sensor network of the AI ​​wheelchair according to the interaction mode, and use the sensor network to collect the multimodal signals of the target user, wherein the multimodal signals include: EEG signals, voice signals and gesture images;

[0083] The spatial filtering algorithm definition module is used to define the spatial filtering algorithm of the EEG signal, and use the spatial filtering algorithm to filter the EEG signal to obtain a spatial filtering signal, wherein the spatial filtering algorithm:

[0084]

[0085] Among them, D a (t) represents the spatial filtering signal of the EEG signal at time t after being filtered by the spatial filtering algorithm, d a (t) represents the EEG signal at time t, d i represents the adjacent potential of the i-th reference electrode of the EEG signal corresponding to the target electrode, L a,i represents the distance between the target electrode and the i-th reference electrode;

[0086] an EEG signal feature analysis module, used to calculate a forward analysis error and a backward analysis error of the spatial filtering signal, determine a signal timing analysis model of the spatial filtering signal based on the forward analysis error and the backward analysis error, calculate a signal spectrum of the spatial filtering signal according to the signal timing analysis model, and extract EEG signal features of the spatial filtering signal according to the signal spectrum;

[0087] A vertical-horizontal component analysis module, used to extract features from the speech signal and the gesture image to obtain language signal features and gesture image features, and calculate the vertical component and horizontal component of the target user according to the EEG signal features, the language signal features and the gesture image features;

[0088] The wheelchair interactive control module is used to collect the external environment of the target user, extract the environmental characteristics of the external environment, calculate the motion path of the target user based on the environmental characteristics, the vertical component and the horizontal component using a preset AI path control algorithm, and perform interactive control of the AI ​​wheelchair according to the motion path.

[0089] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0090] at least one processor; and,

[0091] a memory communicatively connected to the at least one processor; wherein,

[0092] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the multimodal interactive AI wheelchair control method described above.

[0093] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the above-mentioned multimodal interactive AI wheelchair control method.

[0094] Compared with the problems described in the background technology, the present invention obtains the AI ​​wheelchair of the target user, analyzes the physical state of the target user and the user's wheelchair needs, and can better configure and optimize the functions of the AI ​​wheelchair to ensure that it can meet the user's specific needs and provide a safe, comfortable and convenient mobile solution; the present invention determines the interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs to ensure that the interaction mode of the AI ​​wheelchair can maximize the physical state and needs of the target user; further, the present invention configures the sensor network of the AI ​​wheelchair according to the interaction mode to configure an efficient, reliable and user-friendly wheelchair sensor network, thereby supporting the interaction mode of the AI ​​wheelchair. Further, the present invention defines a spatial filtering algorithm for the EEG signal, uses the spatial filtering algorithm to filter the EEG signal, and obtains a spatially filtered EEG signal. These signals have better classification characteristics and can be used for further analysis and application. Further, the present invention calculates the forward analysis error and the backward analysis error of the spatially filtered signal to obtain a spatially filtered EEG signal. The influence of the time filter on the signal is evaluated to provide a basis for the subsequent signal processing. Furthermore, the present invention determines the signal timing analysis model of the spatial filter signal based on the forward analysis error and the backward analysis error, and can determine a model that can accurately describe the timing characteristics of the spatial filter signal. The model can be used for further signal analysis. Next, the present invention extracts features from the voice signal and the gesture image, and obtains language signal features and gesture image features, which can be used as the basis for the subsequent wheelchair operation analysis. The present invention calculates the vertical component and horizontal component of the target user based on the EEG signal features, the language signal features, and the gesture image features, which can realize the estimation of the target user's action intention, thereby providing more accurate control instructions for AI wheelchairs or other auxiliary devices. Finally, the present invention calculates the motion path of the target user based on the environmental features, the vertical component, and the horizontal component using a preset AI path control algorithm, which can calculate a reasonable motion path according to the environmental features and the user's motion intention, thereby guiding the safe and effective movement of the wheelchair or other auxiliary devices. Therefore, the present invention can improve the accuracy of AI wheelchair control. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 A schematic diagram of a flow chart of a multimodal interactive AI wheelchair control method provided by an embodiment of the present invention;

[0096] Figure 2 A functional module diagram of a multimodal interactive AI wheelchair control system provided by an embodiment of the present invention;

[0097] Figure 3A schematic diagram of the structure of an electronic device of a multimodal interactive AI wheelchair control system provided by an embodiment of the present invention;

[0098] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0099] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0100] The embodiment of the present application provides a multimodal interactive AI wheelchair control method. The execution subject of the multimodal interactive AI wheelchair control method includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the multimodal interactive AI wheelchair control method can be executed by software or hardware installed on a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0101] Embodiment 1:

[0102] Reference Figure 1 FIG. 1 is a flow chart of a multimodal interactive AI wheelchair control method provided by an embodiment of the present invention. In this embodiment, the multimodal interactive AI wheelchair control method includes:

[0103] S1. Obtain an AI wheelchair of a target user, analyze the physical state of the target user and the user's wheelchair needs, determine an interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs, configure a sensor network of the AI ​​wheelchair according to the interaction mode, and use the sensor network to collect multimodal signals of the target user, wherein the multimodal signals include: electroencephalogram signals, voice signals, and gesture images.

[0104] The present invention obtains an AI wheelchair for a target user, and analyzing the physical condition of the target user and the user's wheelchair needs can better configure and optimize the functions of the AI ​​wheelchair to ensure that it can meet the user's specific needs and provide a safe, comfortable and convenient mobility solution.

[0105] In detail, the analyzing the target user's physical condition and the user's wheelchair needs includes:

[0106] Acquire user data of the target user;

[0107] Performing data cleaning on the user data to obtain user cleansing data;

[0108] Classifying the user cleansing data to obtain classified user cleansing data, wherein the classified user cleansing data includes basic information, living environment data, user activity data, user function data, nervous system data, sensory ability data, and cognitive ability data;

[0109] Analyze the daily activity needs of the target user based on the basic information, living environment data and user activity data;

[0110] Analyzing the user activity capability of the target user according to the user function data;

[0111] Determining the target user's neurological capacity through the neurological data;

[0112] Analyzing the sensory ability of the target user according to the sensory ability data;

[0113] Analyzing the cognitive ability of the target user based on the cognitive ability data;

[0114] Determine the physical state of the target user by combining the user's activity ability, the nervous system ability, the sensory ability, and the cognitive ability;

[0115] A user wheelchair requirement of the target user is determined according to the physical state and the daily activity requirement.

[0116] Among them, the user data refers to all original information related to the target user, including personal information, health status, behavioral habits, environmental factors and other data; the user cleaned data refers to user data that has been preliminarily processed and cleaned, removing duplicate, erroneous or irrelevant information to ensure the accuracy and availability of the data; the basic information includes the user's basic personal information, such as name, age, gender, height, weight, occupation, etc.; the living environment data refers to environmental characteristics that describe the user's daily life, such as the layout of the living environment, the conditions of the working environment, and the community's barrier-free facilities; the user activity data refers to data recording the type, frequency, duration, intensity of the user's daily activities, such as walking, going up and down stairs, sitting, using transportation, etc.; the user function data refers to information related to the user's physical functions, such as muscle strength, range of joint motion, balance ability, endurance, etc.; the nervous system data refers to information related to the user's nervous system condition, including medical history, neurological examination results, disease diagnosis, etc.; the sensory ability data refers to information about the user's sensory abilities. the information of the user, such as test data of vision, hearing, smell, touch and taste abilities; the cognitive ability data refers to information related to the user's cognitive functions, including data on memory, attention, language ability, problem-solving ability, etc.; the daily activity needs refer to the user's needs for mobility and functions in daily life, such as independent toileting, cooking, shopping, socializing, etc.; the user activity ability refers to the specific functions required by the user when performing daily activities, such as the ability to stand, transfer, manipulate objects, etc.; the nervous system ability refers to the overall functional state of the user's nervous system, including reflex, coordination, muscle control, etc.; the sensory ability refers to the ability of the user's sensory system; the cognitive ability refers to the user's information processing ability, including perception, memory, thinking, judgment, etc.; the physical state refers to the user's overall physical condition, including health status, functional ability, sensory and cognitive ability, etc.; the user wheelchair needs refer to the specific requirements for wheelchairs determined according to the user's physical state and daily activity needs, including requirements for wheelchair type, size, function, control method, etc.

[0117] Optionally, analyzing the user activity ability of the target user based on the user function data may be performed by using a Barthel index.

[0118] Optionally, analyzing the cognitive ability of the target user based on the cognitive ability data may be performed by using the Montreal Cognitive Assessment.

[0119] The present invention determines the interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs, which can ensure that the interaction mode of the AI ​​wheelchair can maximally meet the physical state and needs of the target user.

[0120] In detail, determining the interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs includes:

[0121] Based on the user's wheelchair requirements, analyzing the interactive modes of the AI ​​wheelchair;

[0122] analyzing interaction characteristics of the interactive mode;

[0123] Analyzing the user load and interaction convenience coefficient of the interactive mode on the physical state according to the interactive characteristics;

[0124] Calculating the interaction mode score of the interactive mode according to the user load and the interaction convenience coefficient;

[0125] Based on the interaction mode score, an interaction mode of the AI ​​wheelchair is determined.

[0126] Among them, the interactive mode refers to the various ways in which users can operate and communicate with the AI ​​wheelchair, such as touch screen, gestures, brain signals, voice commands, eye tracking, head control, etc. The interactive feature refers to the description of the specific attributes of each interactive mode, including the complexity of operation, learning curve, reaction time, error rate, user adaptability, etc. The user load refers to the physiological and psychological burden that the user bears when using a certain interactive mode, including fatigue level, attention requirement, operation difficulty, etc. The interaction convenience coefficient refers to the measure of the convenience of the interaction mode for the user, including the intuitiveness, speed, accuracy, etc. of the operation. The interaction mode score refers to the comprehensive evaluation value of the user load and the interaction convenience coefficient. The interaction mode refers to the final interaction mode, including EEG interaction, voice interaction and gesture interaction.

[0127] Optionally, the calculation of the interaction mode score of the interactive mode by using the user load and the interaction convenience coefficient may be implemented by using a hierarchical analysis method.

[0128] According to the interaction mode, the sensor network of the AI ​​wheelchair can be configured to form an efficient, reliable and user-friendly wheelchair sensor network, thereby supporting the interaction mode of the AI ​​wheelchair. In detail, according to the interaction mode, the sensor network of the AI ​​wheelchair can determine the sensors of the AI ​​wheelchair through the interaction mode, including motion sensors, brain signal sensors, environmental sensors, physiological sensors, touch sensors, voice sensors and other sensors, and connect the sensors to a network to obtain a sensor network, wherein the sensor network refers to a sensor network that can realize the collection of user and wheelchair data.

[0129] It should be explained that the EEG signal refers to the electrical signal generated by the activity of brain neurons, the voice signal refers to the sound waves generated by humans through the vibration of the vocal cords and the resonance of the oral, nasal and other vocal tracts, and the gesture image refers to the movement image of the hands and arms, which can be used to convey information or execute commands.

[0130] S2. Define a spatial filtering algorithm for the EEG signal, and use the spatial filtering algorithm to filter the EEG signal to obtain a spatial filtering signal, wherein the spatial filtering algorithm:

[0131]

[0132] Among them, D a (t) represents the spatial filtering signal of the EEG signal at time t after being filtered by the spatial filtering algorithm, d a (t) represents the EEG signal at time t, d i represents the adjacent potential of the i-th reference electrode of the EEG signal corresponding to the target electrode, L a,i represents the distance between the target electrode and the i-th reference electrode.

[0133] The present invention defines a spatial filtering algorithm for the EEG signal, and uses the spatial filtering algorithm to filter the EEG signal to obtain a spatially filtered EEG signal. These signals have better classification characteristics and can be used for further analysis and application.

[0134] It should be explained that the spatial filtering algorithm refers to a filtering technology based on the spatial relationship between electrodes. It is an algorithm that enhances the signal of the area of ​​interest or suppresses interference signals by weighted combination of EEG signals on multiple electrodes. The spatial filtering signal refers to the EEG signal processed by the spatial filtering algorithm. The target electrode refers to the electrode whose signal is to be enhanced when performing spatial filtering. The reference electrode refers to the electrode used to calculate the spatial filtering weight.

[0135] S3. Calculate the forward analysis error and the backward analysis error of the spatial filtering signal, determine the signal timing analysis model of the spatial filtering signal based on the forward analysis error and the backward analysis error, calculate the signal spectrum of the spatial filtering signal according to the signal timing analysis model, and extract the EEG signal characteristics of the spatial filtering signal according to the signal spectrum.

[0136] The present invention can evaluate the influence of the time filter on the signal by calculating the forward analysis error and the backward analysis error of the spatial filtering signal, thereby providing a basis for subsequent signal processing.

[0137] In detail, the calculating of the forward analysis error and the backward analysis error of the spatial filtering signal includes:

[0138] Determining a reference signal of the spatial filtering signal;

[0139] Based on the reference signal, the forward analysis error of the spatial filtering signal is calculated using the following formula:

[0140] E fwd (t) = D ref (t)-D a (t)

[0141] Among them, E fwd (t) represents the forward analysis error of the spatial filtering signal at time t, D ref (t) represents the reference signal of the spatial filtering signal at time t, D a (t) represents the spatial filtered signal of the EEG signal at time t after being filtered by the spatial filtering algorithm;

[0142] Performing inverse filtering on the spatial filtering signal to obtain an inverse filtering signal;

[0143] Based on the inverse filtered signal, the backward analysis error of the spatial filtered signal is calculated using the following formula:

[0144] E bwd (t) = D a (t)-D rev (t)

[0145] Among them, E bwd (t) represents the backward analysis error of the spatial filtering signal at time t, D a (t) represents the spatial filtering signal of the EEG signal at time t after being filtered by the spatial filtering algorithm, D ref (t) represents the inverse filtered signal of the spatial filtered signal at time t.

[0146] Among them, the reference signal refers to a signal used to compare and evaluate the quality of the spatial filtering signal, the forward analysis error refers to the difference between the spatial filtering signal and the reference signal, the inverse filtering signal refers to the signal obtained by subjecting the spatial filtering signal to an inverse filtering process, and the backward analysis error refers to the difference between the spatial filtering signal and its inverse filtering signal.

[0147] The present invention determines the signal timing analysis model of the spatial filtering signal based on the forward analysis error and the backward analysis error, and can determine a model that can accurately describe the timing characteristics of the spatial filtering signal, and the model can be used for further signal analysis. The signal timing analysis model refers to a mathematical model used to analyze the statistical characteristics of a signal changing over time.

[0148] The present invention calculates the signal spectrum of the spatial filtering signal based on the signal timing analysis model, which can be used as a basis for later EEG signal feature extraction.

[0149] In detail, calculating the signal spectrum of the spatial filtering signal according to the signal timing analysis model includes:

[0150] Normalizing the spatial filtering signal to obtain a normalized filtering signal;

[0151] Identifying a stationary coefficient of the normalized filtered signal;

[0152] Based on the stationary coefficient, the normalized filtered signal is differentiated to obtain a differential filtered signal;

[0153] marking potential signals of the differential filtered signal using the signal timing analysis model;

[0154] Combining the potential signal and the differential filter signal to obtain a target EEG signal;

[0155] Dividing the target EEG signal into data segments;

[0156] A preset FFT algorithm is applied to the data segment to obtain the signal spectrum.

[0157] Among them, the normalized filtered signal refers to the signal obtained by scaling the original spatial filtered signal to have a unified standard or range, the smoothness coefficient refers to an indicator for measuring the smoothness of the signal, the differential filtered signal refers to the signal obtained by performing a differential operation on the normalized filtered signal, the potential signal refers to the meaningful signal component identified by the signal timing analysis model in the differential filtered signal, the target EEG signal refers to the final signal obtained after a series of processing (including spatial filtering, normalization, differentiation and potential signal marking), the data segment refers to a series of shorter time series segments into which the target EEG signal is divided, and the signal spectrum refers to the distribution of signal frequency components calculated by the FFT algorithm.

[0158] Optionally, the applying a preset FFT algorithm to the data segment to obtain the signal spectrum is performed by fast Fourier transform.

[0159] The present invention extracts the EEG signal features of the spatial filtering signal according to the signal spectrum, which refers to extracting information with diagnostic or analytical value from the spatial filtering signal based on the signal spectrum analysis. The EEG signal features include frequency features, time features, spatial features and other features.

[0160] S4. Extract features from the speech signal and the gesture image to obtain language signal features and gesture image features, and calculate the vertical component and horizontal component of the target user according to the EEG signal features, the language signal features, and the gesture image features.

[0161] The present invention extracts features from the speech signal and the gesture image, and the obtained language signal features and gesture image features can be used as a basis for subsequent wheelchair operation analysis.

[0162] In detail, the extracting features of the speech signal and the gesture image to obtain language signal features and gesture image features includes:

[0163] Denoising the speech signal to obtain a denoised speech signal;

[0164] Pre-emphasize the denoised speech signal to obtain an emphasized speech signal;

[0165] Splitting the emphasized speech signal into short-time frame signals;

[0166] Marking a signal starting point and a signal ending point of the short-time frame signal;

[0167] Based on the signal starting point and the signal ending point, calculating the Mel frequency cepstrum coefficients of the short time frame signal;

[0168] Analyzing the language signal characteristics of the speech signal through the Mel-frequency cepstral coefficients;

[0169] extracting a gesture area of ​​the gesture image;

[0170] Extracting gesture shape features, gesture movement features, and gesture surface features of the gesture area;

[0171] Based on the gesture shape features, gesture movement features, and gesture surface features, gesture image features of the gesture image are determined.

[0172] Among them, the denoised speech signal refers to the signal obtained by removing the background noise from the original speech signal by applying a denoising algorithm (such as spectral subtraction, noise cancellation, wavelet transform, etc.), the emphasized speech signal refers to the signal processed by applying a pre-emphasis filter to the denoised speech signal, the short-time frame signal refers to dividing the continuous emphasized speech signal into a series of short-time frames, the signal starting point refers to a specific frame marked in the speech signal, representing the beginning of the speech activity, the signal ending point refers to a specific frame marked in the speech signal, representing the end of the speech activity, the Mel-frequency cepstral coefficient refers to a coefficient used to represent the characteristics of the speech signal, and the language signal feature is It refers to the parameters extracted from the speech signal that can represent the language content or speaker characteristics, such as MFCC, pitch, energy, phonemes and other parameters. The gesture area refers to the part representing the gesture action in the image or video identified by the segmentation algorithm. The gesture shape feature refers to the feature that describes the contour or structure of the gesture. The gesture motion feature refers to the feature that describes the change of the gesture over time, such as speed, acceleration, trajectory and other features. The gesture surface feature refers to the feature that describes the surface attributes of the gesture, such as color, texture, skin reflection and other features. The gesture image feature refers to the general term for all features extracted from the gesture image, including shape features, motion features and surface features.

[0173] Furthermore, the extracting of gesture shape features, gesture movement features and gesture surface features of the gesture area includes:

[0174] identifying a gesture contour of the gesture area;

[0175] Analyzing gesture shape features of the gesture area according to the gesture contour;

[0176] Marking the gesture centroid of the gesture area;

[0177] analyzing a movement pattern of the center of mass of the gesture;

[0178] Defining gesture motion features of the gesture area according to the motion pattern;

[0179] Analyzing gesture color distribution in the gesture area;

[0180] Based on the gesture color distribution, a gesture surface feature of the gesture area is determined.

[0181] Among them, the gesture contour refers to the edge or boundary line of the gesture in the image, which defines the shape and size of the gesture area; the gesture centroid refers to the center of gravity or center point of the gesture area; the motion pattern refers to the law of change of the gesture centroid over time, including speed, acceleration, direction, trajectory, etc.; the gesture color distribution refers to the statistical distribution of colors in the gesture area, including color histograms, color moments (such as mean, standard deviation, etc.), distribution in color space, etc.

[0182] Optionally, the gesture centroid for marking the gesture area may be determined by calculating a weighted average position of all pixels in the gesture area.

[0183] The present invention calculates the vertical component and the horizontal component of the target user according to the EEG signal characteristics, the language signal characteristics and the gesture image characteristics, so as to estimate the target user's action intention, thereby providing more accurate control instructions for the AI ​​wheelchair or other auxiliary equipment.

[0184] As an embodiment of the present invention, the calculating the vertical component and the horizontal component of the target user according to the EEG signal feature, the language signal feature and the gesture image feature includes:

[0185] Extracting left brain signal features and right brain signal features of the EEG signal features;

[0186] Determine a left brain vertical signal amplitude and a left brain horizontal signal amplitude of the left brain signal characteristic, and determine a right brain vertical signal amplitude and a right brain horizontal signal amplitude of the right brain signal characteristic;

[0187] Analyzing the left brain signal weight of the left brain signal feature;

[0188] Analyzing the right brain signal weight of the right brain signal feature;

[0189] According to the target user's corresponding physical state, respectively calculating the EEG signal weight, the language signal weight and the gesture image weight of the EEG signal feature, the language signal feature and the gesture image feature;

[0190] According to the left brain vertical signal amplitude, the left brain horizontal signal amplitude, the right brain vertical signal amplitude, the right brain horizontal signal amplitude, the left brain signal weight, the right brain signal weight, the EEG signal weight, the language signal weight and the gesture image weight, the vertical component and the horizontal component of the target user are calculated using the following formula:

[0191] Y=W α (W r R y +W z Z y )+Wβ T y +W γ S y +C

[0192] X=W α (W r R x +W z Z x )+W β T x +W γ S x +C

[0193] Where Y represents the vertical component, W α represents the EEG signal weight, W β represents the language signal weight, W γ represents the gesture image weight, W r represents the right brain signal weight, W z represents the left brain signal weight, R y Represents the vertical signal amplitude of the right brain, R x represents the amplitude of the right brain horizontal signal, Z y Represents the vertical signal amplitude of the left brain, Z x represents the horizontal signal amplitude of the left brain, T y Indicates the language vertical signal feature corresponding to the language signal feature, T x Indicates the language level signal feature corresponding to the language signal feature, S y Indicates the vertical signal feature of the gesture corresponding to the gesture image feature, S x represents the gesture level signal feature corresponding to the gesture image feature, and C represents a constant.

[0194] Among them, the left brain signal feature refers to the feature related to left brain activity extracted from the EEG signal, the right brain signal feature refers to the feature related to right brain activity extracted from the EEG signal, the left brain vertical signal amplitude refers to the signal strength or amplitude related to vertical movement or intention in the left brain signal feature, the left brain horizontal signal amplitude refers to the signal strength or amplitude related to horizontal movement or intention in the left brain signal feature, the right brain vertical signal amplitude refers to the signal strength or amplitude related to vertical movement or intention in the right brain signal feature, the right brain horizontal signal amplitude refers to the signal strength or amplitude related to horizontal movement or intention in the right brain signal feature, and the left brain signal weight refers to the left brain signal feature in the overall The relative importance or influence of the right brain signal characteristics in the overall signal analysis, the right brain signal weight refers to the relative importance or influence of the right brain signal characteristics in the overall signal analysis, the EEG signal weight refers to the relative importance of the EEG signal characteristics in the comprehensive multimodal signal analysis, the language signal weight refers to the relative importance of the language signal characteristics in the comprehensive multimodal signal analysis, the gesture image weight refers to the relative importance of the gesture image characteristics in the comprehensive multimodal signal analysis, the vertical component refers to the quantitative representation of the user's action or intention in the vertical direction calculated according to the multimodal signal characteristics, and the horizontal component refers to the quantitative representation of the user's action or intention in the horizontal direction calculated according to the multimodal signal characteristics.

[0195] S5. Collect the external environment of the target user, extract the environmental characteristics of the external environment, calculate the motion path of the target user using a preset AI path control algorithm based on the environmental characteristics, the vertical component and the horizontal component, and perform interactive control of the AI ​​wheelchair according to the motion path.

[0196] The present invention collects the external environment of the target user, and extracts the environmental features of the external environment, which can be used as the basis for later path planning. The external environment refers to the surrounding space and conditions where the target user is located, including natural environment and artificial environment, and the environmental features refer to representative information extracted from the collected external environment.

[0197] Optionally, the present invention calculates the motion path of the target user based on the environmental characteristics, the vertical component and the horizontal component using a preset AI path control algorithm, and can calculate a reasonable motion path according to the environmental characteristics and the user's movement intention, thereby guiding the safe and effective movement of a wheelchair or other auxiliary equipment.

[0198] In detail, the calculating the motion path of the target user based on the environmental features, the vertical component and the horizontal component using a preset AI path control algorithm includes:

[0199] Determining an initial motion path of the target user according to the vertical component and the horizontal component;

[0200] According to the environmental characteristics, identifying obstacles in the initial motion path, and analyzing physical characteristics and position characteristics of the obstacles;

[0201] According to the physical features and the position features, a preset AI path control algorithm is used to respectively calculate a vertical adjustment parameter of the vertical component and a horizontal adjustment parameter of the horizontal component;

[0202] According to the vertical adjustment parameter and the horizontal adjustment parameter, the vertical component and the horizontal component are adjusted to obtain an optimized vertical component and an optimized horizontal component;

[0203] A motion path of the target user is constructed according to the optimized vertical component and the optimized horizontal component.

[0204] Among them, the initial motion path refers to the direct path of the user's intention calculated based on the vertical component and horizontal component of the target user without considering environmental obstacles. The obstacle refers to an object or area in the environment that prevents the user from moving along the initial motion path. The physical feature refers to the inherent properties of the obstacle, such as size, shape, stability, material, etc. The position feature refers to the specific position of the obstacle in the environment, including the position and distance relative to the user. The AI ​​path control algorithm refers to an algorithm for calculating and optimizing the motion path. The vertical adjustment parameter refers to a parameter for correcting the vertical component in order to avoid obstacles or adapt to environmental characteristics. The horizontal adjustment parameter refers to a parameter for correcting the horizontal component in order to avoid obstacles or adapt to environmental characteristics. The optimized vertical component refers to the vertical component corrected by the vertical adjustment parameter. The optimized horizontal component refers to the horizontal component corrected by the horizontal adjustment parameter. The motion path refers to the path constructed according to the optimized vertical and horizontal components.

[0205] Optionally, the adjusting of the vertical component and the horizontal component may be implemented by a PID controller.

[0206] Compared with the problems described in the background technology, the present invention obtains the AI ​​wheelchair of the target user, analyzes the physical state of the target user and the user's wheelchair needs, and can better configure and optimize the functions of the AI ​​wheelchair to ensure that it can meet the user's specific needs and provide a safe, comfortable and convenient mobile solution; the present invention determines the interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs to ensure that the interaction mode of the AI ​​wheelchair can maximize the physical state and needs of the target user; further, the present invention configures the sensor network of the AI ​​wheelchair according to the interaction mode to configure an efficient, reliable and user-friendly wheelchair sensor network, thereby supporting the interaction mode of the AI ​​wheelchair. Further, the present invention defines a spatial filtering algorithm for the EEG signal, uses the spatial filtering algorithm to filter the EEG signal, and obtains a spatially filtered EEG signal. These signals have better classification characteristics and can be used for further analysis and application. Further, the present invention calculates the forward analysis error and the backward analysis error of the spatially filtered signal to obtain a spatially filtered EEG signal. The influence of the time filter on the signal is evaluated to provide a basis for the subsequent signal processing. Furthermore, the present invention determines the signal timing analysis model of the spatial filter signal based on the forward analysis error and the backward analysis error, and can determine a model that can accurately describe the timing characteristics of the spatial filter signal. The model can be used for further signal analysis. Next, the present invention extracts features from the voice signal and the gesture image, and obtains language signal features and gesture image features, which can be used as the basis for the subsequent wheelchair operation analysis. The present invention calculates the vertical component and horizontal component of the target user based on the EEG signal features, the language signal features, and the gesture image features, which can realize the estimation of the target user's action intention, thereby providing more accurate control instructions for AI wheelchairs or other auxiliary devices. Finally, the present invention calculates the motion path of the target user based on the environmental features, the vertical component, and the horizontal component using a preset AI path control algorithm, which can calculate a reasonable motion path according to the environmental features and the user's motion intention, thereby guiding the safe and effective movement of the wheelchair or other auxiliary devices. Therefore, the present invention can improve the accuracy of AI wheelchair control.

[0207] Embodiment 2:

[0208] like Figure 2 , which is a functional module diagram of a multimodal interactive AI wheelchair control system provided by an embodiment of the present invention.

[0209] The multimodal interactive AI wheelchair control system 200 of the present invention can be installed in an electronic device. According to the functions to be implemented, the multimodal interactive AI wheelchair control system 200 can include a multimodal signal acquisition module 201, a spatial filtering algorithm definition module 202, an EEG signal feature analysis module 203, a vertical-horizontal component analysis module 204 and a wheelchair interactive control module 205. The module of the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0210] In this embodiment, the functions of each module / unit are as follows:

[0211] The multimodal signal acquisition module 201 is used to acquire the AI ​​wheelchair of the target user, analyze the physical state of the target user and the user's wheelchair needs, determine the interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs, configure the sensor network of the AI ​​wheelchair according to the interaction mode, and use the sensor network to collect the multimodal signal of the target user, wherein the multimodal signal includes: EEG signal, voice signal and gesture image;

[0212] The spatial filtering algorithm definition module 202 is used to define the spatial filtering algorithm of the EEG signal, and use the spatial filtering algorithm to filter the EEG signal to obtain a spatial filtering signal, wherein the spatial filtering algorithm:

[0213]

[0214] Among them, D a (t) represents the spatial filtering signal of the EEG signal at time t after being filtered by the spatial filtering algorithm, d a (t) represents the EEG signal at time t, d i represents the adjacent potential of the i-th reference electrode of the EEG signal corresponding to the target electrode, L a,i represents the distance between the target electrode and the i-th reference electrode;

[0215] The EEG signal feature analysis module 203 is used to calculate the forward analysis error and the backward analysis error of the spatial filtering signal, determine the signal timing analysis model of the spatial filtering signal based on the forward analysis error and the backward analysis error, calculate the signal spectrum of the spatial filtering signal according to the signal timing analysis model, and extract the EEG signal feature of the spatial filtering signal according to the signal spectrum;

[0216] The vertical-horizontal component analysis module 204 is used to extract features from the speech signal and the gesture image to obtain language signal features and gesture image features, and calculate the vertical component and horizontal component of the target user according to the EEG signal features, the language signal features and the gesture image features;

[0217] The wheelchair interaction control module 205 is used to collect the external environment of the target user, extract the environmental characteristics of the external environment, calculate the motion path of the target user based on the environmental characteristics, the vertical component and the horizontal component using a preset AI path control algorithm, and perform interactive control of the AI ​​wheelchair according to the motion path.

[0218] In detail, each module described in the multimodal interactive AI wheelchair control system 200 described in the embodiment of the present invention adopts the same technical means as the multimodal interactive AI wheelchair control method described in the accompanying drawings when used, and can produce the same technical effects, which will not be repeated here.

[0219] An embodiment of the present invention provides an electronic device for implementing a multi-modal interactive AI wheelchair control method.

[0220] See also Figure 3 As shown, the electronic device may include a processor 30, a memory 31, a communication bus 32 and a communication interface 33, and may also include a computer program stored in the memory 31 and executable on the processor 30, such as a multimodal interactive AI wheelchair control method program.

[0221] In some embodiments, the processor may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect the various components of the entire electronic device. It runs or executes programs or modules stored in the memory (for example, executing a multimodal interactive AI wheelchair control program, etc.), and calls data stored in the memory to execute various functions of the electronic device and process data.

[0222] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory may also include both an internal storage unit of the electronic device and an external storage device. The memory can be used not only to store application software and various types of data installed in the electronic device, such as code based on a multimodal interactive AI wheelchair control program, but also to temporarily store data that has been output or is to be output.

[0223] The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory and at least one processor, etc.

[0224] The communication interface is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0225] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor through a power management system, so that the power management system can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0226] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0227] The multimodal interactive AI wheelchair control program stored in the memory of the electronic device is a combination of multiple instructions, and when running in the processor, it can achieve:

[0228] Acquire an AI wheelchair of a target user, analyze the physical state of the target user and the user's wheelchair needs, determine an interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs, configure a sensor network of the AI ​​wheelchair according to the interaction mode, and use the sensor network to collect multimodal signals of the target user, wherein the multimodal signals include: electroencephalogram signals, voice signals, and gesture images;

[0229] A spatial filtering algorithm for the EEG signal is defined, and the EEG signal is filtered using the spatial filtering algorithm to obtain a spatial filtering signal, wherein the spatial filtering algorithm:

[0230]

[0231] Among them, D a (t) represents the spatial filtering signal of the EEG signal at time t after being filtered by the spatial filtering algorithm, d a (t) represents the EEG signal at time t, d i represents the adjacent potential of the i-th reference electrode of the EEG signal corresponding to the target electrode, L a,i represents the distance between the target electrode and the i-th reference electrode;

[0232] Calculating a forward analysis error and a backward analysis error of the spatial filtering signal, determining a signal timing analysis model of the spatial filtering signal based on the forward analysis error and the backward analysis error, calculating a signal spectrum of the spatial filtering signal according to the signal timing analysis model, and extracting an EEG signal feature of the spatial filtering signal according to the signal spectrum;

[0233] Extracting features from the speech signal and the gesture image to obtain language signal features and gesture image features, and calculating the vertical component and the horizontal component of the target user according to the EEG signal features, the language signal features, and the gesture image features;

[0234] The external environment of the target user is collected, environmental features of the external environment are extracted, and based on the environmental features, the vertical component and the horizontal component, a motion path of the target user is calculated using a preset AI path control algorithm, and interactive control of the AI ​​wheelchair is performed according to the motion path.

[0235] Specifically, the specific implementation method of the processor for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0236] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or system capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0237] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0238] Acquire an AI wheelchair of a target user, analyze the physical state of the target user and the user's wheelchair needs, determine an interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs, configure a sensor network of the AI ​​wheelchair according to the interaction mode, and use the sensor network to collect multimodal signals of the target user, wherein the multimodal signals include: electroencephalogram signals, voice signals, and gesture images;

[0239] A spatial filtering algorithm for the EEG signal is defined, and the EEG signal is filtered using the spatial filtering algorithm to obtain a spatial filtering signal, wherein the spatial filtering algorithm:

[0240]

[0241] Among them, D a (t) represents the spatial filtering signal of the EEG signal at time t after being filtered by the spatial filtering algorithm, d a (t) represents the EEG signal at time t, d irepresents the adjacent potential of the i-th reference electrode of the EEG signal corresponding to the target electrode, L a,i represents the distance between the target electrode and the i-th reference electrode;

[0242] Calculating a forward analysis error and a backward analysis error of the spatial filtering signal, determining a signal timing analysis model of the spatial filtering signal based on the forward analysis error and the backward analysis error, calculating a signal spectrum of the spatial filtering signal according to the signal timing analysis model, and extracting an EEG signal feature of the spatial filtering signal according to the signal spectrum;

[0243] Extracting features from the speech signal and the gesture image to obtain language signal features and gesture image features, and calculating the vertical component and the horizontal component of the target user according to the EEG signal features, the language signal features, and the gesture image features;

[0244] The external environment of the target user is collected, environmental features of the external environment are extracted, and based on the environmental features, the vertical component and the horizontal component, a motion path of the target user is calculated using a preset AI path control algorithm, and interactive control of the AI ​​wheelchair is performed according to the motion path.

[0245] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0246] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0247] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0248] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0249] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0250] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0251] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim can also be implemented by one unit or system through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A multimodal interactive AI wheelchair control method, characterized in that: The method comprises: Acquire an AI wheelchair of a target user, analyze the physical state of the target user and the user's wheelchair needs, determine an interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs, configure a sensor network of the AI ​​wheelchair according to the interaction mode, and use the sensor network to collect multimodal signals of the target user, wherein the multimodal signals include: electroencephalogram signals, voice signals, and gesture images; A spatial filtering algorithm for the EEG signal is defined, and the EEG signal is filtered using the spatial filtering algorithm to obtain a spatial filtering signal, wherein the spatial filtering algorithm: Among them, D a (t) represents the spatial filtering signal of the EEG signal at time t after being filtered by the spatial filtering algorithm, d a (t) represents the EEG signal at time t, d i represents the adjacent potential of the i-th reference electrode of the EEG signal corresponding to the target electrode, L a,i represents the distance between the target electrode and the i-th reference electrode; Calculating a forward analysis error and a backward analysis error of the spatial filtering signal, determining a signal timing analysis model of the spatial filtering signal based on the forward analysis error and the backward analysis error, calculating a signal spectrum of the spatial filtering signal according to the signal timing analysis model, and extracting an EEG signal feature of the spatial filtering signal according to the signal spectrum; Extracting features from the speech signal and the gesture image to obtain language signal features and gesture image features, and calculating the vertical component and the horizontal component of the target user according to the EEG signal features, the language signal features, and the gesture image features; The external environment of the target user is collected, environmental features of the external environment are extracted, and based on the environmental features, the vertical component and the horizontal component, a motion path of the target user is calculated using a preset AI path control algorithm, and interactive control of the AI ​​wheelchair is performed according to the motion path.

2. The multimodal interactive AI wheelchair control method according to claim 1, characterized in that: The analyzing the target user's physical condition and the user's wheelchair needs includes: Acquire user data of the target user; Performing data cleaning on the user data to obtain user cleansing data; Classifying the user cleansing data to obtain classified user cleansing data, wherein the classified user cleansing data includes basic information, living environment data, user activity data, user function data, nervous system data, sensory ability data, and cognitive ability data; Analyze the daily activity needs of the target user based on the basic information, living environment data and user activity data; Analyzing the user activity capability of the target user according to the user function data; Determining the target user's neurological capacity through the neurological data; Analyzing the sensory ability of the target user according to the sensory ability data; Analyzing the cognitive ability of the target user based on the cognitive ability data; Determine the physical state of the target user by combining the user's activity ability, the nervous system ability, the sensory ability, and the cognitive ability; A user wheelchair requirement of the target user is determined according to the physical state and the daily activity requirement.

3. The multimodal interactive AI wheelchair control method according to claim 2, characterized in that: The determining the interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair requirements includes: Based on the user's wheelchair requirements, analyzing the interactive modes of the AI ​​wheelchair; analyzing interaction characteristics of the interactive mode; Analyzing the user load and interaction convenience coefficient of the interactive mode on the physical state according to the interactive characteristics; Calculating the interaction mode score of the interactive mode according to the user load and the interaction convenience coefficient; Based on the interaction mode score, an interaction mode of the AI ​​wheelchair is determined.

4. The multimodal interactive AI wheelchair control method according to claim 3, characterized in that: The calculating the forward analysis error and the backward analysis error of the spatial filtering signal comprises: Determining a reference signal of the spatial filtering signal; Based on the reference signal, the forward analysis error of the spatial filtering signal is calculated using the following formula: E fwd (t)=D ref (t)-D a (t) Among them, E fwd (t) represents the forward analysis error of the spatial filtering signal at time t, D ref (t) represents the reference signal of the spatial filtering signal at time t, D a (t) represents the spatial filtered signal of the EEG signal at time t after being filtered by the spatial filtering algorithm; Performing inverse filtering on the spatial filtering signal to obtain an inverse filtering signal; Based on the inverse filtered signal, the backward analysis error of the spatial filtered signal is calculated using the following formula: E bwd (t)=D a (t)-D rev (t) Among them, E bwd (t) represents the backward analysis error of the spatial filtering signal at time t, D a (t) represents the spatial filtering signal of the EEG signal at time t after being filtered by the spatial filtering algorithm, D ref (t) represents the inverse filtered signal of the spatial filtered signal at time t.

5. The multimodal interactive AI wheelchair control method according to claim 4, characterized in that: The calculating the signal spectrum of the spatial filtering signal according to the signal timing analysis model comprises: Normalizing the spatial filtering signal to obtain a normalized filtering signal; identifying a stationary coefficient of the normalized filtered signal; Based on the stationary coefficient, the normalized filtered signal is differentiated to obtain a differential filtered signal; marking potential signals of the differential filtering signal using the signal timing analysis model; combining the potential signal and the differential filter signal to obtain a target EEG signal; Dividing the target EEG signal into data segments; A preset FFT algorithm is applied to the data segment to obtain the signal spectrum.

6. The multimodal interactive AI wheelchair control method according to claim 5, characterized in that: The extracting features of the speech signal and the gesture image to obtain language signal features and gesture image features includes: Denoising the speech signal to obtain a denoised speech signal; Pre-emphasize the denoised speech signal to obtain an emphasized speech signal; Splitting the emphasized speech signal into short-time frame signals; Marking a signal starting point and a signal ending point of the short-time frame signal; Based on the signal starting point and the signal ending point, calculating the Mel frequency cepstrum coefficients of the short time frame signal; Analyzing the language signal characteristics of the speech signal through the Mel-frequency cepstral coefficients; extracting a gesture area of ​​the gesture image; Extracting gesture shape features, gesture movement features, and gesture surface features of the gesture area; Based on the gesture shape features, gesture movement features, and gesture surface features, gesture image features of the gesture image are determined.

7. The multimodal interactive AI wheelchair control method according to claim 6, characterized in that: The extracting of gesture shape features, gesture movement features and gesture surface features of the gesture area includes: identifying a gesture contour of the gesture area; Analyzing gesture shape features of the gesture area according to the gesture contour; Marking the gesture centroid of the gesture area; analyzing a movement pattern of the center of mass of the gesture; Defining gesture motion features of the gesture area according to the motion pattern; Analyzing gesture color distribution in the gesture area; Based on the gesture color distribution, a gesture surface feature of the gesture area is determined.

8. The multimodal interactive AI wheelchair control method according to claim 7, characterized in that: The calculating the vertical component and the horizontal component of the target user according to the EEG signal feature, the language signal feature and the gesture image feature includes: Extracting left brain signal features and right brain signal features of the EEG signal features; Determine a left brain vertical signal amplitude and a left brain horizontal signal amplitude of the left brain signal characteristic, and determine a right brain vertical signal amplitude and a right brain horizontal signal amplitude of the right brain signal characteristic; Analyzing the left brain signal weight of the left brain signal feature; Analyzing the right brain signal weight of the right brain signal feature; According to the target user's corresponding physical state, respectively calculating the EEG signal weight, the language signal weight and the gesture image weight of the EEG signal feature, the language signal feature and the gesture image feature; According to the left brain vertical signal amplitude, the left brain horizontal signal amplitude, the right brain vertical signal amplitude, the right brain horizontal signal amplitude, the left brain signal weight, the right brain signal weight, the EEG signal weight, the language signal weight and the gesture image weight, the vertical component and the horizontal component of the target user are calculated using the following formula: Y=W α (W r R y +W z Z y )+W β T y +W γ S y +C X=W α (W r R x +W z Z x )+W β T x +W γ S x +C Where Y represents the vertical component, W α represents the EEG signal weight, W β represents the language signal weight, W γ represents the gesture image weight, W r represents the right brain signal weight, W z represents the left brain signal weight, R y Represents the vertical signal amplitude of the right brain, R x represents the amplitude of the right brain horizontal signal, Z y Represents the vertical signal amplitude of the left brain, Z x represents the horizontal signal amplitude of the left brain, T y Indicates the language vertical signal feature corresponding to the language signal feature, T x Indicates the language level signal feature corresponding to the language signal feature, S y Indicates the vertical signal feature of the gesture corresponding to the gesture image feature, S x represents the gesture level signal feature corresponding to the gesture image feature, and C represents a constant.

9. The multimodal interactive AI wheelchair control method according to claim 8, characterized in that: The calculating the motion path of the target user by using a preset AI path control algorithm based on the environmental features, the vertical component, and the horizontal component includes: Determining an initial motion path of the target user according to the vertical component and the horizontal component; According to the environmental characteristics, identifying obstacles in the initial motion path, and analyzing physical characteristics and position characteristics of the obstacles; According to the physical features and the position features, a preset AI path control algorithm is used to respectively calculate a vertical adjustment parameter of the vertical component and a horizontal adjustment parameter of the horizontal component; According to the vertical adjustment parameter and the horizontal adjustment parameter, the vertical component and the horizontal component are adjusted to obtain an optimized vertical component and an optimized horizontal component; A motion path of the target user is constructed according to the optimized vertical component and the optimized horizontal component.

10. A multimodal interactive AI wheelchair control system, characterized in that: A system for executing the multimodal interactive AI wheelchair control method according to any one of claims 1 to 9, the system comprising: a multimodal signal acquisition module, used to acquire the AI ​​wheelchair of the target user, analyze the physical state of the target user and the user's wheelchair needs, determine the interaction mode of the AI ​​wheelchair based on the physical state and the user's wheelchair needs, configure the sensor network of the AI ​​wheelchair according to the interaction mode, and use the sensor network to collect the multimodal signals of the target user, wherein the multimodal signals include: EEG signals, voice signals and gesture images; A spatial filtering algorithm definition module is used to define the spatial filtering algorithm of the EEG signal, and use the spatial filtering algorithm to filter the EEG signal to obtain a spatial filtering signal, wherein the spatial filtering algorithm: Among them, D a (t) represents the spatial filtering signal of the EEG signal at time t after being filtered by the spatial filtering algorithm, d a (t) represents the EEG signal at time t, d i represents the adjacent potential of the i-th reference electrode of the EEG signal corresponding to the target electrode, L a,i represents the distance between the target electrode and the i-th reference electrode; an EEG signal feature analysis module, used to calculate a forward analysis error and a backward analysis error of the spatial filtering signal, determine a signal timing analysis model of the spatial filtering signal based on the forward analysis error and the backward analysis error, calculate a signal spectrum of the spatial filtering signal according to the signal timing analysis model, and extract EEG signal features of the spatial filtering signal according to the signal spectrum; A vertical-horizontal component analysis module, used to extract features from the speech signal and the gesture image to obtain language signal features and gesture image features, and calculate the vertical component and horizontal component of the target user according to the EEG signal features, the language signal features and the gesture image features; The wheelchair interactive control module is used to collect the external environment of the target user, extract the environmental characteristics of the external environment, calculate the motion path of the target user based on the environmental characteristics, the vertical component and the horizontal component using a preset AI path control algorithm, and perform interactive control of the AI ​​wheelchair according to the motion path.

Images