Cabin barrier-free interaction method and device for disabled people and electronic equipment

Abstract

The application discloses a cockpit barrier-free interaction method and device for the disabled and electronic equipment, and comprises the following steps: inputting the brain electrical signal data, head posture data and voice data of a target passenger into a pre-trained passenger intention recognition model to obtain a first passenger intention and an intention confidence; when the intention confidence is less than a first threshold, a second passenger intention is matched in a historical interaction record, the first passenger intention is corrected according to the second passenger intention to obtain a target passenger intention; when the intention confidence is greater than or equal to the first threshold, the first passenger intention is determined as the target passenger intention; the concentration coefficient of the target passenger is determined according to the brain electrical signal data, if the concentration coefficient is greater than or equal to a second threshold, the corresponding cockpit control instruction is determined according to the target passenger intention, and the vehicle cockpit is controlled. The application improves the convenience and safety of the disabled when they travel by car, and can be widely applied to the technical field of intelligent cockpits.

Description

Technical Field

[0001] This invention relates to the field of smart cockpit technology, and in particular to a cockpit barrier-free interaction method, device and electronic device for people with disabilities. Background Technology

[0002] Currently, in-vehicle interaction technologies for people with disabilities can be mainly divided into two categories: one is basic adaptation type, such as the steering wheel button customization function provided by some car manufacturers, which can only realize simple volume adjustment and answering phone calls, and cannot meet the complex vehicle control needs; the other is single assistance type, such as Tesla's voice control function, which supports some command operations, but is not optimized for the physiological characteristics of people with disabilities. People with physical disabilities cannot trigger interactions other than the wake word, and people with visual impairments have difficulty obtaining screen feedback information.

[0003] In summary, the existing technology has the following drawbacks: 1) Limited interaction dimensions and poor adaptability: Designed only for a certain type of disability (such as only adapting to physical disabilities or only adapting to visual impairments), it cannot meet the complex needs of "physical + language" dual disabilities, and the coverage of the applicable population is not disclosed. 2) Intentional pre-recognition is prone to misoperation: Involuntary head shaking in cerebral palsy patients and delayed instructions in ALS patients are easily misinterpreted by the system as control commands, resulting in a high rate of misoperation; 3) Feedback methods do not match perception abilities: Visually impaired people cannot receive text feedback on the screen, and hearing impaired people cannot receive voice prompts. The feedback information is disconnected from the perception abilities of people with disabilities. 4) Lack of user-friendly hardware adaptation: Auxiliary devices are mostly external modifications, which are cumbersome to install and do not meet automotive safety standards, taking up space in the vehicle and affecting travel.

[0004] The aforementioned problems result in insufficient convenience and safety for people with disabilities when traveling by public transport, and urgently need to be addressed. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0006] Therefore, one objective of this invention is to provide a cabin accessibility interaction method for people with disabilities, which improves the convenience and safety of people with disabilities when traveling by vehicle.

[0007] Another objective of this invention is to provide a cockpit-based barrier-free interactive device for people with disabilities.

[0008] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include: On one hand, embodiments of the present invention provide a cockpit accessibility interaction method for people with disabilities, including the following steps: Acquire the target occupant's EEG signal data, head posture data, voice data, and historical interaction records; The EEG signal data, the head posture data, and the speech data are input into a pre-trained occupant intention recognition model to obtain the first occupant intention and the corresponding intention confidence. When the confidence level of the intent is less than a preset first threshold, the second occupant intent is obtained by matching the head posture data and the voice data in the historical interaction record, and the first occupant intent is corrected according to the second occupant intent to obtain the target occupant intent. When the confidence level of the intent is greater than or equal to the first threshold, the first occupant intent is determined to be the target occupant intent. The focus coefficient of the target occupant is determined based on the EEG signal data. If the focus coefficient is greater than or equal to a preset second threshold, the corresponding cockpit control command is determined based on the target occupant's intention, and the vehicle cockpit is controlled according to the cockpit control command.

[0009] Furthermore, in one embodiment of the present invention, the acquisition of the target occupant's electroencephalogram (EEG) signal data, head posture data, voice data, and historical interaction records specifically includes: The brainwave signal data, head posture data, and voice data are acquired by a smart headband that integrates an EEG sensor, a miniature gyroscope, and a voice acquisition module. The facial data of the target occupant is acquired through the vehicle-mounted camera, the occupant's identity information is identified based on the facial data, and the corresponding historical interaction records are obtained by matching the occupant's identity information in the cockpit interaction database.

[0010] Furthermore, in one embodiment of the present invention, the occupant intention recognition model is trained through the following steps: The test occupants' EEG signal samples, head posture samples, and speech samples were acquired, and the corresponding occupant intention labels were determined through manual annotation. The EEG signal samples, head posture samples, and speech samples are input into a pre-constructed multi-branch spatiotemporal graph convolutional network to obtain the predicted occupant intentions. The loss value is determined based on the predicted occupant intent and the occupant intent label; The parameters of the multi-branch spatiotemporal graph convolutional network are updated using the backpropagation algorithm based on the loss value to obtain the trained occupant intention recognition model. The multi-branch spatiotemporal graph convolutional network includes a temporal convolutional branch for EEG signals, a spatiotemporal graph convolutional branch for head pose, a speech coding branch, a feature fusion layer, and an output layer.

[0011] Furthermore, in one embodiment of the present invention, the step of inputting the EEG signal samples, the head posture samples, and the speech samples into a pre-constructed multi-branch spatiotemporal graph convolutional network to obtain the predicted occupant intention specifically includes: Based on the EEG signal samples, corresponding EEG time-series signal samples are generated, and the EEG time-series signal samples are input into the EEG signal time-series convolution branch to obtain EEG signal features. Based on the head pose sample, a corresponding head pose temporal feature map is generated, and the head pose temporal feature map is input into the head pose spatiotemporal graph convolution branch to obtain head pose features. The speech sample is input into the speech coding branch to obtain speech features; The feature fusion layer performs feature fusion on the EEG signal features, head posture features, and speech features based on a self-attention mechanism to obtain a fused feature vector; The fused feature vector is mapped to the predicted occupant intent through the output layer.

[0012] Furthermore, in one embodiment of the present invention, the step of matching the second occupant intention with the head posture data and the voice data in the historical interaction record, and correcting the first occupant intention according to the second occupant intention to obtain the target occupant intention, specifically includes: Based on the historical interaction records, multiple sets of historical interaction data are determined, including the target occupant's historical head posture, historical voice, and corresponding historical occupant intentions. Calculate the first similarity between the head pose data and the historical head pose, and calculate the second similarity between the speech data and the historical speech, and then determine the matching degree between the head pose data and the speech data and the historical interaction data based on the first similarity and the second similarity; The historical passenger intent corresponding to the historical interaction data with the highest matching degree is determined as the second passenger intent; Determine the first operation object and the first expected result corresponding to the first occupant's intention, and the second operation object and the second expected result corresponding to the second occupant's intention; Calculate the third similarity between the first operation object and the second operation object, and calculate the fourth similarity between the first expected result and the second expected result; When the third similarity is greater than or equal to a preset third threshold, and the fourth similarity is greater than or equal to a preset fourth threshold, the first occupant intention is taken as the target occupant intention; When the third similarity is greater than or equal to the third threshold and the fourth similarity is less than the fourth threshold, the target occupant intention is generated based on the first operation object and the second expected result; When the third similarity is less than the third threshold, the second occupant intention is taken as the target occupant intention.

[0013] Furthermore, in one embodiment of the present invention, determining the focus coefficient of the target occupant based on the electroencephalogram (EEG) signal data specifically includes: The EEG signal data is bandpass filtered to extract the EEG time-domain signal of a preset frequency band; The EEG time-domain signal is converted into the EEG frequency-domain signal by Fourier transform, and the energy integrals of the α-wave band and β-wave band are calculated respectively to obtain the α-wave energy and β-wave energy. The focus coefficient is determined based on the ratio of the β-wave energy to the α-wave energy.

[0014] Furthermore, in one embodiment of the present invention, the cockpit accessibility interaction method further includes the following steps: If the focus coefficient is less than the second threshold, the target occupant is reminded to confirm the target occupant's intention.

[0015] On the other hand, embodiments of the present invention provide a cockpit accessibility interaction device for people with disabilities, comprising: The data acquisition module is used to acquire the target occupant's electroencephalogram (EEG) signal data, head posture data, voice data, and historical interaction records. The intention recognition module is used to input the EEG signal data, the head posture data and the speech data into a pre-trained occupant intention recognition model to obtain the first occupant intention and the corresponding intention confidence. The first intent determination module is used to, when the confidence level of the intent is less than a preset first threshold, match the head posture data and the voice data in the historical interaction record to obtain the second occupant intent, and correct the first occupant intent according to the second occupant intent to obtain the target occupant intent; The second intent determination module is used to determine the first occupant intent as the target occupant intent when the intent confidence level is greater than or equal to the first threshold. The cockpit control module is used to determine the focus coefficient of the target occupant based on the EEG signal data. If the focus coefficient is greater than or equal to a preset second threshold, the module determines the corresponding cockpit control command based on the target occupant's intention and controls the vehicle cockpit according to the cockpit control command.

[0016] On the other hand, embodiments of the present invention provide an electronic device, including: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the above-described method for accessible cockpit interaction for people with disabilities.

[0017] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable computer program that, when executed by a processor, implements the above-described accessible cockpit interaction method for people with disabilities.

[0018] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the above-described accessible cockpit interaction method for people with disabilities.

[0019] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention: This invention acquires the target occupant's EEG signal data, head posture data, voice data, and historical interaction records. The EEG signal data, head posture data, and voice data are input into a pre-trained occupant intention recognition model to obtain a first occupant intention and its corresponding confidence level. When the intention confidence level is less than a preset first threshold, a second occupant intention is obtained by matching the head posture data and voice data against the historical interaction records. The first occupant intention is then corrected based on the second occupant intention to obtain the target occupant intention. When the intention confidence level is greater than or equal to the first threshold, the first occupant intention is determined as the target occupant intention. The target occupant's attention coefficient is determined based on the EEG signal data. If the attention coefficient is greater than or equal to a preset second threshold, the corresponding cockpit control command is determined based on the target occupant intention, and the vehicle cockpit is controlled according to the cockpit control command. This invention identifies passenger intentions based on their EEG signal data, head posture data, and voice data. When the confidence level of the passenger's intention is less than a first threshold, the intention is corrected based on historical interaction records. The passenger's attention coefficient is calculated based on the EEG signal data. If the attention coefficient is greater than or equal to a second threshold, the corresponding cockpit control command is determined based on the passenger's intention. This enables accurate identification of the cockpit control intentions of disabled persons and cockpit control, improving the convenience and safety of disabled persons traveling by vehicle. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating the steps of an accessible cockpit interaction method for people with disabilities, provided in an embodiment of the present invention; Figure 2 A structural block diagram of a barrier-free interactive cockpit device for people with disabilities provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0024] The accessible cockpit interaction method for people with disabilities provided in this invention can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application that implements the accessible cockpit interaction method for people with disabilities, but is not limited to the above forms.

[0025] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0026] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0027] Reference Figure 1 This invention provides a method for accessible cockpit interaction for people with disabilities, specifically including the following steps: S101. Acquire the target occupant's EEG signal data, head posture data, voice data, and historical interaction records; S102. Input the EEG signal data, head posture data and speech data into the pre-trained occupant intention recognition model to obtain the first occupant intention and the corresponding intention confidence. S103. When the confidence level of the intent is less than the preset first threshold, the second occupant intent is obtained by matching the head posture data and voice data in the historical interaction records, and the first occupant intent is corrected according to the second occupant intent to obtain the target occupant intent. S104. When the confidence level of the intent is greater than or equal to the first threshold, the intent of the first occupant is determined to be the intent of the target occupant. S105. Determine the focus coefficient of the target occupant based on the EEG signal data. If the focus coefficient is greater than or equal to the preset second threshold, determine the corresponding cockpit control command based on the target occupant's intention, and control the vehicle cockpit according to the cockpit control command.

[0028] This invention identifies passenger intentions based on their EEG signal data, head posture data, and voice data. When the confidence level of the passenger's intention is less than a first threshold, the intention is corrected based on historical interaction records. The passenger's attention coefficient is calculated based on the EEG signal data. If the attention coefficient is greater than or equal to a second threshold, the corresponding cockpit control command is determined based on the passenger's intention. This enables accurate identification of the cockpit control intentions of disabled persons and cockpit control, improving the convenience and safety of disabled persons traveling by vehicle.

[0029] As a further optional implementation, the target occupant's electroencephalogram (EEG) signal data, head posture data, voice data, and historical interaction records are acquired, specifically including: S1011: A smart headband that integrates an EEG sensor, a miniature gyroscope, and a voice acquisition module acquires EEG signal data, head posture data, and voice data. S1012. Obtain the facial data of the target occupant through the vehicle-mounted camera, identify the occupant's identity information based on the facial data, and match the corresponding historical interaction records in the cockpit interaction database based on the occupant's identity information.

[0030] Specifically, this embodiment of the invention employs a lightweight interactive terminal suitable for people with disabilities, specifically a smart headband integrating an EEG sensor, a miniature gyroscope, and a voice acquisition module. The EEG sensor collects forehead EEG signals, the miniature gyroscope captures head posture, and the voice acquisition module receives voice data. Simultaneously, a vehicle-mounted camera acquires the facial data of the target occupant, identifies the occupant's identity based on the facial data, and matches the occupant's identity information against a cockpit interaction database to obtain corresponding historical interaction records. These historical interaction records store multiple historical cockpit interaction behaviors of the target occupant.

[0031] As an optional further implementation, the occupant intent recognition model is trained through the following steps: S201. Obtain the EEG signal samples, head posture samples, and voice samples of the test occupants, and determine the corresponding occupant intention labels through manual annotation. S202. Input the EEG signal samples, head posture samples, and speech samples into a pre-constructed multi-branch spatiotemporal graph convolutional network to obtain the predicted occupant intentions. S203. Determine the loss value based on the predicted occupant intentions and occupant intention labels; S204. Update the parameters of the multi-branch spatiotemporal graph convolutional network according to the loss value through the backpropagation algorithm to obtain the trained occupant intention recognition model. The multi-branch spatiotemporal graph convolutional network includes a temporal convolutional branch for EEG signals, a spatiotemporal graph convolutional branch for head pose, a speech coding branch, a feature fusion layer, and an output layer.

[0032] Specifically, the process involves acquiring EEG signal samples, head posture samples, and speech samples from test occupants, manually labeling them to determine corresponding occupant intention tags, inputting the EEG signal samples, head posture samples, and speech samples into a pre-constructed multi-branch spatiotemporal graph convolutional network to obtain predicted occupant intentions, determining the loss value based on the predicted occupant intentions and the occupant intention tags, and updating the parameters of the multi-branch spatiotemporal graph convolutional network based on the loss value using the backpropagation algorithm to complete one iteration of training. Training stops when the number of iterations reaches a preset threshold or the loss value falls below a preset threshold, thus obtaining a trained occupant intention recognition model.

[0033] As a further optional implementation, EEG signal samples, head posture samples, and speech samples are input into a pre-constructed multi-branch spatiotemporal graph convolutional network to obtain predicted occupant intentions, specifically including: S2021. Generate corresponding EEG time-series signal samples based on EEG signal samples, and input the EEG time-series signal samples into the EEG signal time-series convolution branch to obtain EEG signal features. S2022. Generate the corresponding head pose temporal feature map based on the head pose sample, and input the head pose temporal feature map into the head pose spatiotemporal graph convolution branch to obtain the head pose features. S2023. Input the speech sample into the speech coding branch to obtain speech features; S2024. The feature fusion layer performs feature fusion on EEG signal features, head posture features and speech features based on the self-attention mechanism to obtain a fused feature vector. S2025. The fused feature vectors are mapped to the predicted occupant intentions through the output layer.

[0034] Specifically, EEG signal features can be extracted by performing temporal convolution processing on EEG signal samples through a temporal convolution branch; head posture samples within 500ms are converted into temporal feature maps, and spatiotemporal convolution processing is performed through a head posture spatiotemporal map convolution branch to capture the correlation between head posture and cockpit control, thus obtaining head posture features; speech signals are encoded through a speech coding branch (such as the Wav2Vec 2.0 model), extracting keywords such as "open window" and "navigation," and combining semantic context to exclude ambiguous commands, thus obtaining speech features; feature fusion layer performs feature fusion on EEG signal features, head posture features, and speech features based on a self-attention mechanism to obtain a fused feature vector, and the output layer maps the fused feature vector to predict occupant intentions.

[0035] The EEG signal data, head posture data, and speech data obtained in the aforementioned steps are input into the trained occupant intention recognition model to obtain multiple occupant intention recognition results and corresponding probability distributions. The occupant intention recognition result with the highest probability is selected as the first occupant intention, and the intention confidence of the first occupant intention is calculated based on the probability distribution.

[0036] When the confidence level of the intent is greater than or equal to the first threshold (e.g., 0.9), the first occupant intent can be directly output as the target occupant intent; when the confidence level of the intent is less than the preset first threshold, the second occupant intent is obtained by matching the head posture data and voice data in the historical interaction records, and the first occupant intent is corrected according to the second occupant intent to obtain the target occupant intent.

[0037] As a further optional implementation, a second occupant intention is obtained by matching head posture data and voice data in historical interaction records, and the first occupant intention is modified based on the second occupant intention to obtain the target occupant intention, which specifically includes: S1031. Determine multiple sets of historical interaction data based on historical interaction records. The historical interaction data includes the target occupant's historical head posture, historical voice, and corresponding historical occupant intentions. S1032. Calculate the first similarity between the head pose data and the historical head pose, and calculate the second similarity between the speech data and the historical speech, and then determine the matching degree between the head pose data and the speech data and the historical interaction data based on the first similarity and the second similarity. S1033. Determine the historical occupant intent corresponding to the historical interaction data with the highest matching degree as the second occupant intent; S1034. Determine the first operation object and the first expected result corresponding to the first occupant's intention, and the second operation object and the second expected result corresponding to the second occupant's intention; S1035. Calculate the third similarity between the first operation object and the second operation object, and calculate the fourth similarity between the first expected result and the second expected result; S1036. When the third similarity is greater than or equal to the preset third threshold and the fourth similarity is greater than or equal to the preset fourth threshold, the first occupant's intention is taken as the target occupant's intention. S1037. When the third similarity is greater than or equal to the third threshold and the fourth similarity is less than the fourth threshold, the target occupant intention is generated based on the first operation object and the second expected result. S1038. When the third similarity is less than the third threshold, the second occupant's intention is taken as the target occupant's intention.

[0038] Specifically, in this embodiment of the invention, the occupant's past head postures (such as the angle of looking down when viewing navigation, or the side profile when adjusting the air conditioning), voice commands, and corresponding cockpit control intentions are bound together to establish a historical interaction record unique to that occupant. The matching degree between the occupant's current head posture data and voice data and historical interaction data is determined through head posture similarity and voice content similarity, thereby finding the closest historical interaction data and determining the corresponding second occupant intention. The real-time identified first occupant intention and the historically matched second occupant intention are compared from two dimensions: the operation object and the expected result. When the operation object similarity is greater than or equal to a third threshold (e.g., 0.9) and the second occupant intention is determined, the second occupant intention is determined. If the fourth similarity is greater than or equal to the fourth threshold (e.g., 0.9), the first occupant's intent identified in real time can be directly used as the target occupant's intent. If the third similarity is greater than or equal to the third threshold but the fourth similarity is less than the fourth threshold, it means that the first occupant's intent and the second occupant's intent operate on the same object but have different expected results. Since the confidence of the first occupant's intent is not high, the target occupant's intent is generated based on the first object and the second expected result. If the third similarity is less than the third threshold, it means that the first occupant's intent and the second occupant's intent operate on different objects. Since the confidence of the first occupant's intent is not high, the second occupant's intent is directly used as the target occupant's intent.

[0039] As a further optional implementation, the focus coefficient of the target occupant is determined based on electroencephalogram (EEG) signal data, specifically including: S1051. Bandpass filtering is performed on the EEG signal data to extract the EEG time-domain signal of the preset frequency band; S1052. Convert the EEG time-domain signal into the EEG frequency-domain signal through Fourier transform, and calculate the energy integrals of the α-wave band and β-wave band respectively to obtain the α-wave energy and β-wave energy. S1053. Determine the focus coefficient based on the ratio of β wave energy to α wave energy.

[0040] Specifically, the EEG signal is first denoised by extracting the 8-30Hz frequency band (only alpha and beta waves are retained) through bandpass filtering. Then, artifact removal is performed to eliminate interference signals generated by blinking, muscle contraction, etc. The time domain signal is then converted to the frequency domain by Fourier transform (FFT) to calculate the energy integral of the alpha wave band (8-13Hz) and the beta wave band (14-30Hz). The focus coefficient is determined based on the ratio of beta wave energy to alpha wave energy.

[0041] Calculate the energy of the beta wave With alpha wave energy The ratio R:

[0042] When R>1, beta waves dominate, and the brain is in an active and focused state; when R=1, alpha and beta waves have roughly equal energy, and the brain is in a relaxed and alert state; when R<1, alpha waves dominate, and the brain tends to be in a relaxed or absent-minded state.

[0043] To obtain an intuitive focus coefficient on a 0-100 scale, this embodiment of the invention performs a non-linear mapping on the energy ratio, resulting in the focus coefficient as follows:

[0044] Where k is an adjustment parameter (can be 1). When R→+∞, C→100 (high concentration), and when R→0, C→0 (complete relaxation).

[0045] As an optional implementation, the cockpit accessibility interaction method further includes the following steps: If the focus coefficient is less than the second threshold, the target occupant is prompted to confirm their intention.

[0046] Specifically, if the attention coefficient is greater than or equal to the preset second threshold (e.g., 50), the corresponding cockpit control command is determined according to the target occupant's intention, and the vehicle cockpit is controlled according to the cockpit control command; if the attention coefficient is less than the second threshold, the target occupant needs to be reminded to confirm the target occupant's intention a second time, and the cockpit is controlled after the target occupant confirms.

[0047] In some optional embodiments, the present invention can provide feedback to the target occupant based on the execution result of the cockpit control. For example, for visually impaired persons, the operation result is fed back through different vibration frequencies based on the side vibration module of the seat (such as "one long vibration" indicates that the window is opened successfully, and "two short vibrations" indicates that the command is invalid), and the specific status is announced by voice. For hearing impaired persons, the operation result is fed back based on the light prompts on the central control screen (green indicates success, red indicates failure) and the tactile feedback of the steering wheel. For persons with both language and physical impairments, the operation result is fed back based on the micro vibrator built into the headband to provide Braille dot matrix feedback (such as "dot matrix 'open' indicates that the air conditioning is turned on").

[0048] The method steps of the embodiments of the present invention have been described above. It can be understood that the embodiments of the present invention recognize the occupant's intention based on the occupant's electroencephalogram (EEG) signal data, head posture data, and voice data. When the confidence level of the occupant's intention is less than a first threshold, the occupant's intention is corrected based on historical interaction records. The occupant's attention coefficient is calculated based on the EEG signal data. If the attention coefficient is greater than or equal to a second threshold, the corresponding cockpit control command is determined based on the occupant's intention. This enables accurate identification of the cockpit control intentions of disabled persons and cockpit control, improving the convenience and safety of disabled persons traveling by vehicle.

[0049] Compared with the prior art, the embodiments of the present invention have the following advantages: 1) Full-scene adaptation and wider coverage: Multimodal fusion technology adapts to single or multiple disability types such as physical disability, visual impairment, and speech impairment, improving the coverage of applicable population; 2) Significantly improved accuracy of intent recognition: Based on historical interaction records, the intents of occupants with low confidence levels are corrected, thereby improving the accuracy of intent recognition; 3) More comprehensive safety redundancy protection: Safety verification is performed based on the attention coefficient to ensure the reliability of the execution of driving safety-related commands and avoid risks caused by misoperation. 4) More user-friendly feedback and hardware design: Multimodal feedback matches different perception capabilities, lightweight terminals meet automotive safety standards, easy installation does not take up valuable space, and improves travel convenience.

[0050] Reference Figure 2 This invention provides a cockpit accessibility interaction device for people with disabilities, comprising: The data acquisition module is used to acquire the target occupant's electroencephalogram (EEG) signal data, head posture data, voice data, and historical interaction records. The intent recognition module is used to input EEG signal data, head posture data and speech data into a pre-trained occupant intent recognition model to obtain the first occupant's intent and the corresponding intent confidence. The first intent determination module is used to match the second occupant intent with head posture data and voice data in historical interaction records when the intent confidence is less than a preset first threshold, and to correct the first occupant intent based on the second occupant intent to obtain the target occupant intent. The second intent determination module is used to determine the first occupant intent as the target occupant intent when the intent confidence level is greater than or equal to the first threshold. The cockpit control module is used to determine the attention coefficient of the target occupant based on EEG signal data. If the attention coefficient is greater than or equal to a preset second threshold, the corresponding cockpit control command is determined according to the target occupant's intention, and the vehicle cockpit is controlled according to the cockpit control command.

[0051] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0052] Reference Figure 3 This invention provides an electronic device, comprising: At least one processor; At least one memory for storing at least one program; When the above-mentioned at least one program is executed by the above-mentioned at least one processor, the above-mentioned at least one processor implements the above-mentioned accessible cockpit interaction method for people with disabilities.

[0053] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0054] This invention also provides a computer-readable storage medium storing a processor-executable computer program that, when executed by a processor, implements the aforementioned accessible cockpit interaction method for people with disabilities.

[0055] This invention provides a computer-readable storage medium that can execute a cockpit accessibility interaction method for people with disabilities provided in the method embodiments of this invention. It can execute any combination of the implementation steps of the method embodiments and has the corresponding functions and beneficial effects of the method.

[0056] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned accessible cockpit interaction method for people with disabilities.

[0057] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0058] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0059] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0060] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0061] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0062] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0063] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0065] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0066] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0067] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0068] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0069] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A cockpit accessibility interaction method for people with disabilities, characterized in that, Includes the following steps: Acquire the target occupant's EEG signal data, head posture data, voice data, and historical interaction records; The EEG signal data, the head posture data, and the speech data are input into a pre-trained occupant intention recognition model to obtain the first occupant intention and the corresponding intention confidence. When the confidence level of the intent is less than a preset first threshold, the second occupant intent is obtained by matching the head posture data and the voice data in the historical interaction record, and the first occupant intent is corrected according to the second occupant intent to obtain the target occupant intent. When the confidence level of the intent is greater than or equal to the first threshold, the first occupant intent is determined to be the target occupant intent. The focus coefficient of the target occupant is determined based on the EEG signal data. If the focus coefficient is greater than or equal to a preset second threshold, the corresponding cockpit control command is determined based on the target occupant's intention, and the vehicle cockpit is controlled according to the cockpit control command.

2. The accessible cockpit interaction method for people with disabilities according to claim 1, characterized in that, The acquisition of the target occupant's electroencephalogram (EEG) signal data, head posture data, voice data, and historical interaction records specifically includes: The brainwave signal data, head posture data, and voice data are acquired by a smart headband that integrates an EEG sensor, a miniature gyroscope, and a voice acquisition module. The facial data of the target occupant is acquired through the vehicle-mounted camera, the occupant's identity information is identified based on the facial data, and the corresponding historical interaction records are obtained by matching the occupant's identity information in the cockpit interaction database.

3. The accessible cockpit interaction method for people with disabilities according to claim 1, characterized in that, The occupant intention recognition model is trained through the following steps: The test occupants' EEG signal samples, head posture samples, and speech samples were acquired, and the corresponding occupant intention labels were determined through manual annotation. The EEG signal samples, head posture samples, and speech samples are input into a pre-constructed multi-branch spatiotemporal graph convolutional network to obtain the predicted occupant intentions. The loss value is determined based on the predicted occupant intent and the occupant intent label; The parameters of the multi-branch spatiotemporal graph convolutional network are updated using the backpropagation algorithm based on the loss value to obtain the trained occupant intention recognition model. The multi-branch spatiotemporal graph convolutional network includes a temporal convolutional branch for EEG signals, a spatiotemporal graph convolutional branch for head pose, a speech coding branch, a feature fusion layer, and an output layer.

4. The accessible cockpit interaction method for people with disabilities according to claim 3, characterized in that, The step of inputting the EEG signal samples, the head posture samples, and the speech samples into a pre-constructed multi-branch spatiotemporal graph convolutional network to obtain the predicted occupant intention specifically includes: Based on the EEG signal samples, corresponding EEG time-series signal samples are generated, and the EEG time-series signal samples are input into the EEG signal time-series convolution branch to obtain EEG signal features. Based on the head pose sample, a corresponding head pose temporal feature map is generated, and the head pose temporal feature map is input into the head pose spatiotemporal graph convolution branch to obtain head pose features. The speech sample is input into the speech coding branch to obtain speech features; The feature fusion layer performs feature fusion on the EEG signal features, head posture features, and speech features based on a self-attention mechanism to obtain a fused feature vector; The fused feature vector is mapped to the predicted occupant intent through the output layer.

5. The accessible cockpit interaction method for people with disabilities according to claim 1, characterized in that, The step of matching the second occupant's intention with the head posture data and the voice data in the historical interaction record, and then modifying the first occupant's intention based on the second occupant's intention to obtain the target occupant's intention, specifically includes: Based on the historical interaction records, multiple sets of historical interaction data are determined, including the target occupant's historical head posture, historical voice, and corresponding historical occupant intentions. Calculate the first similarity between the head pose data and the historical head pose, and calculate the second similarity between the speech data and the historical speech, and then determine the matching degree between the head pose data and the speech data and the historical interaction data based on the first similarity and the second similarity; The historical passenger intent corresponding to the historical interaction data with the highest matching degree is determined as the second passenger intent; Determine the first operation object and the first expected result corresponding to the first occupant's intention, and the second operation object and the second expected result corresponding to the second occupant's intention; Calculate the third similarity between the first operation object and the second operation object, and calculate the fourth similarity between the first expected result and the second expected result; When the third similarity is greater than or equal to a preset third threshold, and the fourth similarity is greater than or equal to a preset fourth threshold, the first occupant intention is taken as the target occupant intention; When the third similarity is greater than or equal to the third threshold and the fourth similarity is less than the fourth threshold, the target occupant intention is generated based on the first operation object and the second expected result; When the third similarity is less than the third threshold, the second occupant intention is taken as the target occupant intention.

6. The accessible cockpit interaction method for people with disabilities according to claim 1, characterized in that, The determination of the target occupant's attention coefficient based on the electroencephalogram (EEG) signal data specifically includes: The EEG signal data is bandpass filtered to extract the EEG time-domain signal of a preset frequency band; The EEG time-domain signal is converted into the EEG frequency-domain signal by Fourier transform, and the energy integrals of the α-wave band and β-wave band are calculated respectively to obtain the α-wave energy and β-wave energy. The focus coefficient is determined based on the ratio of the β-wave energy to the α-wave energy.

7. A cockpit accessibility interaction method for people with disabilities according to any one of claims 1 to 6, characterized in that, The cockpit accessibility interaction method also includes the following steps: If the focus coefficient is less than the second threshold, the target occupant is reminded to confirm the target occupant's intention.

8. A barrier-free interactive cockpit device for people with disabilities, characterized in that, include: The data acquisition module is used to acquire the target occupant's electroencephalogram (EEG) signal data, head posture data, voice data, and historical interaction records. The intention recognition module is used to input the EEG signal data, the head posture data and the speech data into a pre-trained occupant intention recognition model to obtain the first occupant intention and the corresponding intention confidence. The first intent determination module is used to, when the confidence level of the intent is less than a preset first threshold, match the head posture data and the voice data in the historical interaction record to obtain the second occupant intent, and correct the first occupant intent according to the second occupant intent to obtain the target occupant intent; The second intent determination module is used to determine the first occupant intent as the target occupant intent when the intent confidence level is greater than or equal to the first threshold. The cockpit control module is used to determine the focus coefficient of the target occupant based on the EEG signal data. If the focus coefficient is greater than or equal to a preset second threshold, the module determines the corresponding cockpit control command based on the target occupant's intention and controls the vehicle cockpit according to the cockpit control command.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a cockpit accessibility interaction method for people with disabilities as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a cockpit accessibility interaction method for people with disabilities as described in any one of claims 1 to 7.

Images