Voice output device control method, device, and computer readable storage medium

Abstract

The application discloses a control method and device of a voice output device and a computer readable storage medium, and the method comprises the following steps: acquiring an electroencephalogram signal of a brain language function area of a user and a face image of the user; determining a target expression language of the user according to the electroencephalogram signal and the face image; and controlling a broadcasting component to output the target expression language. The accuracy of the expression language of an aphasiac is improved through mutual verification and error correction of the expression language by the electroencephalogram signal and the face image.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a control method, apparatus, and computer-readable storage medium for a voice output device. Background Technology

[0002] Aphasia refers to brain lesions related to language function, causing impairment in the understanding and expression of language in aphasic individuals. Aphasic individuals can communicate with the outside world through sign language, but this communication is limited to those who have systematically learned sign language; they cannot communicate with those who have not, leading to various inconveniences in social activities. In related technologies, brain-computer interface (BCI) technology is used to determine and output the spoken language of aphasic individuals, enabling interaction between them and others. However, relying on BCI technology to determine the spoken language carries the risk of language distortion, and the accuracy of the aphasic individual's spoken language cannot be guaranteed. Summary of the Invention

[0003] This application provides a control method, apparatus, and computer-readable storage medium for a speech output device, aiming to improve the accuracy of speech expression for people with aphasia.

[0004] This application provides a control method for a voice output device, the control method for the voice output device including:

[0005] Acquire electroencephalogram (EEG) signals from the user's language function areas and facial images of the user;

[0006] The user's target expressive language is determined based on the electroencephalogram (EEG) signals and the facial images.

[0007] The broadcasting component is controlled to output the target language.

[0008] Optionally, the step of determining the user's target expressive language based on the electroencephalogram (EEG) signal and the facial image includes:

[0009] The user's first facial movement is determined based on the electroencephalogram (EEG) signal, and the user's second facial movement is determined based on the facial image.

[0010] The user's target expressive language is determined based on the first facial movement and the second facial movement.

[0011] Optionally, the step of determining the user's first facial movement based on the electroencephalogram (EEG) signal includes:

[0012] Obtain the first weight value of the current EEG signal and the second weight value of the output state at the previous moment;

[0013] The output state corresponding to the current EEG signal is determined based on the dot product of the current EEG signal and the first weight value, and the dot product of the output state at the previous moment and the second weight value.

[0014] The user's first facial movement is determined based on the output state corresponding to the current EEG signal.

[0015] Optionally, the step of determining the user's second facial action based on the facial image includes:

[0016] Extract multiple key feature points from the facial image;

[0017] Based on the distance between the key feature points and preset key feature points, the preset key feature points that match each key feature point are determined.

[0018] The preset facial movements corresponding to each of the matched preset key feature points are determined as the user's second facial movements.

[0019] Optionally, the step of determining the user's target expressive language based on the first facial movement and the second facial movement includes:

[0020] The user's first expressive language is determined based on the first facial movement, and the user's second expressive language is determined based on the second facial movement;

[0021] The user's target language is determined based on the similarity between the first and second expressions.

[0022] Optionally, the step of determining the user's target expression language based on the similarity between the first expression language and the second expression language includes:

[0023] When the similarity between the first expression language and the second expression language is greater than a preset value, the first expression language or the second expression language is determined as the user's target expression language.

[0024] Optionally, the step of determining the user's first expressive language based on the first facial movement and determining the user's second expressive language based on the second facial movement includes:

[0025] Determine a preset facial action that matches the first facial action, and use the preset expression language associated with the preset facial action that matches the first facial action as the user's first expression language;

[0026] And, determine a preset facial action that matches the second facial action, and associate the preset facial action that matches the second facial action with a preset expression language, as the user's second expression language.

[0027] Optionally, before the step of determining the user's target expressive language based on the EEG signal and the facial image, the method further includes:

[0028] The EEG signals are preprocessed, which includes, in sequence, electrode localization, removal of EEG signals from non-brain language function regions, filtering, segmentation and baseline calibration, and removal of bad segments and artifacts, in order to determine the user's target language expression based on the preprocessed EEG signals and facial images.

[0029] Furthermore, to achieve the above objectives, the present invention also provides a voice output device comprising:

[0030] Broadcasting components;

[0031] The EEG cap has cortical EEG electrodes on the inside, which are used to collect EEG signals from the language function areas of the user's brain.

[0032] An image acquisition device, wherein the image acquisition device is used to acquire the user's facial image;

[0033] The processor is configured to determine the user's spoken language based on the electroencephalogram (EEG) signal and the facial image, and to control the broadcasting component to output the spoken language.

[0034] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a control program for a voice output device thereon, wherein the control program for the voice output device, when executed by a processor, implements the steps of the above-described control method for the voice output device.

[0035] This application provides a technical solution for controlling a speech output device, including a method, apparatus, and computer-readable storage medium. It captures the electroencephalogram (EEG) signals of an aphasic individual using an EEG cap and captures facial images of the aphasic individual using an image acquisition device. Based on the EEG signals and facial images, the system determines the aphasic individual's intended speech and outputs it through a speaker. Compared to related technologies that rely solely on EEG signals to determine the aphasic individual's speech, leading to potential distortion, this application improves the accuracy of the aphasic individual's speech by cross-validating and correcting the speech using both EEG signals and facial images. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the first embodiment of the control method for the voice output device of the present invention;

[0037] Figure 2 This is a schematic diagram of the voice output device of the present invention.

[0038] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The accompanying drawings are only one embodiment and not the entirety of the invention. Detailed Implementation

[0039] Currently, brain-computer interface (BCI) technology is used to determine and output the spoken language of aphasic individuals, enabling them to interact with others. However, this technology is currently limited to laboratory settings and used for the rehabilitation of aphasic patients. It requires implanting electrodes in the brain, which poses a risk of harm to the aphasic individual. Secondly, the numerous connections involved in BCI technology hinder communication between aphasic individuals and others in mobile settings. Furthermore, the symptoms of aphasic individuals vary; if only BCI technology is used to determine their spoken language, there is a risk of distortion, meaning the aphasic individual may still be unable to accurately convey their intended meaning.

[0040] To address the aforementioned problems, this application proposes a novel control method for a language output device, designed to help aphasic individuals accurately convey spoken language to ordinary people. The control method includes: acquiring electroencephalogram (EEG) signals from the user's language function regions and a facial image of the user; determining the user's target spoken language based on the EEG signals and the facial image; and controlling a broadcasting component to output the target spoken language. Because this application uses the mutual verification and correction of spoken language through EEG signals and facial images, it improves the accuracy of spoken language expression for aphasic individuals.

[0041] In addition, the voice output device of this application includes at least a broadcasting component, an EEG cap, an image acquisition device, and a processor, and each component can be integrated and set up so that communication between aphasic individuals and ordinary people can be realized in both mobile and laboratory settings.

[0042] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0043] like Figure 1 As shown, in the first embodiment of this application, the control method of the voice output device of this application includes the following steps:

[0044] Step S110: Obtain the electroencephalogram (EEG) signals of the user's brain's language function areas and the user's facial image.

[0045] Optionally, the aforementioned user can be a person with aphasia, hearing impairment, or visual impairment. This application uses a person with aphasia as an example.

[0046] Optionally, an EEG cap can be used to acquire EEG signals from the user's language function areas. The inner side of the EEG cap integrates multiple high-density cortical EEG electrodes located in the language function areas of the brain, specifically the cortical regions that control speech and vocalization. By wearing the EEG cap with integrated dry, high-density cortical EEG electrodes, EEG signals from the language function areas of the aphasic patient's brain can be collected. Because the EEG electrodes are integrated into the inner side of the EEG cap and do not need to be implanted in the aphasic patient's brain, the EEG cap is suitable for collecting EEG signals in various scenarios.

[0047] Optionally, the brainwave signals of the language function areas of the aphasic individual can be collected in real time for expressive language analysis; alternatively, the collection time can be determined based on the historical changes in the aphasic individual's brainwave signals, and the brainwave signals of the aphasic individual within a preset time period can be collected for expressive language analysis. The change characteristic can be the fluctuation amplitude of the brainwave signals. For example, when the fluctuation amplitude of historical brainwave signals is outside the preset fluctuation range, it indicates that the aphasic individual has an intention to express themselves. In this case, acquiring the brainwave signals of the aphasic individual within the preset time period for expressive language analysis can reveal the aphasic individual's expressive language.

[0048] Optionally, the facial image can be an image of the entire face of the aphasic person, or a partial facial image, such as an image including the lips and jaw. Of course, if the acquired facial image is an image of the entire face of the aphasic person, the entire facial image can be processed to extract key parts, such as extracting an image containing only the lips and jaw from the overall facial image for speech analysis, avoiding the influence of other parts of the face on the speech analysis results and improving the accuracy of the speech analysis results.

[0049] Optionally, facial images of the aphasic person can be acquired using an image acquisition device mounted on the EEG cap. Optionally, the EEG cap can be designed as a baseball cap or a helmet. For example, the image acquisition device can be located on the inside of the front edge of the baseball cap, above the face, to acquire facial images of the aphasic person.

[0050] Optionally, refer to Figure 2Furthermore, an image acquisition device can be installed on the smart glasses. When a person with aphasia wears the smart glasses, the image acquisition device on the smart glasses captures facial images of the person with aphasia for expression and language analysis. The smart glasses can be VR glasses, MR glasses, etc. The number and placement of the image acquisition devices can be determined according to the actual situation. The smart glasses of this application are easy to wear and suitable for various mobile scenarios, including indoor and outdoor environments; in addition, the smart glasses are discreet in shape, making them less likely to attract prejudiced looks from outsiders.

[0051] Optionally, refer to Figure 2 This application incorporates a processor on the side of the smart glasses. Each EEG electrode transmits its acquired EEG signals to the processor. Simultaneously, the image acquisition device also transmits acquired facial images to the processor. The processor in the smart glasses preprocesses the EEG signals and then integrates the preprocessed EEG signals with the facial images. This integrated processing of EEG signals and facial images via the smart glasses' processor reduces the development cost of language output devices.

[0052] Optionally, refer to Figure 2 The EEG cap and smart glasses of this application can each integrate a separate processor. Each EEG electrode patch is connected to the processor on the EEG cap side, used to transmit the EEG signals collected by each patch to the processor on the EEG cap side. The processor on the EEG cap side is used to collect and preprocess the EEG signals collected by each EEG electrode patch, and then transmit the preprocessed EEG signals to the processor on the smart glasses side. The processor on the smart glasses side is used to perform comprehensive processing of the preprocessed EEG signals and facial images. By integrating separate processors on the EEG cap and smart glasses, data loss during the transmission of EEG signals collected by each EEG electrode patch to the smart glasses for processing is avoided, which would lead to inaccurate expression language analysis results. The EEG cap processor preprocesses the EEG signals collected by each EEG electrode patch before sending them to the smart glasses, ensuring the integrity of the EEG signal data and improving the accuracy of expression language analysis results. At the same time, it reduces the data processing burden on the smart glasses side and improves the data processing speed of the smart glasses.

[0053] Optionally, a signal transmission module is provided between the EEG cap and the smart glasses to transmit EEG signals to the smart glasses. This signal transmission module can be a signal cable or a wireless transmission module.

[0054] For example, when the signal transmission module is a signal cable, one end of the signal cable is fixedly connected to the EEG cap, and the other end is detachably connected to the smart glasses, for example, via a magnetic connection. A storage space is provided at the connection point between the signal cable and the EEG cap. When the signal cable is not connected to the smart glasses, it automatically retracts into this storage space, thus storing the signal cable. Optionally, the signal cable can also be detachably connected to the EEG cap at one end and fixedly connected to the smart glasses at the other end. Alternatively, the signal cable can be either fixedly or detachably connected to both the EEG cap and the smart glasses, depending on the specific situation. This signal cable enables the transmission of EEG signals to the smart glasses.

[0055] For example, the signal transmission module can also be a wireless transmission module, with wireless transmission modules provided in both the EEG cap and the smart glasses, to achieve the transmission of EEG signals.

[0056] Optionally, upon power-on, the smart glasses first initialize, performing initialization checks on each module. Any issues are reported to the system for processing. If all initializations pass, the system begins normal operation. It's important to note that this function requires connection to the EEG cap via a signal transmission module; otherwise, system initialization will fail. A high-density cortical EEG electrode pad is then activated, fitting snugly against the head. This pad is a non-invasive scalp EEG recording electrode with an AgCI coating, as this material records relatively stable EEG signals. Data is then transmitted to the smart glasses' main processor via the signal transmission module for processing.

[0057] Step S120: Determine the user's target expressive language based on the EEG signal and the facial image.

[0058] Optionally, since different aphasic individuals have different circumstances—for example, some aphasic individuals do not move their lips, some are frequently in a state of epilepsy, and some wear masks—the brain's electroencephalogram (EEG) signals of the language function areas can be captured separately to determine the aphasic individual's spoken language, or the facial image of the aphasic individual can be captured separately to determine the aphasic individual's spoken language, or the spoken language can be determined by combining the EEG signals of the language function areas of the aphasic individual's brain with the facial image.

[0059] For example, characteristic information of the aphasic can be obtained; the method for determining the aphasic's expressive language can be determined based on the characteristic information; and the target expressive language of the aphasic can be determined based on the method for determining the expressive language. Optionally, the characteristic information of the aphasic can be pre-classified, and a mapping relationship between the characteristic information categories and the method for determining the expressive language can be established, so that a method for determining the expressive language that matches the characteristics of different aphasic individuals can be determined to accurately determine the aphasic's expressive language.

[0060] Optionally, when the aphasic's characteristic information is the first preset characteristic, the expressed language is determined based on EEG signals; when the aphasic's characteristic information is the second preset characteristic, the expressed language is determined based on facial images to identify secondary facial movements; and when the aphasic's characteristic information is the third preset characteristic, the expressed language is determined based on both EEG signals and facial images. The first, second, and third preset characteristics can be set according to the aphasic's condition. For example, the first preset characteristic could be that the aphasic wears a mask or that the aphasic's lips do not move. The second preset characteristic could be that the aphasic is not wearing a mask. The third preset characteristic could be that the aphasic is not wearing a mask and is frequently in a state of epilepsy.

[0061] For example, when a person with aphasia wears a mask or their lips are still, their spoken language can be determined solely through electroencephalogram (EEG) signals. This method is suitable for communication between a person with aphasia and ordinary people in pandemic scenarios or other situations where mask-wearing is required. When a person with aphasia is not wearing a mask, their spoken language can be determined solely through facial images. This method is suitable for communication between a person with aphasia and ordinary people in ordinary scenarios where mask-wearing is not required. When a person with aphasia is not wearing a mask and is frequently in a state of epilepsy, their spoken language can be determined through both EEG signals and facial images. This method is suitable for communication between a person with aphasia and ordinary people in situations where their facial expressions change significantly.

[0062] Optionally, determining the target spoken language for aphasic individuals based on EEG signals and facial images includes:

[0063] Step S121: Determine the user's first facial movement based on the EEG signal, and determine the user's second facial movement based on the facial image;

[0064] Optionally, the user's first facial movement can be determined using a recurrent convolutional neural network. Specifically, determining the user's first facial movement based on electroencephalogram (EEG) signals includes the following steps:

[0065] Step S1211: Obtain the first weight value of the current EEG signal and the second weight value of the output state at the previous moment.

[0066] Step S1212: Determine the output state corresponding to the current EEG signal based on the dot product of the current EEG signal and the first weight value, and the dot product of the output state at the previous moment and the second weight value.

[0067] Step S1213: Determine the user's first facial movement based on the output state corresponding to the current EEG signal.

[0068] Optionally, a recurrent neural network (RNN) is a network with memory, capable of remembering previous inputs so that the previous input *n* influences the output of the next *n+1* inputs. The new state value can be obtained using the tan hyperbolic function, which is the dot product of the weights associated with the previous state and the dot product of the associated weights and the input state. The tan hyperbolic function is expressed as:

[0069] H t =tanh(W hh .h t-1 +W xh .x t ).

[0070] In this context, the Recurrent Convolutional Neural Network (RNN) is a function of the input vector and the previous state, which leads us to a new state Ht. An RNN consists of at least an input layer, hidden layers, and an output layer. Different layers have corresponding nodes, and the connections between nodes form a directed graph along the time sequence, allowing previous outputs to be used as inputs. t For the current EEG signal, h t-1 W represents the output state corresponding to the EEG signal at the previous moment. hh W is the second weight value of the output state at the previous time step. xh H is the first weight value of the current EEG signal. t This represents the output state corresponding to the current EEG signal.

[0071] This embodiment determines the user's first facial movement based on a recurrent convolutional neural network, thereby improving the accuracy of the first facial movement.

[0072] Alternatively, the user's first facial movement can be determined using a CNN convolutional neural network.

[0073] Optionally, the user's second facial movement can be determined using a CNN convolutional neural network. Specifically, the steps for determining the user's second facial movement based on a facial image include:

[0074] Step S1214: Extract multiple key feature points from the facial image;

[0075] Step S1215: Determine the preset key feature points that match each key feature point based on the distance between the key feature points and preset key feature points.

[0076] Step S1216: Determine the preset facial actions corresponding to each of the matched preset key feature points as the user's second facial actions.

[0077] Optionally, the basic structure of a convolutional neural network generally includes: convolutional layers, activation functions, pooling layers, fully connected layers, and an output layer. In the convolutional layers, settings such as 2D convolution, stride, convolution mode, data padding (e.g., edge padding and dilation padding), receptive field, convolution depth, and kernel size can be configured to perform convolution processing on facial images. Activation functions can be ReLU, ELU, etc.

[0078] Optionally, at least one constraint feature parameter is extracted from each facial image to obtain a feature image corresponding to each constraint feature parameter; a convolutional neural network (CNN) network structure model corresponding to each feature image is trained; a loss function for the feature recognition model is determined based on all the network structure models; and the feature recognition model is established based on the loss function. In subsequent use, the user's second facial action is determined based on this feature recognition model.

[0079] Optionally, the key feature points can be feature points on the lips and jaw. These key feature points are marked. During the feature recognition model process, multiple face samples are stored, and each face sample is labeled with preset key feature points, which can form corresponding facial actions. Multiple key feature points in the currently extracted facial image can be matched with preset key feature points to determine the preset key feature points that match each key feature point. Then, the preset facial actions corresponding to the matched preset key feature points are determined as the user's second facial action.

[0080] Optionally, the cosine distance between a key feature point and preset key feature points can be determined, and the preset key feature points to be matched can be determined based on the cosine distance. For example, preset key feature points with a cosine distance less than a preset value can be determined as matched preset key feature points. Alternatively, the similarity between a key feature point and preset key feature points can be determined, and the preset key feature points to be matched can be determined based on the similarity. For example, preset key feature points with a similarity less than a preset value can be determined as matched preset key feature points.

[0081] Optionally, a second facial movement of the user can also be determined based on an RNN convolutional neural network.

[0082] This embodiment uses a convolutional neural network to determine the user's second facial movements, thereby improving the accuracy of the second facial movements.

[0083] Step S122: Determine the user's target expressive language based on the first facial movement and the second facial movement.

[0084] Optionally, after processing the EEG signals and facial images separately, corresponding facial movements are obtained. The similarity between the first facial movement and the second facial movement is compared. If the similarity is greater than a preset value, the user's target language is determined based on either the first or second facial movement. Alternatively, the first and second facial movements can be compared separately with preset facial movements. If the similarity between the first and preset facial movements is greater than the similarity between the second and preset facial movements, the user's target language is determined based on the first facial movement; otherwise, it is determined based on the second facial movement. By comparing the differences between different facial movements, error correction between facial movements is achieved, making the determined facial movements more accurate.

[0085] Optionally, determining the user's target expressive language based on the first facial movement and the second facial movement includes the following steps:

[0086] Step S1221: Determine the user's first expressive language based on the first facial movement, and determine the user's second expressive language based on the second facial movement;

[0087] Optionally, a mapping relationship between facial movements and spoken language can be pre-established. Based on this mapping relationship and a first facial movement, the user's first spoken language is determined. Based on this mapping relationship and a second facial movement, the user's second spoken language is determined. Specifically, a preset facial movement matching the first facial movement is determined, and a preset spoken language associated with the preset facial movement matching the first facial movement is used as the user's first spoken language; and a preset facial movement matching the second facial movement is determined, and a preset spoken language associated with the preset facial movement matching the second facial movement is used as the user's second spoken language.

[0088] Step S1222: Determine the user's target expression language based on the similarity between the first expression language and the second expression language.

[0089] Optionally, there may be differences between the first and second expressions. Therefore, it is necessary to correct the first and second expressions to obtain the accurate target expression.

[0090] Optionally, the similarity between the first and second expressions can be compared. If the similarity is greater than a preset value, the user's target expression is determined based on either the first or second expression. Alternatively, the first and second expressions can be compared separately with preset expressions. If the similarity between the first and preset expressions is greater than the similarity between the second and preset expressions, the user's target expression is determined based on the first expression; otherwise, it is determined based on the second expression. By comparing the differences between different expressions, error correction can be achieved, making the determined expression more accurate.

[0091] Optionally, the user's first facial movement can be determined based on the electroencephalogram (EEG) signal, and the user's target language can be determined based on the first facial movement. Alternatively, the user's second facial movement can be determined based on the facial image, and the user's target language can be determined based on the second facial movement, thus enabling the determination of the aphasic person's language in different scenarios.

[0092] Step S130: Control the broadcasting component to output the target expression language.

[0093] Optionally, the audio playback component can be mounted on the temples of the smart glasses, specifically on the left or right temples. The left and right temples are detachably connected to the main body of the smart glasses for easy storage when not in use. When the aphasic person's brain signals and facial movements occur, the smart glasses can play the previously processed target speech through speakers located on the left and right temples, thereby helping the aphasic person to output speech with high accuracy.

[0094] Based on the above technical solution, this embodiment improves the accuracy of aphasic individuals' spoken language by using EEG signals and facial images to mutually verify and correct expressed language. Furthermore, the speech output device of this application includes at least a broadcasting component, an EEG cap, an image acquisition device, and a processor, and these components can be integrated to enable communication between aphasic individuals and ordinary people in both mobile and laboratory settings.

[0095] Optionally, after acquiring the EEG signals and before determining the user's target language based on the EEG signals and facial images, preprocessing of the EEG signals is required to improve the accuracy of the target language expression. Optionally, the preprocessing of the EEG signals may include: electrode localization, removal of EEG signals from non-language functional areas, filtering, segmentation and baseline calibration, and removal of bad segments and artifacts. The filtering can employ bandpass filtering at a frequency of 0.1–40 Hz. After preprocessing the EEG signals, the user's target language expression can be determined based on the preprocessed EEG signals and facial images, thus improving the accuracy of the target language expression.

[0096] This invention provides an embodiment of a control method for a voice output device. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0097] like Figure 2 As shown, Figure 2 This is a schematic diagram of the voice output device according to an embodiment of the present invention. The voice output device includes:

[0098] The sound-announcing components can be mounted on the temples of the glasses, and the number of these components can be adjusted according to actual needs. For example, sound-announcing components 4 and 5 can be mounted on the temples.

[0099] The EEG cap has cortical EEG electrodes 7 on its inner side, which are used to collect EEG signals from the language function areas of the user's brain.

[0100] An image acquisition device is provided for acquiring facial images of the user. Optionally, the image acquisition device can be mounted on an EEG cap, or as shown in the image acquisition device... Figure 2 The image acquisition device 1 and image acquisition device 2 are shown as being installed on smart glasses.

[0101] The processor is configured to determine the user's spoken language based on the electroencephalogram (EEG) signal and the facial image, and to control the broadcasting component to output the spoken language.

[0102] Optionally, a signal transmission module is provided between the EEG cap and the smart glasses to transmit EEG signals to the smart glasses. This signal transmission module can be a signal cable or a wireless transmission module. When the signal transmission module is as follows... Figure 2 When the signal line 6 is shown, one end of the signal line 6 is fixedly connected to the EEG cap, and the other end is detachably connected to the smart glasses, for example, by magnetic attraction through the magnetic attachment 3.

[0103] Optionally, a storage space is provided at the connection point between the signal cable 6 and the EEG cap. When the signal cable is not connected to the smart glasses, it automatically retracts into this storage space, thus storing the signal cable. Optionally, the signal cable can also be detachably connected to the EEG cap at one end and fixedly connected to the smart glasses at the other end. Alternatively, the signal cable can be either fixedly connected to both the EEG cap and the smart glasses, or detachably connected, depending on the actual situation. The signal cable enables the transmission of EEG signals to the smart glasses.

[0104] Optionally, the voice output device of this application may further include a memory, which may be a high-speed RAM or a stable memory. The memory may also optionally be a storage device independent of the aforementioned processor.

[0105] Those skilled in the art will understand that Figure 2 The structure of the voice output device shown does not constitute a limitation on the voice output device. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0106] like Figure 2 As shown, the memory, as a storage medium, may include an operating system, a network communication module, a user interface module, and a control program for the voice output device. The operating system is a program that manages and controls the hardware and software resources of the voice output device, the control program for the voice output device, and the operation of other software or programs.

[0107] exist Figure 2 In the illustrated voice output device, the processor can be used to call the control program for the voice output device stored in the memory. Wherein:

[0108] When the processor calls the control program for the voice output device stored in memory, it performs the following operations:

[0109] Acquire electroencephalogram (EEG) signals from the user's language function areas and facial images of the user;

[0110] The user's target expressive language is determined based on the electroencephalogram (EEG) signals and the facial images.

[0111] The broadcasting component is controlled to output the target language.

[0112] When the processor calls the control program for the voice output device stored in memory, it performs the following operations:

[0113] The user's first facial movement is determined based on the electroencephalogram (EEG) signal, and the user's second facial movement is determined based on the facial image.

[0114] The user's target expressive language is determined based on the first facial movement and the second facial movement.

[0115] When the processor calls the control program for the voice output device stored in memory, it performs the following operations:

[0116] Obtain the first weight value of the current EEG signal and the second weight value of the output state at the previous moment;

[0117] The output state corresponding to the current EEG signal is determined based on the dot product of the current EEG signal and the first weight value, and the dot product of the output state at the previous moment and the second weight value.

[0118] The user's first facial movement is determined based on the output state corresponding to the current EEG signal.

[0119] When the processor calls the control program for the voice output device stored in memory, it performs the following operations:

[0120] Extract multiple key feature points from the facial image;

[0121] Based on the distance between the key feature points and preset key feature points, the preset key feature points that match each key feature point are determined.

[0122] The preset facial movements corresponding to each of the matched preset key feature points are determined as the user's second facial movements.

[0123] When the processor calls the control program for the voice output device stored in memory, it performs the following operations:

[0124] The user's first expressive language is determined based on the first facial movement, and the user's second expressive language is determined based on the second facial movement;

[0125] The user's target language is determined based on the similarity between the first and second expressions.

[0126] When the processor calls the control program for the voice output device stored in memory, it performs the following operations:

[0127] When the similarity between the first expression language and the second expression language is greater than a preset value, the first expression language or the second expression language is determined as the user's target expression language.

[0128] When the processor calls the control program for the voice output device stored in memory, it performs the following operations:

[0129] Determine a preset facial action that matches the first facial action, and use the preset expression language associated with the preset facial action that matches the first facial action as the user's first expression language;

[0130] And, determine a preset facial action that matches the second facial action, and associate the preset facial action that matches the second facial action with a preset expression language, as the user's second expression language.

[0131] When the processor calls the control program for the voice output device stored in memory, it performs the following operations:

[0132] The EEG signals are preprocessed, which includes, in sequence, electrode localization, removal of EEG signals from non-brain language function regions, filtering, segmentation and baseline calibration, and removal of bad segments and artifacts, in order to determine the user's target language expression based on the preprocessed EEG signals and facial images.

[0133] Based on the same inventive concept, this application also provides a computer-readable storage medium storing a control program for a voice output device. When the control program for the voice output device is executed by a processor, it implements the various steps of the control method for the voice output device as described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0134] Since the storage medium provided in this application embodiment is the storage medium used to implement the method of this application embodiment, those skilled in the art can understand the specific structure and variations of the storage medium based on the method described in this application embodiment, and therefore will not be repeated here. All storage media used in the method of this application embodiment are within the scope of protection of this application.

[0135] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0136] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, television, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0138] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A control method for a voice output device, characterized in that, The control method for the voice output device includes: Acquire electroencephalogram (EEG) signals from the user's language function areas and facial images of the user; The user's target expressive language is determined based on the electroencephalogram (EEG) signals and the facial images. The broadcasting component is controlled to output the target language expression. The step of determining the user's target expressive language based on the electroencephalogram (EEG) signal and the facial image includes: The user's first facial movement is determined based on the electroencephalogram (EEG) signal, and the user's second facial movement is determined based on the facial image. The user's first expressive language is determined based on the first facial movement, and the user's second expressive language is determined based on the second facial movement; The user's target language is determined based on the similarity between the first and second expressions.

2. The control method for the voice output device as described in claim 1, characterized in that, The step of determining the user's first facial movement based on the electroencephalogram (EEG) signal includes: Obtain the first weight value of the current EEG signal and the second weight value of the output state at the previous moment; The output state corresponding to the current EEG signal is determined based on the dot product of the current EEG signal and the first weight value, and the dot product of the output state at the previous moment and the second weight value. The user's first facial movement is determined based on the output state corresponding to the current EEG signal.

3. The control method for the voice output device as described in claim 1, characterized in that, The step of determining the user's second facial action based on the facial image includes: Extract multiple key feature points from the facial image; Based on the distance between the key feature points and preset key feature points, the preset key feature points that match each key feature point are determined. The preset facial movements corresponding to each of the matched preset key feature points are determined as the user's second facial movements.

4. The control method for the voice output device as described in claim 1, characterized in that, The step of determining the user's target expression language based on the similarity between the first expression language and the second expression language includes: When the similarity between the first expression language and the second expression language is greater than a preset value, the first expression language or the second expression language is determined as the user's target expression language.

5. The control method for the voice output device as described in claim 1, characterized in that, The steps of determining the user's first expressive language based on the first facial movement and determining the user's second expressive language based on the second facial movement include: Determine a preset facial action that matches the first facial action, and use the preset expression language associated with the preset facial action that matches the first facial action as the user's first expression language; And, determine a preset facial action that matches the second facial action, and associate the preset facial action that matches the second facial action with a preset expression language, as the user's second expression language.

6. The control method for the voice output device as described in claim 1, characterized in that, Before the step of determining the user's target expressive language based on the electroencephalogram (EEG) signal and the facial image, the method further includes: The EEG signals are preprocessed, which includes, in sequence, electrode localization, removal of EEG signals from non-brain language function regions, filtering, segmentation and baseline calibration, and removal of bad segments and artifacts, in order to determine the user's target language expression based on the preprocessed EEG signals and facial images.

7. A voice output device, characterized in that, The voice output device includes: Broadcasting components; The EEG cap has cortical EEG electrodes on the inside, which are used to collect EEG signals from the language function areas of the user's brain. An image acquisition device, wherein the image acquisition device is used to acquire the user's facial image; The processor is configured to determine the user's spoken language based on the electroencephalogram (EEG) signal and the facial image, and to control the broadcasting component to output the spoken language. The step of determining the user's target expressive language based on the electroencephalogram (EEG) signal and the facial image includes: The user's first facial movement is determined based on the electroencephalogram (EEG) signal, and the user's second facial movement is determined based on the facial image. The user's first expressive language is determined based on the first facial movement, and the user's second expressive language is determined based on the second facial movement; The user's target language is determined based on the similarity between the first and second expressions.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a control program for a voice output device, which, when executed by a processor, implements the steps of the control method for the voice output device according to any one of claims 1-6.

Images