A training corpus generation method and device, electronic equipment and storage medium

Abstract

The application discloses a training corpus generation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring an audio signal and an electroencephalogram signal of a target object, wherein the electroencephalogram signal is a synchronous electroencephalogram signal of the target object when the target object outputs the audio signal based on a training text; determining a candidate audio segment that meets a preset sound intensity condition based on a time-domain sound intensity envelope of the audio signal; inputting the audio signal into a syllable detection model for syllable detection to obtain a syllable detection result indicating syllable audio segments in the audio signal and corresponding preset syllables and confidence degrees; determining a target syllable audio segment in the candidate audio segment based on the syllable detection result; and generating a training corpus for training an electroencephalogram decoding model of the target object based on the target syllable audio segment, the corresponding preset syllable of the target syllable audio segment and a target electroencephalogram signal segment corresponding to the target syllable audio segment in the electroencephalogram signal. The application improves the construction efficiency of an electroencephalogram-syllable corpus and the accuracy of the training corpus.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for generating training corpora. Background Technology

[0002] Utilizing electroencephalogram (EEG) signals for language decoding is an important research direction in the field of brain-computer interfaces. Its goal is to reconstruct or identify the intended speech content by interpreting the user's brain neural activity. Among these efforts, constructing a high-quality EEG-syllable aligned corpus is fundamental and crucial for training EEG decoding models.

[0003] Because the voiceprint characteristics of different users and the corresponding EEG signals when they produce speech have significant individual specificities, it is usually necessary to build a unique, personalized EEG-syllable corpus for each brain-computer interface user or subject in order to achieve high-precision personal EEG decoding.

[0004] In related technologies, the construction of such personalized EEG-syllable corpora usually relies on a large amount of manual annotation, which reduces the efficiency and quality of EEG-syllable corpus construction. Summary of the Invention

[0005] To address the problems of existing technologies, this application provides a method, apparatus, electronic device, and storage medium for generating training corpora. The technical solution is as follows: On the one hand, a method for generating training corpus is provided, the method comprising: Acquire audio signals and EEG signals of the target object; the EEG signal is the synchronous EEG signal of the target object when it outputs the audio signal based on the training text. Based on the time-domain acoustic intensity envelope of the audio signal, candidate audio segments in the audio signal that satisfy the preset acoustic intensity conditions are determined. The audio signal is input to a syllable detection model for syllable detection to obtain syllable detection results; the syllable detection results indicate the syllable audio segments in the audio signal, as well as the preset syllables and confidence levels corresponding to the syllable audio segments; Based on the syllable detection results, the target syllable audio segment among the candidate audio segments is determined; Based on the preset syllables corresponding to the target syllable audio segment and the target EEG signal segment in the EEG signal corresponding to the target syllable audio segment, a training corpus is generated; the training corpus includes the correspondence between the target EEG signal segment and the corresponding preset syllable, and the training corpus is used to train the EEG decoding model of the target object. On the other hand, a training corpus generation apparatus is provided, the apparatus comprising: The signal acquisition unit is used to acquire the audio signal and EEG signal of the target object; the EEG signal is the synchronous EEG signal of the target object when it outputs the audio signal based on the training text. The candidate audio segment determination unit is used to determine candidate audio segments in the audio signal that meet preset sound intensity conditions based on the time-domain sound intensity envelope of the audio signal. The syllable detection unit is used to input the audio signal into the syllable detection model to perform syllable detection and obtain syllable detection results; the syllable detection results indicate the syllable audio segments in the audio signal and the preset syllables and confidence levels corresponding to the syllable audio segments; The target syllable audio segment determination unit is used to determine the target syllable audio segment among the candidate audio segments based on the syllable detection results; The training corpus generation unit is used to generate training corpus based on preset syllables corresponding to the target syllable audio segment and target EEG signal segments in the EEG signals that correspond to the target syllable audio segment; the training corpus includes the correspondence between the target EEG signal segments and the corresponding preset syllables, and the training corpus is used to train the EEG decoding model of the target object.

[0006] In some implementations, the target syllable audio segment determination unit includes: An audio overlap determination unit is used to determine the audio overlap between the syllable audio segment indicated by the syllable detection result and the candidate audio segment; The candidate syllable audio determination unit is used to select candidate syllable audio segments from the candidate audio segments whose audio overlap meets a preset overlap condition; The target syllable audio determination subunit is used to select a target syllable audio segment that meets a preset confidence threshold from the candidate syllable audio segments based on the confidence level of the preset syllable corresponding to the candidate syllable audio segment.

[0007] In some embodiments, the audio overlap determination unit includes: A time deviation determination unit is used to determine the deviation between the start time of the syllable audio segment and the start time of the candidate audio segment to obtain a first time deviation; and to determine the deviation between the end time of the syllable audio segment and the end time of the candidate audio segment to obtain a second time deviation. The overlap determination subunit is used to determine the audio overlap between the syllable audio segment and the candidate audio segment based on the first time deviation and the second time deviation.

[0008] In some embodiments, the candidate audio segment determination unit includes: The original sound intensity determination unit is used to determine the original sound intensity of the current audio frame based on the instantaneous energy of a first preset number of frequency points within the current audio frame for the current audio frame of the audio signal. The target sound intensity determination unit is used to average the original sound intensity of the current audio frame and the original sound intensity of a second preset number of audio frames preceding the current audio frame to obtain the target sound intensity of the current audio frame. The time-domain acoustic intensity envelope determination unit is used to obtain the time-domain acoustic intensity envelope of the audio signal based on the target sound intensity of each audio frame in the audio signal. The threshold detection unit is used to determine audio segments whose target sound intensity exceeds a preset sound intensity threshold and whose duration exceeds a preset duration, based on the time-domain sound intensity envelope, as candidate audio segments.

[0009] In some implementations, the preset sound intensity threshold is a preset multiple of the target statistical value, and the target statistical value is the statistical value of the target sound intensity within a previous preset time period.

[0010] In some embodiments, the syllable detection model includes an acoustic feature extraction network, a convolutional neural network, a time-delay neural network, and a long short-term memory network cascaded in sequence; the syllable detection unit is specifically used to: input the audio signal into the syllable detection model, and process the audio signal sequentially through the acoustic feature extraction network, the convolutional neural network, the time-delay neural network, and the long short-term memory network of the syllable detection model to obtain the output syllable detection result.

[0011] In some embodiments, the signal acquisition unit is specifically used to: acquire the original audio signal and the synchronized original EEG signal output by the target object based on the training text; perform noise reduction processing on the original audio signal to obtain the audio signal; and perform EEG preprocessing on the original EEG signal to obtain the EEG signal.

[0012] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the training corpus generation method of any of the above aspects.

[0013] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the training corpus generation method as described above.

[0014] On the other hand, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the training corpus generation method of any of the above aspects.

[0015] This application embodiment acquires the audio signal and synchronized EEG signal of the target object, determines candidate audio segments in the audio signal that meet preset conditions based on the temporal intensity envelope of the audio signal, and inputs the audio signal into a syllable detection model to obtain syllable detection results. The syllable detection results indicate the syllable audio segments in the audio signal, as well as the preset syllables and confidence levels corresponding to the syllable audio segments. Then, based on the syllable detection results, the target syllable audio segment in the candidate audio segments is determined. Based on the preset syllables corresponding to the target syllable audio segment and the target EEG signal segments in the EEG signal that correspond to the target syllable audio segment, training corpus for training the EEG decoding model of the target object is generated. The training corpus includes the correspondence between the target EEG signal segments and the corresponding preset syllables, thereby improving the construction efficiency of the EEG-syllable corpus and the accuracy of the training corpus, which is beneficial to improving the accuracy of the EEG decoding model of the target object trained based on the EEG-syllable corpus. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a training corpus generation method provided in an embodiment of this application; Figure 3 This is a schematic diagram of candidate audio segments determined based on the time-domain acoustic intensity envelope of audio signals, provided in an embodiment of this application. Figure 4 This is a schematic diagram of the structure of a syllable detection model provided in an embodiment of this application; Figure 5 This is an example of determining the target syllable audio segment among candidate audio segments based on syllable detection results, provided in an embodiment of this application. Figure 6 This is a schematic diagram of the audio signals before and after noise reduction processing provided in the embodiments of this application; Figure 7 This is an overall schematic diagram of the language decoding process based on electroencephalogram (EEG) signals provided in the embodiments of this application; Figure 8 This is a structural block diagram of a training corpus generation device provided in an embodiment of this application; Figure 9 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0020] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0021] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0022] Utilizing electroencephalogram (EEG) signals for language decoding is an important research direction in the field of brain-computer interfaces. Its goal is to reconstruct or identify the intended speech content by interpreting the user's brain neural activity. Among these efforts, constructing a high-quality EEG-syllable aligned corpus is fundamental and crucial for training EEG decoding models.

[0023] Because the voiceprint characteristics of different users and the corresponding EEG signals when they produce speech have significant individual specificities, it is usually necessary to build a unique, personalized EEG-syllable corpus for each brain-computer interface user or subject in order to achieve high-precision personal EEG decoding.

[0024] Currently, the standard process for constructing such personalized corpora typically relies on extensive manual annotation. Specifically, it requires collecting simultaneous EEG and audio signals from subjects while they are reading or pronouncing words. Subsequently, technicians must repeatedly listen to the audio, manually identifying and annotating the start and end times of each syllable in the continuous audio stream—that is, determining the time boundary of each syllable—and aligning the subject's real-time EEG signals with each syllable in the audio on the timeline to obtain a corpus corresponding to the EEG signals and syllables. However, this highly manual corpus construction method is not only inefficient, but also restricts the rapid and large-scale expansion of the corpus, as the performance of the decoding model is usually positively correlated with the amount of data in the training corpus. Furthermore, the subjective judgment of different annotators may introduce errors, affecting the consistency of the annotation results and thus reducing the quality of the EEG-syllable corpus.

[0025] In view of this, embodiments of this application provide a training corpus generation method. This method acquires the audio signal and synchronized EEG signal of the target object, determines candidate audio segments in the audio signal that meet preset sound intensity conditions based on the temporal intensity envelope of the audio signal, and inputs the audio signal into a syllable detection model to perform syllable detection and obtain syllable detection results. The syllable detection results indicate the syllable audio segments in the audio signal, as well as the preset syllables and confidence levels corresponding to the syllable audio segments. Then, based on the syllable detection results, the target syllable audio segment in the candidate audio segments is determined, and based on the preset syllables corresponding to the target syllable audio segment and the target EEG signal segments in the EEG signal corresponding to the target syllable audio segment, training corpus for training an EEG decoding model of the target object is generated. This improves the construction efficiency of the EEG-syllable corpus and the accuracy of the training corpus, improves the quality of the EEG-syllable corpus, facilitates the rapid expansion of the EEG-syllable corpus, and further improves the accuracy of the EEG decoding model of the target object trained based on the EEG-syllable corpus.

[0026] The technical solutions of the embodiments of this application will be described in detail below.

[0027] Please see Figure 1 The diagram shown is a schematic diagram of an implementation environment provided in this application embodiment. The implementation environment includes an EEG acquisition device 110, an audio acquisition device 120, and a computer device 130.

[0028] The EEG acquisition device 110 can acquire EEG signals, such as electrocorticography (ECoG) signals, from a user or subject using a non-invasive brain-computer interface. The audio acquisition device 120 can acquire the user's or subject's speech audio; for example, the audio acquisition device can be a microphone. The computer device 130 can connect and communicate with both the EEG acquisition device 110 and the audio acquisition device 120, enabling the audio acquisition device 120 to send real-time acquired speech audio to the computer device 130, and the real-time EEG signals acquired by the EEG acquisition device 110 to the computer device 130.

[0029] The computer device 130 can generate an EEG-syllable corpus for the user or subject based on the acquired audio signal and synchronized EEG signal using the training corpus generation method of this application embodiment. This EEG-syllable corpus can be used to train an EEG decoding model for the user or subject. In some embodiments, the computer device 130 can also train a corresponding EEG decoding model based on the user's or subject's EEG-syllable corpus, and use the EEG decoding model to decode the user's or subject's EEG signal to determine the intended speech content.

[0030] In this embodiment of the application, the computer device 130 may be a terminal and / or a server. It may be a single computer device or a system composed of multiple computer devices. When the computer device 130 includes multiple computer devices, the multiple computer devices can communicate with each other through wired or wireless network connections.

[0031] In a specific application scenario, computer device 130 may include a first computer device 131, a second computer device 132, and a third computer device 133. The first computer device 131 is used to generate training corpus for the target object to construct an EEG-syllable corpus for that target object. The second computer device 132 is used to train a model based on the target object's EEG-syllable corpus to obtain an EEG decoding model for the target object. It can also train other necessary models, such as articulation boundary models. The third computer device 133 is used to call the models trained by the second computer device 132, such as the EEG decoding model and the articulation boundary model, to decode the target object's EEG signals to determine the corresponding language content.

[0032] It should be noted that the terminals involved in the embodiments of this application may include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The servers involved in the embodiments of this application may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0033] Please see Figure 2 The diagram shown is a flowchart illustrating a training corpus generation method provided in an embodiment of this application. This method can be applied to... Figure 1 The computer device 130 is described. It should be noted that this specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include: S201, acquire the audio signal and EEG signal of the target object; the EEG signal is the synchronous EEG signal of the target object when it outputs the audio signal based on the training text.

[0034] The target subject can be any brain-computer interface subject or user. Audio signals can be recorded while the target subject reads training text, and the target subject's electroencephalogram (EEG) signals can be acquired simultaneously, thus obtaining the target subject's audio signal and corresponding EEG signal. Understandably, the target subject's audio signal and EEG signal are time-aligned.

[0035] The audio signal of the target object includes at least one syllable. A syllable is the smallest unit of speech that is produced by combining a single vowel phoneme and a consonant phoneme. A single vowel phoneme can also form a syllable on its own. In other words, a syllable includes at least one phoneme.

[0036] In practice, to improve the quality of the training corpus, the training texts need to meet at least one of the following requirements: (1) The training text includes all Chinese syllables, usually 402 to 418 syllables, so as to ensure that the training corpus can fully cover the syllables in Chinese, which is conducive to improving the completeness and stability of the EEG decoding model trained based on the training corpus.

[0037] (2) Each syllable appears at least 60 times in the training text to ensure that each syllable has sufficient training data and to avoid the problem of insufficient and unstable model learning in the later stage due to too few samples of a single syllable. For example, each syllable appears 60 to 120 times in the training text.

[0038] (3) The syllable correlation in the training text conforms to the distribution of modern Chinese, where the syllable correlation can be expressed by the conditional probability P( | ) representation, that is, when the previous syllable When it appears, the next syllable is The probability of this. Thus, when subjects read texts that conform to the rules of natural language, the neural processes of their speech planning and articulation are fluent and natural. This helps to produce purer and stronger EEG signals that are highly correlated with natural speech, thereby improving the signal-to-noise ratio and specificity of EEG signals, and thus improving the quality of training data.

[0039] In practical applications, articles or excerpts that meet at least one of the above requirements can be selected from public text libraries (such as xx Daily) as training texts for target audiences to read aloud.

[0040] S203, Based on the time-domain acoustic intensity envelope of the audio signal, determine the candidate audio segments in the audio signal that satisfy the preset acoustic intensity conditions.

[0041] The time-domain sound intensity envelope of the audio signal is the sound intensity change curve of the audio signal over time. Candidate audio segments in the audio signal that meet the preset sound intensity conditions can include one or more. Each candidate audio segment corresponds to a start time and an end time in the audio signal. Therefore, detecting one or more candidate audio segments through the time-domain sound intensity envelope of the audio signal yields the time boundaries of one or more candidate audio segments. It can be understood that the time boundaries of these one or more candidate audio segments include the time boundaries of syllables, i.e., the start and end times of the syllables in the audio signal. Therefore, this step can also be performed as follows: Figure 7 The diagram shows what is called syllable boundary detection, which may also include the time boundaries of non-syllable sounds such as coughs.

[0042] The preset sound intensity conditions may include a sound intensity exceeding a sound intensity threshold and a duration exceeding a time threshold. The sound intensity threshold and time threshold may be fixed preset values ​​or dynamically calculated changing values.

[0043] In some implementations, to improve the accuracy of audio time boundary detection, determining candidate audio segments in the audio signal that satisfy preset sound intensity conditions based on the time-domain sound intensity envelope of the audio signal may include: For the current audio frame of the audio signal, based on the instantaneous energy of the first preset number of frequency points within the current audio frame, determine the original sound intensity of the current audio frame; Average the original sound intensity of the current audio frame and the original sound intensities of the second preset number of audio frames before the current audio frame to obtain the target sound intensity of the current audio frame; Based on the target sound intensities of the respective audio frames in the audio signal, obtain the time-domain sound intensity envelope of the audio signal; Based on the time-domain sound intensity envelope, determine an audio segment whose target sound intensity exceeds a preset sound intensity threshold and whose duration exceeds a preset duration as a candidate audio segment.

[0044] Specifically, the audio signal can be divided into multiple audio frames of the same length, and then each audio frame is traversed in sequence. For the current audio frame traversed, based on the instantaneous energy of the first preset number of frequency points within the current audio frame, determine the original sound intensity of the current audio frame, and then average it in combination with the original sound intensities of the second preset number of audio frames before the current audio frame. Take the obtained average sound intensity as the target sound intensity of the current audio frame. Furthermore, based on the target sound intensities of each audio frame, obtain the time-domain sound intensity envelope of the audio signal. After that, the time-domain sound intensity envelope can be scanned. Whenever it rises and exceeds the preset sound intensity threshold and the duration exceeds the preset duration (such as 100 ms), mark the audio segment between the start time and the end time (usually when it falls back below the preset sound intensity threshold) as a candidate audio segment. Among them, both the first preset number and the second preset number can be set based on actual experience.

[0045] Such as Figure 3 As shown, it is a schematic diagram of the candidate audio segment determined based on the time-domain sound intensity envelope of the audio signal. Among them, the horizontal axis represents time (unit: second), the vertical axis represents the sound intensity, which is a dimensionless value (after being normalized), the audio signal 31 is a segment of speech corresponding to the training text "You're welcome", the curve 32 represents the time-domain sound intensity envelope of the audio signal 31, and the intervals 33 to 35 respectively represent the detected candidate audio segments; y represents the sound intensity threshold.

[0046] In practical applications, the target sound intensity of an audio frame can be calculated through the following formula:

[0047] Among them, represents the signal of an audio frame; represents the number of frequency points, that is, the first preset number; represents the instantaneous energy of the frequency point, which can also be called the instantaneous power; The reference power can be set based on the acoustic hearing threshold, for example, it can be... ; -1 indicates the second preset number of audio frames preceding this audio frame; Indicates the first The target sound intensity of the audio frame; in, This represents the original sound intensity of the audio frame.

[0048] The above implementation determines the original sound intensity of the current audio frame and filters out rapid, meaningless energy fluctuations (such as consonant plosives and slight inhalation sounds) by averaging M frames. This makes the temporal sound intensity envelope of the obtained audio signal more clearly show the stable intensity of the syllable. Furthermore, by combining a preset sound intensity threshold and a preset duration, one or more stable and meaningful candidate audio segments can be accurately and reliably detected to obtain accurate boundary information.

[0049] In some possible implementations, considering that different people have different voice intensities, and the same person's voice intensity may also change at different times and in different environments, in order to improve the accuracy and stability of candidate audio segment detection, a preset sound intensity threshold is a preset multiple of the target statistical value, which is the statistical value of the target sound intensity within a previous preset time period.

[0050] For example, the target statistical value can be the standard deviation of the target sound intensity within a previous preset time period. The length of this preset time period can be set based on practical experience, such as 10 seconds. The preset multiplier can also be set based on practical experience, such as between 1.2 and 2.0 times, preferably 1.5, thus the preset sound intensity threshold is 1.5 times the standard deviation of the target sound intensity within the previous 10 seconds.

[0051] In specific implementation, during the scanning of the temporal sound intensity envelope, when the current scanning duration is less than the length of a preset time period (e.g., 10 seconds), candidate audio segments can be detected based on the default preset sound intensity threshold. When the current scanning duration is greater than or equal to the length of the preset time period (e.g., 10 seconds), the preset sound intensity threshold is dynamically calculated based on 1.5 times the standard deviation of the target sound intensity within the previous preset time period (e.g., 10 seconds).

[0052] In other possible implementations, in order to improve the detection accuracy of syllable time boundaries by ensuring that the detected candidate audio segments are as close as possible to the audio segments corresponding to syllables, the following formula can be used when determining the candidate audio segments in the audio signal that meet the preset sound intensity conditions based on the time-domain sound intensity envelope of the audio signal:

[0053] in, Indicates duration (unit: seconds); This represents the average sound intensity of the current audio segment, which is the average value of the target sound intensity within the current audio segment; The standard deviation of background noise intensity; This represents the mean value of the background noise intensity. Represents a minimal constant, such as 1e 8. Numerical stability terms to prevent the denominator from being zero; This represents the time threshold (unit: seconds), which can be set based on practical experience, such as 0.3 seconds.

[0054] Specifically, based on the temporal sound intensity envelope, audio segments whose target sound intensity exceeds a fixed sound intensity threshold (which can be set based on practical experience) are detected. The duration T of the audio segment is obtained based on its start and end times. If the duration T is less than 0.1 seconds, the audio segment is determined to be a non-candidate audio segment; conversely, if the duration T is greater than or equal to 0.1 seconds, the average sound intensity I is calculated based on the target sound intensity within the audio segment. This, in turn, determines the degree to which the average sound intensity I deviates from the noise baseline, i.e., calculates... And multiply this by T. If the product is greater than the time threshold... If the audio segment is selected as a candidate audio segment, then the product result is determined to be no greater than the time threshold. Then the audio segment is determined to be a non-candidate audio segment.

[0055] This implementation can effectively distinguish speech from transient noise, adapt to the volume differences of different speakers, and maintain stable detection performance in different noise environments, thus providing reliable speech segments for the determination of syllable boundaries.

[0056] S205, the audio signal is input to the syllable detection model for syllable detection to obtain the syllable detection result; the syllable detection result indicates the syllable audio segment in the audio signal and the preset syllable and confidence level corresponding to the syllable audio segment.

[0057] Here, a syllable audio segment refers to the time interval corresponding to a syllable in an audio signal. The syllable detection model is a pre-trained neural network model that can perform syllable detection on the input audio signal and output a syllable detection result. This result indicates the syllable audio segment in the audio signal, along with the corresponding preset syllable and a confidence level. The confidence level represents the probability that the syllable audio segment belongs to the preset syllable. For example, the syllable detection result can be represented as {( , ),syl,z}, where, ( , ) represents the start and end times of the syllable audio segment, syl represents the preset syllable, and z represents the confidence level.

[0058] Based on this, embodiments of this application may further include a step of training the syllable detection model, specifically including: Obtain the sample audio signal and the corresponding label. The label indicates the reference syllable audio segment in the sample audio signal, the reference preset syllable corresponding to the reference syllable audio segment, and the probability of being the reference preset syllable. It can be understood that the probability of being the reference preset syllable is 0 (i.e. not the reference preset syllable) or 1 (i.e. being the reference preset syllable). The sample audio signal is input into the syllable detection model to be trained to detect syllables and obtain the predicted syllable detection result; the predicted syllable detection result indicates the predicted syllable audio segment in the sample audio signal, as well as the predicted preset syllable and confidence level corresponding to the predicted syllable audio segment; Based on the difference between the predicted syllable detection result and the labeled label corresponding to the sample audio signal, the model parameters of the syllable detection model to be trained are adjusted, and the training continues iteratively until the training termination condition is met, thus obtaining the trained syllable detection model.

[0059] Among them, the reference preset syllable is any one of multiple preset syllables, which can cover all Chinese syllables, such as 402 to 418 syllables in Chinese.

[0060] The adjustment of the model parameters of the syllable detection model to be trained, based on the difference between the predicted syllable detection result and the labeled label corresponding to the sample audio signal, can be achieved by: using a preset loss function, determining the loss function value based on the difference between the predicted syllable detection result and the labeled label, and then using this loss function value to adjust the model parameters of the syllable detection model to be trained. For example, the preset loss function can be the cross-entropy loss function. The model parameters of the syllable detection model to be trained can be adjusted in the direction of reducing the loss function value. For instance, when training the syllable detection model, a backpropagation algorithm (such as stochastic gradient descent (SGD)) can be used to adjust the model parameters in the direction of reducing the loss function value. The training termination condition can be that the number of iterations reaches an iteration threshold (which can be set as needed), the loss function value is less than the loss function threshold (which can be set as needed), or the difference between two adjacent loss function values ​​reaches a difference threshold (which can be set as needed).

[0061] In some possible implementations, the syllable detection model may include an acoustic feature extraction network, a convolutional neural network, a time-delay neural network, and a long short-term memory network cascaded in sequence; the step of inputting the audio signal into the syllable detection model for syllable detection and obtaining the syllable detection result includes: inputting the audio signal into the syllable detection model, and processing the audio signal sequentially through the acoustic feature extraction network, the convolutional neural network, the time-delay neural network, and the long short-term memory network of the syllable detection model to obtain the output syllable detection result.

[0062] like Figure 4 The diagram shows a schematic of a syllable detection model provided in an embodiment of this application. Specifically, when using this syllable detection model for syllable detection, the acoustic feature extraction network of the syllable detection model can be called to extract acoustic features from the audio signal to obtain acoustic features, which may include Mel-frequency cepstral coefficients, Mel-frequency cepstral, etc.; the convolutional neural network of the syllable detection model is called to perform convolution processing on the acoustic features to obtain a local audio feature sequence; the time-delay neural network of the syllable detection model is called to perform deep nonlinear feature transformation and fusion on the local audio feature sequence to obtain global deep audio features; and the long short-term memory network of the syllable detection model is called to perform fusion processing on the global deep audio features to obtain the final output syllable detection result.

[0063] The convolutional neural network can employ multiple layers, such as 6 layers, with its pooling layers using non-linear pooling layers with shared weights. The time-delay neural network can also employ multiple layers, such as 8 layers. The long short-term memory network can be a stacked two-layer long short-term memory network. Using the above syllable detection models can improve the accuracy of syllable detection results, which is beneficial for improving the quality of the training corpus.

[0064] S207, Based on the syllable detection results, determine the target syllable audio segment among the candidate audio segments.

[0065] In this embodiment, the syllable detection result of step S205 is used to correct the candidate audio segments of step S203, thereby determining the target syllable audio segment. The target syllable audio segment is the syllable audio segment among the candidate audio segments that meets the correction condition. The correction condition may be that the audio overlap between the syllable audio segment of the syllable detection result and the candidate audio segment meets the audio overlap threshold. The audio overlap represents the degree of overlap between two audio segments. For example, the audio overlap can be represented by the degree of overlap between two audio segments in the time dimension. The audio overlap threshold can be set based on practical experience.

[0066] In some possible implementations, in order to further improve the accuracy of the target syllable audio segment and thus improve the accuracy of the syllable boundary, step S207 may include the following steps (1) to (3) when determining the target syllable audio segment in the candidate audio segments based on the syllable detection results: (1) Determine the audio overlap between the syllable audio segment indicated by the syllable detection result and the candidate audio segment.

[0067] (2) Select candidate syllable audio segments from the candidate audio segments whose audio overlap meets the preset overlap condition.

[0068] The preset overlap condition indicates overlap, and this preset overlap condition is consistent with the representation method of audio overlap. The candidate syllable audio segment is the audio segment of the syllable selected from the candidate audio segments, and as a candidate, it can be further filtered through subsequent steps (3). The audio overlap between the syllable audio segment and the candidate audio segment meets the preset overlap condition, indicating that the syllable audio segment and the candidate audio segment point to the same syllable.

[0069] For example, when determining the audio overlap between the syllable audio segment indicated by the syllable detection result and the candidate audio segment, the deviation between the start time of the syllable audio segment and the start time of the candidate audio segment can be determined to obtain a first time deviation; and the deviation between the end time of the syllable audio segment and the end time of the candidate audio segment can be determined to obtain a second time deviation; and then, based on the first time deviation and the second time deviation, the audio overlap between the syllable audio segment and the candidate audio segment can be determined.

[0070] Specifically, if the first time deviation does not exceed a preset time deviation threshold and the second time deviation does not exceed the preset time deviation threshold, the audio overlap between the syllable audio segment and the candidate audio segment can be determined to be coincident, meaning that the syllable audio segment and the candidate audio segment point to the same syllable audio, meeting the preset overlap condition (i.e., coincident). In this case, the candidate audio segment is determined as the candidate syllable audio segment. Conversely, if the first time deviation exceeds the preset time deviation threshold and / or the second time deviation exceeds the preset time deviation threshold, the audio overlap between the syllable audio segment and the candidate audio segment can be determined to be non-coincident, thus failing the preset overlap condition, and the candidate audio segment is not considered as the target syllable audio segment. The preset time deviation threshold can be set based on practical experience, for example, as follows: 100 milliseconds. The above method determines the audio overlap between the syllable audio segment and the candidate audio segment by using the first and second time deviations. This can accurately locate the candidate syllable audio segment, thereby improving the accuracy of the target syllable audio segment, which in turn improves the accuracy of the syllable boundary.

[0071] (3) Based on the confidence level of the preset syllable corresponding to the candidate syllable audio segment, select the target syllable audio segment that meets the preset confidence threshold from the candidate syllable audio segments.

[0072] Specifically, the preset syllables corresponding to the candidate syllable audio segments are the preset syllables in the syllable detection results corresponding to syllable audio segments whose audio overlap meets the preset overlap condition. This allows for the acquisition of corresponding confidence levels. Based on the confidence levels of the preset syllables corresponding to each candidate syllable audio segment, target syllable audio segments that meet a preset confidence threshold are selected from the candidate syllable audio segments. These target syllable audio segments can be candidate syllable audio segments whose corresponding confidence levels exceed the preset confidence threshold. The preset confidence threshold can be set based on practical experience, for example, it can be set to 0.9.

[0073] like Figure 5 The image shows an example of determining a target syllable audio segment from candidate audio segments based on syllable detection results. The interval between lines 51-1 and 51-2 defines one candidate audio segment, the interval between lines 52-1 and 52-2 defines another candidate audio segment, and curves 53, 54, and 55 represent three syllable audio segments detected by the syllable detection results. Figure 5 As can be seen, the audio overlap between the candidate audio segments defined by lines 51-1 and 51-2 and the syllable audio segments corresponding to curve segment 53 meets the preset overlap condition. Therefore, the candidate audio segments defined by lines 51-1 and 51-2 can be used as candidate syllable audio segments. If the confidence level of the preset syllable corresponding to curve segment 53 exceeds the preset confidence threshold, the candidate audio segments defined by lines 51-1 and 51-2 will be confirmed as target syllable audio segments. However, the audio overlap between the candidate audio segments defined by lines 52-1 and 52-2 and the syllable audio segments corresponding to curve segment 54 or curve segment 55 does not meet the preset overlap condition. Therefore, the candidate audio segments defined by lines 52-1 and 52-2 will not be used as candidate syllable audio segments and will be discarded.

[0074] S209, Based on the preset syllables corresponding to the target syllable audio segment and the target EEG signal segment in the EEG signal corresponding to the target syllable audio segment, a training corpus is generated; the training corpus includes the correspondence between the target EEG signal segment and the corresponding preset syllable, and the training corpus is used to train the EEG decoding model of the target object.

[0075] Specifically, the time boundary (start time and end time) of the target syllable audio segment can be obtained. Based on this time boundary, a segment of EEG signal that is time-aligned with it can be determined from the EEG signal as the target EEG signal segment corresponding to the target syllable audio segment. Then, training data in the EEG-syllable corpus of the target object can be generated based on the preset syllable corresponding to the target syllable and the target EEG signal segment.

[0076] Specifically, the training corpus can include the correspondence between target EEG signal segments and corresponding preset syllables, where the preset syllables serve as label information for the target EEG signal segment. In practice, the target EEG signal segment can be extracted from the EEG signal, and label information for that segment can be generated based on its corresponding preset syllables to obtain the first training corpus. For example, the generated label information could be (syl, 1), where syl represents the preset syllable, and "1" indicates that the target EEG signal segment is a speaking EEG signal. In practical applications, a second training corpus can also be generated based on the remaining EEG signal segments after extraction. This second training corpus includes the remaining EEG signal segments and label information indicating that the remaining EEG signal segments are non-speaking EEG signals, such as "0". Both the first and second training corpora are stored in the target object's EEG-syllable corpus, thereby improving the completeness of the training corpus in the target object's EEG-syllable corpus.

[0077] In other examples, EEG signals may not be segmented during training corpus generation. In this case, after determining the target EEG signal segment aligned with the target syllable audio segment in the synchronized EEG signal based on the time boundary (start time and end time) of the target syllable audio segment, label information for the corresponding target EEG signal segment is generated based on the preset syllable corresponding to the target syllable audio segment. For example, the label information can be (S,1), where S represents the preset syllable and "1" indicates that the target EEG signal segment is an EEG signal in a speaking state. The label information is marked on the target EEG signal segment of the EEG signal. Label information indicating that the EEG signal segment is an EEG signal in a non-speaking state, such as "0", can also be marked on other EEG signal segments of the EEG signal signal other than the target EEG signal segment. Thus, EEG signals labeled with the above label information are obtained, and the EEG signals labeled with the above label information are used as training corpus in the EEG-syllable corpus. It should be noted that, in the embodiments of this application, there are no specific restrictions on the presentation of the correspondence between the target EEG signal segments and preset syllables in the training corpus. One or more presentation formats can be adopted based on the actual model training needs.

[0078] It should be noted that the EEG signals with labeled information in the EEG-syllable corpus of the target object in this application embodiment can also be referred to as sample EEG signals.

[0079] In some possible implementations, in order to further improve the quality of the training corpus, the aforementioned step S201 of obtaining the audio signal and EEG signal of the target object may include: acquiring the original audio signal and the synchronized original EEG signal output by the target object based on the training text; performing noise reduction processing on the original audio signal to obtain the audio signal; and performing EEG preprocessing on the original EEG signal to obtain the EEG signal.

[0080] Specifically, noise reduction processing of the original audio signal can remove low-frequency signals. For example, a high-pass filter can be used to filter out low-frequency signals below 100Hz, so that subsequent steps S203 and S205 are based on the noise-reduced audio signal. Figure 6 The diagram shows the audio signals before and after noise reduction, where (a) represents the audio signal before noise reduction and (b) represents the audio signal after noise reduction. Noise reduction filters out low-frequency background noise, ensuring that subsequent steps S203 and S205 are based on the noise-reduced audio signal, which significantly improves the accuracy of determining the time boundaries of subsequent syllables.

[0081] EEG preprocessing of the raw EEG signal may include the following operations performed in sequence: (1) Removal of bad channels: Delete electrode data with excessive or insufficient impedance. For example, if the channel impedance is greater than 1M (no connection) or less than 1k (short circuit), the entire channel data is excluded; (2) Power frequency filtering: Remove power frequency interference. Use notch filtering to remove 50 Hz (or 60 Hz) and its harmonic components to suppress the pollution of the neural signal spectrum by power supply noise. In practical applications, if the system is battery powered and there is no obvious power frequency interference, this step is optional; (3) Common mode re-reference: For each time point, use the average (or median) of all channel signals as the reference signal and subtract it from each channel to suppress spatially shared noise components and enhance the distinguishability of local neural activities; (4) Bandpass filtering: Only retain the frequency bands related to speech (such as ECoG). (2) High-frequency γ band), to remove low-frequency drift and high-frequency noise, and focus on neural activities with speech meaning, for example, retaining the frequency band range related to speech as 70~170Hz; (5) Hilbert transform: apply Hilbert transform to the bandpass filtered signal to obtain the analytical signal, and calculate its instantaneous envelope to characterize the change of neural activity intensity over time, thereby obtaining the EEG signal. By performing the above EEG preprocessing on the original EEG signal, the quality of the EEG signal in the training corpus can be significantly improved, thereby improving the quality of the training corpus.

[0082] In this embodiment, the training corpus of the target object is stored in the target object's EEG-syllable corpus, thereby improving the construction efficiency and quality of the target object's EEG-syllable corpus. The training corpus in the EEG-syllable corpus can be used to train the target object's EEG decoding model. The EEG decoding model can be used for the EEG signal processing of the target object. The EEG decoding model can decode the syllable probability distribution corresponding to the EEG signal to be processed of the target object, or the EEG decoding result. The syllable probability distribution indicates the probability that the EEG signal to be processed belongs to each preset syllable. The probability of each preset syllable can also be called the EEG decoding score or confidence level.

[0083] The following is about Figure 7 The technical solution shown is explained below. Figure 7 This is an overall schematic diagram of the language decoding process based on electroencephalogram (EEG) signals provided in an embodiment of this application.

[0084] like Figure 7 The process includes constructing a brainwave-syllable corpus for the target object, a model training phase, and a subsequent real-time decoding and application phase based on the target object's brainwave signals. The construction of the target object's brainwave-syllable corpus can be found in steps S201 to S209 of the embodiments of this application, and will not be repeated here.

[0085] The following describes the process of training the articulation boundary model and EEG decoding model of the target object using the target object's EEG-syllable corpus.

[0086] The training corpus of the target object's EEG-syllable corpus includes sample EEG signals of the target object and label information corresponding to the sample EEG signals. The label information may include a first label, which indicates the target syllable corresponding to the corresponding sample EEG signal among multiple preset syllables. In some embodiments, training an EEG decoding model based on the target object's EEG-syllable corpus may include the following steps: (1) obtaining a training corpus set, wherein the training corpus set includes sample EEG signals of the target object and label information corresponding to the sample EEG signals. The label information includes a first label, which indicates the target syllable corresponding to the sample EEG signal among multiple preset syllables; (2) training a first neural network model for an EEG decoding task based on the sample EEG signals and the corresponding first label to obtain the target object's EEG decoding model; wherein the EEG decoding task is to predict the probability that the corresponding sample EEG signal belongs to each of the preset syllables.

[0087] Specifically, multiple training corpora can be obtained from the target object's EEG-syllable corpus to obtain a training corpus set. For each training corpus, the sample EEG signals are input into the first neural network model for EEG decoding processing to obtain the predicted EEG decoding result corresponding to the sample EEG signal. The predicted EEG decoding result includes the probability that the sample EEG signal belongs to each preset syllable. Based on the difference between the predicted EEG decoding result corresponding to each sample EEG signal and the first label, the model parameters of the first neural network model are adjusted, and the training continues iteratively based on the adjusted model parameters until the training termination condition is met, thus obtaining the trained EEG decoding model.

[0088] For example, adjusting the model parameters of the first neural network model based on the difference between the predicted EEG decoding result and the first label corresponding to each sample EEG signal can be achieved by: using a preset loss function and calculating the EEG decoding loss based on the difference between the predicted EEG decoding result and the first label corresponding to each sample EEG signal, and then using this EEG decoding loss to adjust the model parameters of the first neural network model. The preset loss function may include a cross-entropy loss function. When adjusting the model parameters of the first neural network model using this EEG decoding loss, the model parameters can be adjusted in the direction of reducing the EEG decoding loss for training. For example, when training the first neural network model for an EEG decoding task, a backpropagation algorithm (such as stochastic gradient descent (SGD)) can be used to adjust the model parameters in the direction of reducing the EEG decoding loss. The training termination condition can be that the number of iterations reaches an iteration threshold (which can be set as needed), the EEG decoding loss is less than a loss threshold (which can be set as needed), or the difference between two adjacent EEG decoding losses reaches a difference threshold (which can be set as needed). The first neural network model can be a recurrent neural network, such as a Long Short-Term Memory (LSTM) network.

[0089] The above implementation method uses training data from the target object's EEG-syllable corpus to train the EEG decoding model, thereby achieving personalized modeling for the target object. This can adapt to the differences in neurophysiology among different individuals and improve the accuracy and stability of the EEG decoding model.

[0090] In some implementations, the label information corresponding to the sample EEG signal further includes a second label, which indicates whether the sample EEG signal is an EEG signal in a speaking state. For example, the second label can use "0" to indicate an EEG signal in a non-speaking state and "1" to indicate an EEG signal in a speaking state. Then, training the articulation boundary model based on the EEG-syllable corpus of the target object can include: training a second neural network model on an effective EEG recognition task based on the sample EEG signal and the corresponding second label to obtain the articulation boundary model of the target object; the effective EEG recognition task is to predict whether the corresponding sample EEG information is an EEG signal in a speaking state.

[0091] Specifically, for each training corpus, the sample EEG signals are input into the second neural network model for effective EEG recognition processing to obtain the predicted effective EEG recognition result corresponding to the sample EEG signal. The predicted effective EEG recognition result includes the probability that the sample EEG signal is an EEG signal in a speaking state. Based on the difference between the predicted effective EEG recognition result and the second label corresponding to each sample EEG signal, the model parameters of the second neural network model are adjusted, and the training continues iteratively based on the adjusted model parameters until the training termination condition is met, thus obtaining the trained articulation boundary model.

[0092] For example, adjusting the model parameters of the second neural network model based on the difference between the predicted effective EEG recognition results and the second label corresponding to each sample EEG signal can be achieved by: using a preset loss function and calculating the recognition loss based on the difference between the predicted effective EEG recognition results and the second label corresponding to each sample EEG signal, and then using this recognition loss to adjust the model parameters of the second neural network model. The preset loss function may include a cross-entropy loss function. When adjusting the model parameters of the second neural network model using this recognition loss, the model parameters can be adjusted in the direction of reducing the recognition loss for training. For example, when training the second neural network model for effective EEG recognition, a backpropagation algorithm (such as stochastic gradient descent (SGD)) can be used to adjust the model parameters in the direction of reducing the recognition loss. The training termination condition can be that the number of iterations reaches an iteration threshold (which can be set as needed), the recognition loss is less than a loss threshold (which can be set as needed), or the difference between two adjacent recognition losses reaches a difference threshold (which can be set as needed). The second neural network model can be a deep neural network based on linear discriminant analysis.

[0093] The above implementation method uses training corpus from the target object's EEG-syllable corpus to train the articulation boundary model, thereby achieving personalized modeling for the target object. It can adapt to the differences in articulation habits of different individuals and improve the accuracy and stability of the articulation boundary model.

[0094] In some possible implementations, in order to improve the generalization ability of the articulation boundary model and the EEG decoding model under different noise and individual differences, after acquiring the training corpus, the method may further include: performing data augmentation processing on the sample EEG signals in the training corpus; the data augmentation processing includes one or more of the following combinations: adding random noise, stretching or compressing in the time dimension, perturbing in the frequency dimension, and randomly occluding some features.

[0095] Specifically, adding random noise to the sample EEG signals can simulate unavoidable physiological artifacts (such as slight electromyography and electrocardiography) and environmental noise during the acquisition process. For example, Gaussian white noise can be added.

[0096] The sample EEG signal can be stretched or compressed in the time dimension to perform nonlinear deformation in the time dimension, simulating the natural differences in speech rate among different individuals and the fluctuation of pronunciation duration of the same object in different states. For example, the time axis of the sample EEG signal can be stretched or compressed uniformly within a predetermined scaling range (such as [0.9, 1.1]). Of course, different degrees of stretching and compression can also be applied to different local time periods of the sample EEG signal to simulate the non-uniform changes in syllable duration in speech flow.

[0097] Perturbation of the sample EEG signal in the frequency dimension can simulate the changes in frequency response characteristics caused by individual physiological differences, slight changes in electrode impedance, or fluctuations in brain state. For example, the spectrum of the sample EEG signal can be slightly shifted along the frequency axis, or its amplitude can be slightly and randomly scaled.

[0098] Randomly occluding certain features of the sampled EEG signals can enhance the model's robustness to partial information loss or attentional distraction, simulating transient signal loss or attention shifts that may occur in real-world applications. Examples include temporal random occlusion and channel random occlusion. Temporal random occlusion involves randomly selecting one or more consecutive time segments on the time axis and setting the values ​​of all channels or features within that time segment to zero or the average value of that channel. Channel random occlusion involves randomly selecting a portion of the acquisition channels (e.g., randomly blocking 10%-30% of the electrode channels) and setting their data to zero at all time steps to simulate scenarios of poor electrode contact or partial channel failure.

[0099] It should be noted that the label information corresponding to the enhanced sample EEG signal obtained after data augmentation is consistent with the label information of the sample EEG signal before data augmentation.

[0100] For example, to maximize the data augmentation effect, random noise can be added to a sample EEG signal sequentially, stretching or compressing it in the time dimension, perturbing it in the frequency dimension, and randomly occluding some features.

[0101] The above implementation methods can simulate the diversity and uncertainty of EEG signals in real-world scenarios, allowing the model to "see" richer and more complex EEG signals during training. This forces the model to learn deep features that remain stable even under noise, temporal distortion, frequency perturbations, and partial information loss. Consequently, the trained articulation boundary model can more reliably identify valid speech state EEG segments under various interferences, while the EEG decoding model can decode the enhanced, more diverse, and valid EEG signals. This significantly improves the generalization ability and overall decoding accuracy in complex environments during actual deployment.

[0102] The following describes the process of using the articulation boundary model and EEG decoding model of the target object obtained from the above training to process the EEG signal of the target object to obtain the EEG decoding result. The EEG signal processing process may include the following steps (1) to (3): Step (1): Obtain the EEG signal to be processed from the target subject.

[0103] In some implementations, to improve the accuracy of language content decoding based on EEG signals, the process of acquiring the EEG signal to be processed of the target object may include: acquiring the original EEG signal of the target object; performing EEG preprocessing on the original EEG signal to obtain the EEG signal to be processed of the target object; the EEG preprocessing includes extracting the EEG time-domain envelope of a preset speech-related frequency band.

[0104] Specifically, raw EEG signals can be information on changes in brain neuron activity over time that can be directly acquired using a non-invasive brain-computer interface.

[0105] The preset speech-related frequency band refers to the range of frequencies in the original EEG signal that are related to speech. In this embodiment, the preset speech-related frequency band is 70~170Hz to remove low-frequency drift and high-frequency noise, focusing on neural activity with speech significance. The EEG time-domain envelope of the preset speech-related frequency band is used to characterize the change in the intensity of neural activity within this frequency band over time. During EEG preprocessing, Hilbert transform can be used to extract the EEG time-domain envelope of the preset speech-related frequency band.

[0106] In some examples, the preset speech association frequency band can be set according to the individual neural response characteristics of the target object, so that the EEG preprocessing can be adapted to the neurophysiological differences of different people, thereby improving the individual adaptability of the EEG signal to be processed, which is conducive to improving the accuracy of the final decoded language content.

[0107] In specific implementation, the EEG preprocessing of the raw EEG signal may include the following operations performed sequentially: (a) Removal of bad channels: Deleting electrode data with excessively high or low impedance. For example, if the channel impedance is greater than 1MΩ (no connection) or less than 1kΩ (short circuit), the entire channel data is excluded; (b) Power frequency filtering: Removing power frequency interference, using a notch filter to remove 50 Hz (or 60 Hz) interference. (c) Common-mode re-reference: For each time point, the average (or median) of all channel signals is used as the reference signal and subtracted from each channel to suppress spatially shared noise components and enhance the discriminability of local neural activity; (d) Bandpass filtering: Only the frequency band range related to speech, such as 70~170Hz, is retained to remove low-frequency drift and high-frequency noise and focus on neural activity with speech significance; (e) Hilbert transform: The Hilbert transform is applied to the bandpass filtered signal to obtain the analytic signal, and its instantaneous envelope is calculated to obtain the EEG signal to be processed to characterize the change of neural activity intensity over time.

[0108] By selecting the speech-related frequency band and extracting the EEG temporal envelope, the neural activity representations most relevant to the language temporal structure are initially extracted from the raw EEG signals, thereby effectively improving the efficiency of subsequent processing steps and the accuracy of the final language content decoding.

[0109] Step (2): Call the articulation boundary model of the target object to perform effective EEG recognition processing on the EEG signal to be processed, and obtain effective EEG recognition results.

[0110] The effective EEG recognition result indicates the effective EEG signals in the EEG signals to be processed, and the effective EEG signals are EEG signals in the speaking state. For example, the effective EEG recognition result can be represented as ( , ),in, The start time of an effective EEG signal. The end time of an effective EEG signal.

[0111] Step (3): Call the EEG decoding model of the target object, perform EEG decoding processing based on the effective EEG recognition results, and obtain the EEG decoding results.

[0112] The EEG decoding result indicates the probability that the valid EEG signal belongs to each of the multiple preset syllables. Specifically, the EEG decoding result may include multiple preset syllables and the EEG decoding score of each preset syllable. The EEG decoding score indicates the probability of the corresponding preset syllable, which can also be referred to as the confidence level.

[0113] In some implementations, when calling the EEG decoding model of the target object and performing EEG decoding processing based on the effective EEG recognition results to obtain the EEG decoding results, the process may include: segmenting the EEG signal to be processed based on the effective EEG recognition results to obtain the effective EEG signal; normalizing the effective EEG signal to obtain effective EEG features; and inputting the effective EEG features into the EEG decoding model of the target object for EEG decoding processing to obtain the EEG decoding results.

[0114] Specifically, it can be based on the ( ) in the valid EEG recognition results , The EEG signal to be processed is segmented to extract the starting time. The end time is A segment of brainwave signal was used to obtain an effective brainwave signal.

[0115] The normalization process can be a standardization process performed on each channel or feature dimension, such as Z-score standardization, to make the mean of the data zero and the variance one, so as to obtain effective EEG features. Then, the effective EEG features are input into the EEG decoding model of the target object for EEG decoding processing to obtain the EEG decoding result.

[0116] By utilizing the articulation boundary model of the target object, background neural activity, resting-state signals, and most artifact interference unrelated to language decoding are effectively stripped away. This provides high signal-to-noise ratio and highly correlated effective EEG signals for subsequent EEG decoding steps. This allows the EEG decoding model to perform EEG decoding on effective EEG signals, enabling it to learn more focused neural representations and spatiotemporal patterns directly related to language production. This significantly reduces computational load and the possibility of confusion, thereby effectively improving the accuracy and reliability of syllable probability determination. Consequently, it greatly improves the accuracy and reliability of language content based on EEG signal decoding.

[0117] The following describes how to perform language decoding based on the EEG decoding results to obtain the language content to be output, which may include the following steps (1) to (5): Step (1): Obtain the EEG decoding results at the current time step.

[0118] The EEG decoding result is obtained by decoding the EEG signal of the target object at the current time step. The EEG decoding result includes multiple preset syllables and the corresponding EEG decoding score for each preset syllable. It should be noted that the EEG decoding score corresponding to each preset syllable in the EEG decoding result can also be called the decoding confidence score or confidence level, which represents the probability that the EEG signal to be processed belongs to the corresponding preset syllable.

[0119] Step (2): For each of the preset syllables, expand the historical candidate sentences of the previous time step based on the candidate characters associated with the preset syllable, to obtain an expanded sentence set corresponding to the preset syllable.

[0120] In the embodiments of the present application, a syllable-candidate character mapping relationship library can be pre-constructed or loaded. This mapping relationship library is constructed based on a general Chinese dictionary or a dictionary of a specific field, covering all syllables and common Chinese characters required for the language decoding task in the embodiments of the present application. For example, it includes 401 to 418 syllables and multiple candidate Chinese characters corresponding to each syllable. For example, the candidate characters associated with the syllable "yi" include: "一", "以", "意", "义", "已", and the candidate characters associated with the syllable "shi" can include: "是", "时", "事", "市", "实".

[0121] Then, the candidate characters respectively associated with each preset syllable in the brain decoding result can be determined based on this syllable-candidate character mapping relationship library. It can be understood that the candidate characters associated with each preset syllable can be one or more.

[0122] Among them, the historical candidate sentences of the previous time step refer to the set of candidate sentences screened at the previous moment (such as the t-1 moment) in the language decoding process. Usually, it is an ordered sequence, and each historical candidate sentence has its comprehensive language decoding score in the language decoding process at the t-1 time step, hereinafter referred to as the historical comprehensive language decoding score, and the language content output by the language decoding at the t-1 time step is the candidate sentence selected from this set of candidate sentences. In a specific implementation, a candidate sentence sequence can be maintained for the language decoding process. The candidate sentences in this candidate sentence sequence are updated as the language decoding process progresses. That is, when decoding at the t time step, the candidate sentence sequence stores the candidate sentences screened at the t-1 time step. When the language decoding at the t time step ends and enters the language decoding at the t + 1 time step, the candidate sentence sequence is updated to the candidate sentences screened at the t time step.

[0123] When performing the sentence expansion operation, assume that the historical candidate sentence of the previous time step is represented as , then for a preset syllable syl_k decoded at the current time step t, all candidate characters corresponding to the preset syllable syl_k can be queried according to the syllable-candidate character mapping relationship library. For example, it is denoted as the set ; then for each historical candidate sentence , and each candidate character , perform a string concatenation operation to concatenate the candidate character at the end of the historical candidate sentence, generating an expanded sentence . Traverse all the historical candidate sentences in and All candidate characters in the set will generate all the extended sentences that constitute the set of extended sentences corresponding to the preset syllable syl_k.

[0124] Understandably, the number of extended sentences in the extended sentence set corresponding to a preset syllable is equal to the product of the number of historical candidate sentences and the number of candidate characters corresponding to the preset syllable. Therefore, in order to save computing resources and improve the speed of language decoding, in some examples, for each preset syllable, only the most commonly used first few (such as the first 3 or the first 5) candidate characters can be selected to participate in the extension.

[0125] By performing the above expansion process on each preset syllable in the EEG decoding results, an expanded sentence set corresponding to each preset syllable can be obtained. The expanded sentence set of all preset syllables constitutes the expanded sentence space of the current time step.

[0126] Step (3): For each extended sentence corresponding to the preset syllable, the comprehensive language decoding score of the extended sentence is determined based on the historical comprehensive language decoding score of the previous time step, the first linguistic probability score corresponding to the extended sentence, and the EEG decoding score corresponding to the preset syllable.

[0127] The first linguistic probability score represents the co-occurrence probability of the historical candidate sentence and candidate characters used in the expansion. Specifically, the first linguistic probability score can be represented by the conditional probability of the corresponding candidate character appearing subsequently, given the historical candidate sentence used in the expansion. .

[0128] In some examples, the first linguistic probability score of an expanded sentence can be obtained based on a statistical linguistic model such as an N-gram model. Specifically, when it is necessary to calculate the candidate words... Add to history candidate sentences When calculating the first linguistic probability score of the extended sentence at the end, the N-gram model can extract a string of a certain length from the end of the historical candidate sentence as contextual information for the current prediction, and query the N-gram model for candidate words that appear given that context. The conditional probability is used to obtain the first linguistic probability score of the expanded sentence. In other examples, the first linguistic probability score of the expanded sentence can also be achieved through a linguistic probability model built using deep learning. This model can predict the probability that the next character is a corresponding candidate character based on the input historical candidate sentences. The higher the probability, the more the expansion conforms to human language rules. The linguistic probability score of the corresponding expanded sentence can be obtained based on the probability output by this linguistic probability model. The linguistic probability model built using deep learning can include recurrent neural networks, long short-term memory networks, etc.

[0129] Specifically, each expanded sentence obtained based on the aforementioned steps has the following correspondence: Expanded Sentence - Preset Syllable - EEG Decoding Score - Historical Candidate Sentence - Historical Comprehensive Language Decoding Score - Candidate Character. The EEG decoding score in this correspondence is the EEG decoding score of the preset syllable corresponding to the candidate character at the end of the corresponding expanded sentence in the EEG decoding results. Therefore, after determining the first linguistic probability score corresponding to each expanded sentence, for each expanded sentence, its corresponding comprehensive language decoding score can be calculated based on its first linguistic probability score, EEG decoding score, and historical comprehensive language decoding score.

[0130] In some examples, when calculating the comprehensive language decoding score of an expanded sentence, the historical comprehensive language decoding score used can be the comprehensive language decoding score calculated in the previous time step of the historical candidate sentence upon which the expanded sentence is based. In other examples, when calculating the comprehensive language decoding score of an expanded sentence, the historical comprehensive language decoding score used can also be the comprehensive language decoding score corresponding to the candidate sentence output in the previous time step. In this case, the historical comprehensive language decoding score used when calculating the comprehensive language decoding score for all expanded sentences in the current time step is the same value, which is beneficial to improving language decoding speed. Understandably, when the current time step is at the beginning of a sentence, the historical comprehensive language decoding score is recorded as 0.

[0131] In practice, the comprehensive language decoding score of an extended sentence can be the sum of its corresponding first linguistic probability score, EEG decoding score, and historical comprehensive language decoding score.

[0132] In some implementations, determining the comprehensive language decoding score of the extended sentence based on the historical comprehensive language decoding score of the previous time step, the first linguistic probability score corresponding to the extended sentence, and the EEG decoding score corresponding to the preset syllable may include: obtaining a first weighting coefficient, a second weighting coefficient, and a third weighting coefficient; and performing a weighted summation of the historical comprehensive language decoding score of the previous time step, the first linguistic probability score corresponding to the extended sentence, and the EEG decoding score corresponding to the preset syllable based on the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient to obtain the comprehensive language decoding score of the extended sentence.

[0133] The first weighting coefficient corresponds to the historical comprehensive language decoding score item, the second weighting coefficient corresponds to the linguistic probability score item, and the third weighting coefficient corresponds to the EEG decoding score item, thus expanding the comprehensive language decoding score of the sentence. It can be expressed as the following formula (1): (1) in, This represents the first weighting coefficient. This represents the second weighting coefficient. This represents the third weighting coefficient; This indicates the historical comprehensive language decoding score. This represents the first linguistic probability score. express.

[0134] In some examples, the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient can be fixed values ​​set based on experience.

[0135] In some implementations, obtaining the first weight coefficient, the second weight coefficient, and the third weight coefficient may include: determining the first weight coefficient, the second weight coefficient, and the third weight coefficient based on the current total number of characters in the expanded sentence; wherein the first weight coefficient and the second weight coefficient are both positively correlated with the current total number of characters, and the third weight coefficient is negatively correlated with the current total number of characters.

[0136] Specifically, the first, second, and third weighting coefficients can dynamically change based on the total number of characters in the expanded sentence. The first and second weighting coefficients increase with the total number of characters, meaning that the longer the sentence, the greater the contribution of historical decisions and linguistic patterns to the comprehensive language decoding score at the current time step. For example, the first and second weighting coefficients increase with the total number of characters and approach 1. The third weighting coefficient decreases with the total number of characters, meaning that the longer the sentence, the smaller the contribution of the EEG decoding result to the comprehensive language decoding score at the current time step. For example, the third weighting coefficient decreases with the total number of characters and approaches 0.5.

[0137] In specific implementation, the comprehensive language decoding score of the extended sentence under this implementation method It can be expressed as the following formula (2): (2) in, Indicates the total number of characters in the expanded sentence.

[0138] In the above implementation, by introducing a dynamic weight coefficient that adapts to sentence length, the language decoding process can intelligently and dynamically adjust the contribution of different information sources according to different stages of the decoding process. This is more in line with the laws of cognition and signal processing, improves the accuracy of language decoding, and significantly enhances the stability of long sentence decoding.

[0139] Step (4): Based on the comprehensive language decoding score of the extended sentences corresponding to each preset syllable, determine the candidate sentence sequence for the current time step.

[0140] Specifically, for K preset syllables, each preset syllable corresponds to a set of extended sentences, and each extended sentence can be used to calculate its corresponding comprehensive language decoding score. Merging the extended sentence sets of all preset syllables forms a global candidate pool containing M extended sentences. It can be understood that as the number of decoded syllables increases, if the sentence does not end, the number of obtained extended sentences grows in a tree-like manner. Therefore, in order to reduce computational complexity and improve the efficiency and accuracy of language decoding while ensuring the quality of language decoding, it is necessary to select a controllable and optimal subset of sentences from all possible extended sentence sets generated at the current time step as the "candidate sentence sequence for the current time step," which is used for the final output decision of the current time step and the decoding iteration of the next time step.

[0141] In some implementations, a predetermined number of sentences with the highest comprehensive language decoding scores in the global candidate pool can be selected as the candidate sentence sequence for the current time step. The predetermined number can be set based on practical experience; for example, it can be the top N extended sentences sorted by their comprehensive language decoding scores from highest to lowest as the candidate sentence sequence for the current time step, where N can be any number between 20 and 30.

[0142] Step (5): Perform sentence end detection based on the candidate sentence sequence at the current time step, and determine the language content to be output at the current time step based on the detection result of the sentence end detection.

[0143] Specifically, sentence end detection mainly determines whether a complete sentence has been formed before the current time step, and then outputs the language content based on the result of sentence end detection.

[0144] In some implementations, sentence end detection based on the candidate sentence sequence at the current time step may include: a) For each candidate sentence at the current time step, determine the sentence completeness score corresponding to the candidate sentence based on the historical comprehensive language decoding score and the second linguistic probability score of the previous time step.

[0145] The second linguistic probability score represents the linguistic probability score of the historical candidate sentence corresponding to the candidate sentence as a complete sentence. In specific implementation, the second linguistic probability score can be represented by the conditional probability of the occurrence of an end marker (such as E) indicating the end of a sentence, given a historical candidate sentence. The calculation method of the second linguistic probability score can refer to the aforementioned calculation method of the first linguistic probability score, and will not be repeated here.

[0146] Specifically, the historical comprehensive language decoding score and the second linguistic probability score of the previous time step can be summed to obtain the sentence completeness score of the candidate sentence. The evaluation is "regardless of what you want to say now, does the part that has just been said itself resemble a complete sentence?"

[0147] In practice, the sentence completeness score corresponding to the candidate sentence can be calculated using the following formula (3): (3) in, This indicates the score for sentence completeness; This indicates the historical comprehensive language decoding score of the previous time step; The second linguistic probability score represents the linguistic probability score of the corresponding historical candidate sentence as a complete sentence at the current time step without adding any new words. and This refers to the aforementioned first and second weighting coefficients. This is understandable. The higher the score, the more complete the sentence.

[0148] b) Based on the third linguistic probability score and the EEG decoding score of the corresponding preset syllable, determine the language decoding score when the preset syllable is the beginning of a sentence.

[0149] The third linguistic probability score represents the linguistic probability score of the candidate character corresponding to the preset syllable of the candidate sentence as the beginning of the sentence at the current time step. In specific implementation, the third linguistic probability score can be represented by the conditional probability of a sentence-initial identifier (such as B) appearing before the corresponding candidate character. The calculation method of the third linguistic probability score can refer to the aforementioned calculation method of the first linguistic probability score, and will not be repeated here.

[0150] Specifically, the third linguistic probability score and the corresponding EEG decoding score of the preset syllable can be summed to obtain the language decoding score when the preset syllable is used as the beginning of a sentence. The evaluation is "How good would it be if we started to say a new word now, starting with this preset syllable?"

[0151] In specific implementation, the language decoding score when the preset syllable is the beginning of a sentence can be calculated using the following formula (4). : (4) in, The third linguistic probability score represents the linguistic probability score of the candidate character corresponding to the preset syllable of the candidate sentence as the beginning of the sentence at the current time step; This represents the EEG decoding score of the preset syllable in the EEG decoding results at the current time step; This refers to the aforementioned second weighting coefficient. This represents the aforementioned third weighting coefficient.

[0152] c) Determine the detection score corresponding to the candidate sentence based on the sentence completeness score of the candidate sentence and the language decoding score when the corresponding preset syllable is the beginning of the sentence.

[0153] Specifically, the sentence completeness score of the candidate sentence and the language decoding score when the corresponding preset syllable is the beginning of the sentence can be summed to obtain the detection score corresponding to the candidate sentence. This score represents the comprehensive score of the event that "the historical candidate sentence ends here and the corresponding preset syllable is the beginning of a new sentence".

[0154] In specific implementation, the detection score corresponding to the candidate sentence can be calculated using the following formula (5). : (5) d) Based on the detection score and comprehensive language decoding score corresponding to each candidate sentence, determine the detection result of sentence end detection.

[0155] For example, determining the sentence end detection result based on the detection score and comprehensive language decoding score corresponding to each candidate sentence may include: determining the highest score based on the detection score and comprehensive language decoding score corresponding to each candidate sentence; if the highest score is any of the detection scores, then the sentence end detection result is determined to be a completed sentence; if the highest score is any of the comprehensive language decoding scores, then the sentence end detection result is determined to be an incomplete sentence. In a specific implementation, after obtaining the detection scores of each candidate sentence, all detection scores and the comprehensive language decoding scores of all candidate sentences in the candidate sentence sequence are sorted in descending order. If a detection score is ranked first, then the sentence end detection result is determined to be a completed sentence, that is, the historical candidate sentence in the candidate sentence corresponding to the detection score ranked first is determined to be the previous sentence that has ended, and the preset syllable corresponding to the candidate sentence is determined to be the start of the next sentence; conversely, if the first one is a comprehensive language decoding score, then the sentence end detection result is determined to be an incomplete sentence.

[0156] The above implementation does not rely on a single threshold or fixed rules, but is based on the relative confidence of all predicted hypotheses, which improves the stability of sentence end detection. Moreover, whether a sentence ends depends closely on the integrity of the historical sentence itself and the possibility of the current syllable as a new beginning, which fully considers the context and participates in the global sorting of the current time step, thus improving the accuracy of sentence end detection.

[0157] In some examples, to save computational resources and reduce unnecessary computational consumption to improve the efficiency of language decoding, the process may include, before step b), sorting the sentence completeness scores and comprehensive language decoding scores of each candidate sentence in descending order. If a predetermined number of the top-ranked sentences (e.g., the top 20) contain one or more sentence completeness scores, then steps b) to d) are executed for the candidate sentences corresponding to those one or more sentence completeness scores. This ensures that only high-quality potentially complete sentences can participate in subsequent detection steps, reducing unnecessary computation, saving computational resources, and thus improving the efficiency and accuracy of language decoding.

[0158] In some implementations, determining the output language content of the current time step based on the sentence end detection result may include: if the sentence end detection result indicates that the sentence is incomplete, then the candidate sentence corresponding to the highest score is taken as the output language content of the current time step; if the sentence end detection result indicates that the sentence is complete, then the output language content of the current time step is determined based on the preset syllable corresponding to the highest score as the beginning of a new sentence, and the candidate sentence sequence of the current time step is updated based on the beginning tag.

[0159] Specifically, if the sentence end detection result is that the sentence is incomplete, then the candidate sentence with the highest comprehensive language decoding score can be used as the language content to be output at the current time step and can be output.

[0160] If the sentence end detection result indicates that the sentence is complete, then the language content to be output at the current time step is determined based on the language decoding scores when each of the preset syllables is the beginning of the sentence, and the candidate sentence sequence at the current time step is updated. Specifically, this can be based on the multiple calculated values ​​mentioned above. Sort in descending order and then select the first one. The corresponding sentence is taken as the language content to be output at the current time step, and the number of sentences in the candidate sentence sequence is taken into account. The corresponding sentence is updated in the candidate sentence sequence obtained at the current time step in the aforementioned steps.

[0161] By constructing language decoding as a sequence search process based on dynamic programming of multi-source information, the language decoding process not only depends on the credibility of the current neural signal, but is also continuously constrained and guided by the rules of language structure and semantic boundaries. As a result, the final language decoding result is more in line with human language expression and understanding habits, which greatly improves the accuracy, comprehensibility (intelligibility), and naturalness of the output language content in real communication scenarios, making the language output process based on EEG signals closer to the natural language communication process.

[0162] This application also provides a training corpus generation device. Since the training corpus generation device provided in this application corresponds to the training corpus generation methods provided in the above-mentioned embodiments, the implementation methods of the aforementioned training corpus generation methods are also applicable to the training corpus generation device provided in this embodiment, and will not be described in detail in this embodiment.

[0163] Please see Figure 8 The diagram shows a structural schematic of a training corpus generation device provided in an embodiment of this application. This device has the function of implementing the training corpus generation method described in the above-described method embodiments. This function can be implemented by hardware or by hardware executing corresponding software. Figure 8 As shown, the training corpus generation device 800 may include: The signal acquisition unit 810 is used to acquire the audio signal and EEG signal of the target object; the EEG signal is the synchronous EEG signal when the target object outputs the audio signal based on the training text. The candidate audio segment determination unit 820 is used to determine candidate audio segments in the audio signal that meet preset sound intensity conditions based on the time-domain sound intensity envelope of the audio signal. The syllable detection unit 830 is used to input the audio signal into the syllable detection model to perform syllable detection and obtain syllable detection results; the syllable detection results indicate the syllable audio segments in the audio signal and the preset syllables and confidence levels corresponding to the syllable audio segments; The target syllable audio segment determination unit 840 is used to determine the target syllable audio segment among the candidate audio segments based on the syllable detection result; The training corpus generation unit 850 is used to generate training corpus based on the preset syllables corresponding to the target syllable audio segment and the target EEG signal segment in the EEG signal corresponding to the target syllable audio segment; the training corpus includes the correspondence between the target EEG signal segment and the corresponding preset syllable, and the training corpus is used to train the EEG decoding model of the target object.

[0164] In some embodiments, the target syllable audio segment determination unit 840 includes: An audio overlap determination unit is used to determine the audio overlap between the syllable audio segment indicated by the syllable detection result and the candidate audio segment; The candidate syllable audio determination unit is used to select candidate syllable audio segments from the candidate audio segments whose audio overlap meets a preset overlap condition; The target syllable audio determination subunit is used to select a target syllable audio segment that meets a preset confidence threshold from the candidate syllable audio segments based on the confidence level of the preset syllable corresponding to the candidate syllable audio segment.

[0165] In some embodiments, the audio overlap determination unit includes: A time deviation determination unit is used to determine the deviation between the start time of the syllable audio segment and the start time of the candidate audio segment to obtain a first time deviation; and to determine the deviation between the end time of the syllable audio segment and the end time of the candidate audio segment to obtain a second time deviation. The overlap determination subunit is used to determine the audio overlap between the syllable audio segment and the candidate audio segment based on the first time deviation and the second time deviation.

[0166] In some embodiments, the candidate audio segment determination unit 820 includes: The original sound intensity determination unit is used to determine the original sound intensity of the current audio frame based on the instantaneous energy of a first preset number of frequency points within the current audio frame for the current audio frame of the audio signal. The target sound intensity determination unit is used to average the original sound intensity of the current audio frame and the original sound intensity of a second preset number of audio frames preceding the current audio frame to obtain the target sound intensity of the current audio frame. The time-domain acoustic intensity envelope determination unit is used to obtain the time-domain acoustic intensity envelope of the audio signal based on the target sound intensity of each audio frame in the audio signal. The threshold detection unit is used to determine audio segments whose target sound intensity exceeds a preset sound intensity threshold and whose duration exceeds a preset duration, based on the time-domain sound intensity envelope, as candidate audio segments.

[0167] In some implementations, the preset sound intensity threshold is a preset multiple of the target statistical value, and the target statistical value is the statistical value of the target sound intensity within a previous preset time period.

[0168] In some embodiments, the syllable detection model includes an acoustic feature extraction network, a convolutional neural network, a time-delay neural network, and a long short-term memory network cascaded in sequence; the syllable detection unit 830 is specifically used to: input the audio signal into the syllable detection model, and process the audio signal sequentially through the acoustic feature extraction network, the convolutional neural network, the time-delay neural network, and the long short-term memory network of the syllable detection model to obtain the output syllable detection result.

[0169] In some embodiments, the signal acquisition unit 810 is specifically used to: acquire the original audio signal and the synchronized original EEG signal output by the target object based on the training text; perform noise reduction processing on the original audio signal to obtain the audio signal; and perform EEG preprocessing on the original EEG signal to obtain the EEG signal.

[0170] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0171] This application provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement any of the training corpus generation methods provided in the above method embodiments.

[0172] Furthermore, Figure 9 A schematic diagram of the hardware structure of an electronic device for implementing a training corpus generation method provided in an embodiment of this application is shown. The electronic device may participate in or include the training corpus generation apparatus provided in an embodiment of this application. Figure 9 As shown, the electronic device 90 may include one or more (shown as 902a, 902b, ..., 902n) processors 902 (processors 902 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 904 for storing data, and a transmission device 906 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 9The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device 90 may also include... Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.

[0173] It should be noted that the aforementioned one or more processors 902 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element within the electronic device 90 (or mobile device). As involved in the embodiments of this application, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0174] The memory 904 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method described in the embodiments of this application. The processor 902 executes various functional applications and data processing by running the software programs and modules stored in the memory 904, thereby realizing the above-described training corpus generation method. The memory 904 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 904 may further include memory remotely located relative to the processor 902, and these remote memories can be connected to the electronic device 90 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0175] The transmission device 906 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 90. In one example, the transmission device 906 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In one embodiment, the transmission device 906 may be a radio frequency (RF) module for wireless communication with the Internet.

[0176] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows users to interact with the user interface of an electronic device (or mobile device).

[0177] This application also provides a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a training corpus generation method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any of the training corpus generation methods provided in the above method embodiments.

[0178] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. An electronic device's processor reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform any of the training corpus generation methods provided in the above-described method embodiments.

[0179] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0180] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0181] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0182] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0183] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating training corpus, characterized in that, The method includes: Acquire audio signals and EEG signals of the target object; the EEG signal is the synchronous EEG signal of the target object when it outputs the audio signal based on the training text. Based on the time-domain acoustic intensity envelope of the audio signal, candidate audio segments in the audio signal that satisfy the preset acoustic intensity conditions are determined. The audio signal is input to a syllable detection model for syllable detection to obtain syllable detection results; the syllable detection results indicate the syllable audio segments in the audio signal, as well as the preset syllables and confidence levels corresponding to the syllable audio segments; Based on the syllable detection results, the target syllable audio segment among the candidate audio segments is determined; Based on the preset syllables corresponding to the target syllable audio segment and the target EEG signal segment in the EEG signal corresponding to the target syllable audio segment, a training corpus is generated; the training corpus includes the correspondence between the target EEG signal segment and the corresponding preset syllable, and the training corpus is used to train the EEG decoding model of the target object.

2. The method according to claim 1, characterized in that, The step of determining the target syllable audio segment among the candidate audio segments based on the syllable detection results includes: Determine the audio overlap between the syllable audio segment indicated by the syllable detection result and the candidate audio segment; From the candidate audio segments, select the candidate syllable audio segments whose audio overlap meets the preset overlap condition; Based on the confidence level of the preset syllables corresponding to the candidate syllable audio segments, target syllable audio segments that meet the preset confidence threshold are selected from the candidate syllable audio segments.

3. The method according to claim 2, characterized in that, The audio overlap between the audio segment indicated by the syllable detection result and the candidate audio segment includes: The deviation between the start time of the syllable audio segment and the start time of the candidate audio segment is determined to obtain the first time deviation; The deviation between the end time of the syllable audio segment and the end time of the candidate audio segment is determined to obtain the second time deviation; Based on the first time deviation and the second time deviation, the audio overlap between the syllable audio segment and the candidate audio segment is determined.

4. The method according to claim 1, characterized in that, The step of determining candidate audio segments in the audio signal that satisfy preset sound intensity conditions based on the time-domain sound intensity envelope of the audio signal includes: For the current audio frame of the audio signal, the original sound intensity of the current audio frame is determined based on the instantaneous energy of a first preset number of frequency points within the current audio frame. The target sound intensity of the current audio frame is obtained by averaging the original sound intensity of the current audio frame and the original sound intensity of the second preset number of audio frames preceding the current audio frame. Based on the target sound intensity of each audio frame in the audio signal, the time-domain sound intensity envelope of the audio signal is obtained; Based on the time-domain sound intensity envelope, audio segments whose target sound intensity exceeds a preset sound intensity threshold and whose duration exceeds a preset duration are identified as candidate audio segments.

5. The method according to claim 4, characterized in that, The preset sound intensity threshold is a preset multiple of the target statistical value, and the target statistical value is the statistical value of the target sound intensity within a previous preset time period.

6. The method according to claim 1, characterized in that, The syllable detection model comprises a cascaded acoustic feature extraction network, a convolutional neural network, a time-delay neural network, and a long short-term memory network; the step of inputting the audio signal into the syllable detection model for syllable detection and obtaining the syllable detection result includes: The audio signal is input into the syllable detection model, and the audio signal is processed sequentially by the acoustic feature extraction network, the convolutional neural network, the time delay neural network and the long short-term memory network of the syllable detection model to obtain the output syllable detection result.

7. The method according to any one of claims 1 to 6, characterized in that, The acquisition of the target object's audio and EEG signals includes: Obtain the raw audio signal and synchronized raw EEG signal of the target object based on the training text; The original audio signal is subjected to noise reduction processing to obtain the audio signal; The original EEG signal is subjected to EEG preprocessing to obtain the EEG signal.

8. A training corpus generation device, characterized in that, The device includes: The signal acquisition unit is used to acquire the audio signal and EEG signal of the target object; the EEG signal is the synchronous EEG signal of the target object when it outputs the audio signal based on the training text. The candidate audio segment determination unit is used to determine candidate audio segments in the audio signal that meet preset sound intensity conditions based on the time-domain sound intensity envelope of the audio signal. The syllable detection unit is used to input the audio signal into the syllable detection model to perform syllable detection and obtain syllable detection results; the syllable detection results indicate the syllable audio segments in the audio signal and the preset syllables and confidence levels corresponding to the syllable audio segments; The target syllable audio segment determination unit is used to determine the target syllable audio segment among the candidate audio segments based on the syllable detection results; The training corpus generation unit is used to generate training corpus based on preset syllables corresponding to the target syllable audio segment and target EEG signal segments in the EEG signals that correspond to the target syllable audio segment; the training corpus includes the correspondence between the target EEG signal segments and the corresponding preset syllables, and the training corpus is used to train the EEG decoding model of the target object.

9. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the training corpus generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the training corpus generation method as described in any one of claims 1 to 7.

Images