Semantic analysis method and system for EEG information of autonomous driving safety officer

Abstract

The present invention provides a method and system for semantic analysis of EEG information of an autonomous driving safety officer, which belongs to the field of autonomous driving technology. The method collects driving behavior data and EEG data, performs time-frequency domain analysis to obtain EEG time-frequency feature maps; constructs domain tensors of the driver and the safety officer, decomposes them into core tensors and common adjoint matrices, inputs prediction models LSTM, and trains to obtain source prediction functions; migrates the source prediction functions to the target domain to obtain target domain pseudo-label matrices, and constructs new target domains; uses the source prediction functions at the end of the migration iteration to parse the semantics of the safety officer's EEG information; extracts the core tensors at the end of the iteration, and analyzes to obtain the differences in cognitive states between the driver and the safety officer. The present invention can accurately characterize the differences in cognitive states of users in both manual driving and autonomous driving systems, effectively performs knowledge migration to realize driving semantic analysis of safety officers, and helps to optimize human-vehicle collaboration for the development of autonomous driving, and provides a new way for the construction of brain-computer interaction for safety officers.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method and system for semantic analysis of electroencephalogram information of an autonomous driving safety officer based on tensor transfer learning. Background Art

[0002] The brain is the highest level part of the nervous system. It senses the stimulation of the external environment and induces corresponding EEG signals to control the human body to react, forming a perception-cognition-behavior chain. Driving behavior is a complex behavior of human perception feedback. Under the coupling of different driving conditions, different EEG signals are generated to make different driving decision behaviors. Therefore, the correlation between the driver's EEG cognitive characteristics and driving behavior in different driving situations is analyzed through EEG data and driving behavior data. The driving behavior prediction method based on brain cognition is studied to explore the brain-like driving decision-making process. The analysis of the semantics of the driver's EEG information is of great significance to driving behavior and brain-like research.

[0003] However, with the rapid development of autonomous driving technology, intelligent vehicles can already achieve L3 level autonomous driving, that is, conditional autonomous driving. In this case, the driver only needs to sit behind the steering wheel as a safety officer to monitor the driving environment for human-machine co-driving. Since the safety officer does not need to operate the vehicle and cannot obtain driving behavior data, it is difficult to analyze the complex endogenous semantics of the safety officer's EEG cognitive characteristics and the associated coupling of driving reaction behaviors through feedback, and there is a barrier between the human brain and the "machine brain". Therefore, studying the cognitive state of the safety officer, interpreting the driving behavior mapped by it, and exploring the brain-computer interaction mode of the safety officer's driving behavior under autonomous driving conditions has important theoretical and practical significance for the development of human-vehicle collaborative driving.

[0004] The semantic analysis of the safety officer can be achieved by analyzing the commonalities and differences in the cognitive states of the safety officer in the same driving situation in the two driving systems, extracting effective knowledge, and conducting a transfer study of the endogenous semantics in the manual driving state. Existing studies either collect data from drivers in manual driving or data from safety officers in automatic driving, and lack research on the correlation between the two. In the few comparative experiments, due to differences in driving conditions and spatial positions, the driving situations faced by users in the two systems are not exactly the same, making it difficult to characterize the commonalities and differences of people in the two systems. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for semantic analysis of EEG information of an autonomous driving safety officer based on tensor transfer learning, so as to solve at least one technical problem existing in the above-mentioned background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] On the one hand, the present invention provides a method for semantic analysis of EEG information of an autonomous driving safety officer, comprising:

[0008] A comparative experiment was conducted using a driving simulator and VR equipment to simultaneously collect driving behavior data and EEG data;

[0009] Short-time Fourier transform is used to analyze EEG data in time and frequency domain to obtain EEG time-frequency feature map;

[0010] Construct the domain tensors of the driver and the safety officer, namely the source domain and the target domain;

[0011] The source domain tensor and the target domain tensor are decomposed into core tensors and common adjoint matrices by joint tensor decomposition. The adjoint matrices are concatenated into vectors and input into the prediction model LSTM to train the source prediction function. The source prediction function is transferred to the target domain to obtain the target domain pseudo-label matrix and construct a new target domain tensor.

[0012] Perform a new round of joint tensor decomposition on the new target domain and the source domain, update the target domain tensor, and calculate the MMD distance between the updated target domain and the source domain in each iteration, iterating until the MMD distance is minimum and the companion matrix remains unchanged;

[0013] The source prediction function at the end of the migration iteration parses the semantics of the safety officer's EEG information; the core tensor at the end of the iteration is extracted to analyze the difference in cognitive states between the driver and the safety officer.

[0014] Furthermore, the EEG time-frequency analysis based on short-time Fourier transform can obtain the EEG time-frequency feature map, and its calculation formula is:

[0015]

[0016] Where ω(t) is the window function, x(t) is the signal to be transformed, and X(t,f) is the Fourier transform result of w(t,f)x(τ). As t changes, the window function moves on the time axis and only the position that overlaps with the position highlighted by the window function in time, namely w(t-τ)x(τ), is Fourier transformed.

[0017] Furthermore, for the source domain X s and the target domain X t Joint tensor decomposition can align the source domain and target domain space. The calculation formula is:

[0018]

[0019] In the formula, is the core tensor showing the degree of interaction between different components; U1, U2 and U3 are the left singular matrices of the expanded tensor along the corresponding order.

[0020] Furthermore, the LSTM model uses the one-dimensional vector concatenated by U1, U2 and U3 as input to read the information h at the previous moment. t-1 and the memory information h at the current moment t , and output the input gate, output gate and forget gate information i t , o t and f t , and then obtain the state C of the hidden layer unit at time t t Finally, the source target prediction function is obtained through training. The formulas are:

[0021] i t =σ i [W i ·(h t-1 ,x t )+b i ]

[0022] o t =σ o [W o ·(h t-1 ,x t )+b o ]

[0023] f t =σ f (W f ·(h t-1 ,x t )+b f ]

[0024] In the formula, σ i , σ o and σ f are the activation functions selected for the input gate, output gate, and forget gate respectively; W i , W o and W f are the weights selected for the input gate, output gate, and forget gate respectively; b i 、b o and b f Biases chosen for the input gate, output gate, and forget gate, respectively.

[0025] At time t, the state C of the hidden layer unit t The specific calculation formula is as follows:

[0026]

[0027] Where, ⊙ is the Hadamard product; C t-1 is the state of the hidden layer unit at time t-1; C t Alternative value of W c and bc Calculate The weights and biases used when ; tanh is the hyperbolic tangent function.

[0028] Furthermore, the MMD distance can measure the distribution difference between the source domain and the target domain during the iteration process to obtain the source target prediction function that can be used for migration. To achieve the safety officer driving semantic analysis, the calculation formula is:

[0029]

[0030] Where n s and n t Represent the number of data samples in the source domain and target domain respectively; and Represent the number of data samples belonging to the cth category in the source domain and the target domain respectively; U1, U2 and U3 represent the adjoint matrices of the tensor at the 1st, 2nd and 3rd orders respectively; To regenerate the kernel Hilbert space, the feature space of the original sample needs to be mapped to the regenerated kernel Hilbert space for calculation.

[0031] Furthermore, the core tensor extracted at the end of the iteration and Can reflect the source domain X s and the target domain X t The relationship between the three dimensions and analysis of the cognitive differences between safety officers and drivers.

[0032] In a second aspect, the present invention provides a semantic analysis system for EEG information of an autonomous driving safety officer, comprising:

[0033] The acquisition module is used to complete comparative experiments using driving simulators and VR devices, and to synchronously collect driving behavior data and EEG data;

[0034] An analysis module, used to perform time-frequency domain analysis on EEG data using short-time Fourier transform to obtain EEG time-frequency feature graph;

[0035] The construction module is used to construct the domain tensors of the driver and the safety officer, namely the source domain and the target domain;

[0036] The training iteration module is used to decompose the source domain and the target domain into core tensors and a common adjoint matrix by using joint tensor decomposition; the adjoint matrix is ​​concatenated into a vector and input into the prediction model LSTM, and the source prediction function is obtained by training; the source prediction function is transferred to the target domain to obtain the target domain pseudo-label matrix, and a new target domain is constructed; the steps of constructing a new target domain are repeated for the new target domain and the source domain, and the MMD distance between the new target domain and the source domain is calculated in each iteration process, and it is iterated until the MMD distance is minimized and the adjoint matrix remains unchanged;

[0037] The migration analysis module is used to parse the semantics of the safety officer's EEG information by using the source prediction function when the migration iteration terminates; extract the core tensor at the moment of iteration termination, and analyze the difference in cognitive states between the driver and the safety officer.

[0038] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the method for semantic analysis of EEG information of an autonomous driving safety officer as described above is implemented.

[0039] In a fourth aspect, the present invention provides a computer program product, including a computer program, which, when running on one or more processors, is used to implement the method for semantic analysis of EEG information of an autonomous driving safety officer as described above.

[0040] In a fifth aspect, the present invention provides an electronic device comprising: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the method for semantic analysis of EEG information of an autonomous driving safety officer as described above.

[0041] The beneficial effects of the present invention are as follows: it can collect and compare the cognitive states of drivers and safety officers in manual and automatic driving systems in the same situation, effectively analyze the commonalities and differences between the two, realize the user's "perception-cognition-behavior" domain knowledge transfer in the two systems, and enable EEG to be applied to the interpretation of the safety officer's cognitive state; and construct a brain-computer interaction mode for the safety officer's driving behavior for human-vehicle collaboration.

[0042] Additional advantages of the present invention will be more clearly given in the following description or learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0044] Figure 1 This is a flowchart of semantic analysis of EEG information of a safety officer as described in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below by the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0046] It should be understood by those skilled in the art that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0047] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0048] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0049] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0050] Example 1

[0051] In this embodiment 1, firstly, a semantic analysis system for EEG information of an autonomous driving safety officer is provided, including: an acquisition module, used to complete a comparative experiment using a driving simulator and a VR device, and synchronously collect driving behavior data and EEG data; an analysis module, used to perform time-frequency domain analysis on EEG data using short-time Fourier transform to obtain an EEG time-frequency feature map; a construction module, used to construct domain tensors of the driver and the safety officer, namely, a source domain and a target domain; a training iteration module, used to decompose the source domain and the target domain into a core tensor and a common adjoint matrix using joint tensor decomposition; splicing the adjoint matrix into a vector and inputting it into a prediction model LSTM, and training to obtain a source prediction function; migrating the source prediction function to the target domain to obtain a target domain pseudo-label matrix, and constructing a new target domain; repeating the steps of constructing a new target domain for the new target domain and the source domain, and calculating the MMD distance between the new target domain and the source domain in each iteration process, and iterating until the MMD distance is minimum and the adjoint matrix remains unchanged; a migration analysis module, used to migrate the source prediction function at the end of the iteration to parse the semantics of the EEG information of the safety officer; extracting the core tensor at the end of the iteration, and analyzing to obtain the difference in cognitive states between the driver and the safety officer.

[0052] In this embodiment 1, the above-mentioned system is used to implement a method for semantic analysis of EEG information of an autonomous driving safety officer, including: using an acquisition module to complete a comparative experiment using a driving simulator and a VR device, and synchronously collecting driving behavior data and EEG data; using an analysis module to perform time-frequency domain analysis on EEG data using short-time Fourier transform to obtain an EEG time-frequency feature map; using a construction module to construct domain tensors of the driver and the safety officer, namely, a source domain and a target domain; using a training iteration module to decompose the source domain and the target domain into a core tensor and a common adjoint matrix using a joint tensor decomposition; splicing the adjoint matrix into a vector and inputting it into a prediction model LSTM, and training to obtain a source prediction function; migrating the source prediction function to the target domain to obtain a target domain pseudo-label matrix, and constructing a new target domain; repeating the steps of constructing a new target domain for the new target domain and the source domain, and calculating the MMD distance between the new target domain and the source domain in each iteration process, and iterating until the MMD distance is minimum and the adjoint matrix remains unchanged; using a migration analysis module to migrate the source prediction function at the end of the iteration to parse the semantics of the safety officer's EEG information; extracting the core tensor at the end of the iteration, and analyzing to obtain the difference in cognitive state between the driver and the safety officer.

[0053] The EEG time-frequency analysis based on short-time Fourier transform can obtain the EEG time-frequency characteristic map, and its calculation formula is:

[0054]

[0055] Where ω(t) is the window function, x(t) is the signal to be transformed, and X(t,f) is the Fourier transform result of w(t,f)x(τ). As t changes, the window function moves on the time axis and only the position that overlaps with the position highlighted by the window function in time, namely w(t-τ)x(τ), is Fourier transformed.

[0056] For source domain X s and the target domain X t Joint tensor decomposition can align the source domain and target domain space. The calculation formula is:

[0057]

[0058] In the formula, is the core tensor showing the degree of interaction between different components; U1, U2 and U3 are the left singular matrices of the expanded tensor along the corresponding order.

[0059] The LSTM model uses the one-dimensional vector concatenated by U1, U2 and U3 as input and reads the information h at the previous moment. t-1 and the memory information h at the current moment t , and output the input gate, output gate and forget gate information i t , o t and f t, and then obtain the state C of the hidden layer unit at time t t Finally, the source target prediction function is obtained through training. The formulas are:

[0060] i t =σ i [W i ·(h t-1 ,x t )+b i ]

[0061] o t =σ o [W o ·(h t-1 ,x t )+b o ]

[0062] f t =σ f [W f ·(h t-1 ,x t )+b f ]

[0063] In the formula, σ i , σ o and σ f are the activation functions selected for the input gate, output gate, and forget gate respectively; W i , W o and W f are the weights selected for the input gate, output gate, and forget gate respectively; b i 、b o and b f Biases chosen for the input gate, output gate, and forget gate, respectively.

[0064] At time t, the state C of the hidden layer unit t The specific calculation formula is as follows:

[0065]

[0066] Where, ⊙ is the Hadamard product; C t-1 is the state of the hidden layer unit at time t-1; C t Alternative value of W c and b c Calculate The weights and biases used when ; tanh is the hyperbolic tangent function.

[0067] The MMD distance can measure the distribution difference between the source domain and the target domain during the iteration process to obtain the source target prediction function that can be used for migration. To achieve the safety officer driving semantic analysis, the calculation formula is:

[0068]

[0069] Where n s and n t Represent the number of data samples in the source domain and target domain respectively; and Represent the number of data samples belonging to the cth category in the source domain and the target domain respectively; U1, U2 and U3 represent the adjoint matrices of the tensor at the 1st, 2nd and 3rd orders respectively; To regenerate the kernel Hilbert space, the feature space of the original sample needs to be mapped to the regenerated kernel Hilbert space for calculation.

[0070] The core tensor extracted when the iteration terminates and Can reflect the source domain X s and the target domain X t The relationship between the three dimensions and analysis of the cognitive differences between safety officers and drivers.

[0071] Example 2

[0072] like Figure 1 As shown, in this embodiment 2, a method for semantic analysis of EEG information of autonomous driving safety officers based on tensor transfer learning is provided to solve the problem that it is difficult to accurately characterize the commonalities and differences of the cognitive states of users in the two driving systems in the prior art and that effective knowledge extraction and transfer research cannot be completed. The parsed explicit semantics of the safety officer can be well integrated into the autonomous driving vehicle control system, realizing the joint perception of the human brain and the "machine brain", expanding the depth of human-vehicle integration, and providing a new path for the coordinated development of humans and vehicles.

[0073] The method for constructing a brain-computer interaction model of safety officer driving behavior based on tensor transfer learning includes the following steps:

[0074] Step 1: Design a comparative experiment to simultaneously collect the user's EEG data and driving behavior data in the artificial driving system and record a video. Then let them wear VR glasses to watch their driving videos and collect EEG signal data, and clean the EEG signals in both states.

[0075] Step 2: Combine short-time Fourier transform (STFT) to perform time-frequency domain analysis on the EEG signal data to obtain the time-frequency feature map of the EEG signal.

[0076] Step 3: Based on the EEG signal time-frequency feature map obtained in step 2, construct the "time-frequency-subject" third-order domain tensor of the driver and safety officer in the manual and automatic driving systems respectively as the source domain and target domain The target domain The driving data required in the construction process is assumed to be consistent with manual driving.

[0077] Step 4: Use joint tensor decomposition to obtain the source domain and target domain The core tensor and and the common adjoint matrices U1, U2 and U3.

[0078] Step 5: Construct the source domain label matrix Y based on the driving behavior data collected in step 1 s , concatenate the three adjoint matrices U1, U2, and U3 obtained in step 4 into a vector, and use LSTM to first train on the source domain to obtain the source prediction function

[0079] Step 6: Use the source prediction function trained in step 5 Migrate to target domain X t Predict the target domain pseudo label matrix Building a new target domain

[0080] Step 7: Repeat steps 4-6 to calculate the new target domain generated in each iteration and The MMD distance is minimized until the common adjoint matrices U1, U2 and U3 remain unchanged, and the source prediction function at this moment is migrated. Parse the driving semantics of the safety officer.

[0081] Step 8: Get the core tensor of the last iteration based on step 7 and Analyze the differences in cognitive states between manual and automatic driving systems.

[0082] The specific process of the cleaning treatment in step 1 is as follows:

[0083] 1) The collected EEG data is band-pass filtered from 0.5 Hz to 70 Hz, and the sinusoidal noise of a specific frequency is removed.

[0084] 2) Remove damaged electrode channels and set the EEG data to the average reference to achieve interpolation and supplementation of adjacent electrodes.

[0085] 3) Segment the EEG data and perform baseline correction as needed.

[0086] 4) ICA is used to remove artifacts and obtain the cleaned EEG signal data.

[0087] The specific process of the short-time Fourier transform (STFT) time-frequency domain analysis described in step 2 is as follows:

[0088] 1) Select appropriate short-time Fourier transform parameters, such as time window length, window function type, and frequency range.

[0089] 2) Apply short-time Fourier transform to the cleaned EEG signal data to obtain the two-dimensional function of the signal on the time-frequency plane. The calculation formula is as follows:

[0090]

[0091] In the formula, ω(t) is the window function, x(t) is the EEG signal to be transformed, and X(t,f) is the Fourier transform result of w(t,f)x(τ). As t changes, the window function moves on the time axis and only the place that overlaps with the position highlighted by the window function in time, that is, w(t-τ)x(τ), is Fourier transformed.

[0092] Among them, the time-frequency dimension described in step 3 is the EEG signal time-frequency feature diagram obtained after time-frequency analysis, which has EEG characteristics in both time and frequency directions, and the subject dimension can be represented by the driving operation type of the subject at each moment.

[0093] The joint tensor decomposition described in step 4 is Tucker decomposition, which can decompose the domain tensors of the driver and the safety officer in each time period into the product of one core tensor and three adjoint matrices along each dimension. The specific formula is as follows:

[0094]

[0095] In the formula, is the core tensor showing the degree of interaction between time, frequency, and subjects; U1, U2, and U3 are the left singular matrices of the expanded tensor along the corresponding order.

[0096] The LSTM model described in step 5 is composed of an input layer, an output layer, and a hidden layer, wherein the hidden layer is composed of one or more memory units and three multiplication gate units (input gate, output gate, and forget gate). At time t, the LSTM model reads the information h of the previous time. t-1 and the memory information h at the current moment t At this time, the outputs of the input gate, output gate and forget gate are i t , o t and f t , the calculation formula is as follows:

[0097] i t =σ i [W i ·(h t-1 ,x t )+b i ]

[0098] o t =σ o [W o ·(h t-1 ,x t )+b o ]

[0099] f t =σ f (W f ·(h t-1 ,x t )+b f ]

[0100] In the formula, σ i , σ o and σ f are the activation functions selected for the input gate, output gate, and forget gate respectively; W i , W o and W f are the weights selected for the input gate, output gate, and forget gate respectively; b i 、b o and b f Biases chosen for the input gate, output gate, and forget gate, respectively.

[0101] At time t, the state C of the hidden layer unit t The specific calculation formula is as follows:

[0102]

[0103] Where, ⊙ is the Hadamard product; C t-1 is the state of the hidden layer unit at time t-1; C t Alternative value of W c and b c Calculate The weights and biases used when ; tanh is the hyperbolic tangent function.

[0104] Among them, the MMD distance in step 7 is used to measure the source domain and target domain The difference between Can the target prediction function f be finally confirmed? t The calculation formula is as follows:

[0105]

[0106]

[0107] Where n s and n tRepresent the number of data samples in the source domain and target domain respectively; and Represent the number of data samples belonging to the cth category in the source domain and the target domain respectively; U1, U2 and U3 represent the adjoint matrices of the tensor at the 1st, 2nd and 3rd orders respectively; To regenerate the kernel Hilbert space, the feature space of the original sample needs to be mapped to the regenerated kernel Hilbert space for calculation.

[0108] Among them, the core tensor described in step 8 and Can represent the source domain and target domain The results after dimensionality reduction, where the driving mode of each subject can be obtained after the subject dimension is reduced, which can be used to analyze the correlation between the dimensions under each driving mode.

[0109] Example 3

[0110] This embodiment provides a method for semantic analysis of EEG information of an autonomous driving safety officer based on tensor transfer learning, including the following steps:

[0111] Step 1: Arrange the experimenters to complete the manual driving experiment of the following car scenario in the driving simulator and collect driving data and EEG signal data simultaneously, and then let them wear VR glasses to watch their driving video and collect EEG signals to simulate the autonomous driving experiment. Including:

[0112] Step 1.1: The collected EEG data is band-pass filtered from 0.5 Hz to 70 Hz, and the sinusoidal noise of a specific frequency is removed.

[0113] Step 1.2: Remove damaged electrode channels and set the EEG data to the average reference to achieve interpolation and supplementation of adjacent electrodes.

[0114] Step 1.3: Segment the EEG data into 1s intervals and perform baseline correction as needed.

[0115] Step 1.4: Use ICA to analyze the EEG components and remove artifacts to obtain the cleaned EEG signal data.

[0116] Step 2: Use short-time Fourier transform (STFT) to perform time-frequency domain analysis on the cleaned EEG signal data to obtain the EEG signal time-frequency feature map. Including:

[0117] Step 2.1: Select a time window of 1 s and a moving step of 0.5 s to split the EEG data into several segments with a length of 1 s and an overlap of 0.5 s.

[0118] Step 2.2: Apply short-time Fourier transform to the cleaned EEG signal data. The window function moves on the time axis and performs Fourier transform at the location where the window function highlights the location.

[0119] Step 3: Based on the EEG signal time-frequency feature map extracted in step 2, construct the "time-frequency-subject" third-order domain tensor of the driver and the safety officer, namely the source domain With the target domain

[0120] The driving operation type of the subject dimension can be represented by the acceleration of the driver and the safety officer.

[0121] Step 4: Use joint tensor decomposition to obtain the source domain and target domain The core tensor and And the common adjoint matrices U1, U2 and U3. The adjoint matrices U1, U2 and U3 represent the relationship between time-frequency, frequency-subject and time-subject respectively.

[0122] Step 5: Construct the source domain label matrix Y based on the acceleration in the driving behavior data collected in step 1 s , concatenate the three adjoint matrices into a one-dimensional vector, and use the LSTM model to predict on the source domain to obtain the source prediction function include:

[0123] Step 5.1 LSTM model reads the information h of the previous moment t-1 and the memory information h at the current moment t , and output the input gate, output gate and forget gate information.

[0124] Step 5.2 Calculate the state C of the hidden layer unit at time t based on the input gate, output gate and forget gate information t .

[0125] Step 5.3 Based on the input at time t+1 and the state C of the hidden layer unit at time t t , calculate the candidate value of the current memory cell.

[0126] Step 5.4 Based on the candidate values ​​of the input gate and the memory unit, calculate the state of the memory unit at time t+1 and control its output to the next moment through the output gate, and finally obtain the prediction function source prediction function

[0127] Step 6: Use the source prediction function trained in step 5 Migrate to the target domain Predict the target domain pseudo label matrix and construct a new target domain pseudo label matrix Building a new target domain

[0128] Step 7: Repeat steps 4-6 to calculate the new target domain generated in each iteration and The MMD distance is minimized until the common adjoint matrices U1, U2 and U3 remain unchanged, and the source prediction function at this moment is migrated. Parse the driving semantics of the safety officer.

[0129] Step 8: The core tensor obtained after the iteration stops based on step 7 and The results of dimensionality reduction of the subjects were extracted, and the correlation between the EEG time-frequency features in each driving mode and the differences between the driver and the safety officer in the same situation were analyzed.

[0130] According to the above steps, in the vehicle-following driving scenario, the method described in this embodiment can effectively parse the driving semantics of the safety officer and then construct a brain-computer interaction mode of the safety officer.

[0131] Example 4

[0132] This embodiment 4 provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the above-mentioned method for semantic analysis of EEG information of an autonomous driving safety officer is implemented. The method includes:

[0133] A comparative experiment was conducted using a driving simulator and VR equipment to simultaneously collect driving behavior data and EEG data;

[0134] Short-time Fourier transform is used to analyze EEG data in time and frequency domain to obtain EEG time-frequency feature map;

[0135] Construct the domain tensors of the driver and the safety officer, namely the source domain and the target domain;

[0136] The source domain and target domain are decomposed into core tensors and common adjoint matrices by joint tensor decomposition. The adjoint matrices are concatenated into vectors and input into the prediction model LSTM to obtain the source prediction function through training. The source prediction function is transferred to the target domain to obtain the target domain pseudo-label matrix and construct a new target domain.

[0137] Repeat the steps of constructing a new target domain for the new target domain and the source domain, and calculate the MMD distance between the new target domain and the source domain in each iteration, and iterate until the MMD distance is minimum and the companion matrix remains unchanged;

[0138] The source prediction function at the end of the migration iteration parses the semantics of the safety officer's EEG information; the core tensor at the end of the iteration is extracted to analyze the difference in cognitive states between the driver and the safety officer.

[0139] Example 5

[0140] This embodiment 5 provides a computer program product, including a computer program. When the computer program is executed on one or more processors, it is used to implement the above-mentioned method for semantic analysis of EEG information of an autonomous driving safety officer. The method includes:

[0141] A comparative experiment was conducted using a driving simulator and VR equipment to simultaneously collect driving behavior data and EEG data;

[0142] Short-time Fourier transform is used to analyze EEG data in time and frequency domain to obtain EEG time-frequency feature map;

[0143] Construct the domain tensors of the driver and the safety officer, namely the source domain and the target domain;

[0144] The source domain and target domain are decomposed into core tensors and common adjoint matrices by joint tensor decomposition. The adjoint matrices are concatenated into vectors and input into the prediction model LSTM to obtain the source prediction function through training. The source prediction function is transferred to the target domain to obtain the target domain pseudo-label matrix and construct a new target domain.

[0145] Repeat the steps of constructing a new target domain for the new target domain and the source domain, and calculate the MMD distance between the new target domain and the source domain in each iteration, and iterate until the MMD distance is minimum and the companion matrix remains unchanged;

[0146] The source prediction function at the end of the migration iteration parses the semantics of the safety officer's EEG information; the core tensor at the end of the iteration is extracted to analyze the difference in cognitive states between the driver and the safety officer.

[0147] Example 6

[0148] This embodiment 6 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing the above-mentioned method for semantic analysis of EEG information of an autonomous driving safety officer, the method including:

[0149] A comparative experiment was conducted using a driving simulator and VR equipment to simultaneously collect driving behavior data and EEG data;

[0150] Short-time Fourier transform is used to analyze EEG data in time and frequency domain to obtain EEG time-frequency feature map;

[0151] Construct the domain tensors of the driver and the safety officer, namely the source domain and the target domain;

[0152] The source domain and target domain are decomposed into core tensors and common adjoint matrices by joint tensor decomposition. The adjoint matrices are concatenated into vectors and input into the prediction model LSTM to obtain the source prediction function through training. The source prediction function is transferred to the target domain to obtain the target domain pseudo-label matrix and construct a new target domain.

[0153] Repeat the steps of constructing a new target domain for the new target domain and the source domain, and calculate the MMD distance between the new target domain and the source domain in each iteration, and iterate until the MMD distance is minimum and the companion matrix remains unchanged;

[0154] The source prediction function at the end of the migration iteration parses the semantics of the safety officer's EEG information; the core tensor at the end of the iteration is extracted to analyze the difference in cognitive states between the driver and the safety officer.

[0155] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0159] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative labor on the basis of the technical solution disclosed in the present invention should be included in the scope of protection of the present invention.

Claims

1. A semantic analysis method for EEG information of an autonomous driving safety officer, characterized in that: include: Design a comparative experiment to synchronously collect the user's EEG data and driving behavior data in the manual driving system and record a video. Then let the user wear VR glasses to watch the driving video and collect EEG signal data. Short-time Fourier transform is used to analyze EEG data in time and frequency domain to obtain EEG time-frequency feature map; Construct the domain tensors of the driver and the safety officer, namely the source domain and the target domain; among them, based on the EEG time-frequency feature map obtained by time-frequency domain analysis, construct the "time-frequency-subject" third-order domain tensors of the driver and the safety officer in the manual and automatic driving systems respectively, recorded as the source domain and target domain The source domain and target domain are decomposed into core tensors and common adjoint matrices by joint tensor decomposition. The source domain label matrix is ​​constructed based on the collected driving behavior data. The adjoint matrix is ​​concatenated into a vector and input into the prediction model LSTM. The source prediction function is obtained by training LSTM on the source domain. The source prediction function is transferred to the target domain to obtain the target domain pseudo label matrix and construct a new target domain. s and the target domain X t Joint tensor decomposition can align the source domain and target domain space. The calculation formula is: x = h × 1U1 × 2U2 × 3U3; Where h is the core tensor showing the degree of interaction between different components; U1, U2 and U3 are the left singular matrices of the tensor expanded along the corresponding order; The LSTM model uses the one-dimensional vector concatenated by U1, U2 and U3 as input and reads the information h at the previous moment. t-1 and the memory information h at the current moment t , and output the input gate, output gate and forget gate information i t , o t and f t , and then obtain the state C of the hidden layer unit at time t t Finally, the source target prediction function is obtained through training. The formulas are: i t =σ i [W i ·(h t-1 ,v t )+b i ] the t =s o [W o ·(h t-1 ,v t )+b o ] f t =σ f [W f ·(h t-1 ,v t )+b f ] In the formula, σ i , σ o and σ f are the activation functions selected for the input gate, output gate, and forget gate respectively; W i , W o and W f are the weights selected for the input gate, output gate, and forget gate respectively; b i , b o and b f are the biases selected for the input gate, output gate, and forget gate respectively; At time t, the state C of the hidden layer unit t The specific calculation formula is as follows: Where, ⊙ is the Hadamard product; C t-1 is the state of the hidden layer unit at time t-1; C t Alternative value of W c and b c Calculate The weights and biases selected when ; tanh is the hyperbolic tangent function; Repeat the steps of constructing a new target domain for the new target domain and the source domain, and calculate the MMD distance between the new target domain and the source domain in each iteration, iterating until the MMD distance is minimum and the companion matrix remains unchanged; the MMD distance measures the distribution difference between the source domain and the target domain during the iteration process, and obtains the source target prediction function for migration To achieve the safety officer driving semantic analysis, the calculation formula is: Where n s and n t Represent the number of data samples in the source domain and target domain respectively; and Represent the number of data samples belonging to the cth category in the source domain and the target domain respectively; U1, U2 and U3 represent the adjoint matrices of the tensor at the 1st, 2nd and 3rd orders respectively; To reproduce the kernel Hilbert space, the feature space of the original sample needs to be mapped to the reproduced kernel Hilbert space for calculation; The source prediction function at the end of the migration iteration parses the semantics of the safety officer's EEG information; the core tensor at the end of the iteration is extracted to analyze the difference in cognitive states between the driver and the safety officer.

2. The method for semantic analysis of EEG information of an autonomous driving safety officer according to claim 1, characterized in that: The EEG time-frequency analysis based on short-time Fourier transform can obtain the EEG time-frequency characteristic map, and its calculation formula is: Where ω(t) is the window function, x(t) is the signal to be transformed, and X(t, f) is the Fourier transform result of w(t, f)x(τ). As t changes, the window function moves on the time axis and only the position that overlaps with the position highlighted by the window function in time, namely w(t-τ)x(τ), is Fourier transformed.

3. A semantic analysis system for EEG information of an autonomous driving safety officer, characterized in that: include: The acquisition module designs a comparative experiment to synchronously collect the EEG data and driving behavior data of the user in the manual driving system and record the video. Then, the user is asked to wear VR glasses to watch the driving video and collect EEG signal data. The analysis module uses short-time Fourier transform to perform time-frequency domain analysis on EEG data to obtain EEG time-frequency feature maps; The construction module constructs the domain tensors of the driver and the safety officer, namely the source domain and the target domain. Among them, based on the EEG time-frequency feature map obtained by time-frequency domain analysis, the "time-frequency-subject" third-order domain tensors of the driver and the safety officer in the manual and automatic driving systems are constructed respectively and recorded as the source domain. and target domain The training iteration module uses joint tensor decomposition to decompose the source domain and target domain into core tensors and a common adjoint matrix; constructs a source domain label matrix based on the collected driving behavior data; concatenates the adjoint matrix into a vector and inputs it into the prediction model LSTM, and uses LSTM to train on the source domain to obtain the source prediction function; migrates the source prediction function to the target domain to obtain the target domain pseudo-label matrix, and constructs a new target domain; among them, for the source domain X s and the target domain X t Joint tensor decomposition can align the source domain and target domain space. The calculation formula is: In the formula, is the core tensor showing the degree of interaction between different components; U1, U2 and U3 are the left singular matrices of the tensor expanded along the corresponding order; The LSTM model uses the one-dimensional vector concatenated by U1, U2 and U3 as input and reads the information h at the previous moment. t-1 and the memory information h at the current moment t , and output the input gate, output gate and forget gate information i t , o t and f t , and then obtain the state C of the hidden layer unit at time t t Finally, the source target prediction function is obtained through training. The formulas are: i t =σ i [W i ·(h t-1 ,v t )+b i ] the t =s o [W o ·(h t-1 ,v t )+b o ] f t =σ f [W f ·(h t-1 ,v t )+b f ] In the formula, σ i , σ o and σ f are the activation functions selected for the input gate, output gate, and forget gate respectively; W i , W o and W f are the weights selected for the input gate, output gate, and forget gate respectively; b i 、b o and b f are the biases selected for the input gate, output gate, and forget gate respectively; At time t, the state C of the hidden layer unit t The specific calculation formula is as follows: Where, ⊙ is the Hadamard product; C t-1 is the state of the hidden layer unit at time t-1; C t Alternative value of W c and b c Calculate The weights and biases selected when ; tanh is the hyperbolic tangent function; Repeat the steps of constructing a new target domain for the new target domain and the source domain, and calculate the MMD distance between the new target domain and the source domain in each iteration, iterating until the MMD distance is minimum and the companion matrix remains unchanged; the MMD distance measures the distribution difference between the source domain and the target domain during the iteration process, and obtains the source target prediction function for migration To achieve the safety officer driving semantic analysis, the calculation formula is: Where n s and n t Represent the number of data samples in the source domain and target domain respectively; and Represent the number of data samples belonging to the cth category in the source domain and the target domain respectively; U1, U2 and U3 represent the adjoint matrices of the tensor at the 1st, 2nd and 3rd orders respectively; To reproduce the kernel Hilbert space, the feature space of the original sample needs to be mapped to the reproduced kernel Hilbert space for calculation; In the migration analysis module, the source prediction function at the end of the migration iteration parses the semantics of the safety officer’s EEG information; extracts the core tensor at the end of the iteration, and analyzes the difference in cognitive states between the driver and the safety officer.

4. A computer program product, characterized in that It includes a computer program, which, when running on one or more processors, is used to implement the method for semantic analysis of EEG information of an autonomous driving safety officer as described in claim 1 or 2.

5. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the method for semantic analysis of EEG information of an autonomous driving safety officer as described in claim 1 or 2 is implemented.

6. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the method for semantic analysis of EEG information of an autonomous driving safety officer as described in claim 1 or 2.

Images