Multi-modal data synchronous acquisition device, system and method and medium

Abstract

The embodiment of the invention provides a multi-modal data synchronous acquisition device, system and method and a medium. The equipment comprises an integrated device and an electroencephalogram device, the integrated device comprises a connecting interface, a plurality of electroencephalogram detection point positions and a plurality of human body detection point positions, and the connecting interface is used for being connected with the electroencephalogram device; the n human body detection points are used for detecting human body physiological signals of n channels in an access state; the electroencephalogram device is used for detecting electroencephalogram signals of N-n channels when the electroencephalogram device is connected with the integrated device through the connecting interface; or the N-n electroencephalogram detection points are used for detecting electroencephalogram signals of N-n channels when the electroencephalogram device is not connected with the integrated device through the connection interface; n and n are positive integers, and N is greater than n. According to the embodiment of the invention, the diversity and flexibility of multi-modal data synchronous acquisition are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of signal acquisition technology, and in particular to a multi-modal data synchronous acquisition device, system, method and medium. Background Art

[0002] With the development of sensor technology and signal processing technology, signal changes in the brain and its nervous system can be recorded in real time. By detecting and analyzing changes in EEG signals, the human brain can be better studied.

[0003] In the prior art, a user can wear an electroencephalogram (EEG) device and collect electroencephalogram (EEG) signals through the electroencephalogram (EEG) device. However, in the prior art, the electroencephalogram (EEG) device can only collect EEG signals, but cannot collect other human physiological signals, and the collected signals are single. Summary of the invention

[0004] Embodiments of the present application provide a multimodal data synchronous acquisition device, system, method and medium.

[0005] In a first aspect, a multimodal data synchronous acquisition device is provided, comprising: an integrated device and an EEG device, wherein the integrated device comprises a connection interface, a plurality of EEG detection points and a plurality of human body detection points, the connection interface is used to connect the EEG device, and the EEG device comprises an EEG cap, an EEG helmet or an EEG headband;

[0006] The n human body detection points are used to detect n channels of human body physiological signals when in an access state;

[0007] The EEG device is used to detect EEG signals of Nn channels when the EEG device is connected to the integrated device through the connection interface; or, the Nn EEG detection points are used to detect EEG signals of Nn channels when the EEG device is not connected to the integrated device;

[0008] N and n are both positive integers, and N is greater than n.

[0009] In one possible implementation, the EEG device is used to detect EEG signals of N channels when the integrated device is connected through the connection interface and the human body detection point is not in an access state; or, the N EEG detection points are used to detect EEG signals of N channels when the EEG device is not connected to the integrated device and the human body detection point is not in an access state.

[0010] In a possible implementation, the device further includes: a first indicator light arranged corresponding to the connection interface and a second indicator light arranged corresponding to each of the human body detection points;

[0011] The first indicator light is used to light up when the EEG device is connected to the integrated device through the connection interface;

[0012] The second indicator light is used to light up when the corresponding human body detection point is in a connected state.

[0013] In a possible implementation, the human body detection point includes two electrode points.

[0014] In a possible implementation, the human physiological signal includes an electrocardiogram signal, an electrooculogram signal, or an electromyography signal.

[0015] A second aspect provides a multimodal data synchronous acquisition system, the system comprising a control circuit, an integrated device and an EEG device, the integrated device comprising a connection interface, a plurality of EEG detection points and a plurality of human body detection points, the connection interface is used to connect the EEG device, the EEG device comprises an EEG cap, an EEG helmet or an EEG headband;

[0016] The control circuit comprises: an analog integrated front end and N signal channel branches connected to the analog integrated front end, the N signal channel branches comprise M integrated signal channel branches and NM EEG signal channel branches, M and N are both positive integers and M is greater than N;

[0017] The integrated signal channel branch is used to select the positive human physiological signal and the corresponding negative human physiological signal detected by the human detection point, the positive EEG signal and the corresponding negative EEG signal detected by the EEG device, or the positive EEG signal and the corresponding negative EEG signal detected by the EEG device;

[0018] The EEG signal channel branch is used to select the positive EEG signal detected by the EEG device and the corresponding negative EEG signal or the positive EEG signal detected by the EEG device and the corresponding negative EEG signal;

[0019] The analog integrated front end is used to calculate n human physiological signals based on the selected n positive human physiological signals and the corresponding negative human physiological signals, and to calculate Nn EEG signals based on Nn positive EEG signals and the corresponding negative EEG signals, where n is a positive integer and N is greater than n.

[0020] In a possible implementation, the integrated signal channel branch includes a first integrated signal channel branch and a second integrated signal channel branch, and the electroencephalogram signal channel branch includes a first electroencephalogram signal channel branch and a second electroencephalogram signal channel branch;

[0021] The first integrated signal channel branch is used to select the positive human physiological signal detected by the human body detection point, the positive EEG signal detected by the EEG device, or the positive EEG signal detected by the EEG detection point;

[0022] The second integrated signal channel branch is used to select the negative human physiological signal detected by the human body detection point, the negative EEG signal detected by the EEG device, or the negative EEG signal detected by the EEG detection point;

[0023] The first EEG signal channel branch is used to select the positive EEG signal detected by the EEG device or the positive EEG signal detected by the EEG detection point;

[0024] The second EEG signal channel branch is used to select the negative EEG signal detected by the EEG device or the negative EEG signal detected by the EEG detection point.

[0025] In a possible implementation, the EEG device includes a plurality of EEG detection electrodes, and the human body detection points include a first human body detection point and a second human body detection point;

[0026] The first integrated signal channel branch includes a first switch and a second switch, the first input end of the first switch is connected to the corresponding EEG detection point, the second input end of the first switch is connected to the corresponding EEG detection electrode, the output end of the first switch is connected to the first input end of the second switch, the control end of the first switch is connected to the connection interface, the second input end of the second switch is connected to the corresponding first human body detection point, the output end of the second switch is connected to the analog integrated front end, and the control end of the second switch is connected to the corresponding human body detection point;

[0027] The second integrated signal channel branch includes a third switch and a fourth switch, the first input end of the third switch is connected to the corresponding EEG detection point, the second input end of the third switch is connected to the corresponding EEG detection electrode, the output end of the third switch is connected to the first input end of the fourth switch, the control end of the third switch is connected to the connection interface, the second input end of the fourth switch is connected to the corresponding second human body detection point, the output end of the fourth switch is connected to the analog integrated front end, and the control end of the fourth switch is connected to the corresponding human body detection point.

[0028] In a possible implementation, the EEG device includes a plurality of EEG detection electrodes;

[0029] The first EEG signal channel branch includes a fifth switch, a first input end of the fifth switch is connected to a corresponding EEG detection point, a second input end of the fifth switch is connected to a corresponding EEG detection electrode, an output end of the fifth switch is connected to the analog integrated front end, and a control end of the fifth switch is connected to the connection interface;

[0030] The second EEG signal channel branch includes a sixth switch, a first input end of the sixth switch is connected to a corresponding EEG detection point, a second input end of the sixth switch is connected to a corresponding EEG detection electrode, an output end of the sixth switch is connected to the analog integrated front end, and a control end of the sixth switch is connected to the connection interface.

[0031] A third aspect provides a multimodal data synchronous acquisition method, the method being based on the multimodal data synchronous acquisition device in the first aspect or any possible implementation of the first aspect or the multimodal data synchronous acquisition system in the first aspect or any possible implementation of the first aspect;

[0032] The method comprises:

[0033] Detecting whether the plurality of human body detection points are in an access state;

[0034] If it is detected that n human body detection points are in a connected state, detecting whether the EEG device is connected to the integrated device through the connection interface;

[0035] If it is detected that the EEG device is connected to the integrated device via the connection interface, human physiological signals of n channels are detected via the n human detection points and EEG signals of Nn channels are detected via the EEG device;

[0036] If it is detected that the EEG device is not connected to the integrated device through the connection interface, detecting human physiological signals of n channels through the n human body detection points and detecting EEG signals of Nn channels through Nn EEG detection points;

[0037] N and n are both positive integers, and N is greater than n.

[0038] In a possible implementation, the method further includes:

[0039] If it is detected that the plurality of human body detection points are not in a connected state, detecting whether the EEG device is connected to the integrated device through the connection interface;

[0040] If it is detected that the EEG device is connected to the integrated device via the connection interface, an N-channel EEG signal is detected via the EEG device;

[0041] If it is detected that the electroencephalogram device is not connected to the integrated device through the connection interface, electroencephalogram signals of N channels are detected through the N electroencephalogram detection points.

[0042] A fourth aspect provides an edge computing device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, and when the instructions are executed by the edge computing device, the edge computing device executes the multimodal data synchronization acquisition method in the third aspect or any possible implementation of the third aspect; and / or,

[0043] The edge computing device includes: the multimodal data synchronization acquisition device in the above-mentioned first aspect or any possible implementation of the first aspect, or the multimodal data synchronization acquisition system in the above-mentioned second aspect or any possible implementation of the second aspect.

[0044] The fifth aspect provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the edge computing device where the computer-readable storage medium is located is controlled to execute the multimodal data synchronization acquisition method in the above-mentioned third aspect or any possible implementation of the third aspect.

[0045] In the technical solution provided in the embodiment of the present application, the integrated device includes a connection interface, multiple EEG detection points and multiple human body detection points, the connection interface is used to connect the EEG device; the n human body detection points detect n channels of human physiological signals when in the connected state; the EEG device detects Nn channels of EEG signals when the EEG device is connected to the integrated device through the connection interface, or the Nn EEG detection points detect Nn channels of EEG signals when the EEG device is not connected to the integrated device through the connection interface; the embodiment of the present application improves the diversity and flexibility of synchronous acquisition of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A schematic diagram of the structure of a multimodal data synchronization acquisition device provided in an embodiment of the present application;

[0048] Figure 2 A flowchart of a multi-modal data synchronous acquisition method provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the structure of a multi-modal data synchronous acquisition system provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of the structure of an edge computing device provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of another edge computing device provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of another edge computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0054] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0055] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0056] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0057] Figure 1 A schematic diagram of the structure of a multi-modal data synchronization acquisition device provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the multimodal data synchronous acquisition equipment includes an integrated device 1 and an EEG device 2. The integrated device 1 includes a connection interface 11, multiple EEG detection points 12 and multiple human body detection points 13. The connection interface 11 is used to connect to the EEG device 2.

[0058] In an embodiment of the present application, as an optional solution, the EEG device may include an EEG cap, an EEG helmet or an EEG headband.

[0059] In the embodiment of the present application, the multimodal data synchronous acquisition device can be used to detect EEG signals and human physiological signals. The EEG detection point 12 and the human body detection point 13 are integrated into the integrated device. As an optional solution, the EEG detection point 12 and the human body detection point 13 can both be hole positions. When signal detection is required, the above-mentioned hole positions can be connected to corresponding detection electrodes.

[0060] The number of human body detection points 13 can be set according to actual needs. For example, the number of human body detection points 13 can be M. If n human body physiological signals are to be detected, n human body detection points 13 can be connected to corresponding detection electrodes. At this time, the number of human body detection points 13 in the connected state is n, that is, n human body detection points 13 are in the connected state among M human body detection points 13. M and n are both positive integers and M is greater than or equal to n. Figure 1 As shown, for example, M=8.

[0061] In the embodiment of the present application, the total number of signal channels detected by the multimodal data synchronous acquisition device is N, that is, the total number of detected signals is N, and the number of channels of the detected EEG signal is Nn, where N is a positive integer and N is greater than n. For example, N=32, n=8, then Nn=24.

[0062] In an embodiment of the present application, n human body detection points 13 are used to detect human physiological signals of n channels when in an access state; the EEG device 2 is used to detect EEG signals of Nn channels when connected to the integrated device 1 through the connection interface 11; or, Nn EEG detection points 12 are used to detect EEG signals of Nn channels when the EEG device 2 is not connected to the integrated device 1 through the connection interface 11.

[0063] As an optional solution, the EEG device 2 is used to detect EEG signals of Nn channels when it is connected to the integrated device 1 through the connection interface 11 and the human body detection point 13 is not in an access state; or, N EEG detection points 12 are used to detect EEG signals of N channels when the EEG device 2 is not connected to the integrated device 1 and the human body detection point 13 is not in an access state.

[0064] As an optional solution, the multimodal data synchronous acquisition device also includes a first indicator light 3 corresponding to the connection interface 11 and a second indicator light 4 corresponding to each human body detection point 13. The first indicator light 3 is used to light up when the EEG device 2 is connected to the integrated device 1 through the connection interface 11; the second indicator light 4 is used to light up when the corresponding human body detection point 13 is in a connected state. When the first indicator light 3 can be lit, it indicates that the connection interface 11 can work normally and is valid; when the second indicator light 4 can be lit, it indicates that the human body detection point 13 can work normally and is valid. The colors of the first indicator light 3 and the second indicator light 4 can be set as needed, and are not limited by the comparison of the embodiments of the present application.

[0065] As an optional solution, the human body detection point 13 includes two electrode points, and the electrode points can be hole points.

[0066] In the embodiment of the present application, the EEG device 2 may include multiple EEG detection electrodes. The EEG device 2 can be understood as an EEG acquisition product with fixed measurement points, and the user cannot select which measurement points to collect signals according to needs. In the integrated device 1, the user can select the EEG detection point 12 and the human body detection point 13 for signal collection according to needs.

[0067] As an optional solution, the connection interface 11 may be a display port (Display Port, DP). In practical applications, the connection interface 11 may also be other interfaces, which is not limited in the embodiment of the present application.

[0068] As an optional solution, the human physiological signal includes an electrocardiogram (ECG) signal, an electrooculogram (EOG) signal or an electromyography (EMG) signal. In practical applications, the human physiological signal may also include other types of signals, which are not limited in the embodiments of the present application.

[0069] In the technical solution provided in the embodiment of the present application, the integrated device includes a connection interface, multiple EEG detection points and multiple human body detection points, the connection interface is used to connect the EEG device; the n human body detection points detect n channels of human physiological signals when in the connected state; the EEG device detects Nn channels of EEG signals when the EEG device is connected to the integrated device through the connection interface, or the Nn EEG detection points detect Nn channels of EEG signals when the EEG device is not connected to the integrated device through the connection interface; the embodiment of the present application improves the diversity and flexibility of multimodal data synchronous acquisition by adding human body detection points in the EEG detection equipment.

[0070] In the embodiment of the present application, flexibility of EEG signal detection is achieved by switching between the EEG device and the EEG detection points.

[0071] In the embodiment of the present application, when using an EEG device to collect EEG signals, the user can directly wear the EEG device to complete the collection, so that the user can easily collect EEG signals; when using EEG detection points to collect EEG signals, the user can select the desired EEG detection points by himself, thereby meeting the diverse needs of users. Through the synchronous collection method provided in the embodiment of the present application, the above-mentioned EEG device and EEG detection points can be expanded to synchronously collect human physiological signals. While collecting EEG signals, human physiological signals can also be collected using human detection points. There is no need to upgrade the firmware of the original EEG device, which improves the flexibility of EEG signal collection and realizes the synchronous collection of multimodal signals such as EEG signals and human physiological signals. These synchronously collected signals are convenient for predicting and analyzing human physiological / psychological / mental / healthy states.

[0072] based on Figure 1 The multi-modal data synchronous acquisition device shown or based on the following Figure 3 The multimodal data synchronous acquisition system shown, an embodiment of the present application provides a multimodal data synchronous acquisition method. Figure 2 A flowchart of a multi-modal data synchronous acquisition method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0073] Step 102 , detecting whether a plurality of human body detection points are in an access state, if so, executing step 104 ; if not, executing step 110 .

[0074] If the user needs to perform signal detection at n human body detection points, one end of the connecting line can be inserted into the hole of the human body detection point, and the other end of the detection line is provided with a detection electrode.

[0075] In this step, if one end of the detection line is inserted into the human body detection point, the bipolar control signal of the human body detection point is detected, which indicates that the human body detection point is detected to be in the connected state. If one end of the detection line is not inserted into the human body detection point, the bipolar control signal of the human body detection point cannot be detected, which indicates that the human body detection point is not in the connected state.

[0076] If it is detected that n human body detection points are in the connected state, it is necessary to detect the human physiological signals and EEG signals at the same time, and continue to execute step 104; if it is detected that multiple human body detection points are not in the connected state, it is only necessary to detect the EEG signals, and continue to execute step 110.

[0077] Step 104 , detecting whether the EEG device is connected to the integrated device via the connection interface, if so, executing step 106 ; if not, executing step 108 .

[0078] If the connection line of the EEG device is inserted into the connection interface, the connection control signal of the connection interface is detected, which indicates that the EEG device is detected to be connected to the integrated device through the connection interface, and the process continues to step 106; if the connection line of the EEG device is not inserted into the connection interface, the connection control signal of the connection interface cannot be detected, which indicates that the EEG device is not connected to the integrated device through the connection interface, and the process continues to step 108.

[0079] Step 106: Detect human physiological signals of n channels through n human body detection points and detect EEG signals of Nn channels through an EEG device.

[0080] Each human detection point can detect one channel of human physiological signal, that is, each human detection point can detect one human physiological signal, so n channels of human physiological signals are detected through n human detection points. Since the total number of signal channels detected by the multimodal data synchronous acquisition device is N, it is necessary to detect Nn channels of EEG signals through the EEG device.

[0081] For example, when N=32, if n=8, then Nn=24; for another example, when N=32, if n=7, then Nn=25; for another example, when N=32, if n=6, then Nn=26; for another example, when N=32, if n=1, then Nn=31.

[0082] Step 108: Detect human physiological signals of n channels through n human body detection points and detect EEG signals of Nn channels through Nn EEG detection points.

[0083] Each human body detection point can detect one channel of human physiological signal, that is, each human body detection point can detect one human physiological signal, so n channels of human physiological signals are detected through n human body detection points. Each EEG detection point can detect one channel of EEG signal, that is, each EEG detection point can detect one EEG signal, so N channels of EEG signals are detected through Nn EEG detection points.

[0084] For example, when N=32, if n=8, then Nn=24; for another example, when N=32, if n=7, then Nn=25; for another example, when N=32, if n=6, then Nn=26; for another example, when N=32, if n=1, then Nn=31.

[0085] Step 110 , detecting whether the EEG device is connected to the integrated device via the connection interface, if so, executing step 112 ; if not, executing step 114 .

[0086] If the connection line of the EEG device is inserted into the connection interface, the connection control signal of the connection interface is detected, which indicates that the EEG device is detected to be connected to the integrated device through the connection interface, and the process continues to step 112; if the connection line of the EEG device is not inserted into the connection interface, the connection control signal of the connection interface cannot be detected, which indicates that the EEG device is not connected to the integrated device through the connection interface, and the process continues to step 114.

[0087] Step 112: Detect the N-channel EEG signal through the EEG device.

[0088] Since the total number of signal channels detected by the multimodal data synchronous acquisition device is N, it is necessary to detect N channels of EEG signals through the EEG device. For example, N=32.

[0089] Step 114: Detect N channels of EEG signals through N EEG detection points.

[0090] Each EEG detection point can detect one channel of EEG signal, that is, each EEG detection point can detect one EEG signal, so N channels of EEG signals can be detected through N EEG detection points.

[0091] In an embodiment of the present application, by adding an insertion detection function to a human body detection point, a hard switch between human physiological signal detection or EEG signal detection can be achieved; by adding an insertion detection function to a connection interface, a hard switch between detecting EEG signals through an EEG device or detecting EEG signals through EEG detection points can be achieved.

[0092] In the technical solution provided in the embodiment of the present application, the integrated device includes a connection interface, multiple EEG detection points and multiple human body detection points, the connection interface is used to connect the EEG device; the n human body detection points detect n channels of human physiological signals when in the connected state; the EEG device detects Nn channels of EEG signals when the EEG device is connected to the integrated device through the connection interface, or the Nn EEG detection points detect Nn channels of EEG signals when the EEG device is not connected to the integrated device; the embodiment of the present application improves the diversity and flexibility of multimodal data synchronous acquisition by adding human body detection points in the EEG detection equipment.

[0093] In the embodiment of the present application, flexibility of EEG signal detection is achieved by switching between the EEG device and the EEG detection points.

[0094] Figure 3 A schematic diagram of the structure of a multi-modal data synchronous acquisition system provided in an embodiment of the present application is shown in FIG. Figure 3As shown, the multimodal data synchronous acquisition system includes: a control circuit, an integrated device and an EEG device, the integrated device includes a connection interface, multiple EEG detection points and multiple human body detection points, the connection interface is used to connect the EEG device; the control circuit includes an analog integrated front end and N signal channel branches connected to the analog integrated front end, the N signal channel branches include M integrated signal channel branches and NM EEG signal channel branches, M and N are both positive integers and N is greater than M. The description of the integrated device and the EEG device can be found in Figure 1 The embodiments shown will not be described in detail here.

[0095] The integrated signal channel branch is used to select the positive human physiological signals and the corresponding negative human physiological signals detected by the human body detection points, the positive EEG signals and the corresponding negative EEG signals detected by the EEG device, or the positive EEG signals and the corresponding negative EEG signals detected by the EEG device. The EEG signal channel branch is used to select the positive EEG signals and the corresponding negative EEG signals detected by the EEG device, or the positive EEG signals and the corresponding negative EEG signals detected by the EEG device. The analog integrated front end is used to calculate n human physiological signals based on the selected n positive human physiological signals and the corresponding negative human physiological signals, and calculate Nn EEG signals based on Nn positive EEG signals and the corresponding negative EEG signals, where n is a positive integer and N is greater than n.

[0096] As an optional solution, the integrated signal channel branch includes a first integrated signal channel branch and a second integrated signal channel branch, and the EEG signal channel branch includes a first EEG signal channel branch and a second EEG signal channel branch. The first integrated signal channel branch is used to select positive human physiological signals detected by human body detection points, positive EEG signals detected by an EEG device, or positive EEG signals detected by EEG detection points. The second integrated signal channel branch is used to select negative human physiological signals detected by human body detection points, negative EEG signals detected by an EEG device, or negative EEG signals detected by EEG detection points. The first EEG signal channel branch is used to select positive EEG signals detected by an EEG device, or positive EEG signals detected by an EEG detection point. The second EEG signal channel branch is used to select negative EEG signals detected by an EEG device, or negative EEG signals detected by an EEG detection point.

[0097] like Figure 3 As shown, the EEG device includes a plurality of EEG detection electrodes, and the human body detection points include a first human body detection point and a second human body detection point.

[0098] The first integrated signal channel branch includes a first switch K1 and a second switch K2, the first input end of the first switch K1 is connected to the corresponding EEG detection point 1+, the second input end of the first switch K1 is connected to the corresponding EEG detection electrode 1+, the output end of the first switch K1 is connected to the first input end of the second switch K2, the control end of the first switch K1 is connected to the connection interface, the second input end of the second switch K2 is connected to the corresponding first human body detection point 1+, the output end of the second switch K2 is connected to the analog integrated front end, and the control end of the second switch K2 is connected to the corresponding human body detection point 1.

[0099] The second integrated signal channel branch includes a third switch K3 and a fourth switch K4, the first input end of the third switch K3 is connected to the corresponding EEG detection point 1-, the second input end of the third switch K3 is connected to the corresponding EEG detection electrode 1-, the output end of the third switch K3 is connected to the first input end of the fourth switch K4, the control end of the third switch K3 is connected to the connection interface, the second input end of the fourth switch K4 is connected to the corresponding second human body detection point 1-, the output end of the fourth switch K4 is connected to the analog integrated front end, and the control end of the fourth switch K4 is connected to the corresponding human body detection point 1.

[0100] In the embodiment of the present application, as an optional solution, the first switch K1, the second switch K2, the third switch K3 and the fourth switch K4 may be relays.

[0101] like Figure 3 As shown, the first EEG signal channel branch includes a fifth switch K5, a first input end of the fifth switch K5 is connected to the corresponding EEG detection point N+, a second input end of the fifth switch K5 is connected to the corresponding EEG detection electrode N+, an output end of the fifth switch K5 is connected to the analog integrated front end, and a control end of the fifth switch K5 is connected to the connection interface. The second EEG signal channel branch includes a sixth switch K6, a first input end of the sixth switch K6 is connected to the corresponding EEG detection point N-, a second input end of the sixth switch K6 is connected to the corresponding EEG detection electrode N-, an output end of the sixth switch K6 is connected to the analog integrated front end, and a control end of the sixth switch K6 is connected to the connection interface.

[0102] In the embodiment of the present application, as an optional solution, the fifth switch K5 and the sixth switch K6 may be relays.

[0103] In the embodiment of the present application, the number of integrated signal channel branches is M. Figure 3 Only two integrated signal channels are drawn as an example, and the remaining integrated signal channel branches are not specifically drawn; in the embodiment of the present application, the number of EEG signal channel branches is NM, Figure 3 Only one EEG signal channel branch is drawn as an example, and the other EEG signal channel branches are not specifically drawn.

[0104] like Figure 3 As shown, when a connecting line is inserted into the human body detection point 1, the human body detection point 1 provides a bipolar control signal to the second switch K2 and the fourth switch K4; the second switch K2 conducts the first human body detection point 1+ and the analog integrated front end under the control of the bipolar control signal, so that the first human body detection point 1+ provides a positive human physiological signal to the analog integrated front end; the fourth switch K4 conducts the first human body detection point 1- and the analog integrated front end under the control of the bipolar control signal, so that the first human body detection point 1- provides a negative human physiological signal to the analog integrated front end. The analog integrated front end can calculate the human body physiological signal according to the positive human body physiological signal and the negative human body physiological signal. Specifically, the human body physiological signal = (positive human body physiological signal-ground voltage GND)-(negative human body physiological signal-ground voltage GND).

[0105] like Figure 3As shown, when no connecting wire is inserted into the human body detection point 1, the human body detection point 1 will not provide a bipolar control signal to the second switch K2 and the fourth switch K4; the second switch K2 and the fourth switch K4 remain in a normally closed state when not controlled by the bipolar control signal. At this time, the second switch K2 connects the first switch K1 and the analog integrated front end, and the fourth switch K4 connects the first switch K1 and the analog integrated front end. When the connecting wire of the EEG device is inserted into the connection interface, the connection interface provides a connection control signal to the first switch K1 and the third switch K3; the first switch K1 connects the EEG detection electrode 1+ of the EEG device and the second switch K2 under the control of the connection control signal to connect the EEG detection electrode 1+ to the analog integrated front end, at this time, the EEG detection electrode 1+ provides a positive EEG signal EEG1+ to the analog integrated front end; the third switch K3 connects the EEG detection electrode 1- and the analog integrated front end under the control of the connection control signal, and the EEG detection electrode 1- provides a negative EEG signal EEG1- to the analog integrated front end. When the connection line of the EEG device is not inserted into the connection interface, the connection interface will not provide a connection control signal to the first switch K1 and the third switch K3. The first switch K1 and the third switch K3 remain in a normally closed state when not controlled by the connection control signal. At this time, the first switch K1 turns on the EEG detection point 1+ and the second switch K2 to turn on the EEG detection point 1 and the analog integrated front end, and the EEG detection point 1+ provides a positive EEG signal EEG1+ to the analog integrated front end; the third switch K3 turns on the EEG detection point 1- and the fourth switch K4 to turn on the EEG detection point 1- analog integrated front end, and the EEG detection point 1- provides a negative EEG signal EEG1- to the analog integrated front end. The analog integrated front end can calculate the EEG signal based on the positive EEG signal EEG1+ and the negative EEG signal EEG1-. Specifically, the EEG signal = (positive EEG signal-ground voltage GND)-(negative EEG signal-ground voltage GND). Among them, the negative EEG signal is the reference REF signal.

[0106] like Figure 3As shown, when the connecting line of the EEG device is inserted into the connecting interface, the connecting interface provides a connection control signal to the fifth switch K5 and the sixth switch K6; the fifth switch K5, under the control of the connection control signal, turns on the EEG detection electrode N+ and the analog integrated front end of the EEG device, and at this time, the EEG detection electrode N+ provides a positive EEG signal EEGN+ to the analog integrated front end; the sixth switch K6, under the control of the connection control signal, turns on the EEG detection electrode N- and the analog integrated front end, and the EEG detection electrode N- provides a negative EEG signal EEGN- to the analog integrated front end. When the connection line of the EEG device is not inserted into the connection interface, the connection interface will not provide a connection control signal to the fifth switch K5 and the sixth switch K6. The fifth switch K5 and the sixth switch K6 remain in a normally closed state when not controlled by the connection control signal. At this time, the fifth switch K5 connects the EEG detection point N+ and the analog integrated front end, and the EEG detection point N+ provides a positive EEG signal EEGN+ to the analog integrated front end; the fifth switch K6 connects the EEG detection point N- and the analog integrated front end, and the EEG detection point N- provides a negative EEG signal EEGN- to the analog integrated front end. The analog integrated front end can calculate the EEG signal based on the positive EEG signal EEGN+ and the negative EEG signal EEGN-. Specifically, the EEG signal = (positive EEG signal-ground voltage GND)-(negative EEG signal-ground voltage GND). Among them, the negative EEG signal is the reference REF signal.

[0107] like Figure 3 As shown, the analog integrated front end can calculate n human physiological signals and Nn EEG signals; filter and amplify the n human physiological signals, and perform analog-to-digital conversion on the n human physiological signals after filtering and amplification to generate n digital human physiological signals; filter and amplify the Nn EEG signals, and perform analog-to-digital conversion on the Nn EEG signals after filtering and amplification to generate Nn digital EEG signals.

[0108] like Figure 3 As shown, the control circuit also includes: a main control chip, which is connected to the analog integrated front end. The analog integrated front end can send digital human physiological signals and digital EEG signals to the main control chip. For example, the main control chip can be a microcontroller unit (MCU).

[0109] In the embodiment of the present application, the multimodal data synchronous acquisition system can also realize an offline storage function. Figure 3 As shown, the control circuit also includes: a storage module, which is connected to the main control chip. The main control chip is used to send digital human physiological signals and digital EEG signals to the storage module; the storage module is used to store digital human physiological signals and digital EEG signals.

[0110] As an optional solution, the multimodal data synchronous acquisition system has an offline storage function that can be collected simultaneously with the host computer. In this case, the multimodal data synchronous acquisition system also includes a read-write interface, which is connected to the storage module, and the read-write interface is connected to the host computer through a connecting device. The read-write interface is used to read digital human physiological signals and digital EEG signals from the storage module, and send the digital human physiological signals and digital EEG signals to the host computer through the connecting device, so that the host computer can obtain digital human physiological signals and digital EEG signals from the multimodal data synchronous acquisition system. For example, the connecting device can be a USB data cable, and the read-write interface is a USB interface.

[0111] Specifically, when the host computer starts recording, the offline recording function of the multimodal data synchronous acquisition system is turned on by default, and the user does not need to perform a separate offline recording function turn-on operation. The synchronous acquisition system can record offline data. After the recording of offline data is completed, the user can manually shut down the multimodal data synchronous acquisition system. Among them, the offline data may include digital human physiological signals and digital EEG signals. Thereafter, if the host computer needs to obtain the stored offline data from the multimodal data synchronous acquisition system, the user can connect the connecting device to the read-write interface. For example, if the connecting device is a USB cable, the user can insert the USB interface of the USB cable into the read-write interface. The host computer reads the multimodal data synchronous acquisition system from the storage module through the connecting device and the read-write interface.

[0112] As another optional solution, the multimodal data synchronous acquisition system can realize an offline storage function separated from the host computer. The main control chip is also used to receive an opening instruction input by the user, and execute the step of recording digital human physiological signals and digital EEG signals in response to the opening instruction.

[0113] Disconnection from the host computer means that the host computer and the multimodal data synchronous acquisition system are not connected in communication. In this case, there is no wired communication connection and no wireless communication connection between the host computer and the multimodal data synchronous acquisition system. For example, the user presses the power key of the multimodal data synchronous acquisition system twice in succession to input a start instruction to the multimodal data synchronous acquisition system. At this time, the Bluetooth indicator light corresponding to the power key turns green and flashes, indicating that offline data acquisition has started. The main control chip responds to the start instruction, records digital human physiological signals and digital EEG signals, and sends the digital human physiological signals and digital EEG signals to the storage module; the storage module stores the digital human physiological signals and digital EEG signals.

[0114] As an optional solution, the multimodal data synchronous acquisition system may further include a physiological signal acquisition circuit, which is connected to the main control chip. The physiological signal acquisition circuit is used to collect other physiological signals, for example, other physiological signals may include electrodermal activity (EDA) signals and / or photoplethysmography (PPG) signals. The physiological signal acquisition circuit may send the collected physiological signals to the main control chip. The physiological signal acquisition circuit is not shown in the figure.

[0115] As an optional solution, the multimodal data synchronous acquisition system may further include a power supply circuit, which is used to supply power to the multimodal data synchronous acquisition system. The power supply circuit is not shown in the figure.

[0116] In the technical solution provided in the embodiment of the present application, a multimodal data synchronous acquisition system includes a control circuit, an integrated device and an EEG device. The control circuit includes an analog integrated front end and N signal channel branches connected to the analog integrated front end. The N signal channel branches include M integrated signal channel branches and NM EEG signal channel branches. The analog integrated front end is used to calculate n human physiological signals based on the selected n positive human physiological signals and the corresponding negative human physiological signals, and calculate Nn EEG signals based on the selected Nn positive EEG signals and the corresponding negative EEG signals. The embodiment of the present application improves the diversity and flexibility of multimodal data synchronous acquisition.

[0117] An embodiment of the present application provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the edge computing device where the storage medium is located is controlled to execute an embodiment of the above-mentioned multimodal data synchronization acquisition method.

[0118] An embodiment of the present application provides an edge computing device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the edge computing device, enable the edge computing device to perform the above-mentioned multimodal data synchronization acquisition method.

[0119] Figure 4 A schematic diagram of the structure of an edge computing device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the edge computing device 20 includes: a processor 21, a memory 22, and a computer program 23 stored in the memory 22 and executable on the processor 21. When the computer program 23 is executed by the processor 21, the method for synchronous acquisition of multimodal data applied in the embodiment is implemented. To avoid repetition, they are not described one by one here.

[0120] The edge computing device 20 includes, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that Figure 4 This is only an example of the edge computing device 20 and does not constitute a limitation of the edge computing device 20. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the edge computing device may also include input and output devices, network access devices, buses, etc.

[0121] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0122] The memory 22 may be an internal storage unit of the edge computing device 20, such as a hard disk or memory of the edge computing device 20. The memory 22 may also be an external storage device of the edge computing device 20, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the edge computing device 20. Furthermore, the memory 22 may also include both an internal storage unit of the edge computing device 20 and an external storage device. The memory 22 is used to store computer programs and other programs and data required by the edge computing device 20. The memory 22 may also be used to temporarily store data that has been output or is to be output.

[0123] Figure 5 A schematic diagram of the structure of another edge computing device provided in an embodiment of the present application, such as Figure 5 As shown, in Figure 4 Based on the embodiment shown, the edge computing device 20 may further include a multimodal data synchronous acquisition device 24, which is connected to the processor 21. The multimodal data synchronous acquisition device 24 may be a Figure 1 The multimodal data synchronous acquisition device shown will not be described in detail here.

[0124] Figure 6A schematic diagram of the structure of another edge computing device provided in an embodiment of the present application, such as Figure 6 As shown, in Figure 4 Based on the embodiment shown, the edge computing device 20 may further include a multimodal data synchronous acquisition system 25, which is connected to the processor 21. The multimodal data synchronous acquisition system 25 may be configured as follows: Figure 3 The multi-modal data synchronous acquisition system shown will not be described in detail here.

[0125] As an optional solution, the edge computing device 20 may also include but is not limited to mobile terminals such as head-mounted wearable devices, mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The edge computing device 20 shown is only an example and should not bring any limitations to the functions and occupancy range of the embodiments of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0126] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0129] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (Processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0130] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multimodal data synchronous acquisition device, characterized in that: include: An integrated device and an EEG device, wherein the integrated device comprises a connection interface, a plurality of EEG detection points and a plurality of human body detection points, wherein the connection interface is used to connect the EEG device, and the EEG device comprises an EEG cap, an EEG helmet or an EEG headband; The n human body detection points are used to detect n channels of human body physiological signals when in an access state; The EEG device is used to detect EEG signals of Nn channels when the EEG device is connected to the integrated device through the connection interface; or the Nn EEG detection points are used to detect EEG signals of Nn channels when the EEG device is not connected to the integrated device through the connection interface; N and n are both positive integers, and N is greater than n.

2. The device according to claim 1, characterized in that The EEG device is used to detect EEG signals of N channels when the integrated device is connected via the connection interface and the human body detection point is not in an access state; or, the N EEG detection points are used to detect EEG signals of N channels when the EEG device is not connected to the integrated device and the human body detection point is not in an access state.

3. The device according to claim 1, characterized in that The device further comprises: a first indicator light arranged corresponding to the connection interface and a second indicator light arranged corresponding to each of the human body detection points; The first indicator light is used to light up when the EEG device is connected to the integrated device through the connection interface; The second indicator light is used to light up when the corresponding human body detection point is in a connected state.

4. The device according to claim 1, characterized in that The human body detection points include two electrode points.

5. The device according to any one of claims 1 to 4, characterized in that: The human physiological signal includes an electrocardiogram signal, an electrooculogram signal or an electromyography signal.

6. A multimodal data synchronous acquisition system, characterized in that: The system includes a control circuit, an integrated device and an EEG device, wherein the integrated device includes a connection interface, a plurality of EEG detection points and a plurality of human body detection points, wherein the connection interface is used to connect the EEG device, and the EEG device includes an EEG cap, an EEG helmet or an EEG headband; The control circuit comprises: an analog integrated front end and N signal channel branches connected to the analog integrated front end, the N signal channel branches comprise M integrated signal channel branches and NM EEG signal channel branches, M and N are both positive integers and N is greater than M; The integrated signal channel branch is used to select the positive human physiological signal and the corresponding negative human physiological signal detected by the human detection point, the positive EEG signal and the corresponding negative EEG signal detected by the EEG device, or the positive EEG signal and the corresponding negative EEG signal detected by the EEG device; The EEG signal channel branch is used to select the positive EEG signal detected by the EEG device and the corresponding negative EEG signal or the positive EEG signal detected by the EEG device and the corresponding negative EEG signal; The analog integrated front end is used to calculate n human physiological signals based on the selected n positive human physiological signals and the corresponding negative human physiological signals, and to calculate Nn EEG signals based on Nn positive EEG signals and the corresponding negative EEG signals, where n is a positive integer and N is greater than n.

7. The system according to claim 6, characterized in that The integrated signal channel branch includes a first integrated signal channel branch and a second integrated signal channel branch, and the EEG signal channel branch includes a first EEG signal channel branch and a second EEG signal channel branch; The first integrated signal channel branch is used to select the positive human physiological signal detected by the human body detection point, the positive EEG signal detected by the EEG device, or the positive EEG signal detected by the EEG detection point; The second integrated signal channel branch is used to select the negative human physiological signal detected by the human body detection point, the negative EEG signal detected by the EEG device, or the negative EEG signal detected by the EEG detection point; The first EEG signal channel branch is used to select the positive EEG signal detected by the EEG device or the positive EEG signal detected by the EEG detection point; The second EEG signal channel branch is used to select the negative EEG signal detected by the EEG device or the negative EEG signal detected by the EEG detection point.

8. The system according to claim 7, characterized in that The EEG device includes a plurality of EEG detection electrodes, and the human body detection points include a first human body detection point and a second human body detection point; The first integrated signal channel branch includes a first switch and a second switch, the first input end of the first switch is connected to the corresponding EEG detection point, the second input end of the first switch is connected to the corresponding EEG detection electrode, the output end of the first switch is connected to the first input end of the second switch, the control end of the first switch is connected to the connection interface, the second input end of the second switch is connected to the corresponding first human body detection point, the output end of the second switch is connected to the analog integrated front end, and the control end of the second switch is connected to the corresponding human body detection point; The second integrated signal channel branch includes a third switch and a fourth switch, the first input end of the third switch is connected to the corresponding EEG detection point, the second input end of the third switch is connected to the corresponding EEG detection electrode, the output end of the third switch is connected to the first input end of the fourth switch, the control end of the third switch is connected to the connection interface, the second input end of the fourth switch is connected to the corresponding second human body detection point, the output end of the fourth switch is connected to the analog integrated front end, and the control end of the fourth switch is connected to the corresponding human body detection point.

9. The system according to claim 7, characterized in that The EEG device includes a plurality of EEG detection electrodes; The first EEG signal channel branch includes a fifth switch, a first input end of the fifth switch is connected to a corresponding EEG detection point, a second input end of the fifth switch is connected to a corresponding EEG detection electrode, an output end of the fifth switch is connected to the analog integrated front end, and a control end of the fifth switch is connected to the connection interface; The second EEG signal channel branch includes a sixth switch, a first input end of the sixth switch is connected to a corresponding EEG detection point, a second input end of the sixth switch is connected to a corresponding EEG detection electrode, an output end of the sixth switch is connected to the analog integrated front end, and a control end of the sixth switch is connected to the connection interface.

10. A method for synchronously collecting multimodal data, characterized in that: The method is based on the multimodal data synchronous acquisition device as described in any one of claims 1 to 5, or the multimodal data synchronous acquisition system as described in any one of claims 6 to 9; The method comprises: Detecting whether the plurality of human body detection points are in an access state; If it is detected that n human body detection points are in a connected state, detecting whether the EEG device is connected to the integrated device through the connection interface; If it is detected that the EEG device is connected to the integrated device via the connection interface, human physiological signals of n channels are detected via the n human detection points and EEG signals of Nn channels are detected via the EEG device; If it is detected that the EEG device is not connected to the integrated device through the connection interface, detecting human physiological signals of n channels through the n human body detection points and detecting EEG signals of Nn channels through Nn EEG detection points; N and n are both positive integers, and N is greater than n.

11. The method according to claim 10, characterized in that The method further comprises: If it is detected that the plurality of human body detection points are not in a connected state, detecting whether the EEG device is connected to the integrated device through the connection interface; If it is detected that the EEG device is connected to the integrated device via the connection interface, an N-channel EEG signal is detected via the EEG device; If it is detected that the electroencephalogram device is not connected to the integrated device through the connection interface, electroencephalogram signals of N channels are detected through the N electroencephalogram detection points.

12. An edge computing device, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the edge computing device, enable the edge computing device to perform the multimodal data synchronous acquisition method according to claim 10 or 11; and / or, The edge computing device includes: a multimodal data synchronous acquisition device as described in any one of claims 1 to 5, or a multimodal data synchronous acquisition system as described in any one of claims 6 to 9.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the edge computing device where the computer-readable storage medium is located is controlled to execute the multimodal data synchronous acquisition method according to claim 10 or 11.

Images