Portable multi-modal data synchronization device, sensor and system

Abstract

The embodiment of the invention provides a portable multi-modal data synchronization device, a sensor and a system.The portable multi-modal data synchronization device can comprise a control module, an operation module and a communication module, and the control module can be coupled with the operation module and the communication module; the operation module can be used for sending instructions to the control module; the control module can be used for responding to the instruction and issuing an event mark; the communication module can be used for being connected with the at least two sensors and sending event marks to the at least two sensors. According to the embodiment of the invention, the control module responds to the instruction of the operation module and sends the event mark, and the event mark is sent to the at least two sensors through the communication module, so that the at least two sensors can store the event mark under the condition of offline detection for subsequent data synchronization; the problem that time synchronization of multi-modal data collected by different sensors in an off-line state cannot be achieved is solved.

Description

Technical Field

[0001] The present application relates to human factors engineering and ergonomics technology, and in particular to a portable multimodal data synchronization device, sensor and system. Background Art

[0002] Human factors engineering and ergonomics mainly study the relationship between humans, machines, and the environment, and are widely used in health care, sports rehabilitation, intelligent driving, etc. When studying the relationship between humans, machines, and the environment, human factors engineering needs to collect multimodal data related to humans, machines, and the environment to conduct relevant test analysis such as human factors or ergonomics.

[0003] Usually, the sensor used to detect and obtain multimodal data needs to be connected to a data acquisition terminal (such as a smart phone, a computer, a data logging terminal (datalog), a mobile phone, a tablet computer, a portable computer, etc.) before starting to work. After that, the sensor starts to detect, and during the detection process, the multimodal data obtained by the detection is sent to the data acquisition terminal in real time.

[0004] However, for human factors engineering experiments conducted in some environments such as outdoor environments, sensors are required to collect multimodal data offline, which makes the data collected offline by different sensors face the problem of subsequent inability to synchronize. Summary of the invention

[0005] The embodiments of the present application provide a portable multimodal data synchronization device, sensor, and system, which help solve the problem that data collected offline by different sensors cannot be subsequently synchronized.

[0006] In a first aspect, an embodiment of the present application provides a portable multimodal data synchronization device which may include: a control module, an operation module and a communication module, wherein the control module may be coupled to the operation module and the communication module respectively; the operation module may be used to issue instructions to the control module; the control module may be used to issue event markers in response to the instructions; and the communication module may be used to connect to at least two sensors and issue event markers to the at least two sensors.

[0007] In the technical solution provided in the embodiment of the present application, the portable multimodal data synchronization device sends an event mark to at least two sensors through a communication module. In this way, when the at least two sensors detect multimodal data offline, they can both receive the first type of data and store the detected multimodal data in sequence. In this way, when the subsequent data acquisition terminal reads data from the at least two sensors, it can read the multimodal data and the event mark at the same time, so that the multimodal data detected by the at least two sensors can be synchronized through the event mark.

[0008] In a possible implementation, the operation module may include a start module, and the start module may be used to issue a first instruction to the control module;

[0009] The control module may also be configured to send a first event marker to the at least two sensors via the communication module in response to a first instruction.

[0010] Exemplarily, the first event marker may be one or more. When the portable multimodal data synchronization device sends multiple first event markers to at least two sensors, the possibility that at least two sensors can receive the first event marker can be increased. Exemplarily, the first event marker may be any one or more letters such as a, b, c, d, etc., or any one or more numbers, or a number such as a1, a2, ..., or n1, n2, ..., or a, b, c, d, ..., or a serial number such as 1, 2, 3, ..., and so on.

[0011] In one possible implementation, the control module may be used to issue a first event marker in response to a first instruction, which may include: the control module may be used to issue a first event marker at intervals of a first predetermined time period in response to the first instruction; the communication module may also be used to issue a first event marker to the at least two sensors at intervals of the first predetermined time period.

[0012] In one possible implementation, the startup module may also be used to issue a second instruction to the control module; the control module may also be used to stop issuing the first event mark in response to the second instruction; the communication module may also be used to stop sending the first event mark to the at least two sensors when the control module stops issuing the first event mark.

[0013] In a possible implementation, the control module may also be used to start timing after issuing the first event marker, and stop issuing the first event marker when the timing reaches a second predetermined time period, wherein the second predetermined time period is greater than the first predetermined time period; the communication module may also be used to stop sending the first event marker to the at least two sensors when the control module stops issuing the first event marker.

[0014] In a possible implementation, the device may further include an event triggering module, which may be coupled to the control module and used to send a third instruction to the control module; the control module may also be used to send a second event marker to the at least two sensors through the communication module in response to the third instruction, and the second event marker may include event information, such as increased heart rate, panic, dizziness, headache, too strong light, low temperature, etc. Exemplarily, the control module may preset a correspondence between the third event instruction and the event information, and when the third instruction is received, a second event marker including the event information is generated accordingly.

[0015] Exemplarily, the event trigger module may send different instructions to the control module in response to different operations of the user, so as to correspond to different event information. The control module may pre-store the mapping relationship between the sixth instruction, the seventh instruction, and the eighth instruction and the light is too strong, the headache, and the temperature is too low. The event trigger module may preset the relationship between the first operation, the second operation, and the third operation of the user and the sixth instruction, the seventh instruction, and the eighth instruction. When the light is too strong, the event trigger module sends the sixth instruction to the control module in response to the first operation of the user, and the control module responds to the instruction and issues a second event mark containing the light is too strong according to the mapping relationship; when the subject has a headache, the event trigger module sends the seventh instruction to the control module in response to the second operation of the user, and the control module responds to the instruction and issues a second event mark containing the headache according to the mapping relationship; when the temperature is too low, the event trigger module sends the eighth instruction to the control module in response to the third operation of the user, and the control module responds to the instruction and issues a second event mark containing the temperature is too low according to the mapping relationship; and so on.

[0016] In a possible implementation, the event triggering module may be further configured to send a fourth instruction to the control module; and the control module may be further configured to control the communication module to establish a connection with the at least two sensors in response to the fourth instruction.

[0017] In a possible implementation, the above-mentioned device may also include a voice input module and an artificial intelligence AI module, the voice input module is coupled to the AI ​​module, and the AI ​​module may be coupled to the above-mentioned control module; the voice input module is used to receive voice, and the AI ​​module is used to recognize voice, and when the recognition result matches the target voice, a fifth instruction is issued to the above-mentioned control module; the above-mentioned control module may also be used to respond to the fifth instruction and send a second event marker to the above-mentioned at least two sensors through the above-mentioned communication module, and the second event marker contains event information.

[0018] In a possible implementation, the communication module may be a wireless communication module.

[0019] In a possible implementation manner, the above-mentioned device may further include a storage module, which is coupled to the control module and is used to store the first event mark.

[0020] In a possible implementation, the wireless communication module may be a Bluetooth module or a Wireless Fidelity (WiFi) module.

[0021] In a possible implementation, the device may further include a power module, which may be used to supply power to the control module, the start-up module, the communication module, and other modules. The start-up module may also be used to switch on or off the power supply line between the power module and other modules to control the on / off of the device.

[0022] In a possible implementation, the device may further include a power switch for connecting or disconnecting a power supply line between the power module and other modules to control the switching of the device.

[0023] In one possible implementation, the above-mentioned device may include a power supply port, which can be connected to other electronic devices (such as mobile phones, etc.) through an adapted power cord to obtain electrical energy from other electronic devices to power the above-mentioned control module, starting module, communication module and other modules.

[0024] In a second aspect, an embodiment of the present application provides a sensor, which may include the portable multimodal data synchronization device described in any one of the first aspects above.

[0025] In a third aspect, an embodiment of the present application provides a sensor, which may include a processing module, a detection module, a first communication module, a second communication module and a storage module. The processing module may be coupled to the detection module, the first communication module, the second communication module and the storage module respectively. The first communication module may be used to couple to a data acquisition terminal, and the second communication module may be used to couple to a portable multimodal data synchronization device or a sensor. The detection module may be used to detect a subject and obtain multimodal data. The processing module may be used to send multimodal data to the data acquisition terminal in real time through the first communication module when the first communication module is coupled to the data acquisition terminal. It may also be used to store multimodal data through the storage module when the first communication module is disconnected from the data acquisition terminal, and receive a first event mark sent by the portable multimodal data synchronization device or the sensor through the second communication module, and store the first event mark through the storage module. The first event mark is described in detail in the first aspect above.

[0026] In a possible implementation, the processing module may also be used to receive a second event marker through the second communication module and store the second event marker through the storage module, wherein the second event marker includes event information. The second event marker may be described in detail in the first aspect.

[0027] In a possible implementation manner, the second communication module and the first communication module are the same communication module. For example, the functions of the second communication module and the first communication module are implemented by the same module.

[0028] In a possible implementation, the second communication module may be a wireless communication module.

[0029] In a possible implementation, the wireless communication module may be a Bluetooth module or a WiFi module.

[0030] In a fourth aspect, an embodiment of the present application provides a portable multimodal data synchronization system, which may include a portable multimodal data synchronization device of any one of the first aspect above, and at least two sensors described in any one of the third aspect above, and the portable multimodal data synchronization device is used to send an event marker and a second event marker to at least two sensors.

[0031] The portable multimodal data synchronization device, sensor and system provided in the embodiment of the present application, respond to the instruction of the start module through the control module, issue an event mark, and send it to the sensor through the communication module, so that the sensor can store the event mark in the case of offline detection for subsequent data synchronization, for example, the data acquisition terminal can align the multimodal data collected by different sensors based on the stored event mark, realize multimodal data synchronization, and help solve the problem that the data collected by the sensor in the offline state cannot be synchronized. Further, when the communication module is a wireless communication module, it is more convenient for the testee to wear or carry. Further, the portable multimodal data synchronization device, sensor and system provided in the embodiment of the present application, through the event trigger module and / or voice input module and AI module, can also perform event marking in the outdoor situation, which helps to obtain more effective multimodal data and improve the accuracy of subsequent data synchronization. Further, the portable multimodal data synchronization device and sensor provided in the embodiment of the present application can store event marks through the storage module, which helps to further improve the accuracy of subsequent data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A data collection schematic diagram provided for an embodiment of the present application;

[0034] Figure 2An embodiment of a sensor provided in an embodiment of the present application;

[0035] Figure 3 An embodiment of a portable multimodal data synchronization device provided in an embodiment of the present application;

[0036] Figure 4a Another embodiment of the portable multimodal data synchronization device provided by the embodiment of the present application;

[0037] Figure 4b An embodiment of a portable multimodal data synchronization system provided by an embodiment of the present application;

[0038] Figure 4c Another embodiment of the portable multimodal data synchronization system provided by the embodiment of the present application;

[0039] Figure 5 Another embodiment of the portable multimodal data synchronization device provided by the embodiment of the present application;

[0040] Figure 6 Another embodiment of the portable multimodal data synchronization device provided by the embodiment of the present application;

[0041] Figure 7 Another embodiment of the portable multimodal data synchronization device provided by the embodiment of the present application;

[0042] Figure 8 An example of an application scenario of the portable multimodal data synchronization system provided in an embodiment of the present application;

[0043] Fig. 9 An embodiment of a sensor provided in an embodiment of the present application;

[0044] Fig.10 Another application scenario example of the portable multimodal data synchronization system provided in the embodiment of the present application;

[0045] Fig.11 An example of data synchronization provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0047] like Figure 1As shown in the figure, in the indoor experimental environment, the sensor sends the multimodal data obtained by the detection to the data acquisition terminal in real time during the detection process. The data acquisition terminal receives the multimodal data through the wireless communication module and synchronizes it to the supporting software for related test analysis such as human factors or ergonomics. When the wireless communication module receives the multimodal data, the data acquisition terminal can mark the time for the multimodal data to achieve data synchronization.

[0048] The multimodal data may include environment-related data and / or human-related data. The environment-related data may include any one or more of location data, humidity data, temperature data, chromaticity data, brightness data, noise data, weather data (such as air pressure), road condition data, traffic data, stimulus signal data, and event or signal data. Human body related data may include skin electrical data, skin temperature (SKT) data, pulse data (such as photoplethysmography (PPG)), blood pressure (BP) data, blood oxygen data (such as periphery capillary oxygen saturation (SPO2)), electrocardiogram data, electromyography data (such as electrodermal activity (EDA)), muscle oxygen data, respiratory data (such as respiratory rate (RESP)), biomechanical data (such as single point force-sensitive resistor (FSR) value), near-infrared brain imaging data, electroencephalogram data, transcranial stimulation data, heart rate variability data, heart rate data and behavioral data.

[0049] Among them, sensors may include environmental sensors, biosensors (Biosensors, BIO), electromyography sensors (Electromyography sensor, EMG), electrocardiogram sensors (Electrocardiogram sensor, ECG) and electroencephalogram sensors (Electroencephalogram sensor, EEG). Environmental sensors can be used to collect environment-related data. BIO can be used to collect biological data such as PPG, EDA, SPO2, SKT, RESP, BP, FSR, etc. EMG can be used to collect electromyography data, ECG can be used to collect electrocardiogram data, and EEG can be used to collect electroencephalogram data.

[0050] One embodiment of the sensor is Figure 2As shown, it includes: a processing module, a detection module, a first communication module and a storage module, and the processing module is coupled to the detection module, the first communication module and the storage module respectively. The detection module can be used to detect and obtain multimodal data. The processing module can be used to send the detected multimodal data to the data acquisition terminal in real time through the first communication module. The first communication module can be a wireless communication module, which can be used to send multimodal data to the data acquisition terminal in real time by wireless means. The first communication module can also be a wired communication module, which is used to send multimodal data to the data acquisition terminal in real time by wire. Exemplarily, the wired communication module may include a cable interface. In some embodiments, when the sensor detects multimodal data offline, for example, the first communication module is disconnected from the data acquisition terminal and cannot send multimodal data to the data acquisition terminal in real time, the above-mentioned processing module can detect the obtained multimodal data through the storage module.

[0051] In order to synchronize the multimodal data stored by at least two sensors when they are offline in a data acquisition terminal, an embodiment of the present application provides a portable multimodal data synchronization device to send tags to the sensors.

[0052] Correspondingly, an embodiment of the present application also provides a sensor for connecting to a portable multimodal data synchronization device, receiving a mark sent by the portable multimodal data synchronization device through the connection, and storing the received mark. Exemplarily, the sensor may be based on the sensor provided in the above embodiment, and may add a second communication module for communicating with the portable multimodal data synchronization device to receive the mark sent by the portable multimodal data synchronization device, and the storage module may be further used to store the mark sent by the portable multimodal data synchronization device. Exemplarily, the second communication module for communicating with the portable multimodal data synchronization device may reuse the first communication module of the sensor. Exemplarily, the second communication module may be a wired communication module or a wireless communication module, such as a Bluetooth module (such as Figure 4c As shown), WiFi module, etc.

[0053] A portable multimodal data synchronization device provided by an embodiment of the present application may include: a control module, an operation module and a communication module, wherein the control module may be coupled to the operation module and the communication module respectively; the operation module may be used to issue instructions to the control module; the control module may be used to issue an event mark in response to the instruction; the communication module may be used to connect to at least two sensors and issue an event mark to the at least two sensors. Exemplarily, the operation module may include any one or more of a start module, an event trigger module and a voice input module.

[0054] An embodiment of a portable multimodal data synchronization device is Figure 3 The portable multimodal data synchronization device 101 is shown. Figure 3As shown, the portable multimodal data synchronization device 101 may include a startup module 110, a control module 120 and a communication module 130. The control module 120 is coupled to the startup module 110 and the communication module 130. The startup module 110 may be used to send a first instruction to the control module 120 under the operation of a user (such as a test subject), so as to trigger the control module 120 to issue a first event mark. The first event mark may be described in detail in the above description.

[0055] The communication module 130 can be used to connect to the sensor and send the first event mark to the connected sensor. Exemplarily, the start module 110 can be a button or a switch. The user's operation can be a single click. In the embodiment of the present application, the portable multimodal data synchronization device can be carried and used by the testee, and the sensor receiving the mark can be a sensor worn by the testee, that is, a sensor for detecting the testee.

[0056] Exemplarily, the control module 120 may issue a first event marker in response to the first instruction, or issue a first event marker at intervals of a first predetermined duration. Exemplarily, in the case where the control module 120 issues a first event marker at intervals of a first predetermined duration, for example, the control module 120 issues multiple first event markers in sequence, and the multiple first event markers sent in sequence may be numbers or serial numbers that increase in the order of sending. For example, the control module 120 initially issues a first event marker with a number or serial number of 1, and issues another first event marker with a number or serial number of 2 at an interval of 100ms, and issues another first event marker with a number or serial number of 3 at an interval of 100ms, ..., and issues a first event marker with a number or serial number of n at an interval of 100ms, and so on. The portable multimodal data synchronization device sends multiple first event markers, which can avoid the problem of the sensor not being successfully received when only one first event marker is sent, and ensure that at least two sensors successfully receive the first event marker.

[0057] In some embodiments, when the control module 120 issues a first event marker every first predetermined time interval in response to the first instruction, the control module 120 may also be used to stop issuing the first event marker when the time after the first first event marker is issued reaches a second predetermined time. Exemplarily, the second predetermined time (e.g., 2s) is greater than the first predetermined time (e.g., 100ms). Alternatively, the start module 110 may also be used to send a second instruction to the control module 120 under the operation of the user, and the control module 120 may also be used to stop issuing the first event marker in response to the second instruction. Exemplarily, the user's operation may be a single click, a double click, etc.

[0058] In some embodiments, the start module may be further configured to send a second instruction to the control module 120 under the operation of the user, and the control module 120 may be further configured to stop sending marks in the first mark group in response to the second instruction.

[0059] Exemplarily, the functions of the control module 120 may be implemented by a field programmable gate array (FPGA), a microcontroller unit (MCU), a central processing unit (CPU), or the like.

[0060] The communication module 130 can be used to couple at least two sensors and send the first event marker or the first event marker group issued by the control module 120 to the at least two sensors. Exemplarily, the communication module 130 can be a wireless communication module, such as a Bluetooth communication module, a wireless fidelity (WiFi) communication module, etc., which can be used to wirelessly connect the sensor and send the first event marker to the sensor in a wireless manner. Exemplarily, the communication module 130 can be a wired communication module, which is used to wiredly connect at least two sensors and send the first event marker to the at least two sensors in a wired manner.

[0061] In some embodiments, the portable multimodal data synchronization device may include two communication modules, one of which may be a wired communication module for coupling with a sensor whose second communication module is a wired communication module, and the other communication module may be a wireless communication module for coupling with a sensor whose second communication module is a wireless communication module. In this way, the portable multimodal data synchronization device can send tags to different sensors in both wired and wireless ways. Exemplarily, when the communication module in the portable multimodal data synchronization device is a Bluetooth module, correspondingly, the second communication module in at least two sensors for receiving tags sent by the portable multimodal data synchronization device also needs to be a Bluetooth module. Alternatively, when the communication module in the portable multimodal data synchronization device is a WiFi module, the second communication module in at least two sensors for receiving tags sent by the portable multimodal data synchronization device may be a WiFi module; and so on.

[0062] Another embodiment of the portable multimodal data synchronization device is Figure 4a The portable multimodal data synchronization device 102 is shown. Figure 4a As shown, the portable multimodal data synchronization device 102 is different from the above-mentioned embodiment in that it includes a wireless communication module 131 and also includes an event triggering module 140 .

[0063] The event trigger module 140 can be used to send a third instruction to the control module 120 under the operation of the user (test subject). The control module 120 is also used to respond to the third instruction and issue a second event mark. The second event mark may include event information, which can be detailed in the above description. For example, when the user (test subject) has a special event such as a sudden increase in heartbeat, a headache, etc., the event trigger module 140 can be operated (such as a single click or double click) to issue a third instruction to instruct the control module 120 to issue a second event mark. Exemplarily, the event trigger module 140 can be a component such as a button or a switch.

[0064] In this embodiment, the control module 120 may also be configured to control the wireless communication module 131 to establish a wireless connection with the sensor in response to the first instruction, so as to transmit the mark.

[0065] The wireless communication module 131 can be used to wirelessly connect to the sensor and send the first event marker and / or the second event marker to the connected sensor wirelessly. Figure 4b As shown, the portable multimodal data synchronization device 102 multicasts the first event marker and / or the second event marker to multiple sensors through the wireless communication module 131.

[0066] In some embodiments, the event trigger module 140 may also be used to send a fourth instruction to the control module 120 to control the wireless communication module 131 to establish a wireless connection with the sensor. The control module 120 may also be used to control the wireless communication module 131 to establish a wireless connection with the sensor in response to the fourth instruction to transmit the tag.

[0067] Another embodiment of the portable multimodal data synchronization device is Figure 4c The portable multimodal data synchronization device 103 shown. Figure 4c As shown, the portable multimodal data synchronization device 103 is different from the above embodiment in that the module for the portable multimodal data synchronization device 103 to communicate with the sensor is a Bluetooth module 132. Accordingly, the second communication module in the sensor 1 to the sensor n is a Bluetooth module to receive the mark sent by the portable multimodal data synchronization device 103 via Bluetooth. Exemplarily, the mark may be the above-mentioned first event mark and / or the second event mark.

[0068] During use, the user (or the person being tested) can send a first instruction to the control module 120 by clicking the start module 110 , and the control module 120 controls the Bluetooth module 132 to establish Bluetooth connections with sensors 1 to n in response to the first instruction.

[0069] When the Bluetooth module 132 successfully establishes a connection with the Bluetooth modules in sensors 1 to n, the Bluetooth modules in sensors 1 to n are in a scanning state, ready to receive broadcast or multicast information from the portable multimodal data synchronization device 103 at any time.

[0070] After the control module 120 sends the first event mark to the Bluetooth module 132 in response to the first instruction, the Bluetooth module 132 sends a data packet containing the first event mark to sensors 1 to n through the established Bluetooth connection. Sensors 1 to n receive the data packet containing the first event mark through the Bluetooth modules of each group, unpack the data packet through their respective processing modules to obtain the first event mark, and store the first event mark in their respective storage modules. In some embodiments, the priority of the sensor receiving the data packet from the portable multimodal data synchronization device is higher than the priority of operations such as detecting multimodal data or storing multimodal data.

[0071] The user sends a second instruction to the control module 120 by double-clicking the start module 110. In response to the second instruction, the control module 120 stops sending the first event mark and shuts down the Bluetooth module 132 to reduce power consumption. When the user encounters the above-mentioned special event, he can click the event trigger module 140 to send a third instruction to the control module 120. The control module 120 can respond to the third instruction to control the Bluetooth module 132 to establish a Bluetooth connection with sensors 1 to n. After the connection is established, the control module 120 sends a second event mark to the Bluetooth module 132. The Bluetooth module 132 sends the second event mark to sensors 1 to n through the established Bluetooth connection. Sensors 1 to n receive the second event mark through their respective Bluetooth modules and store them in their respective storage modules.

[0072] In some embodiments, the control module 120 may only stop issuing the first event mark in response to the second instruction without shutting down the Bluetooth module 132. In this way, the control module 120 may only issue the second event mark in response to the third instruction, which can improve the reaction speed and accuracy of the second event mark.

[0073] Another embodiment of the portable multimodal data synchronization device is Figure 5 The portable multimodal data synchronization device 104 is shown. Figure 5 As shown, the portable multimodal data synchronization device 104 is different from the above embodiment in that it further includes a storage module 150 for storing the above first event marker and / or second event marker as a backup to ensure the accuracy of subsequent data synchronization.

[0074] Another embodiment of the portable multimodal data synchronization device is Figure 6 The portable multimodal data synchronization device 105 shown. Figure 6As shown, the portable multimodal data synchronization device 105 is different from the above embodiments in that it further includes an artificial intelligence (AI) module 160 and a voice input module 170. The voice input module 170 is coupled to the control module 120 via the AI ​​module 160.

[0075] The voice input module 170 may be used to convert voice into electrical signals, and the AI ​​module 160 may be used to perform voice recognition on the electrical signals, and when the recognition result matches the target voice, a fifth instruction is issued to the control module 120 .

[0076] When the wireless communication module 131 is in working state and establishes a wireless connection with at least two sensors, the control module 120 may send a second event mark to the wireless communication module 131 in response to the fifth instruction. When the wireless communication module 131 stops working, the control module 120 may first control the wireless communication module 131 to establish a wireless connection with the sensor in response to the fifth instruction. After the connection is established, the control module 120 sends a second event mark to the wireless communication module 131. The wireless communication module 131 sends the second event mark to the at least two sensors through the established wireless connection.

[0077] In some embodiments, the portable multimodal data synchronization device may include an event trigger module, an AI module, a voice input module, and a storage module. Figure 7 A portable multimodal data synchronization device 106 is shown.

[0078] In some embodiments, the portable multimodal data synchronization device 106 may further include a housing to protect the portable multimodal data synchronization device 106. For example, the parts of the startup module and the event triggering module for user operation may be through the housing for user operation. For example, the part of the housing corresponding to the voice input module may be provided with a through hole so that the voice input module can receive clearer voice.

[0079] In some embodiments, the portable multimodal data synchronization device may include more or fewer modules than the above embodiments. For example, the portable multimodal data synchronization device may further include a power module, or may further include a power switch, or may further include a power port.

[0080] In some embodiments, when the user turns on the power switch, the wireless communication module in the portable multimodal data synchronization device can be turned on at the same time. When the sensor worn by the test subject is turned on, the wireless communication module can automatically establish a wireless connection with the sensor worn by the test subject after being turned on.

[0081] In some embodiments, when a user inserts an adapted power cable into the power port, the portable multimodal data synchronization device can be powered on.

[0082] An example of a usage scenario of the portable multimodal data synchronization device provided in the above embodiment is as follows: Figure 8 As shown, the test subject conducts a simple human factors engineering experiment while riding outdoors. Six EMGs (three EMGs 201, EMG 202 and EMG 203 are shown in the figure) and a chest-worn ECG 204 can be attached to the body, and the portable multimodal data synchronization device 100 can be fixed on the bicycle (the portable multimodal data synchronization device 100 can also be carried on the user, such as in the test subject's pocket, etc., which is not limited here).

[0083] When the subject is conducting the experiment while riding a bicycle, the seven sensors on the body can be turned on first to put the seven sensors into wireless broadcasting state.

[0084] Then, the testee can turn on the power switch of the portable multimodal data synchronization device 100, turn on the portable multimodal data synchronization device 100, double-click the event trigger button on the portable multimodal data synchronization device 100, and wirelessly connect to the seven sensors. The testee can understand the connection status through the status of the light emitting diodes (LEDs) on the seven sensors.

[0085] After the portable multimodal data synchronization device 100 is successfully connected to the seven sensors, the experiment begins, and the test subject can click the start button on the portable multimodal data synchronization device 100, and the portable multimodal data synchronization device 100 sends the first event mark to the seven sensors through the wireless connection with the seven sensors. The seven sensors receive the first event mark and store it in the storage module.

[0086] If the testee encounters the above special event during riding, he can click the event trigger button to trigger the portable multimodal data synchronization device 100 to issue a second event mark, and send it to the seven sensors through wireless connection. If the testee is not convenient to operate the event trigger button, he can say the above special event (such as words recorded in the physical examination) to trigger the portable multimodal data synchronization device 100 to issue a second event mark through voice. The seven sensors receive the first event mark and store it in the storage module.

[0087] After completing the experimental operation, the subject can double-click the start button to end the experiment. The subject can further turn off the portable multimodal data synchronization device 100 and the seven sensors.

[0088] A sensor provided in an embodiment of the present application may also include the functions of any portable multimodal data synchronization device provided in the above embodiments.

[0089] Another embodiment of the sensor is Fig. 9 The sensor 301 is shown. Fig. 9 As shown, the sensor 301 may include: a storage module 310, a detection module 320, a first communication module 330, a processing module 340, a start module 110, a control module 120, a wireless communication module 131, an event triggering module 140, an AI module 160 and a voice input module 170. Among them, the control module 120 is coupled to the storage module 310, and the control module 120 is used to store the first event mark and / or the second event mark sent in the storage module 310. The processing module 340 can be used to send the multimodal data detected by the detection module through the first communication module 330, and can also be used to store the multimodal data through the storage module 310.

[0090] The sensor provided in the embodiment of the present application has both the functions of the sensor itself and the event marking function, and can save storage modules and communication resources while integrating the functions of a portable multimodal data synchronization device, and can also save space, making the sensor more portable.

[0091] In some embodiments, the processing module 340 may reuse the control module 120, or the control module 120 may reuse the processing module 340. Alternatively, for example, the functions of the processing module 340 and the control module 120 may be simultaneously implemented by one functional module to further reduce resource loss, save space, and further improve portability.

[0092] Exemplarily, the storage module in the above-mentioned portable multimodal data synchronization device embodiment and sensor embodiment can be a solid state disk (SSD), a secure digital memory (SD) card, and the like.

[0093] Exemplarily, the sensor may be a relatively large sensor such as EEG, functional Near-Infrared Spectroscopy (Fnirs), etc.

[0094] An example of a usage scenario of the sensor provided in the above embodiment is as follows: Fig.10 As shown, Figure 8The difference of the illustrated embodiment is that the sensor EEG 302 is used to send the first event marker and / or the second event marker to the seven sensors, namely, ECG 204, EMG 201, 202 and 203. The wireless communication module in the sensor EEG 302 may be a Bluetooth module or a WiFi module, and the working principle of the event marker is similar to that of the above-mentioned portable multimodal data synchronization device embodiment, except that the sensor EEG 302 may also detect and store multimodal data.

[0095] The portable multimodal data synchronization system provided in the embodiment of the present application may include any portable multimodal data synchronization device provided in the above embodiment and at least two sensors, such as Figure 4b , Figure 4c and Figure 8 shown.

[0096] In some embodiments, the portable multimodal data synchronization system provided by the embodiments of the present application may include at least one sensor provided by the above embodiments and a sensor having the function of a portable multimodal data synchronization device, such as Fig.10 shown.

[0097] During the subsequent data analysis process, the data acquisition terminal can obtain multimodal data detected by the sensor, as well as the first event marker and / or the second event marker by reading data stored in the sensor, or the sensor and the storage module in the sensor having event marking function and sending function.

[0098] Afterwards, the data acquisition terminal or the corresponding software installed therein, based on the multimodal data and the first event marker and / or the second event marker, uses the first event marker and / or the second event marker to match the time axis position. For example, assume that the portable multimodal data synchronization device simultaneously sends a first event marker such as a to sensor 1 and sensor 2 that detect multimodal data offline. After sensor 1 detects D56 and stores it, it receives marker a and stores it. After sensor 2 detects D81 and stores it, it receives marker a and stores it. When the offline detection of sensor 1 and sensor 2 is completed, the subsequent data acquisition terminal reads markers such as a from the storage modules of sensor 1 and sensor 2. Fig.11 The data shown. Fig.11As shown, the data obtained by the data acquisition terminal has no time information, so the data detected by sensor 1 cannot be synchronized (or aligned) with the data detected by sensor 2. However, since sensor 1 and sensor 2 receive tag a at the same time, it can be concluded based on tag a that data D58 of sensor 1 and data D81 of sensor 2 are data detected at the same time, data D59 of sensor 1 and data D82 of sensor 2 are data detected at the same time, and so on, for other data detected by sensor 1 and sensor 2. Similarly, assuming that the portable multimodal data synchronization device simultaneously sends a second event marker to sensor 1 and sensor 2 that detect multimodal data offline, the data acquisition terminal can not only use the second event marker to synchronize the multimodal data detected by sensor 1 and sensor 2, but also obtain the corresponding multimodal data when the event occurs. For example, when Fig.11 The mark a in is the second event mark, which indicates an event of too high temperature. Then the data D59 and the subsequent data of sensor 1 and the data D82 and the subsequent data of sensor 2 are multimodal data under too high temperature, and so on.

[0099] Then, the data acquisition terminal or the corresponding software installed therein can further perform other operations, such as adjusting the time axis corresponding to the data of each sensor by stretching or shortening, and adding or removing part of the data accordingly, and saving, etc.

[0100] In some embodiments, the data acquisition terminal may further read the marks stored in the storage module in the portable multimodal data synchronization device or the second sensor as a reference in data analysis to improve the accuracy of data synchronization.

[0101] It should be understood that the division of the modules in the above embodiments is only a division of logical functions, and in actual implementation, they can be fully or partially integrated into one physical entity, or physically separated. Moreover, these modules can be implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware.

[0102] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0103] Those skilled in the art will appreciate that the functions of each module described in the embodiments disclosed herein can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0104] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0105] The above is only a specific implementation of the present application. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A portable multimodal data synchronization device, characterized in that: include: A control module, an operation module and a communication module, wherein the control module is coupled to the start module and the communication module respectively; The operation module is used to issue instructions to the control module; The control module is used to issue an event mark in response to the instruction; The communication module is used to connect with at least two sensors and send the event mark to the at least two sensors.

2. The device according to claim 1, characterized in that The operation module includes a start module, and the start module is used to issue a first instruction to the control module; The control module is further configured to send a first event marker to the at least two sensors through the communication module in response to the first instruction.

3. The device according to claim 2, characterized in that The control module is used to issue a first event mark in response to the first instruction, including: The control module is used for issuing a first event mark at a first predetermined time interval in response to the first instruction; The communication module is further configured to send one of the first event markers to the at least two sensors at intervals of the first predetermined time period.

4. The device according to claim 2, characterized in that The startup module is also used to issue a second instruction to the control module; The control module is further configured to stop issuing the first event mark in response to the second instruction; The communication module is further configured to stop sending the first event marker to the at least two sensors when the control module stops sending the first event marker.

5. The device according to claim 2, characterized in that The control module is further configured to start timing after issuing the first first event mark, and stop issuing the first event mark when the timing reaches a second predetermined time length, wherein the second predetermined time length is greater than the first predetermined time length; The communication module is further configured to stop sending the first event marker to the at least two sensors when the control module stops sending the first event marker.

6. The device according to any one of claims 1 to 5, characterized in that The operation module further includes an event triggering module, which is coupled to the control module and configured to issue a third instruction to the control module; The control module is further configured to send a second event marker to the at least two sensors through the communication module in response to the third instruction, where the second event marker includes event information.

7. The device according to claim 6, characterized in that The event triggering module is further used to send a fourth instruction to the control module; The control module is further configured to control the communication module to establish the connection with the at least two sensors in response to the fourth instruction.

8. The device according to claim 6, characterized in that It also includes a voice input module and an artificial intelligence (AI) module, wherein the voice input module is coupled to the AI ​​module, and the AI ​​module is coupled to the control module; The voice input module is used to receive voice, and the AI ​​module is used to recognize the voice, and when the recognition result matches the target voice, a fifth instruction is issued to the control module; The control module is further configured to send a second event marker to the at least two sensors through the communication module in response to the fifth instruction, where the second event marker includes event information.

9. The device according to any one of claims 1 to 5, characterized in that: The communication module is a wireless communication module. The device further includes a storage module, which is coupled to the control module and is used to store the first event mark.

10. A sensor, characterized in that: It includes a processing module, a detection module, a first communication module, a second communication module and a storage module, wherein the processing module is coupled to the detection module, the first communication module, the second communication module and the storage module respectively, the first communication module is used to couple to a data acquisition terminal, and the second communication module is used to couple to a portable multimodal data synchronization device or a sensor; The detection module is used to detect the test subject and obtain multimodal data; The processing module is used to send the multimodal data to the data acquisition terminal in real time through the first communication module when the first communication module is coupled to the data acquisition terminal, and is also used to store the multimodal data through the storage module when the first communication module is disconnected from the data acquisition terminal, and The second communication module receives a first event marker sent by a sensor of the portable multimodal data synchronizer, and stores the first event marker through the storage module.

11. The sensor according to claim 10, characterized in that The processing module is further configured to receive a second event marker through the second communication module and store the second event marker through the storage module, wherein the second event marker includes event information.

12. The sensor according to claim 10, characterized in that The second communication module and the first communication module are the same communication module.

13. The sensor according to any one of claims 10 to 12, characterized in that The second communication module is a wireless communication module.

14. A portable multimodal data synchronization system, characterized in that: The portable multimodal data synchronization device comprises a sensor according to any one of claims 1 to 9, and at least two sensors according to any one of claims 10 to 13, wherein the portable multimodal data synchronization device is used to send an event marker and a second event marker to at least two of the sensors.

Images