Multimodal man-machine environment data acquisition synchronization method, edge device, system and medium

Abstract

The embodiment of the invention provides a multi-modal man-machine environment data acquisition synchronization method, an edge device, a system and a medium, and the method is applied to the multi-modal man-machine environment data acquisition synchronization edge device. The multi-mode man-machine environment data acquisition synchronization edge device is connected with an upper computer and at least one man-machine environment data acquisition device used for acquiring man-machine environment data. The time of the multi-modal man-machine environment data acquisition synchronization edge device is calibrated by the multi-modal man-machine environment data acquisition synchronization edge device according to the time of the upper computer; and based on the calibrated time, the multi-modal man-machine environment data acquisition synchronization edge device controls each man-machine environment data acquisition device to send man-machine environment data based on the clock of the multi-modal man-machine environment data acquisition synchronization edge device. According to the technical scheme provided by the embodiment of the invention, the multi-modal man-machine environment data detected by the man-machine environment data acquisition equipment can be synchronized in the transmission and analysis processing process.

Description

Technical Field

[0001] This application relates to the fields of human factors engineering and ergonomics, and in particular to a synchronization method, edge device, and system for multimodal human-machine environment data acquisition. Background Technology

[0002] Human factors engineering and ergonomics primarily study the interrelationships between humans, machines, and the environment, and are widely applied in fields such as cognitive neuroscience, emotion recognition, and workload assessment. Human factors engineering and ergonomics typically require the collection of human-machine-environment data for related testing and analysis. However, because multimodal human-machine-environment data may be collected by different devices, the collected data exhibits significant temporal discrepancies, making it difficult to meet the synchronization requirements of multimodal human-machine-environment data across different fields. Summary of the Invention

[0003] This application provides a method, system, tester, and medium for synchronizing multimodal human-machine environment data acquisition, so as to achieve multimodal human-machine environment data synchronization.

[0004] In a first aspect, embodiments of this application provide a synchronization method for multimodal human-machine environment data acquisition, applied to a multimodal human-machine environment data acquisition synchronization edge device. This multimodal human-machine environment data acquisition synchronization edge device is connected to a host computer and at least one human-machine environment data acquisition device used to acquire multimodal human-machine environment data. The synchronization method includes: the multimodal human-machine environment data acquisition synchronization edge device calibrating its own time according to the host computer's time; and the multimodal human-machine environment data acquisition synchronization edge device controlling each human-machine environment data acquisition device to send human-machine environment data based on its clock, according to the calibrated time.

[0005] In one possible implementation, the aforementioned human-machine environment data acquisition device transmits human-machine environment data based on the clock of the multimodal human-machine environment data acquisition synchronization edge device, which may include: The multimodal human-machine environment data acquisition synchronization edge device takes over the clock of the human-machine environment data acquisition device and controls the human-machine environment data acquisition device to send human-machine environment data based on the clock of the multimodal human-machine environment data acquisition synchronization edge device.

[0006] In one possible implementation, after the aforementioned human-machine environment data acquisition device transmits human-machine environment data based on the clock of the multimodal human-machine environment data acquisition synchronization edge device, it may further include: The multimodal human-machine environment data acquisition synchronization edge device receives human-machine environment data sent by the human-machine environment data acquisition device based on the clock of the multimodal human-machine environment data acquisition synchronization edge device, and sends the human-machine environment data to the host computer.

[0007] In one possible implementation, the aforementioned human-machine environment data acquisition device transmits human-machine environment data based on the clock of the multimodal human-machine environment data acquisition synchronization edge device, which may include: The multimodal human-machine environment data acquisition synchronization edge device sends synchronization pulses to the human-machine environment data acquisition device to control the clock of the human-machine environment data acquisition device to synchronize with the synchronization pulses, and detects human-machine environment data and sends human-machine environment data to the host computer based on the synchronized pulses.

[0008] In one possible implementation, the multimodal human-machine environment data acquisition synchronization edge device calibrates its own time according to the host computer's time, which may include: The multimodal human-machine environment data acquisition synchronization edge device sends a request to the host computer, which is used to obtain the timestamp; Multimodal human-machine environment data acquisition synchronous edge device receives timestamps sent by host computer according to request; Multimodal human-machine environment data acquisition and synchronization edge devices calibrate local time based on timestamps.

[0009] In one possible implementation, the aforementioned multimodal human-machine environment data acquisition and synchronization edge device calibrates its local time based on a timestamp, which may include: Multimodal human-machine environment data acquisition synchronous edge device estimates the duration from the time of timestamp generation to the time of reception, and calibrates the local time based on the duration and timestamp.

[0010] In one possible implementation, the aforementioned multimodal human-machine environment data acquisition and synchronization edge device, after calibrating its local time based on a timestamp, may include: Determine if the number of requests sent has reached the preset value. If not, execute the multimodal human-machine environment data acquisition synchronization edge device to send a request to the host computer again; if the preset value has been reached, the calibration is completed.

[0011] Secondly, embodiments of this application provide a multimodal human-machine environment data acquisition synchronization edge device, including: a calibration module and a clock synchronization module; wherein, the calibration module is used to calibrate the time of the multimodal human-machine environment data acquisition synchronization edge device according to the time of the host computer; the clock synchronization module is used to control the human-machine environment data acquisition device to send human-machine environment data based on the time calibrated by the calibration module and the clock of the multimodal human-machine environment data acquisition synchronization edge device.

[0012] Thirdly, embodiments of this application provide a multimodal human-machine environment data acquisition and synchronization system, including a host computer, at least one human-machine environment data acquisition device for acquiring multimodal human-machine environment data, and a multimodal human-machine environment data acquisition and synchronization edge device as described in any of the second aspects above.

[0013] Fourthly, embodiments of this application provide an edge computing device, including a processor and a memory, wherein the memory stores programs or instructions that can run on the processor, and when the programs or instructions are executed by the processor, they implement the method as described in any of the first aspects.

[0014] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, wherein when the computer program is run on a computer, the computer performs the method described in any of the first aspects.

[0015] The multimodal human-machine environment data acquisition synchronization method, edge device, system, and medium provided in this application embodiment achieve time synchronization of each device in the multimodal human-machine environment data acquisition synchronization system by calibrating the time of the multimodal human-machine environment data acquisition synchronization edge device using a host computer and synchronizing the clocks of the multimodal human-machine environment data acquisition synchronization edge device with the accessed human-machine environment data acquisition device. This enables the multimodal human-machine environment data detected by the human-machine environment data acquisition device to be synchronized during transmission, analysis, and processing. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of the structure of a multimodal human-machine environment data acquisition and synchronization system provided in this application embodiment; Figure 2 A schematic diagram of another multimodal human-machine environment data acquisition and synchronization system provided in this application embodiment; Figure 3 A schematic diagram of the structure of another multimodal human-machine environment data acquisition and synchronization system provided in this application embodiment; Figure 4 A schematic flowchart of a synchronization method for multimodal human-machine environment data acquisition provided in this application embodiment; Figure 5 for Figure 4A schematic diagram illustrating one implementation of step 501 in the method shown; Figure 6 for Figure 4 A schematic diagram of another implementation of step 501 in the method shown; Figure 7 This is a schematic diagram of the structure of an edge computing device provided in an embodiment of this application. Detailed Implementation

[0018] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0019] In the synchronization method for multimodal human-machine environment data acquisition provided in this application embodiment, the human-machine environment data may include data of different modalities such as human factors data, machine data, human-machine interaction data, environmental data, human-environment interaction data, and data of human-machine interaction and environment. Among them, human factors data may include peripheral physiological (BIO) data and physiological electrical (EXG) data. Peripheral physiological data may include: photoplethysmogram (PPG), electrodermal activity (EDA), respiratory rate (RESP), skin temperature (SKT), peripheral capillary oxygen saturation (SPO2), blood pressure (BP), force-sensing resistor (FSR) detection data (used to detect pressure-related signals), functional near-infrared spectroscopy (fNIRS), inertial measurement unit (IMU) detection data (used for motion monitoring), heart rate (HR), and pulse pressure (Pulse pressure). Data such as pressure (PP) and physiological electrical data can include data from different modalities such as electromyography (EMG), root mean square (RMS, which can be used to analyze signal amplitude characteristics), electrocardiography (ECG), electrooculography (EOG), and electroencephalography (EEG).

[0020] In the multimodal human-machine environment data acquisition and synchronization system provided in this application embodiment, multiple different modalities of human-machine environment data can be acquired by a single human-machine environment data acquisition device, or different modalities of human-machine environment data can be acquired by different human-machine environment data acquisition devices. Because multimodal human-machine environment data is abundant, and given that a single type of multimodal human-machine environment data is obtained from a single human-machine environment data acquisition device, the multimodal human-machine environment data acquisition synchronization system provided in this application embodiment can include multiple human-machine environment data acquisition devices. How to achieve data synchronization among the host computer, the multimodal human-machine environment data acquisition synchronization edge device, and the human-machine environment data acquisition device in the multimodal human-machine environment data acquisition synchronization system is the problem solved by the multimodal human-machine environment data acquisition synchronization method provided in this application embodiment.

[0021] The synchronization method for multimodal human-machine environment data acquisition provided in this application solves the problem of multimodal human-machine environment data synchronization by synchronizing the time of each device in the multimodal human-machine environment data acquisition synchronization system.

[0022] The multimodal human-machine environment data acquisition and synchronization system provided in this application embodiment can, as follows: Figure 1 As shown, it includes a host computer 11, a multimodal human-machine environment data acquisition and synchronization edge device 12, and a human-machine environment data acquisition device 13. The multimodal human-machine environment data acquisition and synchronization edge device 12 is connected to the host computer 11 and the human-machine environment data acquisition device 13, respectively.

[0023] The host computer 11 can be a desktop computer, portable computer, etc., whose hardware and software performance can meet the data analysis and processing needs of the human-machine environment. The host computer 11 can also be a computer, mobile phone, tablet computer, etc. In some embodiments, the host computer can be as follows: Figure 2 As shown, the software platform interacts with the multimodal human-machine environment data acquisition and synchronization edge device to calibrate the time of the multimodal human-machine environment data acquisition and synchronization edge device. The software platform can also analyze and process the multimodal human-machine environment data detected by the human-machine environment data acquisition device 13. For example, this software platform can be human-machine environment data analysis and processing software (such as ergonomic experimental software). In this case, the hardware and software performance of the host computer needs to meet the requirements of the human-machine environment data analysis and processing software for operation and the analysis and processing of human-machine environment data.

[0024] The multimodal human-machine environment data acquisition and synchronization edge device 12 can be an edge computing device deployed on the edge to reduce privacy risks, etc.

[0025] There can be multiple human-machine environment data acquisition devices 13, such as devices for detecting peripheral physiological data and devices for detecting physiological electrical data. The human-machine environment data acquisition device 13 can connect its clock circuit interface to the clock circuit interface of the multimodal human-machine environment data acquisition synchronization edge device 12 via a wired connection to achieve time synchronization with the multimodal human-machine environment data acquisition synchronization edge device 12. For ease of description, the connection between the clock circuit interfaces of the human-machine environment data acquisition device 13 and the multimodal human-machine environment data acquisition synchronization edge device 12 will be referred to as the clock synchronization line.

[0026] The multimodal human-machine environment data acquisition and synchronization edge device 12 can connect to the host computer 11 via wired or wireless means (such as Wi-Fi, Bluetooth, infrared, etc.) to obtain the system time of the host computer 11 and calibrate its own time based on the obtained system time.

[0027] The host computer 11 is also used to analyze and process the human-machine environment data collected by the human-machine environment data acquisition device 13. Specifically, the human-machine environment data analysis and processing software installed in the host computer 11 can be used to analyze and process the human-machine environment data.

[0028] In some embodiments, the multimodal human-machine environment data acquisition and synchronization edge device 12 may also include a marking function, and / or a data acquisition and transmission function, such as collecting human-machine environment data from the human-machine environment data acquisition device, packaging the human-machine environment data, and sending it to the host computer.

[0029] In some embodiments, the host computer can be further deployed with an Artificial Intelligence (AI) module and a model training module. The large amount of human-machine environment data collected by the human-machine environment data acquisition device can be used as training data for the intelligent model of the AI ​​module. Through the deep mining and learning of the data by the model training module, accurate models of emotion, load, cognition, etc. can be established, and these models can be used for prediction and evaluation, providing intelligent solutions for research and application in related fields.

[0030] Another multimodal human-machine environment data acquisition and synchronization system provided in this application embodiment is as follows: Figure 2As shown, the system includes a host computer 21, a multimodal human-machine environment data acquisition synchronization edge device 22, human-machine environment data acquisition devices a1 to an, and human-machine environment data acquisition devices b1 to bn. Among these, human-machine environment data acquisition devices a1 to an are non-third-party devices (e.g., manufactured or designed by the same manufacturer or company as the software platform in the host computer and the multimodal human-machine environment data acquisition synchronization edge device), and can be a single set of human-machine environment data acquisition devices for one subject, or multiple sets of human-machine environment data acquisition devices for multiple subjects. Human-machine environment data acquisition devices b1 to bn are third-party devices, and can also be one or more sets of human-machine environment data acquisition devices. For example, a single set of human-machine environment data acquisition devices can include 16 human-machine environment data acquisition devices, of which 8 devices can be devices such as EDA sensors, SKT sensors, and HR sensors used to detect and obtain 8 types of BIO data, and 8 devices can be devices such as EEG sensors, EMG sensors, and ECG sensors used to detect and obtain 8 types of EXG data.

[0031] The multimodal human-machine environment data acquisition and synchronization edge device 22 includes a synchronization module 221, a marking module 222, and a data acquisition module 223 connected by wires.

[0032] The synchronization module 221 includes a calibration module and a clock synchronization module. The calibration module is used to calibrate the time of the multimodal human-machine environment data acquisition synchronization edge device 22 according to the system time of the host computer 21. The clock synchronization module is used to control the time of the human-machine environment data acquisition devices a1 to an to synchronize with the time of the multimodal human-machine environment data acquisition synchronization edge device 22 based on the calibrated time, thereby realizing the synchronization of all devices in the multimodal human-machine environment data acquisition synchronization system (including the host computer 21, the multimodal human-machine environment data acquisition synchronization edge device 22, and the human-machine environment data acquisition devices a1 to an).

[0033] The marking module 222 is used to send event codes to the human-machine environment data acquisition devices b1 to bn upon event triggering, so that the human-machine environment data acquisition devices b1 to bn mark the detected human-machine environment data, thereby enabling the host computer 21 to synchronize the human-machine environment data according to the event codes when analyzing and processing the data. In some embodiments, the marking module 222 can also be used to send event codes to the human-machine environment data acquisition devices a1 to an upon event triggering. The marking module 222 includes a manual marking module, an application programming interface (API) marking module, a transistor-transistor logic (TTL) marking module, an optical marking module, and an acoustic marking module.

[0034] The manual marking module can be a button-operated marker, serving as a signal input method to facilitate manual recording of events at specific times and sending corresponding event codes to the human-machine environment data acquisition devices b1 to bn. The API marking module supports interface with ergonomic experimental software, Python, E-prime, and other experimental design software via API serial ports. Users can send corresponding event codes to the human-machine environment data acquisition devices b1 to bn, achieving automatic event marking and facilitating accurate recording of key events during experiments. The TTL marking module can synchronize event markings from third-party devices, such as eye-tracking and third-party EEG devices, via a DB15 interface, enabling collaborative marking across multiple devices and facilitating the integration and comprehensive analysis of data from different sources. The optical marking module is used for marking based on light thresholds. When a change in light intensity reaches a set threshold, it records the corresponding event information and sends the corresponding event code to the human-machine environment data acquisition devices b1 to bn. The acoustic marking module marks based on sound thresholds. When external sound reaches a set threshold, the system automatically records the event and sends the corresponding event code to the human-machine environment data acquisition devices b1 to bn.

[0035] The data acquisition module 223 can simultaneously acquire human-machine environment data detected by human-machine environment data acquisition devices a1 to an and human-machine environment data acquisition devices b1 to bn, and package and provide it to the host computer 21.

[0036] The multimodal human-machine environment data acquisition and synchronization edge device 22, through a calibration module, a clock synchronization module and / or a marking module 222, and a data acquisition module 223, can simultaneously meet the multimodal human-machine environment data synchronization needs of multiple subjects (e.g., 10).

[0037] The host computer 21 includes a software platform and an AI module.

[0038] This software platform can simultaneously and completely receive and analyze multimodal human-computer environment data from multiple subjects. The platform includes a visualization module and a batch processing and analysis module. The visualization module provides an intuitive interface for visualizing the multimodal human-computer environment data of a single subject. Users can view waveforms, curves, and other visualizations of EEG, fNIRS, and BIO data in real time. The interface displays key features of each modality. This visualization interface also supports simultaneous visualization of multimodal human-computer environment data from multiple subjects, integrating data from multiple subjects into a single interface for convenient group comparison analysis and group behavior research. For example, the visualization interface can simultaneously display waveforms of multimodal human-computer environment data from two subjects. Alternatively, it can simultaneously display data from 10 subjects, facilitating data comparison and analysis in multi-subject experimental scenarios (such as team collaborative research and group emotion analysis). The batch processing and analysis module provides batch processing functionality for human-machine environment data, enabling batch operations on large amounts of collected human-machine environment data, such as data format conversion, data preprocessing (e.g., filtering, noise reduction), and feature extraction.

[0039] The AI ​​module possesses model training and prediction capabilities, and can include a model training module and a real-time prediction module. The model training module can train the intelligent model of the real-time prediction module using multimodal human-machine environment data collected by the multimodal human-machine environment data acquisition synchronous edge device 22, or multimodal human-machine environment data detected by human-machine environment data acquisition devices a1 to an and b1 to bn. Through in-depth mining and learning of human-machine environment data, accurate models of emotion, load, cognition, etc., are established, and these models are used for prediction and evaluation, providing intelligent solutions for research and applications in related fields.

[0040] In some embodiments, the multimodal human-machine environment data acquisition and synchronization edge device 22 may omit the clock synchronization module and achieve data synchronization through the marking module.

[0041] In some embodiments, the multimodal human-machine environment data acquisition and synchronization edge device 22 may omit the marking module, and correspondingly, the multimodal human-machine environment data acquisition and synchronization system may omit the human-machine environment data acquisition devices b1 to bn.

[0042] In some embodiments, when the data acquisition module in the multimodal human-machine environment data acquisition synchronization edge device 22 is connected to the human-machine environment data acquisition device of only one subject, and the multimodal human-machine environment data acquisition synchronization edge device 22 is wirelessly connected to the host computer 21, the multimodal human-machine environment data acquisition synchronization edge device 22 and the subject's acquisition device can be designed as a wearable device.

[0043] In some embodiments, the hardware of the multimodal human-machine environment data acquisition synchronization edge device 12 may be a hub.

[0044] Another human-machine environment data acquisition and synchronization system provided in this application embodiment is as follows: Figure 3 As shown, it includes a host computer 41, a hub 42, an EEG sensor 431, and an EEG sensor 432.

[0045] Hub 42 includes a calibration module and a clock module. The clock module is used to generate the time of Hub 42, and the calibration module is used to obtain the system time of host computer 41 from host computer 41 and calibrate the clock module according to the system time of host computer 41 so that the time generated by the clock module is consistent with the system time of host computer 41.

[0046] Both EEG sensor 431 and EEG sensor 432 include a clock module. The clock modules of EEG sensor 431 and EEG sensor 432 are connected to the clock module of Hub 42. Hub 42 achieves time synchronization with EEG sensor 431 and EEG sensor 432 through its own clock module and the clock modules of EEG sensor 431 and EEG sensor 432, thereby enabling time synchronization of all devices in the multimodal human-machine environment data acquisition and synchronization system.

[0047] In some embodiments, the multimodal human-machine environment data acquisition and synchronization edge device may further include a power module or a power management module for providing power to the multimodal human-machine environment data acquisition and synchronization edge device. In some embodiments, the power module or power management module may also provide power to the human-machine environment data acquisition device.

[0048] In some embodiments, the multimodal human-machine environment data acquisition and synchronization edge device may further include four hardware interfaces: a clock synchronization interface, a marking interface, a charging interface, and a data transmission interface. The clock synchronization interface is used to connect to the human-machine environment data acquisition device via a clock synchronization line. The clock synchronization line can support multiple devices connected simultaneously (e.g., one to two or three devices). The marking interface is used to send event codes to the human-machine environment data acquisition device to add to the multimodal human-machine environment data detected by the device. The charging interface is used to power the multimodal human-machine environment data acquisition and synchronization edge device, or further to power the device. The data transmission interface may include interfaces such as Universal Serial Bus (USB) or Bluetooth, used for communication with a host computer. In some embodiments, the multimodal human-machine environment data acquisition and synchronization edge device may further include a data transmission interface for communicating with the human-machine environment data acquisition device, used to receive the multimodal human-machine environment data detected by the device.

[0049] This application provides a method for synchronizing human-machine environment data acquisition in a multimodal manner, as follows: Figure 4 As shown, the method, applicable to any of the multimodal human-machine environment data acquisition and synchronization edge devices provided in the above embodiments, includes: Step 501: The multimodal human-machine environment data acquisition and synchronization edge device calibrates its own time based on the host computer's time. The host computer's time can be its real-time system time. For example, the host computer can proactively send its real-time system time to the multimodal human-machine environment data acquisition and synchronization edge device, which then calibrates its own time accordingly. Alternatively, the multimodal human-machine environment data acquisition and synchronization edge device can proactively request the host computer's real-time system time and then calibrate its own time based on the system time returned by the host computer.

[0050] For example, step 501 may be as follows Figure 5 As shown, it includes: Step 601: The multimodal human-machine environment data acquisition and synchronization edge device sends a request to the host computer to obtain a timestamp. For example, the multimodal human-machine environment data acquisition and synchronization edge device can send the request to the host computer via wired (e.g., via USB port) or wireless communication. This timestamp can be data identifying the host computer's real-time system time. This real-time system time can be the moment the host computer generated the timestamp.

[0051] Step 602: The multimodal human-machine environment data acquisition synchronization edge device receives the timestamp sent by the host computer according to the request. For example, the software platform in the host computer can generate the timestamp according to the request and send a response signal carrying the timestamp through the communication function of the host computer. The multimodal human-machine environment data acquisition synchronization edge device receives the response signal through wired or wireless communication and decodes the response signal to obtain the timestamp.

[0052] Step 603: Multimodal human-machine environment data acquisition and synchronization edge devices calibrate local time according to timestamps.

[0053] In some embodiments, step 603 may include: The multimodal human-machine environment data acquisition synchronization edge device estimates the duration from the time the timestamp is generated to the time it is received, and calibrates the local time based on this duration and the timestamp.

[0054] For example, the multimodal human-machine environment data acquisition synchronization edge device records the time of sending a request (e.g., T1) and the time of receiving a response signal (T2), and calculates the round-trip time (RTT), such as RTT = T2 - T1. Based on RTT / 2, the one-way delay, i.e., the duration from the timestamp generation time to the receiving time, is estimated, yielding the time deviation between the multimodal human-machine environment data acquisition synchronization edge device and the host computer: T' = RTT + time in the timestamp - T2, or, T' = T2 - (RTT + time in the timestamp). Then, the multimodal human-machine environment data acquisition synchronization edge device uses the time deviation to calibrate its local time, such as adjusting the current time to t + T' or t - T'.

[0055] In some embodiments, steps 601-602 may be repeated multiple times (e.g., 10 times) to obtain multiple time deviations, and then the local time is calibrated using the average of the time deviations to improve accuracy.

[0056] For example, another implementation of step 501 may be as follows: Figure 6 As shown, it includes: Step 701: Multimodal human-machine environment data acquisition synchronization edge device sends a timestamp request to the host computer and records the sending time.

[0057] Step 702: The host computer returns a response, which includes a timestamp.

[0058] Step 703: Multimodal human-machine environment data acquisition synchronization edge device receives response and records the reception time.

[0059] Step 704: The multimodal human-machine environment data acquisition synchronization edge device calculates the time deviation based on the timestamp, sending time, and receiving time. For details, please refer to the relevant description in the aforementioned embodiments.

[0060] Step 705: The multimodal human-machine environment data acquisition synchronization edge device determines whether step 701 has reached the preset number of times. If yes, the next step is to continue to step 706; otherwise, the next step is to continue to step 701.

[0061] Step 706: The multimodal human-machine environment data acquisition synchronization edge device calculates the average time deviation. For example, assuming the preset number of times is 8, correspondingly, the time deviations calculated in step 704 are 8, such as T1', T2', T3', T4', T5', T6', T7', and T8'. Then, step 706 can obtain the average time deviation by dividing the sum of the multiple time deviations by the number of time deviations, such as (T1'+T2'+T3'+T4'+T5'+T6'+T7'+T8') / 8.

[0062] Step 707: The multimodal human-machine environment data acquisition synchronization edge device calibrates the local time using the average value of the time deviation. For details, please refer to the relevant description in the aforementioned embodiments.

[0063] In some embodiments, steps 601-603 may be repeated multiple times to improve calibration accuracy. For example, step 603 may be followed by: Determine if the number of requests sent has reached a preset value (e.g., 12 times). If it has not reached the preset value, then execute the multimodal human-machine environment data acquisition synchronization edge device to send a request to the host computer again; if it has reached the preset value, then the calibration is completed.

[0064] Through the above calibration, the time deviation between the multimodal human-machine environment data acquisition synchronization edge device and the host computer can be reduced to 1 millisecond or even less, and the calibration accuracy is higher.

[0065] Step 502: The multimodal human-machine environment data acquisition synchronization edge device controls each human-machine environment data acquisition device to send human-machine environment data based on the clock of the multimodal human-machine environment data acquisition synchronization edge device, based on the calibrated time.

[0066] For example, this step may include: the multimodal human-machine environment data acquisition synchronization edge device takes over the clock of the human-machine environment data acquisition device and controls the human-machine environment data acquisition device to send human-machine environment data based on the clock of the multimodal human-machine environment data acquisition synchronization edge device. For example, the multimodal human-machine environment data acquisition synchronization edge device is connected to multiple human-machine environment data acquisition devices through a clock synchronization interface. When the multimodal human-machine environment data acquisition synchronization edge device detects the multiple human-machine environment data acquisition devices, it simultaneously sends control signals to the multiple human-machine environment data acquisition devices through the clock synchronization interface. In response to the control signals, the multiple human-machine environment data acquisition devices disconnect their internal clock modules, such as stopping receiving the clock source provided by the internal clock module, and instead receive clock pulses provided by the clock module of the multimodal human-machine environment data acquisition synchronization edge device through a synchronization line connected to the clock synchronization interface of the multimodal human-machine environment data acquisition synchronization edge device, and use the clock pulses to send multimodal human-machine environment data, thereby achieving synchronization with the multimodal human-machine environment data acquisition synchronization edge device. After the clock of the human-machine environment data acquisition device is taken over by the multimodal human-machine environment data acquisition synchronization edge device, the device begins to detect the subject, encodes the detected signals based on the clock of the multimodal human-machine environment data acquisition synchronization edge device to generate multimodal human-machine environment data, and sends the multimodal human-machine environment data based on the synchronized pulse.

[0067] Accordingly, after the multimodal human-machine environment data acquisition synchronization edge device controls the human-machine environment data acquisition device to send human-machine environment data based on the clock of the multimodal human-machine environment data acquisition synchronization edge device, it may further include: the multimodal human-machine environment data acquisition synchronization edge device receiving the human-machine environment data sent by the human-machine environment data acquisition device based on the clock of the multimodal human-machine environment data acquisition synchronization edge device, and sending the human-machine environment data to the host computer. For example, when multiple human-machine environment data acquisition devices are connected to the multimodal human-machine environment data acquisition synchronization edge device, the multimodal human-machine environment data acquisition synchronization edge device receives the multimodal human-machine environment data detected by the multiple human-machine environment data acquisition devices, reassembles the data packets, and sends them to the host computer through the data transmission interface. For example, the multimodal human-machine environment data acquisition and synchronization edge device combines multimodal human-machine environment data detected at the same time into a single data packet. For instance, it combines BIO and EXG data detected at 10:00 into one data packet, BIO and EXG data detected at 10:01 into another, and so on. The data packets are then sent to the host computer according to their combination order or time sequence. Centralized transmission of multimodal human-machine environment data through this edge device ensures the order and accuracy of the data packets, making it suitable for scenarios requiring strict multimodal human-machine environment data fusion.

[0068] In some other embodiments, step 502 may include: the multimodal human-machine environment data acquisition synchronization edge device sends a synchronization pulse to the human-machine environment data acquisition device, controls the clock of the human-machine environment data acquisition device to synchronize with the synchronization pulse, and detects human-machine environment data and sends human-machine environment data to the host computer based on the synchronized pulse.

[0069] For example, when the multimodal human-machine environment data acquisition synchronization edge device detects the access of a human-machine environment data acquisition device, it sends a periodic pulse wave (such as a 1kHz-10kHz square wave) greater than 1000Hz (i.e., a synchronization pulse) to the accessed human-machine environment data acquisition device through a clock synchronization interface as a high-frequency clock reference signal for the human-machine environment data acquisition device. The clock module within the accessed human-machine environment data acquisition device can be a phase-locked loop (PLL), including a phase detector, a loop filter, and a voltage-controlled oscillator (VCO). The phase detector detects the phase difference between the VCO's output signal and the clock reference signal. The loop filter filters the phase detector's output signal to eliminate noise and interference. The VCO adjusts the frequency and phase of its output clock signal according to the loop filter's output signal to maintain consistency with the high-frequency clock reference signal provided by the multimodal human-machine environment data acquisition synchronization edge device, thus achieving clock synchronization between all accessed human-machine environment data acquisition devices and the multimodal human-machine environment data acquisition synchronization edge device.

[0070] After the clocks of the human-machine environment data acquisition device connected to the multimodal human-machine environment data acquisition synchronization edge device are synchronized, the detection of the subject begins. The detected signals are encoded based on the synchronized clock to generate multimodal human-machine environment data, and the multimodal human-machine environment data is transmitted based on the synchronized pulses.

[0071] In this embodiment, multiple human-machine environment data acquisition devices directly send multimodal human-machine environment data to the host computer through their respective data transmission interfaces (such as USB and Bluetooth). Since the clocks of all the human-machine environment data acquisition devices sending multimodal human-machine environment data are synchronized with the multimodal human-machine environment data acquisition synchronization edge device, and the multimodal human-machine environment data acquisition synchronization edge device is synchronized with the host computer, the multimodal human-machine environment data received by the host computer has a unified time reference, allowing for data alignment and fusion via the host computer's system time. In the technical solution provided in this embodiment, the multimodal human-machine environment data acquisition synchronization edge device does not need to collect multimodal human-machine environment data or repackage it, supporting independent device operation and reducing the processing burden on the multimodal human-machine environment data acquisition synchronization edge device. This is suitable for distributed acquisition scenarios, and the synchronization accuracy is ensured by a high-frequency pulse clock reference signal.

[0072] In some embodiments, a human-machine environment data acquisition device that is connected to a multimodal human-machine environment data acquisition synchronization edge device and whose clock is synchronized with that of the multimodal human-machine environment data acquisition synchronization edge device can also send the detected multimodal human-machine environment data to the multimodal human-machine environment data acquisition synchronization edge device, which will then repackage the data and send it to the host computer.

[0073] The multimodal human-machine environment data acquisition and synchronization edge device provided in this application embodiment can be found in the relevant description of the multimodal human-machine environment data acquisition and synchronization edge device in any of the foregoing embodiments.

[0074] The multimodal human-machine environment data acquisition and synchronization system provided in this application embodiment can be found in the relevant descriptions of the multimodal human-machine environment data acquisition and synchronization system in any of the foregoing embodiments.

[0075] The edge computing device provided in this application embodiment may include a processor, a memory, and a computer program / instructions. There may be one or more processors, coupled to the memory, and the computer program is stored in the memory. When the processor calls and executes the computer program in the memory, the edge computing device can perform the synchronization method for human multimodal human-machine environment data acquisition provided in any of the above embodiments.

[0076] like Figure 7 As shown, the edge computing device 80 includes a processor 81, a memory 82, and a computer program 83. The computer program 83 is stored in the memory 82, and the processor 81 is coupled to the memory 82. When the processor 81 calls and executes the computer program 83 in the memory 82, the edge computing device 80 can execute the synchronization method for multimodal human-machine environment data acquisition provided in any of the above embodiments.

[0077] It should be understood that the division of modules in the above embodiments is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in hardware; alternatively, some modules can be implemented in software through processing element calls, while others are implemented in hardware. For example, the processor can be a separate processing element or integrated into a chip within the electronic device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or implemented independently. During implementation, each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0078] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0079] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for synchronizing multi-modal human-machine environment data collection, the method comprising: The application is applied to a multi-modal man-machine environment data collection synchronization edge device, which is connected with an upper computer and at least one man-machine environment data collection device for collecting multi-modal man-machine environment data respectively, and the method comprises the following steps of: The multi-modal man-machine environment data collection synchronization edge device calibrates the time of the multi-modal man-machine environment data collection synchronization edge device according to the time of the upper computer; The multi-modal man-machine environment data collection synchronization edge device controls each man-machine environment data collection device to send man-machine environment data based on the clock of the multi-modal man-machine environment data collection synchronization edge device based on the calibrated time.

2. The method of claim 1, wherein, The control of the man-machine environment data collection device to send man-machine environment data based on the clock of the multi-modal man-machine environment data collection synchronization edge device comprises the following steps of: The multi-modal man-machine environment data collection synchronization edge device controls the man-machine environment data collection device to send the man-machine environment data based on the clock of the multi-modal man-machine environment data collection synchronization edge device by taking over the clock of the man-machine environment data collection device.

3. The method of claim 2, wherein, After the control of the man-machine environment data collection device to send man-machine environment data based on the clock of the multi-modal man-machine environment data collection synchronization edge device, the method further comprises the following steps of: The multi-modal man-machine environment data collection synchronization edge device receives the man-machine environment data sent by the man-machine environment data collection device based on the clock of the multi-modal man-machine environment data collection synchronization edge device, and sends the man-machine environment data to the upper computer.

4. The method of claim 1, wherein, The control of the man-machine environment data collection device to send man-machine environment data based on the clock of the multi-modal man-machine environment data collection synchronization edge device comprises the following steps of: The multi-modal man-machine environment data collection synchronization edge device controls the clock of the man-machine environment data collection device to be synchronized with the synchronization pulse by sending the synchronization pulse to the man-machine environment data collection device, and detects the man-machine environment data based on the synchronized pulse and sends the man-machine environment data to the upper computer.

5. The method according to any one of claims 1 to 4, characterized in that, The calibration of the time of the multi-modal man-machine environment data collection synchronization edge device according to the time of the upper computer comprises the following steps of: The multi-modal man-machine environment data collection synchronization edge device sends a request to the upper computer, and the request is used to acquire a time stamp; The multi-modal man-machine environment data collection synchronization edge device receives the time stamp sent by the upper computer according to the request; The multi-modal man-machine environment data collection synchronization edge device calibrates the local time according to the time stamp.

6. The method of claim 5, wherein, The calibration of the local time of the multi-modal man-machine environment data collection synchronization edge device according to the time stamp comprises the following steps of: The multi-modal man-machine environment data collection synchronization edge device estimates the time length from the generation time to the receiving time of the time stamp, and calibrates the local time according to the time length and the time stamp.

7. The method of claim 5, wherein, After the calibration of the local time of the multi-modal man-machine environment data collection synchronization edge device according to the time stamp, the method further comprises the following steps of: determining whether the number of times of sending the request reaches a preset value, if the preset value is not reached, the multi-modal man-machine environment data acquisition synchronization edge device sends a request to the host computer again; if the preset value is reached, the calibration is completed.

8. A multi-modal human-machine environment data collection synchronization edge device, comprising: The method comprises the following steps: The calibration module is configured to calibrate the time of the multi-modal man-machine environment data acquisition synchronization edge device according to the time of the host computer. The clock synchronization module is configured to control the man-machine environment data acquisition device to send man-machine environment data based on the clock of the multi-modal man-machine environment data acquisition synchronization edge device based on the calibrated time of the calibration module.

9. A multimodal human-machine environment data collection synchronization system, comprising: The system comprises a host computer, at least one man-machine environment data acquisition device for acquiring multi-modal man-machine environment data, and the multi-modal man-machine environment data acquisition synchronization edge device according to claim 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program runs on the computer, the computer executes the method according to any one of claims 1-7.

Images