Multi-modal data synchronization edge computing device and synchronization system based on human intelligence

Abstract

According to the multi-modal data synchronization edge computing device based on human intelligence and the synchronization system provided by the embodiment of the invention, the behavior and / or voice of the testee can be recognized through the AI module, and the event marking information is sent out when the recognition result is matched with the target behavior and / or the target voice; according to the event marking method and device, the tested person can accurately trigger the event marking as long as making a predetermined target behavior or sending a predetermined target voice under the condition that the tested person is stimulated in vision, hearing, touch, taste and the like, so that the inaccuracy of test analysis caused by manually triggering the event marking by a person implementing an experiment is avoided, the objectivity of the event marking is effectively improved, and the test efficiency is improved. Therefore, the influence of the stimulation event on the multi-modal data can be accurately analyzed.

Description

Technical Field

[0001] The present application relates to human factors engineering and ergonomics technology, and in particular to a multimodal data synchronization edge computing device and synchronization system based on human factors intelligence. Background Art

[0002] Human factors engineering and ergonomics mainly study the relationship between humans, machines, and the environment, and are widely used in health care, sports rehabilitation, intelligent driving, and other scenarios. When studying the relationship between humans, machines, and the environment, human factors engineering needs to collect data related to humans, machines, and the environment, conduct human factors or ergonomics related test analysis, and the person conducting the experiment needs to manually trigger event markers during the test to analyze the impact of stimulus events on multimodal data.

[0003] However, the event markers triggered by the people who implement the experiment are not necessarily objective, making it impossible to accurately analyze the impact of stimulus events on multimodal data. Summary of the invention

[0004] The embodiments of the present application provide a multimodal data synchronization edge computing device and synchronization system based on human intelligence, which recognizes the behavior and / or voice of the test subject through an AI module and objectively triggers event marking, thereby helping to accurately determine the impact of stimulus events on multimodal data.

[0005] In a first aspect, an embodiment of the present application provides a multimodal data synchronization edge computing device based on human intelligence, including: an artificial intelligence AI module, a control module and a first interface, the AI ​​module and the first interface are coupled to the control module; the AI ​​module is used to couple with a video capture device and / or a voice input device, and is used to identify a video signal obtained by the video capture device, and issue a first event marking information when the recognition result matches the target behavior, and / or, is used to identify a voice signal obtained by the voice input device, and issue a second event marking information when the recognition result matches the target voice; the control module is used to perform communication protocol encapsulation processing on the first event marking information and / or the second event marking information, and send it to the first interface; the first interface is used to couple with a host computer and send the first event marking information and / or the second event marking information to the host computer.

[0006] In a possible implementation, the above-mentioned device may also include a wireless communication module, which may be coupled to the above-mentioned control module. The above-mentioned control module may also be used to send the above-mentioned first event marker information and / or second event marker information to at least one sensor through the wireless communication module. The at least one sensor may be a sensor worn by the test subject.

[0007] In a possible implementation, when the at least one sensor detects multimodal data offline, the control module may send the first event marker information and / or the second event marker information to the at least one sensor via the wireless communication module.

[0008] In a possible implementation, the at least one sensor detects the multimodal data offline in a case where the at least one sensor has no wired or wireless connection to the host computer and does not send the detected multimodal data to the host computer in real time.

[0009] In a possible implementation, the apparatus may further include a storage module, and the storage module may be coupled to the control module. The control module may be further configured to store the first event marker information and / or the second event marker information in the storage module.

[0010] In a possible implementation, the device may further include a second interface, which may be coupled to the control module and may also be used to couple to a third-party software terminal; the control module may also be used to send the first event marking information and / or the second event marking information to the third-party software terminal through the second interface. Exemplarily, the third-party software terminal may refer to a terminal where the third-party software is installed. In a possible implementation, the third-party software terminal may be the host computer.

[0011] In a possible implementation, the control module may also be used to obtain an event marking protocol from the host computer through the first interface, and may send the event marking protocol to the third-party software terminal through the second interface, wherein the event marking protocol may be used to define the first event marking information and / or the second event marking information. Exemplarily, the event marking protocol may be a specification for defining event marking information, such as defining which fields are used to represent event marking information, and defining different event contents corresponding to different values ​​of certain fields.

[0012] In a possible implementation, the control module can also be used to connect to an optical module, receive an optical signal sent by the optical module, and generate third event marking information when the optical signal exceeds a first threshold, and can send the third event marking information to the host computer through the first interface.

[0013] In a possible implementation, the above-mentioned device may also include an audio interface, which can be coupled to the control module and can be used to send the received audio signal to the control module; the control module can also be used to generate fourth event marking information when the audio signal exceeds the second threshold, and can send the fourth event marking information to the above-mentioned host computer through the above-mentioned first interface.

[0014] In a possible implementation, the above-mentioned device may also include a third interface, which can be coupled to the control module, and can also be used to couple the first external device to convert the level of the fifth event marker information sent by the first external device into a TTL level supported by the control module; the control module can also be used to receive the fifth event marker information through the third interface, and can send the fifth event marker information to the host computer through the above-mentioned first interface.

[0015] In a possible implementation, the above-mentioned device may also include a fourth interface, which can be coupled to the control module and can also be used to couple a second external device; the control module can also be used to send the above-mentioned first event marking information and / or second event marking information to the second external device through the fourth interface.

[0016] In a second aspect, an embodiment of the present application provides a multimodal data synchronization system based on human intelligence, which may include a host computer, at least one sensor, and a multimodal data edge computing device as described in any one of the first aspects above. The multimodal data synchronization edge computing device also includes a video capture device and / or a voice input device;

[0017] The at least one sensor may be used to send the collected multimodal data to the host computer;

[0018] The video capture device and / or the voice input device may be used to trigger the multimodal data edge computing device multimodal data synchronization edge computing device event marker, and the event marker is used to generate first event marker information and / or second event marker information;

[0019] The multimodal data edge computing device and the multimodal data synchronization edge computing device may be used to send the first event marking information and / or the second event marking information to the host computer;

[0020] The host computer may be used to synchronize and analyze the multimodal data and the first event marker information and / or the second event marker information.

[0021] In a possible implementation, the multimodal data edge computing device and the multimodal data synchronization edge computing device may send the first event marking information and / or the second event marking information to the at least one sensor when the at least one sensor is offline.

[0022] In a possible implementation, the system may further include an optical module; the optical module may be used to trigger the multimodal data edge computing device and the multimodal data synchronization edge computing device to perform optical event marking, and the optical event marking may be used to generate third event marking information;

[0023] The multimodal data edge computing device and the multimodal data synchronization edge computing device may also be used to send third event marking information to the host computer;

[0024] The host computer may also be used to synchronize and analyze the multimodal data, the first event marker information and / or the second event marker information, and the third event marker information.

[0025] The host computer may also be used to trigger the multimodal data edge computing device and the multimodal data synchronization edge computing device to perform acoustic event marking, where the acoustic event marking is used to generate fourth event marking information;

[0026] The multimodal data edge computing device and the multimodal data synchronization edge computing device may also be used to send fourth event marking information to the host computer;

[0027] The host computer may also be used to synchronize and analyze the multimodal data, the first event marker information and / or the second event marker information, and the fourth event marker information.

[0028] In a possible implementation, the system may further include a first external device;

[0029] The first external device can be used to send fifth event marking information to the multimodal data edge computing device and the multimodal data synchronization edge computing device;

[0030] The multimodal data edge computing device and the multimodal data synchronization edge computing device may also be used to receive the fifth event marking information and send the fifth event marking information to the host computer;

[0031] The host computer may also be used to synchronize and analyze the multimodal data, the first event marker information and / or the second event marker information, and the fifth event marker information.

[0032] In a possible implementation, the system may further include a second external device;

[0033] The above-mentioned multimodal data edge computing device and the multimodal data synchronization edge computing device can also be used to send the above-mentioned first event marking information and / or second event marking information to a second external device.

[0034] In a possible implementation, the at least one sensor may send the multimodal data to the host computer wirelessly.

[0035] The multimodal data synchronization edge computing device and synchronization system based on human intelligence provided by the embodiment of the present application can recognize the behavior and / or voice of the testee through the AI ​​module, and send out event marking information when the recognition result matches the target behavior and / or target voice, so that the testee can accurately trigger the event mark as long as the predetermined target behavior or the predetermined target voice is made when the visual, auditory, tactile, taste, etc. are stimulated, thereby avoiding the inaccuracy of the test analysis caused by the personnel implementing the experiment triggering the event mark artificially, effectively improving the objectivity of the event mark, and thus helping to accurately analyze the impact of the stimulus event on the multimodal data. In addition, the host computer can synchronize the data sent by different sensors after the host computer receives the event mark information according to the event mark information, and then synchronize the data sent by different sensors before the host computer receives the event mark information. Therefore, the event mark information also helps to achieve the synchronization of multimodal data sent by different sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 An embodiment of a multimodal data synchronization edge computing device based on human intelligence provided in an embodiment of the present application;

[0038] Figure 2a Another embodiment of the multimodal data synchronization system based on human intelligence provided in the embodiment of the present application;

[0039] Figure 2b Another embodiment of the multimodal data synchronization system based on human intelligence provided in the embodiment of the present application;

[0040] Figure 2c Another embodiment of the multimodal data synchronization system based on human intelligence provided in the embodiment of the present application;

[0041] Figure 3 Another embodiment of the multimodal data synchronization system based on human factor intelligence provided by the embodiment of the present application;

[0042] Figure 4 Another embodiment of the multimodal data synchronization system based on human intelligence provided in the embodiment of the present application;

[0043] Figure 5Another embodiment of the multimodal data synchronization system based on human intelligence provided in the embodiment of the present application. DETAILED DESCRIPTION

[0044] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0045] In the embodiment of the present application, the multimodal data may include environment-related data and / or human-related data. The environment-related data may include any one or more of position data, humidity data, temperature data, chromaticity data, brightness data, noise data, weather data (such as air pressure), road condition data, traffic data, stimulus signal data, and event or signal event marker data. Human body related data may include skin electrical data, skin temperature (SKT) data, pulse data (such as photoplethysmography (PPG)), blood pressure (BP) data, blood oxygen data (such as periphery capillary oxygen saturation (SPO2)), such as skin electrical response (Electrodermal Activity, EDA), electrocardiogram data, electromyography data, muscle oxygen data, respiratory data (such as respiratory rate (RESP)), biomechanical data (such as single point force-sensitive resistor (FSR) value), near-infrared brain imaging data, electroencephalogram data, transcranial stimulation data, heart rate variability data, heart rate data and behavioral data.

[0046] In the embodiment of the present application, the event marking information may refer to information about an event that causes a change in human functions by stimulating human vision, hearing, vibration (touch), smell, taste, etc. For example, such an event may be that sound exceeds a sound threshold, stimulating human hearing, thereby causing a change in human functions, or light exceeds a light threshold, stimulating human vision, thereby causing a change in human functions, or temperature exceeds a temperature threshold, stimulating the human body, causing a change in human functions, and so on. Exemplarily, the event marking information may also be information generated by a certain behavior triggering the generation of corresponding information, and so on. Exemplarily, different events represented by the event marking information may be defined by different values ​​of one or more fields.

[0047] An embodiment of a multimodal data synchronization edge computing device based on human intelligence is Figure 1 The multimodal data synchronization edge computing device 100 shown. Figure 1As shown, the multimodal data synchronization edge computing device 100 may include a control module 110, an environmental data acquisition module 121, a biosensor (Biosensors, BIO) data acquisition module 122, an EXG data acquisition module 123, an EEG data acquisition module 124, a first interface 131, a second interface 132, a third interface 141 and a fourth interface 142, etc.

[0048] The environment data acquisition module 121 may be used to connect to an environment sensor and receive environment data sent by the environment sensor.

[0049] The BIO data acquisition module 122 can be used to connect to the BIO sensor and receive the BIO data sent by the BIO sensor, such as PPG, EDA, SPO2, SKT, RESP, BP, FSR and other biological data.

[0050] The EXG data acquisition module 123 can be used to connect to EXG sensors, such as electromyography sensor (EMG), electrocardiogram sensor (ECG), etc., and receive bioelectric data such as electromyography data and electrocardiogram data sent by the EXG sensor.

[0051] The EEG data acquisition module 124 may be used to connect to an electroencephalogram sensor (EEG) to receive bioelectric data such as electroencephalogram data sent by the EEG.

[0052] BIO, EMG, ECG and EEG can be worn on the subject to detect the subject and obtain BIO data, electromyography data, electrocardiogram data and electroencephalogram data, etc.

[0053] The first interface 131 can be used to be connected to the host computer, and data such as environmental data, BIO data, electromyographic data, electrocardiographic data and electroencephalographic data are transmitted to the host computer. The host computer records these data through supporting software. Exemplarily, the first interface 131 can be a type A (Type-A) USB interface, a type B (Type-B) USB interface, a type C (Type-C) USB interface, etc. Among them, the host computer can be a server, an industrial computer, or a terminal such as a mobile phone or a computer. Software for data reception, processing, analysis, and / or acquisition control can be installed on the host computer. After the host computer receives the event marker information and the multimodal data collected by the sensor through the first interface 131, the multimodal data collected at the same time can be synchronized based on the time axis of the event marker information through the software, so as to realize the processing and analysis of data of multiple different modes collected at the same time, so as to obtain the impact of the stimulus event that generates the event marker on various aspects of human body function. For example, to test the impact of fatigue on the driver's physical function, the test subject can wear or attach BIO, ECG, EEG and other sensors to collect the test subject's multimodal data and send it to the host computer. When the test subject is fatigued, the event marker is triggered and the event marker information is sent to the host computer. The host computer aligns the multimodal data through the event marker information, and can obtain the corresponding relationship between the multimodal data, such as the corresponding ECG data, EMG data, EEG data, etc. when the blood pressure value is what, and according to the position of the event marker information, the corresponding values ​​of the test subject's multimodal data at different times in the fatigue state can be obtained, and then these multimodal data are analyzed to determine the state of the corresponding human body function and the possible consequences in the driving state.

[0054] The control module 110 can be used to generate event marking information (the event marking information generated by light triggering can be collectively referred to as third event marking information) when the light emitted by the optical module exceeds the threshold, and send the event marking information through the first interface 131, the second interface 132 and the fourth interface 142.

[0055] The control module 110 can also be used to perform communication protocol processing on the event marker information, such as encapsulating the corresponding communication protocol packet header for the event marker information, so that the event marker information can meet the communication protocol requirements used by the first interface 131, the second interface 132 and the fourth interface 142, and send it to the first interface 131, the second interface 132 and the fourth interface 142. For example, the control module 110 can encapsulate the event marker information into a data packet that complies with the USB communication protocol, or process it into a data packet that can be sent by a serial peripheral interface (Serial Peripheral interface, SPI), and so on.

[0056] The first interface 131 is also used to transmit the event marking information to the host computer.

[0057] For example, the test subject can visually perceive the light emitted by the optical module. When the light exceeds the threshold, it can stimulate the test subject's function to change. When the multimodal data synchronization edge computing device 100 collects multimodal data, the control module 110 generates event marking information under the triggering of the stimulus event, and synchronizes the multimodal data and event marking information to the host computer. The supporting software of the host computer can analyze the relationship between the stimulus event and the human body function based on the event marking information and the collected multimodal data, and determine the impact of the stimulus event on the human body function.

[0058] The second interface 132 is coupled to the control module 110, and the control module 110 can also be used to send the event marker information to other software (third-party software) installed in the host computer through the second interface 132. Exemplarily, the second interface 132 can be a type A USB interface, a type B USB interface, a type C USB interface, etc.

[0059] In some embodiments, the second interface 132 can be used for third-party software to obtain the above event marking protocol from the supporting software, so that the third-party software can parse the event marking information sent by the control module 110 through the event marking protocol to obtain the meaning or content of the event marking information.

[0060] In some embodiments, the supporting software in the host computer can send the event marking protocol to the control module through the first interface 131. After the control module 110 parses the content sent by the first interface 131, it determines that the event marking protocol is to be sent to the third-party software, and then encapsulates the event marking protocol through the communication protocol and sends it to the third software through the second interface 132. For example, the multimodal data synchronization edge computing device 100 and the supporting software are designed or produced by the same design manufacturer or the same manufacturer. The multimodal data synchronization edge computing device 100 and the supporting software are provided with the same event marking protocol, configuration protocol, etc., so that the supporting software can parse the event marking information sent by the multimodal data synchronization edge computing device 100 and obtain the meaning of the event marking information or the content to be expressed. If the user uses third-party software from different design manufacturers or manufacturers to interact with the multimodal data synchronization edge computing device 100, the event marking protocol can be obtained from the supporting software through the multimodal data synchronization edge computing device 100, and the event marking protocol can be used to parse the event marking information sent by the multimodal data synchronization edge computing device 100.

[0061] In some embodiments, the third-party software may issue a request to obtain the event marking protocol through the second interface 132 in response to the user's operation. After receiving the request, the control module 110 sends it to the host computer through the first interface 131. In response to the request, the supporting software sends the event marking protocol to the control module 110 through the first interface 131. After processing, the control module 110 sends the event marking protocol to the third-party software through the second interface 132.

[0062] The fourth interface 142 can be used to connect to the first external device and / or the second external device, and can be used to convert the transistor-transistor logic (TTL) level of the event mark information sent by the control module 110 into the TTL level supported by the first external device and / or the second external device, and then transmit it to the first external device and / or the second external device. Exemplarily, the fourth interface 142 can communicate with the control module 110 via SPI. The first external device can be a sensor that supports input event marking and supports output event marking (such as an environmental sensor, a BIO sensor, etc.), and the second external device can be a sensor that supports input event marking (such as an environmental sensor, a BIO sensor, etc.). For example, when the first external device and the second external device are not designed or produced by the same manufacturer as the multimodal data synchronization edge computing device based on human intelligence, the signal logic level used by the circuit board may be inconsistent. In order for the first external device and the second external device to be able to cooperate with the multimodal data synchronization edge computing device with different logic levels, the above-mentioned first interface 131 and / or the second interface 132 can be set in the multimodal data synchronization edge computing device based on human intelligence.

[0063] In some embodiments, the first external device and / or the second external device may be further configured to detect multimodal data.

[0064] In some embodiments, the first external device and / or the second external device may be further configured to send the detected multimodal data to the host computer in real time.

[0065] The third interface 141 can be used to connect to the first external device, to receive the event marker information sent by the first external device (the event marker information sent by the first external device can be collectively referred to as the fifth event marker information), and to convert the event marker information into a TTL level supported by the multimodal data synchronization edge computing device 100, and then send it to the control module 110. Exemplarily, the third interface 141 can communicate with the control module 110 via SPI.

[0066] The control module 110 can also be used to process the event marking information sent by the third interface 141 through the transmission protocol, and then transmit it to the supporting software of the host computer through the first interface 131, transmit it to other software of the host computer through the second interface 132, and transmit it to the second external device through the fourth interface 142.

[0067] Exemplarily, the functions of the control module 12 can be implemented by a field programmable gate array (FPGA), a microcontroller unit (MCU), a central processing unit (CPU), etc.

[0068] In some embodiments, the multimodal data synchronization edge computing device based on human intelligence may include more or fewer modules and / or interfaces than the multimodal data synchronization edge computing device 100. For example, the multimodal data synchronization edge computing device based on human intelligence may include any one or more modules of the environmental data acquisition module 121, the BIO data acquisition module 122, the EXG data acquisition module 123, and the EEG data acquisition module 124, the EXG data acquisition module 123 and the EEG data acquisition module 124 may be combined into one module, or the EXG data acquisition module 123 may be divided into an electromyography data acquisition module and an electrocardiogram data acquisition module, or there may be multiple BIO data acquisition modules 122, or the multimodal data synchronization edge computing device based on human intelligence may further include a power supply module, or the multimodal data synchronization edge computing device based on human intelligence may further include an optical port for converting the light emitted by the optical module into an electrical signal, and so on.

[0069] In some embodiments, the multimodal data synchronization edge computing device based on human intelligence may further include an artificial intelligence (AI) module based on the above embodiments.

[0070] Another embodiment of the multimodal data synchronization edge computing device based on human intelligence is Figure 2a The multimodal data synchronization edge computing device 101 shown. As shown in FIG. 2 , the multimodal data synchronization edge computing device 101 includes: a control module 110 , a first interface 131 , a second interface 132 , a fourth interface 142 and an AI module 150 .

[0071] The AI ​​module 150 is used to connect to a video capture device such as a camera, and can recognize the collected video signals through the AI ​​model. The video capture device is used to capture the video signal of the test subject in the monitoring area in real time and transmit the video signal to the AI ​​module 150.

[0072] In some embodiments, the AI ​​module 150 can also be used to perform denoising, enhancement, decoding and other processing on the video signal to improve the quality and clarity of the video image, thereby achieving the purpose of improving the accuracy of AI recognition.

[0073] In some embodiments, the AI ​​module 150 performs AI recognition on the video signal, which may include one or more of video segmentation, feature extraction, target detection, target tracking, behavior analysis, and event recognition.

[0074] Video segmentation may include segmenting a video into frames of images.

[0075] Feature extraction may include using a computer vision algorithm to extract key action features of the subject from an image of each frame obtained by video segmentation.

[0076] Object detection may include key action features extracted based on features, and using object detection algorithms to accurately identify and locate specific actions of subjects in the video.

[0077] The target may include the action detected on the target, and track between consecutive frames obtained by video segmentation to ensure the continuity and accuracy of the target.

[0078] Behavior analysis and event recognition may include in-depth analysis of the target's behavior based on its trajectory and action, identifying abnormal behavior or specific events, and generating event marking information (the event marking information generated by the video trigger may be collectively referred to as the first event marking information) to achieve behavioral event marking of the subject. Exemplarily, the event marking information may be some fields, one or more of which may be used to represent the target action, such as the subject making a patting action on the head. Exemplarily, when the AI ​​module 150 identifies the target action, the value of the one or more fields is set to a predetermined value to complete the behavioral event marking.

[0079] The AI ​​module 150 may also be used to send the event marker information to the control module 110. In some embodiments, the AI ​​module 150 may also encapsulate the event marker information into a data packet that can be received by the control module 110. For example, the AI ​​module 150 may communicate with the control module 110 via SPI. Accordingly, the AI ​​module 150 may encapsulate the event marker information into a data packet that can be received by SPI and send it to the control module 110.

[0080] After receiving the data packet, the control module 110 can parse out the event marking information, and encapsulate the event marking information into data packets that can be transmitted by the first interface 131, the second interface 132, and the fourth interface 142, and send them to the first interface 131, the second interface 132, and the fourth interface 142 respectively. The first interface 131 can also be used to send the event marking information of the AI ​​module 150 to the supporting software of the host computer, and the second interface 132 can also be used to send the event marking information of the AI ​​module 150 to other software of the host computer. The fourth interface 142 can also be used to send the event marking information of the AI ​​module 150 to a second external device.

[0081] For example, the AI ​​model used for AI recognition in the AI ​​module 150 can be Figure 2b The AI ​​software shown is trained. The AI ​​software in the host computer can first build a deep learning model, and use a large amount of labeled data to train the deep learning model, so that it can learn the mapping relationship between the input data and the correct output recognition result during the training process, so that it has the ability to accurately analyze and recognize new videos captured by the camera, and can recognize and classify objects in the video, and recognize target behaviors. After the AI ​​software has trained a preliminary model, that is, after the model has the ability to analyze and recognize, it is evaluated and optimized to improve the accuracy and efficiency of the model. For example, the AI ​​software in the host computer can use the initially trained model to recognize the video sent by the camera through the multimodal data synchronization edge computing device 101 to determine whether the model meets the requirements.

[0082] Once the model is trained, Figure 2b As shown, the trained model is sent from the AI ​​software in the host computer to the control module 110 in the multimodal data synchronization edge computing device 102 through the first interface 131. Then, the control module 110 sends the model to the AI ​​module. In the subsequent experimental process, the AI ​​module can use the model to perform real-time analysis and recognition of the received video stream, and mark the event when the recognition result matches the target behavior, and feed back the event marking information to the control module 110.

[0083] In some embodiments, the multimodal data synchronization edge computing device based on human intelligence may have fewer interfaces than the multimodal data synchronization edge computing device 101, such as only any one or two of the first interface, the second interface, and the fourth interface.

[0084] In some embodiments, Figure 2a The AI ​​module 150 in the multimodal data synchronization edge computing device 101 shown can be as shown in FIG. Figure 2cAs shown, it is used to connect with a microphone, recognize the voice signal introduced by the microphone, and generate event marking information (the event marking information generated by voice triggering can be collectively referred to as second event marking information) when the recognition result matches the target voice, to achieve voice event marking. The way in which the AI ​​module 150 performs voice event marking can be similar to the principle of video event marking in the above embodiment.

[0085] In the above embodiments, the multimodal data synchronization edge computing device based on human factors intelligence can be applied to any human factors engineering experiment occasions that require human active triggering through the AI ​​module. For example, the multimodal data synchronization edge computing device can identify the behavior of the test subject during the human factors engineering experiment through the AI ​​module without the implementation of the experimenter. When the test subject makes a set behavior such as a gesture, expression or eye contact, it can trigger the AI ​​module to mark the event and realize the human factors engineering experiment.

[0086] Another embodiment of the multimodal data synchronization edge computing device based on human intelligence is Figure 3 The multimodal data synchronization edge computing device 103 shown. Figure 3 As shown, the difference between this embodiment and the above embodiment is that the AI ​​module 150 can be used for both video event marking and voice event marking, and the multimodal data synchronization edge computing device 103 can further include a left channel interface 161 and a right channel interface 162 for receiving a sound source from the host computer. The sound source of the host computer is used to stimulate the hearing of the test subject. When the volume of the sound source exceeds the threshold, the multimodal data synchronization edge computing device 103 can mark the sound event (that is, mark the event that the sound exceeds the threshold), generate event marking information (the event marking information generated by the sound trigger can be collectively referred to as the fourth event marking information), and send it to the supporting software, other software, first external device and second external device of the host computer through the first interface 131, the second interface 132 and the fourth interface 142 respectively.

[0087] like Figure 3 As shown, when the host computer is online, the sensors worn by the test subject can send the collected data (such as biotype data and bioelectric type data) in real time through electromagnetic waves. The host computer can receive the data in real time through the wireless communication module and send it to the supporting software for analysis and other processing.

[0088] Another embodiment of the multimodal data synchronization edge computing device based on human intelligence is Figure 4 The multimodal data synchronization edge computing device 104 shown. Figure 4As shown, the difference between the multimodal data synchronization edge computing device 104 and the above-mentioned embodiment is that it further includes a wireless communication module 170 and a storage module 180, and the AI ​​module 150 can be used not only for video event marking but also for voice event marking.

[0089] The storage module 180 is connected to the control module 110 and can be used to store event marker information. For example, when the control module 110 stores the first event marker information (the first event marker information may be the event information of the target behavior) or the second event marker information (the second event marker information may be the event information of the target voice) issued by the AI ​​module in the storage module 180, when the light emitted by the light module exceeds the threshold, the control module 110 generates the third event marker information (the third event marker information may be the event information of the light exceeding the threshold), and sends the third event marker information to the storage module 180 for storage. For another example, when the sound volume received by the left channel interface 161 and the right channel interface 162 exceeds the threshold, the control module 110 stores the generated fourth event marker information (the fourth event marker information may be the event information of the sound volume exceeding the threshold) in the storage module 180. For another example, the control module 110 stores the fifth event marker information received by the third interface 141 from the first external device in the storage module 180, and so on.

[0090] The wireless communication module 170 is used to transmit any one or more of the above-mentioned first event marker information, the above-mentioned second event marker information, the above-mentioned third event marker information and the above-mentioned fourth event marker information by wireless means. For example, when the sensor worn by the test subject is in an offline storage state and cannot send the detected data to the host computer in real time, the control module 110 can send the generated third event marker information and / or fourth event marker information, and / or the first event marker information and / or second event marker information received from the AI ​​module 150 to the sensor through the wireless communication module 170. When there are multiple sensors, the wireless communication module 170 can synchronize the event marker information to the multiple sensors by multicasting. When there are multiple test subjects, such as Figure 5 As shown, the wireless communication module 170 can also synchronize the event marker information to the sensors worn by the multiple test subjects by means of multicast.

[0091] In this embodiment, the multimodal data synchronization edge computing device 104 stores event marking information through the storage module 180. When the event marking information sent by the wireless communication module 170 is lost, or when the wireless communication module 170 sends the event marking information but the sensor does not receive it, it can provide backup event marking information for the subsequent host computer to analyze and process the multimodal data, thereby improving the reliability of data synchronization of the multimodal data synchronization edge computing device.

[0092] Exemplarily, the wireless communication module may be a Bluetooth module (such as a Bluetooth Low Energy (BLE) module), a Wireless Fidelity (WiFi) module, a Near Field Communication (NFC) module, or the like.

[0093] In the above embodiment, the multimodal data synchronization edge computing device based on human intelligence uses a storage module and / or a wireless communication module. After the software AI of the host computer has trained the AI ​​model and deployed it to the AI ​​module, the multimodal data synchronization edge computing device can further perform offline event marking on external devices in the absence of a host computer to achieve data synchronization.

[0094] The multimodal data synchronization edge system based on human intelligence provided in the embodiment of the present application may include the multimodal data synchronization edge computing device based on human intelligence provided in any of the above embodiments, and any one or more of a host computer, an optical module, at least one sensor, a first external device, and a second external device. Figure 1 As shown, the multimodal data synchronization edge system based on human factors intelligence may include a multimodal data synchronization edge computing device 101, an environmental sensor, a BIO sensor, an EXG sensor, an EEG sensor, an optical module, a host computer, a first external device, and a second external device. Figure 2a As shown, the multimodal data synchronization edge system based on human intelligence may include a multimodal data synchronization edge computing device 101, a video capture device, a host computer, and a second external device. Figure 2b As shown, the multimodal data synchronization edge system based on human intelligence may include a multimodal data synchronization edge computing device 102 and a host computer. Figure 2c As shown, the multimodal data synchronization edge system based on human intelligence may include a multimodal data synchronization edge computing device 101, a voice input device, a host computer, and a second external device. Figure 3 As shown, the multimodal data synchronization edge system based on human intelligence may include a multimodal data synchronization edge computing device 103, a sensor, an optical module, a host computer, a first external device, and a second external device. Figure 4 As shown, the multimodal data synchronization edge system based on human factors intelligence may include a multimodal data synchronization edge computing device 104, a sensor, an optical module, a first external device and a second external device.

[0095] It should be understood that the division of the modules in the above embodiments is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or physically separated. And these modules can be implemented in the form of some modules called by software through processing elements, and some modules can be implemented in the form of hardware. For example, the processor can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently.

[0096] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0097] Those skilled in the art will appreciate that the modules described in the embodiments disclosed herein can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether the functions of these modules are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0098] In several embodiments provided in the present application, if any function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0099] The above is only a specific implementation of the present application. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multimodal data synchronization edge computing device based on human intelligence, characterized in that: include: An artificial intelligence AI module, a control module and a first interface, wherein the AI ​​module and the first interface are coupled to the control module; The AI ​​module is used to couple with a video capture device and / or a voice input device, to recognize a video signal acquired by the video capture device, and to issue a first event marking information when a recognition result matches a target behavior, and / or to recognize a voice signal acquired by the voice input device, and to issue a second event marking information when a recognition result matches a target voice. The control module is used for performing communication protocol encapsulation processing on the first event marking information and / or the second event marking information, and sending the encapsulation processing to the first interface; The first interface is used to couple to a host computer and send the first event marking information and / or the second event marking information to the host computer.

2. The device according to claim 1, characterized in that It also includes a wireless communication module, which is coupled to the control module. The control module is also used to send the first event marking information and / or the second event marking information to at least one sensor through the wireless communication module.

3. The device according to claim 2, characterized in that It also includes a storage module, which is coupled to the control module. The control module is also used to store the first event marking information and / or the second event marking information in the storage module.

4. The device according to any one of claims 1 to 3, characterized in that Also includes a second interface, the second interface is coupled to the control module and is also used to couple to a third-party software terminal; The control module is further configured to send the first event marking information and / or the second event marking information to the third-party software terminal via the second interface.

5. The device according to claim 4, characterized in that The control module is also used to obtain an event marking protocol from the host computer through the first interface, and send the event marking protocol to a third-party software terminal through the second interface, wherein the event marking protocol is used to define the first event marking information and / or the second event marking information.

6. The device according to any one of claims 1 to 3, characterized in that The control module is also used to connect to the optical module, receive the optical signal sent by the optical module, and generate third event marking information when the optical signal exceeds the first threshold, and send the third event marking information to the host computer through the first interface.

7. The device according to any one of claims 1 to 3, characterized in that: Also includes an audio interface, the audio interface is coupled to the control module and is used to send the received audio signal to the control module; The control module is further configured to generate fourth event marking information when the audio signal exceeds a second threshold, and send the fourth event marking information to the host computer through the first interface.

8. The device according to any one of claims 1 to 3, characterized in that It also includes a third interface, the third interface is coupled to the control module, and is also used to couple to a first external device, and convert the level of the fifth event mark information sent by the first external device into a TTL level supported by the control module; The control module is further configured to receive the fifth event marker information through the third interface, and The fifth event marking information is sent to the host computer through the first interface.

9. The device according to any one of claims 1 to 3, characterized in that: Also includes a fourth interface, the fourth interface is coupled to the control module and is also used to couple to a second external device; The control module is further configured to send the first event marking information and / or the second event marking information to the second external device through the fourth interface.

10. A multimodal data synchronization system based on human intelligence, characterized in that: It comprises a host computer, at least one sensor, and a multimodal data synchronization edge computing device as claimed in any one of claims 1 to 9, and further comprises a video capture device and / or a voice input device; The at least one sensor is used to send the collected multimodal data to the host computer; The video capture device and / or the voice input device is used to trigger the multimodal data synchronization edge computing device event marker of the multimodal data edge computing device, and the event marker is used to generate first event marker information and / or second event marker information; The multimodal data edge computing device and the multimodal data synchronization edge computing device are used to send the first event marking information and / or the second event marking information to the host computer; The host computer is used to synchronize and analyze the multimodal data and the first event marker information and / or the second event marker information.

11. The system according to claim 10, characterized in that The multimodal data edge computing device and the multimodal data synchronization edge computing device send the first event marking information and / or the second event marking information to the at least one sensor when the at least one sensor is offline.

12. The system according to claim 10 or 11, characterized in that Also includes optical modules; The optical module is used to trigger the multimodal data edge computing device and the multimodal data synchronization edge computing device to perform optical event marking, and the optical event marking is used to generate third event marking information; The multimodal data edge computing device multimodal data synchronization edge computing device is also used to send the third event marking information to the host computer; The host computer is also used to synchronize and analyze the multimodal data, the first event marker information and / or the second event marker information, and the third event marker information.

13. The system according to claim 10 or 11, characterized in that: The host computer is also used to trigger the multimodal data edge computing device and the multimodal data synchronization edge computing device to perform acoustic event marking, and the acoustic event marking is used to generate fourth event marking information; The multimodal data edge computing device multimodal data synchronization edge computing device is also used to send the fourth event marking information to the host computer; The host computer is also used to synchronize and analyze the multimodal data, the first event marking information and / or the second event marking information, and the fourth event marking information.

14. The system according to claim 10 or 11, characterized in that Also included is a first external device; The first external device is used to send fifth event marking information to the multimodal data edge computing device multimodal data synchronization edge computing device; The multimodal data edge computing device multimodal data synchronization edge computing device is also used to receive the fifth event marking information and send the fifth event marking information to the host computer; The host computer is also used to synchronize and analyze the multimodal data, the first event marking information and / or the second event marking information, and the fifth event marking information.

15. The system according to claim 10 or 11, characterized in that Also included is a second external device; The multimodal data edge computing device and the multimodal data synchronization edge computing device are also used to send the first event marking information and / or the second event marking information to the second external device.

16. The system according to claim 10 or 11, characterized in that The at least one sensor sends the multimodal data to the host computer wirelessly.

Images