Multi-modal data transmission method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses a multi-mode data transmission method. According to the method, a global clock is introduced into terminal equipment, so that data of different modals are printed with global timestamps while being collected, and data alignment processing based on the global timestamps is locally realized on the terminal equipment. And for the aligned multi-modal data, a predefined multi-modal data transmission protocol is adopted to efficiently organize and compress the synchronized multi-modal data with the same global timestamp into a single and structured data packet, and the single and structured data packet is sent through a network connection. And the server side can extract the synchronous multi-modal data only through the global timestamp at the head of the data packet, so that accurate data alignment is realized. According to the method, the data transmission efficiency can be effectively improved, and the data analysis and data alignment processes of the server side are greatly simplified. The multi-mode data transmission device, the storage medium and the electronic equipment in the embodiment of the specification also have the above beneficial effects.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to a multimodal data transmission method, apparatus, storage medium, and electronic device. Background Technology

[0002] Smart glasses and other head-mounted smart terminal devices integrate various sensors, such as cameras, microphones, inertial measurement units (IMUs), and ambient light sensors, continuously generating massive amounts of multimodal data (video, audio, motion posture, position, etc.). Currently, the industry-standard data processing workflow involves the various sensors on the terminal device operating independently, and the resulting data streams being transmitted to the server independently. After receiving this data, the server needs to perform complex data alignment and correlation logic. This approach results in low transmission efficiency, high server-side resource consumption, and an inability to guarantee high-precision time alignment. Summary of the Invention

[0003] One or more embodiments of this specification provide a multimodal data transmission method, apparatus, storage medium, and electronic device that can at least partially solve the above-mentioned technical problems.

[0004] Firstly, a multimodal data transmission method is provided, applicable to terminal devices, the method comprising: In response to a data acquisition request, multimodal data is synchronously acquired based on a global clock, and a global timestamp is added to the acquired data to obtain target data of at least one modality. According to a preset multimodal data transmission protocol, target data of at least one modality having the same global timestamp are encoded into a data packet, and the global timestamp is carried in the data header of the data packet; The data packet is transmitted to the server so that the server can extract the target data of different modalities that have been synchronized based on the global timestamp carried in the data packet.

[0005] As an optional implementation of the method described in the first aspect, the terminal device includes a global clock source; in response to a data acquisition request, multimodal data synchronous acquisition is performed based on the global clock, specifically including: In response to the data acquisition request, a unified hardware clock is provided to the multimodal data acquisition module of the terminal device through the global clock source, and the multimodal data acquisition module is triggered to synchronously acquire data through the hardware clock.

[0006] As an optional implementation of the method described in the first aspect, the method further includes: Before packaging the target data of the at least one modality, the target data of the at least one modality is aligned based on the global timestamp.

[0007] Specifically, based on the global timestamp, aligning the target data of the at least one modality includes: A time window is set for each of the global timestamps, and within the time window, the target data of at least one modality with the global timestamp is collected and stored in a buffer. The target data of at least one modality within the same time window in the buffer are used as synchronously acquired data.

[0008] As an alternative implementation of the method described in the first aspect, the data packet includes the data header and data blocks, each data block corresponding to a modality of the target data.

[0009] Specifically, the data block includes a sub-data header, which includes the data type, data length, and compression algorithm identifier of the data block.

[0010] Specifically, the data packet also includes a cyclic redundancy check code at the end.

[0011] Secondly, a multimodal data transmission device is provided, suitable for terminal equipment, including: The multimodal data acquisition module is used to respond to data acquisition requests, perform multimodal data synchronous acquisition based on a global clock, and add a global timestamp to the acquired data to obtain target data of at least one modality. The data synchronization module is used to provide the global clock for the multimodal data acquisition module; The data processing and encapsulation module is used to encode target data of at least one modality having the same global timestamp into a data packet according to a preset multimodal data transmission protocol, and to carry the global timestamp in the data header of the data packet; The data transmission module is used to transmit the data packet to the server, so that the server can extract the target data of different modalities that have been synchronized based on the global timestamp carried in the data packet.

[0012] As an optional embodiment of the apparatus described in the second aspect, the data synchronization module includes a global clock source; the data synchronization module is specifically used for: In response to the data acquisition request, a unified hardware clock is provided to the multimodal data acquisition module through the global clock source, and the multimodal data acquisition module is triggered to synchronously acquire data through the hardware clock.

[0013] As an optional embodiment of the apparatus described in the second aspect, the data processing and encapsulation module is specifically used for: Before packaging the target data of the at least one modality, the target data of the at least one modality is aligned based on the global timestamp.

[0014] Specifically, the data processing and encapsulation module is used for: A time window is set for each of the global timestamps, and within the time window, the target data of at least one modality with the global timestamp is collected and stored in a buffer. The target data of at least one modality within the same time window in the buffer are used as synchronously acquired data.

[0015] As an alternative implementation of the apparatus described in the second aspect, the data packet includes the data header and data blocks, each data block corresponding to a modality of the target data.

[0016] Specifically, the data block includes a sub-data header, which includes the data type, data length, and compression algorithm identifier of the data block.

[0017] Specifically, the data packet also includes a cyclic redundancy check code at the end.

[0018] Thirdly, a terminal device is provided, which includes the aforementioned multimodal data transmission device.

[0019] Fourthly, a smart glasses device is provided, which includes the aforementioned multimodal data transmission device.

[0020] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when the computer program is run on an electronic device, causes the electronic device to perform the above-described multimodal data transmission method.

[0021] Sixthly, an electronic device is provided, comprising: At least one memory for storing programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the multimodal data transmission method described above.

[0022] The beneficial effects of the multimodal data transmission method described in the embodiments of this specification are as follows: This method introduces a global clock into the terminal device, enabling data from different modalities to be timestamped globally upon acquisition. Data alignment based on this global timestamp is then performed locally on the terminal device. For the aligned multimodal data, a predefined multimodal data transmission protocol is used to efficiently organize and compress synchronized multimodal data with the same global timestamp into a single, structured data packet, which is then sent via a network connection. The server can extract the synchronized multimodal data simply by examining the global timestamp in the packet header, achieving accurate data alignment. This method effectively improves data transmission efficiency and significantly simplifies the server-side data parsing and alignment process.

[0023] The multimodal data transmission device, storage medium, and electronic device described in the embodiments of this specification also have the above-mentioned beneficial effects. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A schematic flowchart of a multimodal data transmission method described in an embodiment of this specification is shown as an example.

[0026] Figure 2 A schematic diagram illustrating the structure of a data packet as described in an embodiment of this specification is shown as an example.

[0027] Figure 3 A schematic diagram of the structure of a multimodal data transmission device according to an embodiment of this specification is shown as an example.

[0028] Figure 4 An exemplary schematic diagram of the structure of an electronic device provided in an embodiment of this specification is shown. Detailed Implementation

[0029] First, it should be noted that the terminology used in the embodiments of this invention is for the purpose of describing specific embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0030] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of the embodiments. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0032] With the development of artificial intelligence and augmented reality (AR) technologies, head-mounted smart devices such as smart glasses are gradually becoming important information collection and interaction terminals. These devices integrate a variety of sensors, such as cameras, microphones, inertial measurement units (IMUs), and ambient light sensors, and can continuously generate massive amounts of multimodal data (video, audio, motion posture, position, etc.).

[0033] Currently, the industry-standard data processing workflow is as follows: various sensors in smart glasses and other terminal devices operate independently, and the resulting data streams (such as video streams, audio streams, and IMU sensor data streams) are also transmitted independently to the server via a wireless network. After receiving this data, the server needs to perform complex preprocessing steps, including: Data stream matching: Associate different modal data belonging to the same time point.

[0034] Timestamp alignment: Relies on the local timestamps of each data stream and performs later synchronization through methods such as Network Time Protocol (NTP). The accuracy is greatly affected by network jitter and system latency, and is usually only at the millisecond level or even worse.

[0035] Data fusion: The aligned data is input into subsequent processing modules (such as multimodal AI models) for analysis and inference.

[0036] This approach has the following drawbacks: 1. Low synchronization accuracy: Independent transmission and subsequent software synchronization methods cannot guarantee high-precision time alignment (especially under microsecond-level requirements). Some subsequent processing modules have extremely high requirements for data synchronization accuracy, such as visual-inertial odometry (VIO) applications. Insufficient accuracy will directly lead to a decrease in AI model performance and distortion of analysis results.

[0037] 2. Complex server-side processing: The server side needs to deploy additional data alignment and association logic, which increases the complexity and computational overhead of the system.

[0038] 3. Low transmission efficiency: Multiple independent data streams require the establishment and maintenance of multiple transmission connections, resulting in high protocol header overhead and difficulty in cross-modal joint compression optimization.

[0039] 4. High resource consumption: The complex server-side processing leads to higher computing resource consumption and latency.

[0040] In view of the above, one or more embodiments of this specification provide a multimodal data transmission method, apparatus, storage medium, and electronic device that can at least partially overcome the above-mentioned technical defects.

[0041] The multimodal data transmission method, apparatus, storage medium, and electronic device described in one or more embodiments of this specification will be further described in detail below with reference to the accompanying drawings and specific embodiments. However, this detailed description does not constitute a limitation on the embodiments of this specification.

[0042] One or more embodiments of this specification provide a multimodal data transmission method suitable for terminal devices. Please refer to... Figure 1 , Figure 1 A flowchart illustrating the method is shown, as follows. Figure 1 As shown, the method includes steps S100 to S104.

[0043] S100: In response to a data acquisition request, multimodal data is synchronously acquired based on a global clock, and a global timestamp is applied to the acquired data to obtain target data of at least one modality.

[0044] The terminal device described in this embodiment can be a wearable / portable device equipped with a multimodal data acquisition module and a certain computing capability, such as smart glasses, smart action cameras, etc.

[0045] The aforementioned multimodal data acquisition module includes sensors of different modalities, such as cameras, microphones, and IMUs.

[0046] The aforementioned data acquisition request is initiated by the user by triggering the terminal device in response to the target scenario requiring data collection. The multimodal data acquisition module in the terminal device responds to the user's request by calling the corresponding sensors to collect data in at least one modality of the target scenario, obtaining target data in at least one modality. The terminal device locally sets up a high-precision global clock source. In response to the data acquisition request, this global clock source, triggered by hardware or via the system terminal, unifies the global clock for the multimodal data acquisition module, ensuring that the module marks the data with a unified global timestamp while acquiring it.

[0047] Specifically, assuming the global hardware clock provided by the global clock source is T0, the various sensors in the multimodal data acquisition module are triggered by hardware to simultaneously acquire data at the global hardware clock T0, and the acquired target data is stamped with a global timestamp T0.

[0048] S102: According to the preset multimodal data transmission protocol, target data of at least one mode with the same global timestamp are encoded into a data packet, and the global timestamp is carried in the data header of the data packet.

[0049] In some implementations, the target data of at least one modality may be aligned based on a global timestamp before being packaged.

[0050] Specifically, a time window can be set for each global timestamp. Within the time window, target data of at least one modality with that global timestamp is collected and stored in a buffer. The system waits for target data of different modalities with the same global timestamp to arrive within this time window. When the time window ends, the target data of at least one modality within the same time window in the buffer is used as synchronously collected data, and a preset multimodal data transmission protocol is invoked to encode the target data of at least one modality with the same global timestamp into a data packet. This data packet includes a header and data blocks. The header carries the global timestamp, and each data block corresponds to a specific modality of target data. During packaging, the target data of each modality is compressed into data blocks according to the compression algorithm for different modalities and assembled into the corresponding positions in the data packet.

[0051] In some implementations, the data packet may also include a cyclic redundancy check (CRC) code at the tail for error detection.

[0052] Please refer to Figure 2 , Figure 2A schematic diagram of a data packet structure is shown. This data packet includes a PacketHeader field, a Data Chunk field, and a Cyclic Redundancy Check (CRC) Footer field. The PacketHeader field mainly includes information such as the Sync Flag, protocol version number, global timestamp, packet sequence number, payload type, and length. The Data Chunk field contains data chunks of different modalities, such as video frames, audio frames, and IMU data.

[0053] In some implementations, each data block may also have its own sub-header, which contains data type, data length, compression algorithm identifier, etc.

[0054] S104: Transmit the data packet to the server so that the server can extract the target data of different modalities that have been synchronized based on the global timestamp carried in the data packet.

[0055] The target data of different modalities in the data packet is synchronized during packaging. Therefore, the server only needs to parse the data packet according to the parsing protocol and obtain the global timestamp in the data packet header to get the target data of different modalities corresponding to the global timestamp, without any additional matching calculations.

[0056] The above describes the multimodal data transmission method provided in the embodiments of this specification. This method uses a global clock on the terminal device side to assign a global timestamp to the collected target data of different modalities. It also proposes a custom streaming transmission protocol to uniformly encapsulate and jointly encode the originally independently transmitted audio, video, IMU, and other multimodal data on the device terminal side, transmitting them to the server side via a single path. This solution fundamentally addresses the core pain point of insufficient time alignment accuracy of multimodal data on the server side, greatly reducing server-side processing complexity and transmission overhead, and significantly improving the accuracy and real-time performance of upper-layer AI applications.

[0057] Corresponding to the multimodal data transmission method described above, one or more embodiments of this specification propose a multimodal data transmission apparatus suitable for terminal devices. It should be noted that the aforementioned multimodal data transmission method can be implemented using this apparatus, but is not limited to it.

[0058] like Figure 3 As shown, the device includes: The multimodal data acquisition module 301 is used to respond to a data acquisition request, perform multimodal data synchronous acquisition based on a global clock, and add a global timestamp to the acquired data to obtain target data of at least one modality.

[0059] The data synchronization module 302 is used to provide a global clock for the multimodal data acquisition module.

[0060] The data processing and encapsulation module 303 is used to encode target data of at least one mode with the same global timestamp into a data packet according to a preset multimodal data transmission protocol, and to carry the global timestamp in the data header of the data packet.

[0061] The data transmission module 304 is used to transmit data packets to the server so that the server can extract the target data of different modalities that have been synchronized based on the global timestamp carried in the data packet.

[0062] The terminal device described in this embodiment can be a wearable / portable device equipped with a multimodal data acquisition module and a certain computing capability, such as smart glasses, smart action cameras, etc.

[0063] Optionally, the multimodal data acquisition module 301 described above includes sensors of different modes, such as cameras, microphones, IMUs, etc.

[0064] Optionally, the aforementioned data synchronization module 302 includes a global clock source. Specifically, the data synchronization module 302 is used to: respond to a data acquisition request, based on the global clock source, to unify the global clock for the multimodal data acquisition module of the terminal device via hardware triggering or system terminal method, ensuring that the multimodal data acquisition module 301 marks the data with a unified global timestamp while acquiring data. Specifically, assuming the global hardware clock provided by the global clock source is T0, the multimodal data acquisition module is triggered by hardware triggering to simultaneously acquire data at the global hardware clock T0, and to mark the acquired target data with a global timestamp T0.

[0065] Optionally, the data processing and encapsulation module 303 described above is specifically used to: align the target data of at least one modality based on a global timestamp before packaging the target data of at least one modality.

[0066] Specifically, the data processing and encapsulation module 303 can set a time window for each global timestamp. Within the time window, the collected target data of at least one modality with that global timestamp is stored in a buffer. During this time window, it waits for target data of different modalities with the same global timestamp to arrive. When the time window ends, the target data of at least one modality within the same time window in the buffer is used as synchronously collected data, and a preset multimodal data transmission protocol is invoked to encode the target data of at least one modality with the same global timestamp into a data packet. This data packet includes a data header and data blocks. The data header carries the global timestamp, and each data block corresponds to a type of target data. During packaging, the target data of a different modality is compressed into data blocks according to the compression algorithm for different modalities and assembled into the corresponding positions in the data packet.

[0067] Optionally, the aforementioned data packet includes a header and data blocks, with each data block corresponding to a specific modality of target data. For example... Figure 2 As shown, the header fields mainly include information such as the synchronization flag, protocol version number, global timestamp, packet sequence number, payload type, and length. The data block field contains data blocks of different modalities, such as video frames, audio frames, and IMU data.

[0068] Optionally, a data block may also include a sub-header, which includes the data type, data length, and compression algorithm identifier of the data block.

[0069] Optionally, the data packet may also include a cyclic redundancy check (CRC) code at the end for error detection.

[0070] Below, with Figure 3 The implementation principle of the device is illustrated using the multimodal data transmission device shown as an example. Taking a module as a software functional unit as an example, the multimodal data acquisition module 301 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the multimodal data acquisition module 301 may include code running on multiple hosts / virtual machines / containers. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically proximate data centers. Typically, a region may include multiple AZs.

[0071] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0072] As an example of a hardware functional unit, the multimodal data acquisition module 301 may include at least one computing device, such as a server. Alternatively, the multimodal data acquisition module 301 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0073] The multimodal data acquisition module 301 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the multimodal data acquisition module 301 includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multimodal data acquisition module 301 includes multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0074] In other embodiments, the multimodal data acquisition module 301 can be used to execute any step in the multimodal data transmission method described above; the data synchronization module 302 can be used to execute any step in the multimodal data transmission method described above; the data processing and encapsulation module 303 can be used to execute any step in the multimodal data transmission method described above; and the data transmission module 304 can be used to execute any step in the multimodal data transmission method described above. The steps implemented by the multimodal data acquisition module 301, data synchronization module 302, data processing and encapsulation module 303, and data transmission module 304 can be specified as needed. By implementing different steps in the multimodal data transmission method through the multimodal data acquisition module 301, data synchronization module 302, data processing and encapsulation module 303, and data transmission module 304, all functions of the multimodal data transmission device described above can be achieved.

[0075] In this implementation, the device can also be applied to computing devices such as computers and servers, or to a cluster of computing devices including at least one computing device, to achieve specific multimodal data transmission functions.

[0076] Corresponding to the multimodal data transmission device described above, one or more embodiments of this specification propose a terminal device that applies the multimodal data transmission device described above to implement the multimodal data transmission method described above.

[0077] Corresponding to the multimodal data transmission device described above, one or more embodiments of this specification propose a smart glasses that utilizes the aforementioned multimodal data transmission device to implement the aforementioned multimodal data transmission method.

[0078] One or more embodiments described in this specification also provide an electronic device. Please refer to... Figure 4 The electronic device includes a bus 401, a processor 402, a memory 403, and a communication interface 404. The processor 402, memory 403, and communication interface 404 communicate via the bus 401. This electronic device can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the electronic device.

[0079] Bus 401 can be a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 4The bus 401 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 401 may include a path for transmitting information between various components of an electronic device (e.g., processor 402, memory 403, and communication interface 404).

[0080] Processor 402 may include any one or more processors such as CPU, graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0081] Memory 403 may include volatile memory, such as random access memory (RAM). Memory 403 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0082] The memory 403 stores executable program code, and the processor 402 executes the executable program code to implement the aforementioned multimodal data transmission method.

[0083] Communication interface 404 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between electronic devices and other devices or communication networks.

[0084] One or more embodiments of this specification provide a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the multimodal data transmission method described above.

[0085] The computer-readable storage medium can be any available medium that an electronic device can store, or a data storage device such as a data center that contains one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives). The computer-readable storage medium includes instructions that direct the electronic device to perform the aforementioned multimodal data transmission method.

[0086] It is understood that the structures illustrated in the embodiments of this specification do not constitute a specific limitation on the system of the embodiments of this specification. In other embodiments of the specification, the above system may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0087] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0088] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0089] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

Claims

1. A multimodal data transmission method, applicable to terminal devices, the method comprising: In response to a data acquisition request, multimodal data is synchronously acquired based on a global clock, and a global timestamp is added to the acquired data to obtain target data of at least one modality. According to a preset multimodal data transmission protocol, target data of at least one modality having the same global timestamp are encoded into a data packet, and the global timestamp is carried in the data header of the data packet; The data packet is transmitted to the server so that the server can extract the target data of different modalities that have been synchronized based on the global timestamp carried in the data packet.

2. The method according to claim 1, wherein, The terminal device includes a global clock source; in response to a data acquisition request, it performs multimodal data synchronous acquisition based on the global clock, specifically including: In response to the data acquisition request, a unified hardware clock is provided to the multimodal data acquisition module of the terminal device through the global clock source, and the multimodal data acquisition module is triggered to synchronously acquire data through the hardware clock.

3. The method according to claim 1, further comprising: Before packaging the target data of the at least one modality, the target data of the at least one modality is aligned based on the global timestamp.

4. The method according to claim 3, wherein, Aligning the target data of at least one modality based on the global timestamp specifically includes: A time window is set for each of the global timestamps, and within the time window, the target data of at least one modality with the global timestamp is collected and stored in a buffer. The target data of at least one modality within the same time window in the buffer are used as synchronously acquired data.

5. The method according to claim 1, wherein, The data packet includes a data header and data blocks, with each data block corresponding to a modality of the target data.

6. The method according to claim 5, wherein, The data block includes a sub-data header, which includes the data type, data length, and compression algorithm identifier of the data block.

7. The method according to claim 5, wherein, The data packet also includes a cyclic redundancy check code at the end.

8. A multimodal data transmission device, suitable for terminal equipment, comprising: The multimodal data acquisition module is used to respond to data acquisition requests, perform multimodal data synchronous acquisition based on a global clock, and add a global timestamp to the acquired data to obtain target data of at least one modality. The data synchronization module is used to provide the global clock for the multimodal data acquisition module; The data processing and encapsulation module is used to encode target data of at least one modality having the same global timestamp into a data packet according to a preset multimodal data transmission protocol, and to carry the global timestamp in the data header of the data packet; The data transmission module is used to transmit the data packet to the server, so that the server can extract the target data of different modalities that have been synchronized based on the global timestamp carried in the data packet.

9. The apparatus according to claim 8, wherein, The data synchronization module includes a global clock source; the data synchronization module is specifically used for: In response to the data acquisition request, a unified hardware clock is provided to the multimodal data acquisition module through the global clock source, and the multimodal data acquisition module is triggered to synchronously acquire data through the hardware clock.

10. The apparatus according to claim 8, wherein, The data processing and encapsulation module is specifically used for: Before packaging the target data of the at least one modality, the target data of the at least one modality is aligned based on the global timestamp.

11. The apparatus according to claim 10, wherein, The data processing and encapsulation module is specifically used for: A time window is set for each of the global timestamps, and within the time window, the target data of at least one modality with the global timestamp is collected and stored in a buffer. The target data of at least one modality within the same time window in the buffer are used as synchronously acquired data.

12. The apparatus according to claim 8, wherein, The data packet includes a data header and data blocks, with each data block corresponding to a modality of the target data.

13. The apparatus according to claim 12, wherein, The data block includes a sub-data header, which includes the data type, data length, and compression algorithm identifier of the data block.

14. The apparatus according to claim 12, wherein, The data packet also includes a cyclic redundancy check code at the end.

15. A terminal device, the terminal device comprising a multimodal data transmission apparatus as described in any one of claims 8 to 14.

16. A smart glasses device comprising a multimodal data transmission device as claimed in any one of claims 8 to 14.

17. A computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the method as claimed in any one of claims 1 to 7.

18. An electronic device comprising: At least one memory for storing programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1 to 7.

Images