A Method and System for Aligning IMU Data and Video Data

Abstract

The present invention discloses a method for aligning IMU data and video data, including: acquiring video stream data collected by a camera and IMU data stream collected by an IMU sensor; aligning the first IMU data in the IMU data stream with the I-frame of the video stream data; when synchronously storing the aligned video stream data and IMU data stream, acquiring the video frame number and identification number, and establishing a corresponding relationship between the video frame number and the identification number; aligning the video stream data and the IMU data stream according to the corresponding relationship. The present invention also discloses a system for aligning IMU data and video data. The method and system for aligning IMU data and video data according to the present invention ensure the alignment of IMU data and video frames at various times by aligning IMU data and video frames in the initial stage and generating a corresponding relationship on the subsequent generated data, making the subsequent action determination of users more accurate.

Description

Technical Field

[0001] The present invention relates to information processing technology, and particularly to a method and system for aligning IMU data and video data. Background Art

[0002] With the development of intelligent fitness technology, it has been widely applied to families, public fitness service centers and various gyms. Intelligent fitness provides multi-dimensional and all-round software and hardware products, creating a new intelligent fitness venue. In a system platform, fitness services are completed, greatly reducing the time, space and economic costs brought by fitness.

[0003] In intelligent fitness technology, in order to accurately capture the movements of users, the existing technology uses IMU technology or video acquisition technology to capture data. When it is necessary to simultaneously use these two technologies to collect intuitive and accurate data, the alignment of the two becomes a difficult problem in the industry. Summary of the Invention

[0004] The technical problem to be solved by the present invention is the lack of technology for aligning IMU data and video data in the prior art. The purpose is to provide a method and system for aligning IMU data and video data to solve the above problems.

[0005] The present invention is achieved by the following technical solutions:

[0006] On the one hand, this embodiment provides a method for aligning IMU data and video data, including:

[0007] Obtaining video stream data collected by a camera and IMU data stream collected by an IMU sensor;

[0008] Aligning the first IMU data in the IMU data stream with the I-frame of the video stream data;

[0009] When synchronously storing the aligned video stream data and the IMU data stream, obtaining a video frame number and an identification number, and establishing a corresponding relationship between the video frame number and the identification number; the video frame number is formed by sequentially numbering the video frames in the video stream data according to the time axis; the identification number is the position of the IMU data in the IMU data stream relative to the first IMU data;

[0010] Aligning the video stream data and the IMU data stream according to the corresponding relationship.

[0011] In the prior art, the inventor found that when generally using IMU technology to obtain user movement data, a large number of IMU sensors are usually required to be arranged on the user to ensure that the overall movement of the user can be reflected. When using video acquisition technology for image processing to collect user movement data, since the image data lacks depth data, it is easy to have errors in the recognition of the collection points. Therefore, the inventor adopted a method combining IMU technology and video acquisition technology to reflect the user's movement. However, the inventor found that through this method, comprehensive judgment requires comprehensive alignment of the data collected by the video acquisition and the data collected by the IMU sensors to ensure that the IMU data and the video data are synchronized during the user's movement.

[0012] When implementing this embodiment, to ensure the synchronization of the IMU data and the video data, through the alignment of the first IMU data and the I-frame, the video stream data and the IMU data stream are aligned at the initial stage. When synchronously storing these two types of data, a corresponding relationship is generated between the video frame number and the identification number. Since the difference between the video sampling frequency and the IMU data sampling frequency is generally very large, in order to clearly define the corresponding relationship between the video frame and the IMU data in subsequent data processing, in this embodiment, the method of corresponding the video frame number and the identification number is adopted. Among them, the video frame number is generated by numbering the video frames in chronological order, and the identification number is the position of the IMU data relative to the first IMU data. The position relative to the first IMU data described here is generated by initializing the identification number of the first IMU data and incrementing the identification number each time a new IMU data is generated. For example, the order of the video frame numbers is that the number of the I-frame is 0, the number of the first frame after the I-frame is 1, and so on. The generation of the identification number is by initializing the identification number corresponding to the first IMU data to 0, the first data after the first IMU data is 1, and so on.

[0013] In this embodiment, through the corresponding relationship between the video frame number and the identification number, the video frame and the IMU data can be corresponded in subsequent processing, which is convenient for checking, correction, and processing. For example, the IMU sampling frequency is 100, and the video frame rate is 25. When storing the first IMU data, the frame number of the video is 1. When storing the second IMU data, no new frame number is received, and the frame number of the video is still 1… When storing the fifth IMU data, a new frame number of 5 is received, and at this time the video frame number is 2. By aligning the IMU data and the video frame at the initial stage and generating a corresponding relationship on the subsequent generated data in this embodiment, the alignment of the IMU data and the video frame at various times is ensured, making the subsequent determination of the user's actions more accurate.

[0014] Further, aligning the first piece of IMU data in the IMU data stream with the I-frame of the video stream data includes:

[0015] When it is necessary to store the IMU data stream, parse the video stream data to obtain the basic information of the video frames in the video stream data;

[0016] If it is detected from the basic information that the corresponding video frame is an I-frame, start storing the IMU data stream and the video stream data simultaneously.

[0017] Further, obtaining the video frame number and the identification number and establishing the corresponding relationship between the video frame number and the identification number includes:

[0018] Generate a video frame number when storing the video stream data, and send the video frame number to the thread that receives the IMU data stream;

[0019] When the thread that receives the IMU data stream stores the IMU data stream, synchronously store the video frame number and the identification number to form the corresponding relationship.

[0020] Further, it further includes:

[0021] Obtain the sampling frequency of the IMU data stream as the first frequency, and obtain the sampling frequency of the video stream data as the second frequency;

[0022] Check and correct the corresponding relationship according to the ratio of the first frequency and the second frequency, and align the video stream data and the IMU data stream according to the corrected corresponding relationship.

[0023] Further, checking and correcting the corresponding relationship according to the ratio of the first frequency and the second frequency includes:

[0024] Perform a ratio operation on the first frequency and the second frequency to generate a check ratio, and obtain the number of IMU data corresponding to each video frame in the corresponding relationship;

[0025] When the difference between the check ratio and the number of IMU data is greater than a preset value, it is considered that the alignment of the video frame corresponding to the number of IMU data fails, and data increase or decrease is performed on the IMU data corresponding to the video frame according to the check ratio to complete the correction.

[0026] On the other hand, this embodiment provides a system for aligning IMU data and video data, including:

[0027] A camera configured to collect video stream data;

[0028] An IMU sensor, configured to collect an IMU data stream;

[0029] An acquisition unit, configured to acquire the video stream data and the IMU data stream;

[0030] A preliminary alignment unit, configured to align the first IMU data in the IMU data stream with the I-frame of the video stream data;

[0031] A subsequent alignment unit, configured to, when synchronously storing the aligned video stream data and the IMU data stream, acquire a video frame number and an identification number, and establish a correspondence between the video frame number and the identification number; the video frame number is formed by sequentially numbering the video frames in the video stream data according to the time axis; the identification number is the position of the IMU data in the IMU data stream relative to the first IMU data; align the video stream data and the IMU data stream according to the correspondence.

[0032] Further, the preliminary alignment unit is further configured to:

[0033] When it is necessary to store the IMU data stream, parse the video stream data to obtain the basic information of the video frames in the video stream data;

[0034] If it is detected from the basic information that the corresponding video frame is an I-frame, start storing the IMU data stream and the video stream data simultaneously.

[0035] Further, the subsequent alignment unit is further configured to:

[0036] Generate a video frame number when storing the video stream data, and send the video frame number to the thread receiving the IMU data stream;

[0037] The thread receiving the IMU data stream synchronously stores the video frame number and the identification number to form the correspondence when storing the IMU data stream.

[0038] Further, it further includes a verification unit, configured to:

[0039] Acquire the sampling frequency of the IMU data stream as the first frequency, and acquire the sampling frequency of the video stream data as the second frequency;

[0040] Verify and correct the correspondence according to the ratio of the first frequency and the second frequency, and align the video stream data and the IMU data stream according to the corrected correspondence.

[0041] Further, the verification unit is further configured to:

[0042] Perform a ratio operation on the first frequency and the second frequency to generate a verification ratio, and obtain the number of IMU data corresponding to each video frame in the corresponding relationship;

[0043] When the difference between the verification ratio and the number of IMU data is greater than a preset value, it is considered that the alignment of the video frame corresponding to the number of IMU data fails, and data increase and decrease are performed on the IMU data corresponding to the video frame according to the verification ratio to complete the correction.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] A method and system for aligning IMU data and video data according to the present invention ensure the alignment of IMU data and video frames at various times by aligning IMU data and video frames in the initial stage and generating a corresponding relationship on the subsequent generated data, making the subsequent action determination of users more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, form a part of this application, and do not limit the embodiments of the present invention. In the drawings:

[0047] Figure 1 It is a schematic diagram of the method steps of the embodiment of the present invention;

[0048] Figure 2 It is a schematic diagram of the system architecture of the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the embodiments and the drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and do not limit the present invention.

[0050] Embodiment

[0051] Please refer to Figure 1 , which is a flowchart of a method for aligning IMU data and video data provided by an embodiment of the present invention. The method for aligning IMU data and video data can be applied to Figure 2 in a system for aligning IMU data and video data. Further, the method for aligning IMU data and video data may specifically include the content described in the following steps S1-S4.

[0052] S1: Obtain video stream data collected by a camera and IMU data stream collected by an IMU sensor;

[0053] S2: Align the first IMU data in the IMU data stream with the I-frame of the video stream data;

[0054] S3: When synchronously storing the aligned video stream data and the IMU data stream, obtain the video frame number and the identification number, and establish the corresponding relationship between the video frame number and the identification number; the video frame number is formed by sequentially numbering the video frames in the video stream data according to the time axis; the identification number is the position of the IMU data in the IMU data stream relative to the first IMU data;

[0055] S4: Align the video stream data and the IMU data stream according to the corresponding relationship.

[0056] In the prior art, the inventor found that when generally using the IMU technology to obtain user motion data, a large number of IMU sensors usually need to be arranged on the user to ensure the overall reaction of the user's motion. When using the video acquisition technology to process images for user motion data acquisition, since the image data lacks depth data, it is easy to have errors in the recognition of the acquisition points. Therefore, the inventor adopted a method combining the IMU technology and the video acquisition technology to reflect the user's motion. However, the inventor found that through this method, comprehensive judgment requires comprehensive alignment of the video acquisition data and the IMU sensor acquisition data to ensure that the IMU data and the video data are synchronized during the user's motion.

[0057] When this embodiment is implemented, to ensure the synchronization of the IMU data and the video data, through the alignment of the first IMU data and the I-frame, the video stream data and the IMU data stream are aligned initially. When synchronously storing these two types of data, a corresponding relationship is generated for the video frame number and the identification number. Since the difference between the video sampling frequency and the IMU data sampling frequency is generally very large, in order to clarify the corresponding relationship between the video frame and the IMU data in subsequent data processing, in this embodiment, the method of corresponding the video frame number and the identification number is adopted. Among them, the video frame number is generated by sequentially numbering the video frames according to the time sequence, and the identification number is the position of the IMU data relative to the first IMU data. The position relative to the first IMU data here is generated by initializing the identification number of the first IMU data and incrementing the identification number each time a new IMU data is generated. For example, the order of the video frame numbers is that the number of the I-frame is 0, the number of the first frame after the I-frame is 1, and so on. The generation of the identification number is by initializing the identification number corresponding to the first IMU data to 0, the first data after the first IMU data is 1, and so on.

[0058] In this embodiment, through the correspondence between the video frame number and the identification number, the video frames and the IMU data can be corresponded in subsequent processing, facilitating checking, correction, and processing. For example, the IMU sampling frequency is 100, and the video frame rate is 25. When storing the first piece of IMU data, the frame number of the video is 1. When storing the second piece of IMU data, no new frame number is received, and the frame number of the video remains 1... When storing the fifth piece of IMU data, a new frame number of 5 is received, and at this time the video frame number is 2. By aligning the IMU data and the video frames in the initial stage and generating the corresponding relationship on the subsequent generated data in this embodiment, the alignment of the IMU data and the video frames at various times is ensured, making the subsequent user action determination more accurate.

[0059] In one embodiment, aligning the first piece of IMU data in the IMU data stream with the I-frame of the video stream data includes:

[0060] When it is necessary to store the IMU data stream, parse the video stream data to obtain the basic information of the video frames in the video stream data;

[0061] If it is detected from the basic information that the corresponding video frame is an I-frame, start storing the IMU data stream and the video stream data simultaneously.

[0062] When implementing this embodiment, as a more specific implementation method, this embodiment determines whether the starting frame is an I-frame by obtaining the basic information of the video frames, and aligns the first piece of IMU data with the I-frame, so that the video frames and the IMU data are aligned from the very beginning.

[0063] In one embodiment, obtaining the video frame number and the identification number and establishing the correspondence between the video frame number and the identification number includes:

[0064] Generate a video frame number when storing the video stream data, and send the video frame number to the thread receiving the IMU data stream;

[0065] When storing the IMU data stream, the thread receiving the IMU data stream synchronously stores the video frame number and the identification number to form the correspondence.

[0066] When implementing this embodiment, since the storage of the video frames and the IMU data is synchronous, the generation of the video frame number and the generation of the identification number are also synchronous. Generate the video frame number when storing the video stream data, and the thread receiving the IMU data stream receives the video frame number and synchronously stores the video frame number and the identification number to form the correspondence.

[0067] In one embodiment, it further includes:

[0068] Obtain the sampling frequency of the IMU data stream as the first frequency, and obtain the sampling frequency of the video stream data as the second frequency;

[0069] Verify and correct the corresponding relationship according to the ratio of the first frequency to the second frequency, and align the video stream data and the IMU data stream according to the corrected corresponding relationship.

[0070] When this embodiment is implemented, the inventor found that the ratio of the sampling frequency of the IMU data stream to the sampling frequency of the video stream data can characterize the alignment of the video stream data and the IMU data stream. For example, when the IMU sampling frequency is 100Hz, if the video frame rate is 25Hz, then one video frame will correspond to four IMU data. Therefore, this embodiment uses this rule to verify the alignment of the video stream data and the IMU data stream. Of course, in the above example, only the most ideal situation is given, that is, the ratio of the first frequency to the second frequency is a positive integer. If the ratio is not an integer, it can be judged by setting a difference threshold.

[0071] In this embodiment, the correction of the IMU data stream is mainly carried out by deleting and adding IMU data. First, round the ratio. If the number of IMU data corresponding to a video frame is more than the ratio, at least one IMU data corresponding to the video frame is deleted until the ratio is satisfied. If the number of IMU data corresponding to a video frame is less than the ratio, at least one IMU data corresponding to the video frame is added until the ratio is satisfied.

[0072] In one embodiment, verifying and correcting the corresponding relationship according to the ratio of the first frequency to the second frequency includes:

[0073] Perform a ratio operation on the first frequency and the second frequency to generate a verification ratio, and obtain the number of IMU data corresponding to each video frame in the corresponding relationship;

[0074] When the difference between the verification ratio and the number of IMU data is greater than a preset value, it is considered that the alignment of the video frame corresponding to the number of IMU data fails, and data addition and subtraction are performed on the IMU data corresponding to the video frame according to the verification ratio to complete the correction.

[0075] In a more specific embodiment, the following method is used for data alignment:

[0076] When data needs to be stored, start parsing the video stream data captured by the camera. The video stream is transmitted using RTSP (Real-Time Streaming Protocol), which contains the basic information of each frame of the video. When it is detected from the basic information that the frame is an I-frame, notify each thread receiving IMU data to start storing data, and at the same time, the thread receiving video data also starts storing the video. This ensures that the first I-frame of the video is aligned with the first piece of IMU data.

[0077] When storing the video, the video frame number (count relative to the first frame) is numbered, and this number is sent to each thread receiving IMU data. The thread receiving IMU data stores the video frame number corresponding to this piece of IMU data while storing the IMU data (for example, if the IMU sampling frequency is 100 and the video frame rate is 25, when storing the first piece of IMU data, the frame number of the video is 1, when storing the second piece of IMU data, no new frame number is received, and the frame number of the video is still 1..., when storing the fifth piece of IMU data, a new frame number of 5 is received, and at this time the video frame number is 2). This ensures that each subsequent piece of IMU data is aligned with the corresponding frame of the video in time.

[0078] Please refer to Figure 2 Based on the same inventive concept, a system for aligning IMU data and video data is also provided. The system includes:

[0079] A camera configured to collect video stream data;

[0080] An IMU sensor configured to collect IMU data stream;

[0081] An acquisition unit configured to acquire the video stream data and the IMU data stream;

[0082] A preliminary alignment unit configured to align the first piece of IMU data in the IMU data stream with the I-frame of the video stream data;

[0083] A subsequent alignment unit configured to, when synchronously storing the aligned video stream data and the IMU data stream, acquire a video frame number and an identification number, and establish a correspondence between the video frame number and the identification number; the video frame number is formed by sequentially numbering the video frames in the video stream data according to the time axis; the identification number is the position of the IMU data in the IMU data stream relative to the first piece of IMU data; align the video stream data and the IMU data stream according to the correspondence.

[0084] In one embodiment, the preliminary alignment unit is further configured to:

[0085] When it is necessary to store the IMU data stream, parse the video stream data to obtain the basic information of the video frames in the video stream data;

[0086] If it is detected from the basic information that the corresponding video frame is an I-frame, start storing the IMU data stream and the video stream data simultaneously.

[0087] In one embodiment, the later alignment unit is further configured to:

[0088] Generate a video frame number when storing the video stream data, and send the video frame number to the thread that receives the IMU data stream;

[0089] The thread that receives the IMU data stream synchronously stores the video frame number and the identification number when storing the IMU data stream to form the corresponding relationship.

[0090] In one embodiment, it further includes a verification unit, which is configured to:

[0091] Obtain the sampling frequency of the IMU data stream as the first frequency, and obtain the sampling frequency of the video stream data as the second frequency;

[0092] Verify and correct the corresponding relationship according to the ratio of the first frequency and the second frequency, and align the video stream data and the IMU data stream according to the corrected corresponding relationship.

[0093] In one embodiment, the verification unit is further configured to:

[0094] Perform a ratio operation on the first frequency and the second frequency to generate a verification ratio, and obtain the number of IMU data corresponding to each video frame in the corresponding relationship;

[0095] When the difference between the verification ratio and the number of IMU data is greater than a preset value, it is considered that the alignment of the video frame corresponding to the number of IMU data fails, and data increase or decrease is performed on the IMU data corresponding to the video frame according to the verification ratio to complete the correction.

[0096] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0097] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces, devices or units, and can also be electrical, mechanical or other forms of connection.

[0098] The units described as separate components may or may not be physically separated. Obviously, those of ordinary skill in the art can realize that, in combination with the units and algorithm steps of the examples described in the embodiments disclosed herein, they can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0099] In addition, the functional units in each embodiment of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0100] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a grid device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0101] The specific embodiments described above further elaborate on the object, technical solution and beneficial effects of the present invention. It should be understood that the above description is only the specific embodiments of the present invention and is not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for aligning IMU data and video data, characterized in that, including: obtaining video stream data collected by a camera and IMU data stream collected by an IMU sensor; aligning the first IMU data in the IMU data stream with the I-frame of the video stream data; when synchronously storing the aligned video stream data and the IMU data stream, obtaining a video frame number and an identification number, and establishing a corresponding relationship between the video frame number and the identification number; the video frame number is sequentially numbered according to the time axis for the video frames in the video stream data; the identification number is the position of the IMU data in the IMU data stream relative to the first IMU data; aligning the video stream data and the IMU data stream according to the corresponding relationship.

2. The method for aligning IMU data and video data according to claim 1, wherein, Aligning the first IMU data in the IMU data stream with the I-frame of the video stream data includes: when it is necessary to store the IMU data stream, parsing the video stream data to obtain basic information of video frames in the video stream data; if it is detected from the basic information that the corresponding video frame is an I-frame, start storing the IMU data stream and the video stream data simultaneously.

3. The method for aligning IMU data and video data according to claim 1, wherein Obtaining a video frame number and an identification number, and establishing a corresponding relationship between the video frame number and the identification number includes: generating a video frame number when storing the video stream data, and sending the video frame number to the thread receiving the IMU data stream; the thread receiving the IMU data stream synchronously stores the video frame number and the identification number to form the corresponding relationship when storing the IMU data stream.

4. A method for aligning IMU data and video data according to claim 1, characterized in that, further including: obtaining the sampling frequency of the IMU data stream as the first frequency, and obtaining the sampling frequency of the video stream data as the second frequency; checking and correcting the corresponding relationship according to the ratio of the first frequency and the second frequency, and aligning the video stream data and the IMU data stream according to the corrected corresponding relationship.

5. A method for aligning IMU data and video data according to claim 4, characterized in that, Checking and correcting the corresponding relationship according to the ratio of the first frequency and the second frequency includes: performing a ratio operation on the first frequency and the second frequency to generate a check ratio, and obtaining the number of IMU data corresponding to each video frame in the corresponding relationship; when the difference between the check ratio and the number of IMU data is greater than a preset value, it is considered that the alignment of the video frame corresponding to the number of IMU data fails, and data increase and decrease are performed on the IMU data corresponding to the video frame according to the check ratio to complete the correction.

6. A system for aligning IMU data and video data, characterized in that, including: a camera configured to collect video stream data; an IMU sensor configured to collect an IMU data stream; an obtaining unit configured to obtain the video stream data and the IMU data stream; a preliminary alignment unit configured to align the first IMU data in the IMU data stream with the I-frame of the video stream data; A post-alignment unit, configured to obtain a video frame number and an identification number and establish a correspondence between the video frame number and the identification number when synchronously storing the aligned video stream data and the IMU data stream; the video frame number is formed by sequentially numbering video frames in the video stream data according to a time axis; the identification number is the position of the IMU data in the IMU data stream relative to the first IMU data; align the video stream data and the IMU data stream according to the correspondence.

7. The system for aligning IMU data and video data according to claim 6, wherein The preliminary alignment unit is further configured to: When it is necessary to store the IMU data stream, parse the video stream data to obtain basic information of video frames in the video stream data; If it is detected from the basic information that the corresponding video frame is an I-frame, start storing the IMU data stream and the video stream data simultaneously.

8. The system for aligning IMU data and video data according to claim 6, wherein The post-alignment unit is further configured to: Generate a video frame number when storing the video stream data and send the video frame number to the thread receiving the IMU data stream; The thread receiving the IMU data stream synchronously stores the video frame number and the identification number to form the correspondence when storing the IMU data stream.

9. The system for aligning IMU data and video data according to claim 6, wherein It further includes a verification unit, configured to: Obtain the sampling frequency of the IMU data stream as a first frequency and obtain the sampling frequency of the video stream data as a second frequency; Verify and correct the correspondence according to the ratio of the first frequency and the second frequency, and align the video stream data and the IMU data stream according to the corrected correspondence.

10. The system for aligning IMU data and video data according to claim 9, characterized in that, The verification unit is further configured to: Perform a ratio operation on the first frequency and the second frequency to generate a verification ratio, and obtain the number of IMU data corresponding to each video frame in the correspondence; When the difference between the verification ratio and the number of IMU data is greater than a preset value, it is considered that the alignment of the video frame corresponding to the number of IMU data fails, and data increase or decrease is performed on the IMU data corresponding to the video frame according to the verification ratio to complete the correction.

Images