A method and system for acquiring training data for multimodal large models

Abstract

This invention discloses a method and system for acquiring training data for a multimodal large model, belonging to the field of artificial intelligence and data processing technology. The method involves acquiring the spatial location information and temporal synchronization parameters of each modal device to construct a modal distribution matrix; calculating the spatiotemporal alignment error between modal devices to establish a modal collaboration metric function; selecting highly collaborative modal pairs based on the collaboration metric function to construct a set of acquisition link combinations; evaluating the spatial coverage and temporal stability of each modal combination to obtain a modal stability score; adjusting the sampling frequency and exposure parameters according to the score to generate a modal acquisition strategy; and finally executing the strategy to complete multimodal collaborative acquisition and generate a training dataset. This invention has advantages such as adaptive acquisition strategy, high spatiotemporal alignment accuracy, and comprehensive modal coverage, and can dynamically adapt to different acquisition environments and modal characteristics.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and data processing technology, specifically to a method and system for acquiring training data for multimodal large models. Background Technology

[0002] In recent years, with the widespread application of multimodal large models in fields such as image and text understanding, video generation, and speech recognition, the demand for high-quality, highly correlated multimodal training data has increased dramatically. Traditional data acquisition methods often rely on public datasets or manual annotation, which suffer from problems such as limited data types, poor annotation consistency, and low modality alignment accuracy, making it difficult to meet the requirements of large model training for data diversity and high correlation.

[0003] In practical applications, such as autonomous driving, smart healthcare, or industrial monitoring, it is often necessary to simultaneously collect image, audio, text, and sensor data to train large-scale perception-decision integrated models. However, due to the complexity of the acquisition environment, time synchronization errors, spatial alignment offsets, and inconsistent acquisition intensities between different modal data sources can easily lead to modal mismatch, resulting in a significant decrease in model training performance.

[0004] Especially in industrial pipeline inspection scenarios, different sensors (such as infrared cameras, lidar, ultrasonic probes, etc.) are prone to the phenomenon of "cross-modal gap" due to differences in installation location and operating frequency: that is, a certain modality lacks effective data at a critical moment, while other modalities capture target information, which seriously affects the integrity and effectiveness of the final training data. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for acquiring training data for multimodal large models, so as to overcome the shortcomings of the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for acquiring multimodal large model training data, comprising: S100. Acquire the spatial location information and time synchronization parameters of each heterogeneous modal device in the environment to be collected, and construct the modal distribution matrix M, where each modal device corresponds to a position vector and a time offset; S200. Based on the modal distribution matrix M, calculate the spatiotemporal alignment error between each modal device, and establish a modal cooperation metric function R to represent the degree of cooperation matching between modal pairs; S300. Based on the collaborative metric function R, select the mode pair combination with the minimum alignment error from the candidate mode devices to establish the acquisition link combination set C; S400. For each mode combination in the acquisition link combination set C, evaluate its mode coverage and mode gap risk within the target acquisition area to obtain the mode stability score S; S500: Based on the modal stability score S, adjust the sampling frequency and exposure parameters of the modal acquisition device to generate the modal acquisition strategy P; S600: Execute the modal acquisition strategy P to perform multimodal collaborative acquisition and obtain the training dataset.

[0007] Preferably, the step S100 of acquiring the spatial location information of each heterogeneous modal device includes: At least three 3D calibration points with known world coordinates are set up in the environment to be acquired. The visual modal device is controlled to capture images of the calibration points, and the image coordinates of the calibration points are obtained through image recognition. Combined with the coordinates of the calibration points, the position coordinates and orientation angle of the modal device are solved by the EPnP algorithm. The image reprojection error is converged to less than 0.5 pixels by the Levenberg-Marquardt nonlinear least squares optimization algorithm.

[0008] Preferably, the step S100 of acquiring time synchronization parameters includes: Generate a time reference signal; Each modal device receives the time reference signal, records its local timestamp, and calculates the time offset Δt; Repeat the time offset sampling at least 10 times and take the average value; Record a continuous sampling timestamp sequence and calculate the time drift Δd per unit time using linear least squares fitting.

[0009] Preferably, the establishment of the modal cooperative metric function R in S200 includes: Calculate the three-dimensional Euclidean distance and attitude angle difference between the modal device pairs; Spatial error weighting function Characterizing spatial alignment error; Calculate the time offset difference and drift difference between modal device pairs, and construct a time error function. ; The spatial error matrix and the temporal error matrix are normalized and then weighted and combined using a fusion weight λ to obtain the spatiotemporal alignment error matrix. ; Finally, the modal collaborative metric function is calculated. And set upper and lower limits for synergy to truncate and correct.

[0010] Preferably, the S300 establishes the acquisition link combination set C, which includes: All modal device pairs in the modal cooperation metric function are screened, and a cooperation threshold Rth is set. Modal device pairs with R(k,l) greater than Rth are selected as candidate sets. For candidate mode pairs, spatial redundancy detection is performed. If the three-dimensional spatial distance is less than 0.3 meters and the attitude angle difference is less than 5 degrees, it is considered redundant. For candidate mode pairs, if the time offset difference is less than 50 microseconds and the time drift difference is less than 5 microseconds per second, it is considered redundant. For redundant mode pairs, only the pair with the highest degree of cooperation is retained, and the rest are discarded; Construct an undirected graph structure where nodes represent modal devices and edge weights are the degree of collaboration R(k,l). Select modal combination paths using the maximum weighted matching or minimum edge cover algorithm to form a set of acquisition link combinations C.

[0011] Preferably, the modal coverage evaluation in S400 adopts a method based on high-dimensional space kernel density estimation, including: The target acquisition area was divided into a three-dimensional grid of sampling points with a spacing of 0.2 meters to 0.5 meters; A modality sensing function K(i,g) is constructed, using a three-dimensional Gaussian kernel function and weighted by orientation influence factors; Calculate the maximum modal sensing response value R(g) for each sampling point, and take the average value of R(g) for all sampling points to obtain the spatial coverage score Cs, with a value range of [0,1].

[0012] Preferably, the modal window period risk assessment in S400 adopts a local temporal stability scoring method, including: Extract the sampling timestamp sequence of each modal device within a preset time window T; The standard deviation σ of the time interval is calculated using a sliding window of length L and a step size of 0.5 seconds. The average of the standard deviations of all windows is taken as the local volatility of this mode. The maximum variability σmax of all modes is used to calculate the time stability score; The time stability score is Ct = 1 - min(1, σmax / σmax-allow), where σmax-allow is 100 milliseconds.

[0013] Preferably, the generation of the modal acquisition strategy P in S500 includes: The stability score (S) is divided into three levels: high, medium, and low. Different sampling frequencies and exposure times are adjusted according to the level. The intermediate-level mode combination uses a linear enhancement strategy, with a frequency increase ratio η1 = (S - 0.5) / 0.3 × Δfmax and an exposure reduction ratio η2 = (S - 0.5) / 0.3 × Δemax; where Δfmax is the maximum allowable increase in sampling frequency and Δemax is the maximum reduction in exposure time. The low-stability combination activates a keyframe enhancement strategy, increasing the sampling frequency to 1.5 times the original frequency and increasing the exposure time Δeadd; The score S is mapped to the final frequency and exposure parameters using mapping functions F(S) and E(S), and a modal acquisition strategy P with key-value pair structure is generated.

[0014] Preferably, the modal acquisition strategy P executed by S600 includes: After the modal acquisition strategy P is sent to the modal device, the parameters are loaded. Construct a unified global acquisition timeline and perform offset correction based on the time offset of each device; Synchronous triggering and acquisition of modal devices are performed according to the time schedule; frame-level alignment is achieved for non-equivalent frequency devices using interpolation or buffering methods. Organize modal data from the same sampling time point into a co-frame; All collaborative frames are organized in chronological order to form the training dataset D.

[0015] This invention also provides a multimodal large model training data acquisition system, comprising: The synchronization module acquires the spatial location information and time synchronization parameters of each heterogeneous modal device in the environment to be collected, and constructs a modal distribution matrix M, in which each modal device corresponds to a position vector and a time offset; The collaborative evaluation module calculates the spatiotemporal alignment error between each modal device based on the modal distribution matrix M, and establishes a modal collaboration metric function R to represent the degree of collaboration matching between modal pairs. The link construction module selects mode pair combinations with the minimum alignment error from candidate mode devices based on the collaborative metric function R, and establishes a set of acquisition link combinations C; The modal stability analysis module evaluates the modal coverage and modal gap risk within the target acquisition area for each modal combination in the acquisition link combination set C, and obtains a modal stability score S. The acquisition strategy generation module adjusts the sampling frequency and exposure parameters of the modal acquisition device based on the modal stability score S, and generates a modal acquisition strategy P. The data generation module executes the modal acquisition strategy P to perform multimodal collaborative acquisition and obtain the training dataset.

[0016] The technical effects and advantages provided by the present invention in the above technical solution are as follows: 1. This invention constructs a modal distribution matrix, integrating multi-dimensional features such as spatial location, orientation, temporal offset, and temporal drift, to achieve accurate modeling of heterogeneous modal devices within a unified spatial and temporal reference frame. Furthermore, through spatiotemporal error assessment and the construction of a modal collaboration metric function, it effectively identifies the collaborative capabilities between modal devices, avoiding the problems of traditional solutions that rely on manual experience for modal deployment and are difficult to quantitatively optimize, thus improving the spatiotemporal matching accuracy and system scalability of modal combinations.

[0017] 2. This invention achieves a comprehensive assessment of modal coverage capability and window period risk in the acquisition process by introducing modal coverage analysis based on high-dimensional kernel density estimation and a stability scoring model based on local time window volatility. Based on the scoring results, it adaptively adjusts the sampling frequency and exposure parameters to generate an optimized modal acquisition strategy. Compared to existing fixed parameter configuration methods, this method can dynamically adapt to different acquisition environments and modal characteristics, improving the spatiotemporal consistency, diversity, and effectiveness of multimodal training data, thereby significantly enhancing the basic data quality and generalization ability of large model training. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 This is a flowchart of the method of the present invention.

[0020] Figure 2 This is a flowchart of the system modules of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1, please refer to Figure 1 As shown in this embodiment, a method for acquiring training data for a multimodal large model includes: S100. Obtain the spatial location information and time synchronization parameters of each heterogeneous modal device in the environment to be collected, and construct the modal distribution matrix M, where each modal device corresponds to a position vector and a time offset.

[0023] In multimodal large model training data acquisition methods, to achieve collaborative data acquisition from multiple modal devices such as image, speech, text, depth, laser point cloud, infrared, and inertial navigation, a unified spatiotemporal reference system must first be established. The following are detailed steps in this embodiment of the invention regarding the acquisition of spatial location information and time synchronization parameters of modal devices, and the construction of the modal distribution matrix.

[0024] Mapping different types of modal acquisition devices (such as high-definition cameras, microphone arrays, lidar, temperature and humidity sensors, etc.) to a unified spatial coordinate system facilitates accurate modeling of their spatial relationships.

[0025] Calibration environment preparation: Set up a standard three-dimensional calibration device in the environment to be collected, including at least 3 calibration points P1(x1,y1,z1), P2(x2,y2,z2), and P3(x3,y3,z3) with known world coordinate positions. The distance error between the calibration points shall not exceed 1 mm.

[0026] Equipment image acquisition: Control each visual modal device (such as an RGB camera) to capture static images of the calibration device, and identify the image coordinates of the calibration points through image processing methods.

[0027] 3D position determination: Based on the known world coordinates and image coordinates, construct a perspective projection model to solve the position (X,Y,Z) and orientation angle (α,β,γ) of each device in the world coordinate system, namely pitch, yaw and roll.

[0028] The attitude angle is solved using the Efficient Perspective-n-Point (EPnP) algorithm, and the image reprojection error (per unit pixel) is minimized to less than 0.5 by Levenberg-Marquardt nonlinear least squares optimization.

[0029] Result encapsulation: Combine the spatial position and attitude angle of each device into a six-dimensional spatial vector V. s =[X,Y,Z,Pitch,Yaw,Roll], and store it in the corresponding attribute table of the device.

[0030] Time synchronization parameters are obtained by using a high-precision time synchronization device to eliminate time offset of multimodal devices at the sampling time and ensure that data frames are consistent at the same time.

[0031] Time reference signal generation: A master clock timing device is set up in the acquisition environment, and a global reference clock T0 is generated using GPS or IEEE 1588 protocol, with a time accuracy better than 10 microseconds.

[0032] Modal device response: Each modal device receives T0 and records its local system time Ti, and calculates its time offset Δt = Ti - T0. This process is repeated N times (N ≥ 10), and the average value is taken to eliminate the impact of burst delays.

[0033] Time drift calculation: Each device records its data frame timestamp sequence {t1, t2, ..., tn} during continuous sampling and fits the difference curve between it and the ideal time sequence. The sampling frequency drift Δd is calculated using the linear least squares method, defined as the rate of change of time error per unit time (unit: microseconds / second).

[0034] Result encapsulation: Time synchronization vector Vt=[Δt,Δd], representing the time offset and drift information of the device.

[0035] To facilitate subsequent matrix construction and distance calculation, the above information is uniformly encoded into feature vector form, as follows: Spatial position and orientation vectorization: For each modal device, its spatial coordinates (X,Y,Z) and orientation angle (Pitch,Yaw,Roll) are combined into a six-dimensional modal space vector Vs.

[0036] Time synchronization parameter vectorization: Encode the time offset Δt and time drift Δd into a two-dimensional time synchronization vector Vt.

[0037] Complete vector representation: Each modal device is represented as an eight-dimensional vector V=[X,Y,Z,Pitch,Yaw,Roll,Δt,Δd].

[0038] Construct a modal distribution matrix M to describe the relative distribution of all modal devices in the spatial and temporal dimensions.

[0039] Matrix structure definition: The modal distribution matrix M is an N×8 two-dimensional array, where N is the total number of modal devices, and each row is an eight-dimensional vector V of a modal device.

[0040] Matrix generation process: Traverse all devices and fill in their eight-dimensional vectors into each row of matrix M in sequence.

[0041] For example: The first row is the vector [X1,Y1,Z1,Pitch1,Yaw1,Roll1,Δt1,Δd1] of modal device 1; The second row is the vector [X2,Y2,Z2,Pitch2,Yaw2,Roll2,Δt2,Δd2] of modal device 2; ...... The modality distribution matrix M will be used for subsequent calculation of spatial Euclidean distance between modes and evaluation of temporal synchronization residuals. It is the basic input parameter for calculating the collaborative metric function of mode pairs.

[0042] S200. Based on the modal distribution matrix M, calculate the space-time alignment error between each modal device, and establish a modal cooperation metric function R to represent the degree of cooperation matching between modal pairs.

[0043] Based on the spatial position vectors obtained from the modal distribution matrix, the spatial alignment deviation between any two modal devices is accurately calculated.

[0044] Calculation of three-dimensional spatial position difference: assuming modal device Spatial position is a three-dimensional vector The position of modal device A1 is Then the three-dimensional Euclidean distance between the two The calculation method is as follows: This value is used to measure the absolute difference in position between two devices in space.

[0045] Attitude and orientation difference calculation: Assume the equipment The orientation angle is A1 is Then the three-axis attitude difference vector for: Each item is expressed in degrees (°), with a maximum permissible attitude error of 30°. The three-dimensional spatial distance and the three-axis attitude difference are combined to form the spatial error value. Using a weighted summation method: ; where α and β are the spatial position weight coefficient and attitude angle weight coefficient, respectively (e.g., α=0.7, β=0.3), which can be adjusted according to the scene's sensitivity to position or orientation.

[0046] For each pair of modal devices Repeat the above calculations to generate the spatial error matrix. Where N is the number of modal devices, Indicates equipment Spatial alignment error between A1 and A1.

[0047] Time alignment error is used to reflect the differences in the time reference and stability of modal devices in data acquisition.

[0048] Time offset difference calculation: Let the time offsets of modal devices Ak and A1 be respectively... and The unit is microseconds (μs), and the time offset error is defined as: ; Time drift difference calculation: Let the time drift (offset per second) of the modal device be... and The unit is microseconds per second (μs / s), and the time drift error is defined as: .

[0049] Define the time alignment error as: ; where γ and δ are the weight coefficients of time offset and time drift respectively, and the recommended values are γ = 0.6 and δ = 0.4.

[0050] Fill the calculation results of the time errors between all pairs of devices into the time error matrix , and the element in the k-th row and l-th column of the matrix represents the time alignment error between Ak and A1.

[0051] Since the dimensions and numerical ranges of the space error and time error are different, they need to be normalized before fusion calculation.

[0052] Let the maximum value in the space error matrix be Maxs and the maximum value in the time error matrix be Maxt, then the normalization process is as follows: Normalized space error matrix: ; Normalized time error matrix: ; The normalized value ranges are both [0, 1].

[0053] Let the fusion weight λ ∈ [0, 1], which is used to control the relative importance of the space error and time error. Then the formula for calculating the fused space-time error is: ; The recommended default value of λ is 0.5, indicating that the weights of the space and time errors are equal.

[0054] Obtain the final space-time alignment error matrix , and each element represents the alignment error score of the corresponding modal device pair in the comprehensive space-time dimension.

[0055] To facilitate the decision support for subsequent modal device combination and optimization strategies, the space-time alignment error matrix is further mapped to a modal collaboration metric function R, which is used to characterize the degree of collaborative matching.

[0056] The modal collaboration metric function R(k, l) is defined as: ; The value range of this function is [0, 1], and the larger the value, the higher the space-time collaboration degree between the modal device pairs.

[0057] To avoid the influence of extreme values on the overall metric, a minimum collaboration threshold Rmin (such as 0.1) and a maximum collaboration threshold Rmax (such as 0.95) can be set to truncate and correct the function results: If R(k, l) < Rmin, then set it to Rmin; If R(k, l) > Rmax, then set it to Rmax.

[0058] The final function R takes the modal device number pair (k, l) as input and returns the corresponding collaborative metric R(k, l). This value will be used for sorting, filtering, and optimization operations in the next stage of the modal combination strategy.

[0059] S300. Based on the collaborative metric function R, select the mode pair combination with the minimum alignment error from the candidate mode devices and establish the acquisition link combination set C.

[0060] Based on the established modal collaboration metric function, this embodiment further proposes a modal pair combination optimization method based on collaboration metric ranking, redundancy detection, and graph structure analysis to improve the spatiotemporal consistency and modal complementarity of multimodal training data acquisition. This method can construct a set of modal acquisition link combinations that meet the acquisition accuracy requirements, achieving a highly collaborative and high-coverage modal collaborative acquisition strategy.

[0061] Effective modal pairs with high synergy in the spatiotemporal dimension are selected from all modal device pairs to provide a data foundation for subsequent combination.

[0062] Input data: Use the modal collaboration metric function R(k,l) constructed above, where k and l represent modal device numbers, R(k,l)∈[0,1], and the larger the value, the higher the degree of collaboration matching.

[0063] Screening strategy: Set a coordination threshold Rth to remove device pairs with low coordination. It is recommended that Rth range from 0.6 to 0.8, and set flexibly according to the spatial-temporal synchronization accuracy requirements of the acquisition task.

[0064] Execution process: Iterate through all modal device pairs in turn. If R(k,l)≥Rth, then add the modal pair to the candidate modal pair set S. Finally, the candidate modal pair set S={(k1,l1),(k2,l2),...,(kn,ln)} is obtained.

[0065] Redundant combinations with highly similar spatial locations and temporal synchronization are identified among candidate modal pairs to avoid duplicate acquisition and improve the effectiveness and diversity of the acquisition link.

[0066] Spatial redundancy detection: If the three-dimensional spatial distance between any pair of devices in two modal pairs (k,l) and (m,n) is less than the set minimum discrimination threshold Dmin (recommended Dmin=0.3 meters), and the attitude difference is less than 5 degrees, then it is determined to be a spatial redundancy combination.

[0067] Time redundancy detection: If both the time offset error and the time drift error are less than the set time redundancy threshold (e.g., offset difference less than 50 microseconds, drift difference less than 5 microseconds per second), it is determined to be time redundancy.

[0068] Redundancy removal strategy: For each pair of modal pairs that meet the redundancy condition, retain the pair with the higher collaboration metric and remove the rest to ensure combinatorial diversity.

[0069] In order to select a combination of modal devices that can fully cover the target area and have the best collaborative performance, this step constructs a modal coverage map and applies graph structure analysis methods to optimize the modal path combination.

[0070] Modal cover graph definition: Construct an undirected graph G(V,E), where each node V represents a modal device, and edge E represents the cooperative connection relationship between two modal devices. If device pair (k,l) belongs to the candidate modal pair set S, then connect Vk and V1 in the graph with edge weight R(k,l).

[0071] Coverage Analysis: A modal coverage index Ci is introduced, representing the weighted sum of the edge weights connected to node Vi, used to measure the data acquisition contribution of the device in that modality. Ci = ∑R(i,j) is calculated for all devices, where j represents all devices connected to i.

[0072] Based on the maximum weighted matching algorithm or the minimum edge cover algorithm, select several non-overlapping modal pair path combinations from graph G to maximize the sum of the total cooperative metric values ​​and ensure coverage of the preset modal types (such as image, audio, depth, etc.).

[0073] After completing the modal path optimization, a set of acquisition link combinations is finally constructed to guide the execution of actual multimodal collaborative acquisition tasks.

[0074] Link definition: Each acquisition link is defined as a combination of modal devices arranged in a highly coordinated order, with clear spatial distribution and temporal alignment characteristics, and meets the modal complementarity required for a specific acquisition task.

[0075] Combination set formation: All selected modal paths are combined and encapsulated into a collection link combination set C={L1,L2,...,Li,...,Lm}, where each Li represents a collection link, which consists of at least two modal devices with a coordination degree higher than the threshold.

[0076] Based on the target area of ​​the data collection task, verify whether each link meets the spatiotemporal coverage requirements. If the coverage of a certain link is insufficient, readjust the weight threshold Rth or reconstruct the execution path until all task areas are fully covered.

[0077] S400. For each modal combination in the acquisition link combination set C, evaluate its modal coverage and modal gap risk within the target acquisition area to obtain a modal stability score S.

[0078] To improve the spatial comprehensiveness and temporal continuity of multimodal data acquisition within the target area, this embodiment provides a modal stability scoring method that integrates spatial coverage analysis and temporal stability assessment. Specifically, for each modal combination in the acquisition link combination set C, its modal coverage and modal gap risk are evaluated, and a comprehensive modal stability score S is calculated accordingly for dynamic adjustment and optimization of acquisition links.

[0079] By establishing a kernel density function model within the acquisition area, the sensing coverage capability of each mode combination in the spatial dimension is evaluated.

[0080] The target acquisition area is divided into a three-dimensional grid of equally spaced points, denoted as G={g1,g2,...,gi,...,gn}, where each sampling point gi corresponds to a fixed spatial coordinate (xi,yi,zi). The recommended spacing between sampling points is 0.2 meters to 0.5 meters to meet spatial resolution requirements.

[0081] For each modal device in the modal combination, its spatial position (X, Y, Z) and orientation angle (Pitch, Yaw, Roll) are extracted to construct a sensing function K(i, g), which describes the device's response capability to the target point g. A larger value for K(i, g) indicates a stronger sensing capability of the modal device at that point. The sensing function is defined using a three-dimensional Gaussian kernel function as follows: Where: d is the Euclidean distance between the device location and the sampling point g; σ is the kernel function diffusion parameter, with a recommended value of 1.0; A(θi,g) is the orientation weighting coefficient, defined as a nonlinear mapping of the cosine of the angle between the device orientation and the sampling point direction, with a mapping value ∈ [0,1].

[0082] For all modal devices in the modal combination, calculate their response capability at each sampling point, and take the maximum response value at each sampling point as the comprehensive sensing value of that point: R(g)=max{K(1,g),K(2,g),...,K(m,g)}; then, sum up the R(g) of all sampling points and take the average, which is defined as the spatial coverage score Cs of the modal combination, with a value range of [0,1]. The larger the value, the more comprehensive the coverage.

[0083] By analyzing the sampling time interval fluctuations of modal devices, the continuity of their time dimension is quantified, and the temporal stability of modal combinations is evaluated. Specifically: For each modal device in the modal combination, extract its sampling timestamp sequence {t1, t2, ..., tn} within a preset acquisition window T (e.g., 10 seconds), where tn represents the sampling time of the nth frame.

[0084] The complete time series is divided into fixed-length sliding time windows, with a window length of L (e.g., 1 second) and a sliding step size of 0.5 seconds. Within each window, the standard deviation σk of the sampling time interval is calculated, resulting in the σ sequence {σ1, σ2, ..., σm} corresponding to all windows, which is used to describe the temporal volatility.

[0085] Take the average value of the σ sequence for each modal device. This reflects its time-sampling stability. For all devices in the modal combination... The maximum value σmax is taken as the time fluctuation index of this combination. The time stability score Ct is defined as: Ct=1-min(1,σmax / σmax-allow); where σmax-allow is the maximum allowed time fluctuation limit (e.g., 100 milliseconds). Ct∈[0,1], the larger the value, the better the time stability and the lower the risk of the gap period.

[0086] To obtain the overall modal combination stability score, the spatial coverage score Cs and the temporal stability score Ct are weighted and fused to calculate the modal stability score S. The formula is defined as follows: S = A × Cs + B × Ct; where A and B are the weight coefficients for spatial coverage and temporal stability, respectively, satisfying A + B = 1; the recommended values ​​are A = 0.6 and B = 0.4, which are suitable for most application scenarios with heavy spatial protection and light temporal delay; for scenarios with higher real-time requirements, A = 0.4 and B = 0.6 can be adjusted.

[0087] The range of S is [0,1]. The higher the value, the better the modal combination performs in terms of spatial perception capability and temporal sampling stability, making it suitable for the final acquisition strategy deployment stage.

[0088] S500: Based on the modal stability score S, adjust the sampling frequency and exposure parameters of the modal device to generate a modal acquisition strategy P.

[0089] After obtaining the modal stability score for each modal combination, in order to further improve the acquisition efficiency and data quality, this embodiment adaptively adjusts the sampling frequency and exposure parameters of each modal device based on the score results, and constructs an optimized modal acquisition strategy to guide the scheduling and execution in the subsequent data acquisition process.

[0090] Modal combinations are managed hierarchically according to their stability scores to match different levels of parameter optimization strategies. Let the range of the modal stability score S be [0,1], divided into three level intervals, defined as follows: High stability rating: S≥0.8; Medium stability class: 0.5 ≤ S < 0.8; Low stability level: S < 0.5; The grading can be adjusted according to the actual scenario, but the boundary values ​​must be fixed to avoid overlapping scores.

[0091] For each mode combination in the acquisition link combination set C, its stability level is labeled L∈{High,Medium,Low} according to the interval in which its score S falls, providing a basis for subsequent parameter adjustment strategies.

[0092] Different parameter adjustment strategies are adopted for modal combinations with different stability levels to enhance acquisition performance and control resource consumption.

[0093] High stability modal combination (S≥0.8): This type of modal device has shown stability in terms of spatial coverage and temporal continuity, and no parameter adjustment is required. It maintains its original sampling frequency f0 and exposure time e0 to ensure the balance between system energy efficiency and computational load.

[0094] Medium-stability mode combinations (0.5 ≤ S < 0.8): To improve the data accuracy of medium-stability combinations, a linear enhancement strategy is used for parameter fine-tuning. The sampling frequency boost ratio η1 = (S - 0.5) ÷ 0.3 × Δfmax; The reduction in exposure time is η2 = (S - 0.5) ÷ 0.3 × Δemax; Where Δfmax is the maximum allowable increase in sampling frequency (e.g., 20%), and Δemax is the maximum reduction in exposure time (e.g., 15%). After adjustment, the sampling frequency is f = f0 × (1 + η1), and the exposure time is e = e0 × (1 - η2).

[0095] Low-stability mode combinations (S<0.5): Low-stability combinations exhibit spatial blind spots or sampling discontinuities, requiring a focus on enhancing data acquisition density and image quality. A keyframe enhancement strategy is enabled, increasing the sampling frequency of keyframes selected in consecutive time slices by 1.5 times; At the same time, the exposure time is extended by Δeadd to improve the signal-to-noise ratio; Δeadd is recommended not to exceed 30%.

[0096] This strategy can dynamically adjust the keyframe selection segment through a local anomaly detection algorithm.

[0097] To achieve automated parameter adjustment, this step constructs a mapping function between the stability score S and the frequency and exposure parameters, providing continuous adjustment capability.

[0098] Define the frequency adjustment function F(S) as a piecewise linear function: if S≥0.8, then F(S)=f0; if 0.5≤S<0.8, then F(S)=f0×(1+0.6×(S-0.5)); if S<0.5, then F(S)=f0×1.5 (key frame).

[0099] Define the exposure adjustment function E(S) as an inverse mapping function: if S≥0.8, then E(S)=e0; if 0.5≤S<0.8, then E(S)=e0×(1-0.5×(S-0.5)); if S<0.5, then E(S)=e0+Δeadd.

[0100] The two mapping functions above are used to convert the stability score of each mode combination into specific parameter values ​​for the corresponding device.

[0101] Based on the obtained sampling frequency and exposure parameters, acquisition strategies are further constructed for each modal device. The modal acquisition strategy P is a key-value structure set, in the following form: P={Device No.1:[Frequency F1, Exposure E1], Device No.2:[Frequency F2, Exposure E2], ...}; Each entry records the final sampling frequency and exposure time configuration of a modal device, and the data is updated in a fixed time window.

[0102] The data acquisition strategy P can be exported as a configuration file, used to send parameters to the data acquisition execution unit. It supports dynamic updates during runtime, adjusting acquisition behavior based on real-time scoring.

[0103] S600: Execute the modal acquisition strategy P to perform multimodal collaborative acquisition and obtain the training dataset.

[0104] After generating the parameters for the modal acquisition strategy P, this embodiment further executes the acquisition strategy, synchronously controlling each heterogeneous modal device to carry out multimodal collaborative acquisition tasks based on a cooperative scheduling mechanism, and finally obtaining a multimodal training dataset D that meets the requirements of spatiotemporal alignment. The training dataset has temporal consistency, spatial coordination, and modal diversity, providing high-quality basic data for large model training.

[0105] The acquisition strategy is distributed and parameters are loaded. This step first distributes the sampling frequency and exposure parameters contained in the modal acquisition strategy P to the corresponding modal devices one by one. Each device, upon receiving its corresponding parameters, completes local configuration loading, including: Set the frame rate parameter f_img and exposure time e_img for the image modality device; Set the sampling rate f_audio and recording period for the audio modal device; Set the trigger frequency and integration time for modal devices such as laser, radar, infrared, and depth sensors.

[0106] All devices use a unified set of control commands for configuration updates. The command format includes device identifiers, parameter types, parameter values, and verification fields to ensure the correctness and consistency of parameter transmission.

[0107] After the parameters are configured, the acquisition scheduling management unit constructs a time scheduling table Ts based on the modal coordination metric function R and the synchronization offset Δt, which is used to coordinate the acquisition operations of each modality. The specific implementation method is as follows: Define a global acquisition timeline for all modal devices; Based on the time offset Δt between devices, offset correction is performed to generate a sequence of synchronized sampling time points; For devices with inconsistent sampling frequencies, nearest neighbor interpolation or sample buffering is used to perform time-aligned frame interpolation for low-frequency sampling modes.

[0108] All modal devices acquire data frames under a global time synchronization reference to ensure the temporal consistency of multimodal data at each sampling moment. The sampling operation is controlled by an event-triggered method, and within each time slice, all modal devices are triggered to acquire data according to the schedule.

[0109] During the sampling process, the data frames generated by each modal device are marked according to their modality type and timestamp information, forming structured raw acquired data. Subsequently, multimodal data frames within the same time slice are aggregated into "coordinated frames," the structure of which is as follows: Cooperative frames ={timestamp t k Image frame: Audio clip: Depth map: Infrared image: Laser point cloud: }

[0110] Collaborative frames are the basic building blocks of a dataset, ensuring precise temporal alignment and semantic spatial consistency across all modalities.

[0111] After the data acquisition task is completed, all collaborative frames are organized in chronological order to construct a multimodal training dataset D. Dataset D contains the following structure: Data index table: records the timestamps and modal content of all collaborative frames; Data content directory: Divided into subdirectories according to modality type, storing data such as images, audio, and depth; Metadata file: contains auxiliary information such as device identifier, parameter configuration, acquisition time, and intermodal synchronization error.

[0112] The dataset export format supports HDF5, TFRecord, or custom binary formats, and features high-speed read / write performance and multi-task training compatibility.

[0113] Example 2, please refer to Figure 2As shown in this embodiment, a multimodal large model training data acquisition system includes: The synchronization module acquires the spatial location information and time synchronization parameters of each heterogeneous modal device in the environment to be collected, and constructs a modal distribution matrix M, in which each modal device corresponds to a position vector and a time offset; The collaborative evaluation module calculates the spatiotemporal alignment error between each modal device based on the modal distribution matrix M, and establishes a modal collaboration metric function R to represent the degree of collaboration matching between modal pairs. The link construction module selects mode pair combinations with the minimum alignment error from candidate mode devices based on the collaborative metric function R, and establishes a set of acquisition link combinations C; The modal stability analysis module evaluates the modal coverage and modal gap risk within the target acquisition area for each modal combination in the acquisition link combination set C, and obtains a modal stability score S. The acquisition strategy generation module adjusts the sampling frequency and exposure parameters of the modal acquisition device based on the modal stability score S, and generates a modal acquisition strategy P. The data generation module executes the modal acquisition strategy P to perform multimodal collaborative acquisition and obtain the training dataset.

[0114] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for acquiring training data for a multimodal large model, characterized in that... :include: S100. Acquire the spatial location information and time synchronization parameters of each heterogeneous modal device in the environment to be collected, and construct the modal distribution matrix M, where each modal device corresponds to a position vector and a time offset; S200. Based on the modal distribution matrix M, calculate the spatiotemporal alignment error between each modal device, and establish a modal cooperation metric function R to represent the degree of cooperation matching between modal pairs; S300. Based on the collaborative metric function R, select the mode pair combination with the minimum alignment error from the candidate mode devices to establish the acquisition link combination set C; S400. For each mode combination in the acquisition link combination set C, evaluate its mode coverage and mode gap risk within the target acquisition area to obtain the mode stability score S; S500: Based on the modal stability score S, adjust the sampling frequency and exposure parameters of the modal acquisition device to generate the modal acquisition strategy P; S600: Execute the modal acquisition strategy P to perform multimodal collaborative acquisition and obtain the training dataset.

2. The method for acquiring multimodal large model training data according to claim 1, characterized in that... The step S100 of acquiring the spatial location information of each heterogeneous modal device includes: At least three 3D calibration points with known world coordinates are set up in the environment to be acquired. The visual modal device is controlled to capture images of the calibration points, and the image coordinates of the calibration points are obtained through image recognition. Combined with the coordinates of the calibration points, the position coordinates and orientation angle of the modal device are solved by the EPnP algorithm. The image reprojection error is converged to less than 0.5 pixels by the Levenberg-Marquardt nonlinear least squares optimization algorithm.

3. The method for acquiring training data for a multimodal large model according to claim 1, characterized in that... The S100 acquisition of time synchronization parameters includes: Generate a time reference signal; Each modal device receives the time reference signal, records its local timestamp, and calculates the time offset Δt; Repeat the time offset sampling at least 10 times and take the average value; Record a continuous sampling timestamp sequence and calculate the time drift Δd per unit time using linear least squares fitting.

4. The method for acquiring training data for a multimodal large model according to claim 1, characterized in that... The establishment of the modal cooperative metric function R in S200 includes: Calculate the three-dimensional Euclidean distance and attitude angle difference between the modal device pairs; Spatial error weighting function Characterizing spatial alignment error; Calculate the time offset difference and drift difference between modal device pairs, and construct a time error function. ; The spatial error matrix and the temporal error matrix are normalized and then weighted and combined using a fusion weight λ to obtain the spatiotemporal alignment error matrix. ; Finally, the modal collaborative metric function is calculated. And set upper and lower limits for synergy to truncate and correct.

5. The method for acquiring training data for a multimodal large model according to claim 1, characterized in that... The S300 establishes a set of acquisition link combinations C, which includes: All modal device pairs in the modal cooperation metric function are screened, and a cooperation threshold Rth is set. Modal device pairs with R(k,l) greater than Rth are selected as candidate sets. For candidate mode pairs, spatial redundancy detection is performed. If the three-dimensional spatial distance is less than 0.3 meters and the attitude angle difference is less than 5 degrees, it is considered redundant. For candidate mode pairs, if the time offset difference is less than 50 microseconds and the time drift difference is less than 5 microseconds per second, it is considered redundant. For redundant mode pairs, only the pair with the highest degree of cooperation is retained, and the rest are discarded; Construct an undirected graph structure where nodes represent modal devices and edge weights are the degree of collaboration R(k,l). Select modal combination paths using the maximum weighted matching or minimum edge cover algorithm to form a set of acquisition link combinations C.

6. The method for acquiring multimodal large model training data according to claim 1, characterized in that... The modal coverage evaluation in the S400 adopts a method based on high-dimensional space kernel density estimation, including: The target acquisition area was divided into a three-dimensional grid of sampling points with a spacing of 0.2 meters to 0.5 meters; A modality sensing function K(i,g) is constructed, using a three-dimensional Gaussian kernel function and weighted by orientation influence factors; Calculate the maximum modal sensing response value R(g) for each sampling point, and take the average value of R(g) for all sampling points to obtain the spatial coverage score Cs, with a value range of [0,1].

7. The method for acquiring training data for a multimodal large model according to claim 1, characterized in that... The modal window period risk assessment in S400 adopts a local temporal stability scoring method, including: Extract the sampling timestamp sequence of each modal device within a preset time window T; The standard deviation σ of the time interval is calculated using a sliding window of length L and a step size of 0.5 seconds. The average of the standard deviations of all windows is taken as the local volatility of this mode. The maximum variability σmax of all modes is used to calculate the time stability score; The time stability score is Ct = 1 - min(1, σmax / σmax-allow), where σmax-allow is 100 milliseconds.

8. The method for acquiring training data for a multimodal large model according to claim 1, characterized in that... The generation of the modal acquisition strategy P in S500 includes: The stability score (S) is divided into three levels: high, medium, and low. Different sampling frequencies and exposure times are adjusted according to the level. The intermediate-level mode combination uses a linear enhancement strategy, with a frequency increase ratio η1 = (S - 0.5) / 0.3 × Δfmax and an exposure reduction ratio η2 = (S - 0.5) / 0.3 × Δemax; where Δfmax is the maximum allowable increase in sampling frequency and Δemax is the maximum reduction in exposure time. The low-stability combination activates a keyframe enhancement strategy, increasing the sampling frequency to 1.5 times the original frequency and increasing the exposure time Δeadd; The score S is mapped to the final frequency and exposure parameters using mapping functions F(S) and E(S), and a modal acquisition strategy P with key-value pair structure is generated.

9. The method for acquiring training data for a multimodal large model according to claim 1, characterized in that... The S600 execution modal acquisition strategy P includes: After the modal acquisition strategy P is sent to the modal device, the parameters are loaded. Construct a unified global acquisition timeline and perform offset correction based on the time offset of each device; Synchronous triggering and acquisition of modal devices are performed according to the time schedule; frame-level alignment is achieved for non-equivalent frequency devices using interpolation or buffering methods. Organize modal data from the same sampling time point into a co-frame; All collaborative frames are organized in chronological order to form the training dataset D.

10. A multimodal large model training data acquisition system, used to implement the multimodal large model training data acquisition method according to any one of claims 1-9, characterized in that... :include: The synchronization module acquires the spatial location information and time synchronization parameters of each heterogeneous modal device in the environment to be collected, and constructs a modal distribution matrix M, in which each modal device corresponds to a position vector and a time offset; The collaborative evaluation module calculates the spatiotemporal alignment error between each modal device based on the modal distribution matrix M, and establishes a modal collaboration metric function R to represent the degree of collaboration matching between modal pairs. The link construction module selects mode pair combinations with the minimum alignment error from candidate mode devices based on the collaborative metric function R, and establishes a set of acquisition link combinations C; The modal stability analysis module evaluates the modal coverage and modal gap risk within the target acquisition area for each modal combination in the acquisition link combination set C, and obtains a modal stability score S. The acquisition strategy generation module adjusts the sampling frequency and exposure parameters of the modal acquisition device based on the modal stability score S, and generates a modal acquisition strategy P. The data generation module executes the modal acquisition strategy P to perform multimodal collaborative acquisition and obtain the training dataset.

Images