A method, device and medium for generating a multimodal image data set

Abstract

The present application discloses a method, device and medium for generating a multimodal image data set, the method comprising determining an image feature extraction strategy based on preprocessed multimodal image data; analyzing the correlation and interaction mode between each modality image data, and determining the influence weight of each modality image data on the synthetic image; establishing an association relationship between each modality image data, and obtaining modality association information; obtaining original image data of a first type of sensor modality and deep image data of a second type of sensor modality, and inputting an image feature extraction strategy to obtain an image feature fusion result; obtaining a synchronization timestamp and sampling frequency between the first type of sensor modality and the second type of sensor modality; performing model optimization of a multimodal fusion model; and generating a target image data set using the optimized multimodal fusion model. The present application solves the problem that a single modality image data set is difficult to meet the high-precision recognition requirements of a deep learning model in a complex environment.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal data processing, and in particular to a method, device and medium thereof for generating a multimodal image data set. Background Art

[0002] In the training process of deep learning models, traditional image dataset generation methods usually rely on the collection of information from a single modality. This method cannot fully utilize the complementarity between information from different modalities, and single-modality datasets limit the generalization and robustness of deep learning models, especially in complex environments. For example, the performance of the trained model degrades significantly when faced with lighting changes, occlusions, etc. Therefore, there is a problem that single-modality image datasets are difficult to meet the high-precision recognition requirements of deep learning models in complex environments. Summary of the invention

[0003] In order to solve the problem that a single modality image dataset is difficult to meet the high-precision recognition requirements of deep learning models in complex environments, the present application provides a method, device and medium for generating a multimodal image dataset.

[0004] In the first aspect, the invention objective of the present application is achieved by adopting the following technical solutions:

[0005] A method for generating a multimodal image dataset, comprising:

[0006] Collect multimodal image data from different sources and perform preprocessing; determine image feature extraction strategy based on the preprocessed multimodal image data;

[0007] Analyze the correlation and interaction patterns between the image data of each modality, and determine the influence weight of each modality image data on the synthetic image; divide the multi-modality of different sources into the first type of sensor modality and the second type of sensor modality, and establish the correlation relationship between the image data of each modality to obtain modality correlation information;

[0008] Based on the modality association information, the original image data of the first sensor modality and the depth image data of the second sensor modality are acquired, and the image feature extraction strategy is input to obtain the image feature fusion result;

[0009] Based on the synchronization timestamp and sampling frequency between the first type of sensor modality and the second type of sensor modality; inputting the synchronization timestamp, sampling frequency and image feature fusion result into a preset multimodal fusion model for model optimization;

[0010] Generate target image dataset using optimized multimodal fusion model.

[0011] By adopting the above technical scheme, multimodal image data from different sources include but are not limited to visible light RGB images, near-infrared images, lidar point cloud data and audio recording data; the target image dataset is a multimodal image dataset containing rich diversity, which can meet the high-precision recognition requirements of deep learning models in complex environments; specifically, the present application analyzes the correlation and interaction patterns between the image data of each modality, calculates the influence weight of each modality image data on the synthetic image, and establishes the correlation relationship between the modalities, so as to fully explore and utilize the complementarity between different modal information, and at the same time, in order to improve the generalization ability and robustness of the model, classifies the multimodal data into the first sensor modality and the second sensor modality. The image feature fusion result of multiple modal features is obtained by combining the two types of sensor modalities, so as to make the subsequently generated image data sets more diversified. The synchronization between the modal data of the first type of sensor modality and the modal data of the second type of sensor modality is analyzed in the time dimension and the space dimension through the synchronization timestamp and sampling frequency between the two types of sensor modalities. The synchronization timestamp, sampling frequency and image feature fusion results are then input into the preset multimodal fusion model for model optimization. The model parameters are adjusted through the iterative optimization process, and finally a high-quality image data set with rich diversity is generated. The final image data set enables the trained model to maintain good performance when facing illumination changes, occlusion and other situations.

[0012] In a preferred example of the present application: the image feature extraction strategy determined based on the pre-processed multimodal image data specifically includes:

[0013] Based on the preprocessed multimodal image data, characteristic information of each modality image data is obtained, wherein the characteristic information includes modality type, resolution, color depth, and data volume;

[0014] Analyze the role of each modality image data in the composite image and determine the importance level of each modality image data;

[0015] The graphical information of each modality image data is associated with the importance level, and an image feature extraction strategy suitable for the current group of multimodality image data is generated.

[0016] By adopting the above technical solution, the characteristic information of each modal image data is obtained in a refined manner, so as to have a comprehensive understanding of the characteristics of different modal data, so that the image feature extraction strategy can more accurately reflect the characteristics of each modal data; at the same time, by analyzing the role of each modal image data in the synthetic image, its importance level to the synthetic image is determined, which not only considers the physical properties of the data itself, but also combines its importance in practical applications to ensure that the generated image feature extraction strategy can give priority to key information; then the graphic information of each modal image data is associated with its importance level to generate an image feature extraction strategy suitable for the current group of multimodal image data, thereby ensuring a reasonable weight distribution between different modal data and improving the accuracy of feature extraction.

[0017] In a preferred example of the present application: the obtaining of modality association information specifically includes:

[0018] Inputting data of the first sensor modality and the second sensor modality into a preliminary multimodal fusion model;

[0019] Determine the dependency and complementarity between different modal data based on the characteristic information and importance level of each modality image data; select and mark key modality data based on the preset synthetic image quality standard;

[0020] Calculate the influence weight of each modality image data on the synthesized image according to the key modality data and the corresponding characteristic information, and the timestamps of the first sensor modality and the second sensor modality;

[0021] According to the dependency and complementarity between different modal data and the influence weight of each modal image data on the synthetic image, the association relationship between the modal image data is established to form modal association information.

[0022] By adopting the above technical scheme, the first type of sensor modality is a visual sensor modality, and the corresponding image data includes RGB image data, grayscale image data, infrared image data, hyperspectral image data and hyperspectral image data, etc.; the second type of sensor modality is a non-visual sensor modality, and the corresponding image data includes depth image data (from laser radar (LiDAR), structured light camera or stereo vision system), point cloud data, radar data, ultrasonic data, thermal imaging data, etc.; the present invention judges the dependency and complementarity between different modal data based on the characteristic information and importance level of each modal image data, which is conducive to discovering the intrinsic connection between the modalities, and then by selecting and marking key modal data, it is conducive to prioritizing the retention and utilization of information that is crucial to the quality of the synthetic image during the data fusion process, and then by calculating the influence weight of each modal image data on the synthetic image, the association relationship between the image data of each modality is established, and detailed modal association information for guiding the subsequent image feature fusion process is formed.

[0023] In a preferred example of the present application: the step of inputting the synchronization timestamp, sampling frequency and image feature fusion result into a preset multimodal fusion model for model optimization specifically includes:

[0024] Adjust the parameter settings in the multimodal fusion model according to the determined influence weights, optimize the image feature fusion process, and obtain an optimized image feature fusion result;

[0025] Inputting the synchronization timestamp, sampling frequency and optimized image feature fusion result into an image feature extraction strategy to generate first feature mapping information of each modality image data;

[0026] Utilizing the first feature mapping information, based on the characteristic information of each modality image data and its influence weight, combined with the synchronization timestamp and sampling frequency, the model optimization parameters are calculated;

[0027] Based on the model optimization parameters, the final multimodal fusion model is obtained.

[0028] By adopting the above technical scheme, the parameter settings in the multimodal fusion model are adjusted according to the determined influence weights, the image feature fusion process is optimized, and it is ensured that the model can adaptively adjust its internal parameters under different data conditions. By introducing time and space information such as synchronization timestamps and sampling frequencies, combined with the optimized image feature fusion results, the first feature mapping information of each modality image data is generated to further refine the representation of image features and enhance the diversity and representativeness of the data set; at the same time, the first feature mapping information is used to calculate the model optimization parameters based on the characteristic information of each modality image data and its influence weight, combined with the synchronization timestamp and sampling frequency, so as to ensure the scientificity and systematicness of the model optimization process, so that the optimized multimodal fusion model can better adapt to various complex scenarios.

[0029] In a preferred example of the present application, the use of an optimized multimodal fusion model to generate a target image dataset specifically includes:

[0030] Based on the model parameter setting of the final multimodal fusion model, the preset complex scene simulation data is associated with the first feature mapping information, and according to the type of the complex scene simulation data, the output of the multimodal fusion model is adjusted to generate second feature mapping information adapted to different scenes;

[0031] The first feature mapping information and the second feature mapping information corresponding to each modality image data are combined to generate a target image data set.

[0032] By adopting the above technical solution, based on the model parameter setting of the final multimodal fusion model, the preset complex scene simulation data is associated with the first feature mapping information, ensuring that the generated image data set not only contains data under conventional environments, but also covers a variety of complex scenes (such as illumination changes, occlusions, etc.), thereby improving the generalization ability of the trained deep learning model in practical applications; according to the type of complex scene simulation data, the output of the multimodal fusion model is adjusted to generate second feature mapping information adapted to different scenes. In this way, the model can dynamically respond to the needs of different scenes.

[0033] In a preferred example of the present application, the calculation of the influence weight of each modality image data on the synthesized image specifically includes:

[0034] Based on formula (1), the influence weight W of each modality image data on the synthetic image is calculated: i :

[0035] W i =α×T i / T avg +β×R i / R avg +γ×F i / Favg +δ×(1-Δt i / Δt max )

[0036] Where i is the mode type; T i is the thermal sensitivity of the modality image data; T avg is the average thermal sensitivity of all modality image data; R i is the resolution of the modality image data; R avg is the average resolution of all modality image data; F i is the data size of the modality image data; F avg is the average data size of all modal image data; Δt i is the timestamp difference of the modality image data; Δt max is the maximum timestamp difference of all modality image data; α, β, γ, δ are preset weight coefficients.

[0037] By adopting the above technical solution, the resolution and data size of the modal image data are used to balance the importance of each modal image data, so as to effectively avoid excessive influence of a certain modal data on the synthesis result due to too high or too low resolution or data size. Formula (1) takes into account the timestamp difference of the modal image data, especially the maximum timestamp difference, to ensure the time synchronization between different modal data.

[0038] In a preferred example of the present application, the calculation model optimization parameters specifically include:

[0039] The model optimization parameter P is calculated based on formula (2): final :

[0040]

[0041] Where P ini are the initial multimodal fusion model parameters; μ is the learning rate, which is used to control the optimization step size; n is the number of modality image data; i is the modality type; W i is the influence weight of each modality image data on the synthetic image; TS i is the synchronization timestamp of the modality image data; TS avg is the average synchronization timestamp of all modality image data; SF i is the sampling frequency of the modality image data; SF avg is the average sampling frequency of image data of all modalities.

[0042] By adopting the above technical solution, formula (2) avoids the problem of too fast or too slow convergence by controlling the optimization step size through an appropriate learning rate, introduces the influence weight of each modality image data on the synthetic image, and comprehensively considers the temporal and spatial characteristics of each modality image data, so that the optimized multimodal fusion model can efficiently process multimodal data.

[0043] In the second aspect, the invention objective of the present application is achieved by adopting the following technical solutions:

[0044] A system for generating a multimodal image dataset, applied to the method for generating a multimodal image dataset as described above, the system comprising:

[0045] Data collection and preprocessing module, used to collect multimodal image data from different sources and perform preprocessing;

[0046] A strategy determination module, used for determining an image feature extraction strategy based on preprocessed multimodal image data;

[0047] A modality association analysis module is used to analyze the correlation and interaction mode between the image data of each modality, determine the influence weight of each modality image data on the synthetic image, and establish the association relationship between the image data of each modality to obtain modality association information; an image feature fusion module is used to classify the multi-modality of different sources into a first type of sensor modality and a second type of sensor modality based on the modality association information; obtain the original image data of the first type of sensor modality and the depth image data of the second type of sensor modality, and input the image feature extraction strategy to obtain the image feature fusion result;

[0048] A model optimization module, for inputting the synchronization timestamp, sampling frequency and image feature fusion result into a preset multimodal fusion model for model optimization based on the synchronization timestamp and sampling frequency between the first type of sensor modality and the second type of sensor modality;

[0049] The image dataset generation module is used to generate a target image dataset using an optimized multimodal fusion model.

[0050] In the third aspect, the invention objective of the present application is achieved by adopting the following technical solutions:

[0051] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for generating a multimodal image data set when executing the computer program.

[0052] In a fourth aspect, the invention objective of the present application is achieved by adopting the following technical solutions:

[0053] A computer program product comprises a computer program / instruction, which implements the steps of the above-mentioned method for generating a multimodal image dataset when executed by a processor.

[0054] In summary, the present application includes at least one of the following beneficial technical effects:

[0055] 1. Integrate multimodal image data from different sources and perform refined processing. By classifying multimodal data into the first sensor modality (such as visual sensor) and the second sensor modality (such as radar sensor), and combining the original image data and deep image data of the two to perform image feature fusion, the generated image dataset is more diverse, which helps to train deep learning models with stronger generalization capabilities;

[0056] 2. Introducing the synchronization timestamp and sampling frequency as optimization parameters in the multimodal fusion model ensures the synchronization of the two modal data in time and space. This not only improves the authenticity and accuracy of the synthesized image, but also makes the trained model more robust and stable in complex environments such as lighting changes and occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of a method for generating a multimodal image dataset in one embodiment of the present application. DETAILED DESCRIPTION

[0058] The present application is further described in detail below in conjunction with the accompanying drawings.

[0059] In one embodiment, if Figure 1 As shown, the present application discloses a method for generating a multimodal image dataset, which specifically includes the following steps:

[0060] S1: Collect multimodal image data from different sources and perform preprocessing; determine the image feature extraction strategy based on the preprocessed multimodal image data.

[0061] In this embodiment, multimodal image data refers to data obtained from multiple different types of sensors, such as visible light cameras, infrared cameras, laser radars (LiDAR), ultrasonic sensors, etc.; image preprocessing includes but is not limited to data cleaning, format conversion, resizing, noise removal, etc. to ensure that data from different sources are compatible and effectively used; image feature extraction strategy refers to selecting a suitable algorithm or model to extract key information from the image based on the characteristics of the preprocessed data, such as edge detection, texture analysis, color distribution, etc.

[0062] Specifically, step S1 includes:

[0063] S11: Based on the preprocessed multimodal image data, characteristic information of each modality image data is obtained, where the characteristic information includes modality type, resolution, color depth, and data volume.

[0064] Specifically, the modality type refers to the type of sensor or data source, such as RGB camera, infrared camera, LiDAR, ultrasonic sensor, etc.; resolution refers to the number of pixels in the image, usually expressed as width × height; color depth refers to the number of bits of color information for each pixel, which determines the richness of the color; data size refers to the size of the image file or the amount of data it contains, and the unit can be bytes.

[0065] S12: Analyze the role of each modality image data in the synthesized image, and determine the importance level of each modality image data.

[0066] Specifically, the composite image is the final output image formed by the fusion of multiple modal image data; role analysis refers to the evaluation of the specific contribution and function of each modal data in generating the composite image; the importance level is determined by assigning a relative importance score to each modal image data based on the results of the role analysis.

[0067] S13: Associating the graphic information of each modality image data with the importance level, and generating an image feature extraction strategy suitable for the multimodal image data of the current group.

[0068] Specifically, graphic information refers to the description of image content, including but not limited to visual features such as edges, textures, and shapes; the image feature extraction strategy is based on comprehensive considerations and develops specific algorithms or models to extract the most useful image features.

[0069] S2: Analyze the correlation and interaction mode between the image data of each modality, determine the influence weight of each modality image data on the synthetic image, and establish the correlation relationship between the image data of each modality to obtain modality correlation information.

[0070] In this embodiment, correlation analysis refers to studying the similarities and differences between different modal data and understanding how they complement each other. Statistical methods or machine learning algorithms (such as Pearson correlation coefficient, mutual information, etc.) can be used to calculate the similarity between the modalities; the interactive mode refers to exploring the potential connections and cooperation methods between the modalities, such as time synchronization, spatial consistency, etc.; the influence weight is used to quantify the importance of each modality's contribution to the final composite image; the modal association information is used to record the relationship between the modalities and to guide the subsequent data fusion process.

[0071] Specifically, step S2 includes:

[0072] S21: Inputting data of the first type of sensor modality and the second type of sensor modality into a preliminary multimodal fusion model.

[0073] Specifically, the preliminary multimodal fusion model refers to a model framework suitable for preliminary fusion, which needs to be further optimized.

[0074] S22: Determine the dependency and complementarity between different modality data based on the characteristic information and importance level of each modality image data.

[0075] Specifically, the statistical tool of Pearson correlation coefficient can be used to calculate the similarity and dependence between modes; heat maps or network diagrams can also be drawn through visualization tools to intuitively display the relationship between modes; complementarity evaluation includes difference analysis and joint contribution evaluation, among which difference analysis refers to comparing the uniqueness and irreplaceability of different modal data; joint contribution evaluation takes into account the gain effect after combining two or more modes.

[0076] S23: Select and mark key modality data based on a preset synthetic image quality standard.

[0077] In this embodiment, the synthetic image quality standard is an indicator system for evaluating the quality of the synthetic image, such as clarity, contrast, noise level, etc.; the key modal data refers to the modal data that has a significant impact on the quality of the final synthetic image.

[0078] Specifically, according to application requirements, specific synthetic image quality standards are formulated, such as PSNR (peak signal-to-noise ratio), SSIM (structural similarity index), etc., and key data are selected and marked, such as labeling selected key modal data.

[0079] S24: Calculate the influence weight of each modality image data on the synthesized image according to the key modality data and the corresponding characteristic information, and the timestamps of the first sensor modality and the second sensor modality.

[0080] In this embodiment, the influence weight is used to quantify the importance of each modality data's contribution to the final synthesized image.

[0081] Specifically, the influence weight W of each modality image data on the synthesized image is calculated based on formula (1): i :

[0082] W i =α×T i / T avg +β×R i / R avg +γ×F i / F avg +δ×(1-Δt i / Δt max )

[0083] Where i is the mode type; T i is the thermal sensitivity of the modality image data; T avg is the average thermal sensitivity of all modality image data; R i is the resolution of the modality image data; R avg is the average resolution of all modality image data; F i is the data size of the modality image data; F avg is the average data size of all modal image data; Δt i is the timestamp difference of the modality image data; Δt max is the maximum timestamp difference of all modality image data; α, β, γ, δ are preset weight coefficients, which are used to adjust the relative importance of each factor.

[0084] S25: According to the dependency and complementarity between different modal data and the influence weight of each modal image data on the synthetic image, an association relationship between the modal image data is established to form modal association information.

[0085] In this embodiment, the association relationship describes the mode and strength of interaction between image data of each modality; the modality association information is a structured document or database that records detailed association rules and parameters.

[0086] Specifically, association rule definition refers to defining specific association rules based on previous dependencies, complementarities, and weight calculation results; creating a structured database or chart to clearly display the association rules between the modalities. In actual application, it is necessary to verify the effectiveness of the established association relationship through experiments, and make necessary adjustments based on feedback. At the same time, it is necessary to regularly update and maintain the modal association information to ensure that it always meets the latest data characteristics and application scenario requirements.

[0087] S3: Based on the modality association information, the multi-modal data from different sources are divided into the first sensor modality and the second sensor modality; based on the original image data of the first sensor modality and the depth image data of the second sensor modality, and inputting the image feature extraction strategy, the image feature fusion result is obtained.

[0088] In this embodiment, the classification standard is to divide the modalities into two categories according to the standards defined in the modal association information. For example, the first category provides two-dimensional visual information and the other category provides three-dimensional depth information. The original image data refers to image data that has not been depth processed, such as RGB images. The depth image data is information generated by the depth sensor to represent the distance of the object.

[0089] The first type of sensor modality is the visual sensor modality, and the corresponding image data includes RGB image data, grayscale image data, infrared image data, hyperspectral image data, and hyperspectral image data, etc.; the second type of sensor modality is the non-visual sensor modality, and the corresponding image data includes depth image data (from laser radar (LiDAR), structured light camera or stereo vision system), point cloud data, radar data, etc.

[0090] S4: Based on the synchronized timestamp and sampling frequency between the first type of sensor modality and the second type of sensor modality; the synchronized timestamp, sampling frequency and image feature fusion results are input into a preset multimodal fusion model for model optimization.

[0091] In this embodiment, the synchronization timestamp records the time point when each sensor collects data at the same moment, which is used to ensure the consistency and comparability of the data; the sampling frequency refers to the number of samples collected per unit time, reflecting the speed of data update; the multimodal fusion model is an intelligent algorithm model that can integrate multiple types of data and output comprehensive results.

[0092] Specifically, step S4 includes:

[0093] S41: adjusting parameter settings in the multimodal fusion model according to the determined influence weights, optimizing the image feature fusion process, and obtaining an optimized image feature fusion result.

[0094] In this embodiment, the influence weight is used to quantify the importance of the contribution of each modal data to the final synthesized image; parameter setting refers to various configuration parameters within the multimodal fusion model, such as learning rate, activation function, etc.; the image feature fusion process refers to the process of combining image features from different modalities.

[0095] Specifically, the influence weights (refer to W) of each modality image data calculated previously are collected. i ), adjust the key parameters in the multimodal fusion model according to the influence weights. For example, for the modalities with higher importance, higher weights or looser learning rates can be assigned; the optimization process includes using gradient descent or other optimization algorithms, retraining the multimodal fusion model based on the adjusted parameter settings, applying the optimized model to the test data set, and evaluating its performance indicators (such as accuracy, loss value, etc.) to ensure that the optimization effect is significant.

[0096] S42: Inputting the synchronization timestamp, sampling frequency and optimized image feature fusion result into the image feature extraction strategy to generate first feature mapping information of each modality image data.

[0097] Specifically, the first feature mapping information refers to the feature representation extracted from the original data and mapped to the new space. The first feature mapping information provides low-level and mid-level features and retains the time and frequency characteristics of the original data. The first feature mapping information includes low-level visual features (boundary information, texture features and color distribution in the image), spatial features (spatial coordinates of pixels or regions), time features (for video or time series data), frequency features (such as Fourier transform results for converting images from the spatial domain to the frequency domain and wavelet transform results for providing multi-resolution analysis) and modality-specific features (such as thermal imaging features of infrared cameras and depth information from radar sensors).

[0098] S43: Utilizing the first feature mapping information, based on the characteristic information of each modality image data and its influence weight, combined with the synchronization timestamp and the sampling frequency, the model optimization parameters are calculated.

[0099] Specifically, the model optimization parameters refer to the optimal parameter configuration used to guide the operation of the multimodal fusion model after optimization.

[0100] This embodiment calculates the model optimization parameter P based on formula (2): final :

[0101]

[0102] Where P ini are the initial multimodal fusion model parameters; μ is the learning rate, which is used to control the optimization step size; n is the number of modality image data; i is the modality type; W i is the influence weight of each modality image data on the synthetic image; TS i is the synchronization timestamp of the modality image data; TS avg is the average synchronization timestamp of all modality image data; SF i is the sampling frequency of the modality image data; SF avg is the average sampling frequency of image data of all modalities.

[0103] S44: Optimize the parameters based on the model to obtain the final multimodal fusion model.

[0104] Specifically, the final multimodal fusion model is deployed to the actual application scenario and starts processing new multimodal data.

[0105] S5: Generate target image dataset using optimized multimodal fusion model.

[0106] In this embodiment, the target image dataset refers to a new image set generated after multimodal fusion, which is intended to be used for training or testing deep learning models.

[0107] Specifically, step S5 includes:

[0108] S51: Based on the model parameter setting of the final multimodal fusion model, the preset complex scene simulation data is associated with the first feature mapping information, and according to the type of the complex scene simulation data, the output of the multimodal fusion model is adjusted to generate the second feature mapping information suitable for different scenes.

[0109] Specifically, complex scene simulation data refers to pre-set simulation data used to simulate various complex real-life scenes (such as lighting changes, occlusions, weather conditions, etc.); the second feature mapping information includes mapping information of high-level semantic features and cross-modal fusion features, including object recognition, contextual information of scene understanding, high-level semantic features of behavior recognition, fusion features (such as comprehensive feature identification), trajectory tracking, and spatiotemporal features of time detection (suitable for dynamic scene analysis).

[0110] S52: Generate a target image data set by combining the first feature mapping information and the second feature mapping information corresponding to each modality image data.

[0111] In this embodiment, the first feature mapping information and the second feature mapping information corresponding to each modal image data are integrated; ensuring that the two feature mapping information are aligned in time and space, and by combining different first and second feature mapping information, rich and diverse image samples are generated, which may include different scenes, lighting conditions, weather conditions, etc.

[0112] Specifically, the first and second feature mapping information are combined to generate a target image dataset, which provides strong support for training high-quality deep learning models.

[0113] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0114] In one embodiment, a system for generating a multimodal image dataset is provided. The system for generating a multimodal image dataset corresponds to a method for generating a multimodal image dataset in the above embodiment.

[0115] A system for generating a multimodal image dataset includes a data collection and preprocessing module, a strategy determination module, a modality association analysis module, an image feature fusion module, a model optimization module, and an image dataset generation module. The detailed description of each functional module is as follows:

[0116] Data collection and preprocessing module, used to collect multimodal image data from different sources and perform preprocessing;

[0117] A strategy determination module, used for determining an image feature extraction strategy based on preprocessed multimodal image data;

[0118] The modal association analysis module is used to analyze the correlation and interaction mode between the image data of each modality, determine the influence weight of each modality image data on the synthetic image, and establish the correlation relationship between the image data of each modality to obtain modal association information; the image feature fusion module is used to divide the multi-modality of different sources into the first sensor modality and the second sensor modality based on the modal association information; obtain the original image data of the first sensor modality and the depth image data of the second sensor modality, and input the image feature extraction strategy to obtain the image feature fusion result;

[0119] A model optimization module is used to obtain a synchronization timestamp and a sampling frequency between the first sensor modality and the second sensor modality; and input the synchronization timestamp, the sampling frequency and the image feature fusion result into a preset multimodal fusion model for model optimization;

[0120] The image dataset generation module is used to generate a target image dataset using an optimized multimodal fusion model.

[0121] For the specific definition of the system for generating a multimodal image dataset, please refer to the definition of a method for generating a multimodal image dataset in the above text, which will not be repeated here; each module in the above-mentioned system for generating a multimodal image dataset can be implemented in whole or in part by software, hardware and a combination thereof; each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0122] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0123] S1: Collect multimodal image data from different sources and perform preprocessing; determine image feature extraction strategy based on the preprocessed multimodal image data;

[0124] S2: Analyze the correlation and interaction mode between the image data of each modality, determine the influence weight of each modality image data on the synthetic image; divide the multi-modality of different sources into the first sensor modality and the second sensor modality, and establish the correlation relationship between the image data of each modality to obtain modality correlation information;

[0125] S3: acquiring the original image data of the first sensor modality and the depth image data of the second sensor modality based on the modality association information, and inputting the image feature extraction strategy to obtain the image feature fusion result;

[0126] S4: obtaining a synchronization timestamp and a sampling frequency between the first sensor modality and the second sensor modality; inputting the synchronization timestamp, the sampling frequency and the image feature fusion result into a preset multimodal fusion model for model optimization;

[0127] S5: Generate target image dataset using optimized multimodal fusion model.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0129] In one embodiment, in particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program / instruction, which implements the steps of the above-mentioned method for generating a multimodal image data set when executed by a processor. In such an embodiment, the computer program can be downloaded and installed from a network through a communication module, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), various functions defined in the present invention are performed.

[0130] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0131] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the features thereof may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating a multimodal image dataset, characterized in that: include: Collect multimodal image data from different sources and perform preprocessing; Determining an image feature extraction strategy based on preprocessed multimodal image data; Analyze the correlation and interaction patterns between the image data of each modality, and determine the influence weight of each modality image data on the synthetic image; divide the multi-modality of different sources into the first type of sensor modality and the second type of sensor modality, and establish the correlation relationship between the image data of each modality to obtain modality correlation information; Based on the modality association information, the original image data of the first sensor modality and the depth image data of the second sensor modality are acquired, and the image feature extraction strategy is input to obtain the image feature fusion result; Based on synchronized timestamps and sampling frequencies between the first type of sensor modality and the second type of sensor modality; Inputting the synchronization timestamp, sampling frequency and image feature fusion results into a preset multimodal fusion model for model optimization; Generate target image dataset using optimized multimodal fusion model; The image feature extraction strategy is determined based on the preprocessed multimodal image data, specifically including: Based on the preprocessed multimodal image data, characteristic information of each modality image data is obtained, wherein the characteristic information includes modality type, resolution, color depth, and data volume; Analyze the role of each modality image data in the composite image and determine the importance level of each modality image data; associating the graphical information of each modality image data with the importance level to generate an image feature extraction strategy applicable to the current group of multimodality image data; The obtaining of the modal association information specifically includes: Inputting data of the first sensor modality and the second sensor modality into a preliminary multimodal fusion model; Determine the dependency and complementarity between different modal data based on the characteristic information and importance level of each modality image data; select and mark key modality data based on the preset synthetic image quality standard; Calculate the influence weight of each modality image data on the synthesized image according to the key modality data and the corresponding characteristic information, and the timestamps of the first sensor modality and the second sensor modality; According to the dependency and complementarity between different modal data and the influence weight of each modal image data on the synthetic image, the association relationship between the image data of each modality is established to form modal association information; The step of inputting the synchronization timestamp, sampling frequency and image feature fusion result into a preset multimodal fusion model for model optimization specifically includes: Adjust the parameter settings in the multimodal fusion model according to the determined influence weights, optimize the image feature fusion process, and obtain an optimized image feature fusion result; Inputting the synchronization timestamp, sampling frequency and optimized image feature fusion result into an image feature extraction strategy to generate first feature mapping information of each modality image data; Utilizing the first feature mapping information, based on the characteristic information of each modality image data and its influence weight, combined with the synchronization timestamp and sampling frequency, the model optimization parameters are calculated; Based on the model optimization parameters, the final multimodal fusion model is obtained.

2. A method for generating a multimodal image dataset according to claim 1, characterized in that: The method of generating a target image dataset by using an optimized multimodal fusion model specifically includes: Based on the model parameter setting of the final multimodal fusion model, the preset complex scene simulation data is associated with the first feature mapping information, and according to the type of the complex scene simulation data, the output of the multimodal fusion model is adjusted to generate second feature mapping information adapted to different scenes; The first feature mapping information and the second feature mapping information corresponding to each modality image data are combined to generate a target image data set.

3. The method for generating a multimodal image dataset according to claim 1, characterized in that: The calculating the influence weight of each modality image data on the synthesized image specifically includes: Based on formula (1), the influence weight W of each modality image data on the synthetic image is calculated: i : W i =α×T i / T avg +β×R i / R avg +γ×F i / F avg +δ×(1-Δt i / Δt max ) Where i is the mode type; T i is the thermal sensitivity of the modality image data; T avg is the average thermal sensitivity of all modality image data; R i is the resolution of the modality image data; R avg is the average resolution of all modality image data; F i is the data size of the modality image data; F avg is the average data size of all modal image data; Δt i is the timestamp difference of the modality image data; Δt max is the maximum timestamp difference of all modality image data; α, β, γ, δ are preset weight coefficients.

4. The method for generating a multimodal image dataset according to claim 1, characterized in that: The calculation model optimization parameters specifically include: The model optimization parameter P is calculated based on formula (2): final : Where P ini are the initial multimodal fusion model parameters; μ is the learning rate, which is used to control the optimization step size; n is the number of modality image data; i is the modality type; W i is the influence weight of each modality image data on the synthetic image; TS i is the synchronization timestamp of the modality image data; TS avg is the average synchronization timestamp of all modality image data; SF i is the sampling frequency of the modality image data; SF avg is the average sampling frequency of image data of all modalities.

5. A system for generating a multimodal image dataset, characterized in that: Applied to a method for generating a multimodal image dataset as claimed in any one of claims 1 to 4, the system comprising: Data collection and preprocessing module, used to collect multimodal image data from different sources and perform preprocessing; A strategy determination module, used for determining an image feature extraction strategy based on preprocessed multimodal image data; A modality association analysis module is used to analyze the correlation and interaction mode between the image data of each modality, determine the influence weight of each modality image data on the synthetic image, and establish the association relationship between the image data of each modality to obtain modality association information; an image feature fusion module is used to classify the multi-modality of different sources into a first type of sensor modality and a second type of sensor modality based on the modality association information; obtain the original image data of the first type of sensor modality and the depth image data of the second type of sensor modality, and input the image feature extraction strategy to obtain the image feature fusion result; A model optimization module, for inputting the synchronization timestamp, sampling frequency and image feature fusion result into a preset multimodal fusion model for model optimization based on the synchronization timestamp and sampling frequency between the first type of sensor modality and the second type of sensor modality; An image data set generation module, used to generate a target image data set using an optimized multimodal fusion model; The image feature extraction strategy is determined based on the preprocessed multimodal image data, specifically including: Based on the preprocessed multimodal image data, characteristic information of each modality image data is obtained, wherein the characteristic information includes modality type, resolution, color depth, and data volume; Analyze the role of each modality image data in the composite image and determine the importance level of each modality image data; associating the graphical information of each modality image data with the importance level to generate an image feature extraction strategy applicable to the current group of multimodality image data; The obtaining of the modal association information specifically includes: Inputting data of the first sensor modality and the second sensor modality into a preliminary multimodal fusion model; Determine the dependency and complementarity between different modal data based on the characteristic information and importance level of each modality image data; select and mark key modality data based on the preset synthetic image quality standard; Calculate the influence weight of each modality image data on the synthesized image according to the key modality data and the corresponding characteristic information, and the timestamps of the first sensor modality and the second sensor modality; According to the dependency and complementarity between different modal data and the influence weight of each modal image data on the synthetic image, the association relationship between the image data of each modality is established to form modal association information; The step of inputting the synchronization timestamp, sampling frequency and image feature fusion result into a preset multimodal fusion model for model optimization specifically includes: Adjust the parameter settings in the multimodal fusion model according to the determined influence weights, optimize the image feature fusion process, and obtain an optimized image feature fusion result; Inputting the synchronization timestamp, sampling frequency and optimized image feature fusion result into an image feature extraction strategy to generate first feature mapping information of each modality image data; Utilizing the first feature mapping information, based on the characteristic information of each modality image data and its influence weight, combined with the synchronization timestamp and sampling frequency, the model optimization parameters are calculated; Based on the model optimization parameters, the final multimodal fusion model is obtained.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of a method for generating a multimodal image dataset as claimed in any one of claims 1 to 4.

7. A computer program product, characterized in that Comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of a method for generating a multimodal image dataset as described in any one of claims 1 to 4 are implemented.

Images