A multimodal data enhancement method based on dynamic semantic fusion

Abstract

The present invention discloses a multimodal data enhancement method based on dynamic semantic fusion, comprising the following steps: S1, multimodal data collection and preprocessing; S2, dynamic background, attribute and action generation; S3, semantic fusion and enhancement; S4, multi-view and multi-scene synthesis; S5, self-supervised training and feedback mechanism; S6, dynamic adjustment and optimization; through innovative dynamic background generation, attribute and action adjustment, semantic fusion and multi-view scene synthesis technology, combined with self-supervised training and feedback mechanism, it is not only possible to generate real, diverse and semantically consistent data samples, providing a more comprehensive training foundation for deep learning models, but also to adapt to different training needs and application scenarios by dynamically adjusting data enhancement strategies.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a multimodal data enhancement method based on dynamic semantic fusion. Background Art

[0002] In the field of deep learning and artificial intelligence, the quality and diversity of data directly affect the performance and generalization ability of the model, especially in areas such as visual recognition, natural language processing, and 3D model processing.

[0003] At present, multimodal data processing technology based on deep learning is often used, which comprehensively utilizes different types of data such as images, texts, videos and 3D models, and improves the effect of data processing and model training by integrating complementary information. However, this method has the following shortcomings: 1) The authenticity and diversity of the generated data are insufficient, which makes it difficult to meet the needs of complex application scenarios; 2) There is a lack of effective mechanism to dynamically adjust the data generation strategy to adapt to changes in the model training process; 3) There are still deficiencies in the integration and semantic fusion of multimodal data, especially how to ensure the semantic consistency and complementarity between different modal data. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal data enhancement method based on dynamic semantic fusion, which can not only generate real, diverse and semantically consistent data samples, but also dynamically adjust the data enhancement strategy to adapt to different training needs and application scenarios.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] A multimodal data enhancement method based on dynamic semantic fusion includes the following steps:

[0007] S1. Multimodal data collection and preprocessing: Collect multimodal data of objects from various data sources of objects, and extract key information and features through preprocessing technology;

[0008] The multimodal objects in step S1 include images, videos, 3D models, and texts. Data is collected from different sources to extract useful information and features, providing a wide range of raw data for all subsequent operations. The preprocessing in step S1 includes semantic segmentation of images, semantic segmentation of videos, simplification of 3D models, and semantic analysis of texts. Through preprocessing technology, the multimodal data is standardized and simplified, making subsequent operations more efficient and more targeted, providing a good foundation for building a powerful deep learning model.

[0009] S2, dynamic background, attribute and action generation: based on the data obtained after preprocessing in step S1, dynamically generate a background for the object according to given environmental conditions and dynamic factors, and adjust the attributes and actions of the object to further increase the diversity and authenticity of the data;

[0010] S3, semantic fusion and enhancement: placing the preprocessed object in step S1 into the background generated in step S2 to obtain a fused image, and adjusting and fusion according to the attributes or actions of the object and the characteristics of the background to ensure natural matching and visual consistency between the object and the background;

[0011] S4, multi-view and multi-scene synthesis: performing multi-view rendering and multi-scene synthesis on the fused image in step S3 to obtain diversified data, thereby improving the adaptability and understanding ability of the deep learning model to scene changes;

[0012] S5, self-supervised training and feedback mechanism: Based on the data set generated in steps S1 to S4, the deep learning model of the object is trained, and the training strategy and parameters are dynamically adjusted according to the performance of the model in combination with the feedback mechanism, so as to optimize the learning process and final performance of the model;

[0013] S6, dynamic adjustment and optimization: According to the model performance and the requirements of specific tasks, the dynamic background, attribute and action generation in step S2 and the semantic fusion and enhancement in step S3 are continuously optimized to improve the generalization ability and performance of the model;

[0014] From the precise collection and preprocessing of multimodal data to complex semantic fusion, from detailed multi-view scene construction to intelligent self-supervised training and feedback adjustment, each step S1 to S6 enhances the deep learning model's ability to understand and adapt to the complexity of the real world, thereby creating a highly flexible and dynamic data generation framework, which not only improves the model's performance on specific tasks, but also provides a rich, multi-dimensional training environment for deep learning models, ultimately realizing a more intelligent and adaptive artificial intelligence system.

[0015] Preferably, the formula for semantic segmentation of the image in step S1 is:

[0016] S(I)=sum from n=1to N of(w_n*f_n(I))

[0017] Where I represents the image,

[0018] n represents the nth feature,

[0019] N represents the total number of features,

[0020] f_n(I) represents the extraction function of the nth feature,

[0021] w_n represents the weight corresponding to each function.

[0022] For a given image I, N different feature extraction functions f_n(I) are applied, each function has a corresponding weight w_n. Through this weighted sum method, the information of multiple features is fused to achieve more accurate semantic segmentation.

[0023] Preferably, the formula for semantic segmentation of the video in step S1 is:

[0024] S(V)=sum from t=1to T of sum from n=1to N of(w_n,t*f_n,t(V))

[0025] Among them, V represents video,

[0026] n represents the nth feature,

[0027] N represents the feature,

[0028] t represents the time point,

[0029] f_n,t(V) represents the nth feature extraction function applied at time point t,

[0030] w_n,t represents the weight corresponding to each function.

[0031] Video data includes not only spatial information but also temporal dimension. f_n,t(V) represents the nth feature extraction function applied at time point t, and w_n,t is the corresponding weight. Through this weighted sum method, the dynamic changes between video frames are taken into account, so as to better perform semantic segmentation.

[0032] Preferably, the formula for simplifying the 3D model in step S1 is:

[0033] P(M)=minimize over M'of(L(M,M')+lambda*R(M'))

[0034] M represents the original model,

[0035] M' represents a simplified model,

[0036] L(M,M') represents the difference between the original model M and the simplified model M'.

[0037] R(M') represents the model complexity, which is adjusted by the regularization parameter lambda.

[0038] 3D model simplification is an optimization process whose goal is to find a simplified model M' so that the weighted sum of the difference L(M,M') between the original model M and the simplified model M' plus the complexity R(M') of the simplified model (adjusted by the regularization parameter lambda) is minimized, thereby reducing the complexity of the model while retaining the main features and appearance of the 3D model.

[0039] Preferably, the formula for semantic analysis of the text in step S1 is:

[0040] A(T)=union from k=1to K of(phi_k(T))

[0041] T stands for text,

[0042] K represents the total number of information types that need to be extracted.

[0043] phi_k(T) represents the extraction function.

[0044] Through a series of functions phi_k(T), key attributes or action information in the text T are extracted, multi-dimensional semantic information is obtained from the text, descriptive content is generated, and it is associated with other modal data (such as images and videos), so as to facilitate subsequent work.

[0045] Preferably, the formula for generating the dynamic background in step S2 is:

[0046] G_b(C,D)=beta*g(C)+delta*h(D)

[0047] Where C represents the environmental conditions,

[0048] g(C) represents the function that generates the background according to the environmental condition C,

[0049] D represents the dynamic factor,

[0050] h(D) represents the function that adjusts the background according to the dynamic factor D.

[0051] beta and delta represent the corresponding weight parameters, which are used to adjust the influence of environmental conditions and dynamic factors on the background generation results;

[0052] The above formula reflects that background generation is a process that takes into account the influence of multiple factors. By adjusting the weight parameters, the contribution of different factors to the background generation results can be controlled.

[0053] The formula for generating attributes and actions in step S2 is:

[0054] G_p(K)=sum from k=1to K of(alpha_k*psi_k(K))

[0055] k represents the kth attribute or action,

[0056] K represents the total number of attributes or actions that need to be adjusted or generated.

[0057] psi_k(K) represents the function that adjusts or generates the kth attribute or action

[0058] alpha_k is the corresponding weight parameter.

[0059] The above formula represents the importance of each attribute or action adjustment function. Through this weighted summation method, the adjustment results of multiple attributes or actions are combined to generate objects with rich attributes and dynamic behaviors.

[0060] Preferably, the formula for semantic fusion and enhancement in step S3 is:

[0061] F(P,B)=integral of(p(P)*q(B))over P and B

[0062] in,

[0063] P represents an object,

[0064] p(P) represents the probability distribution function of the object attribute P. This distribution defines the influence weight of different attributes on how the object blends with the background.

[0065] B stands for background,

[0066] q(B) represents the probability distribution function of the background B. This distribution takes into account the contribution of different background features to the fidelity of object fusion.

[0067] The above formula seeks, through integral operations, those fusion modes that make the interaction between the object and the background most natural among all possible combinations of object attributes P and background B. It is a function that measures the naturalness and realism of the fusion.

[0068] Through this complex fusion enhancement process, it is possible to create scenes that are both consistent with visual perception and physically feasible, providing a richer and more realistic training environment for deep learning models. This improves the performance of the model on specific tasks, as well as enhances the model's adaptability and generalization capabilities to complex scenes in the real world.

[0069] Preferably, the formula for multi-view and multi-scene synthesis in step S4 is:

[0070] V(I',theta,phi)=gamma*rot(I',theta,phi)

[0071] in:

[0072] I' represents the image after semantic fusion and enhancement, that is, the image that integrates object attributes and dynamic background.

[0073] Theta pitch angle defines the vertical offset of the viewing angle.

[0074] phi: yaw angle, defines the horizontal deviation of the viewing angle.

[0075] rot(I',theta,phi) represents the rotation operation of image I' according to the given pitch angle and yaw angle, simulating the observation effect of the scene from different spatial positions.

[0076] Gamma is a scaling factor used to resize images to fit specific visual needs or to match the specifications of a model input.

[0077] The above formula describes how to generate images of objects or scenes observed from different perspectives through rotation and scaling operations.

[0078] Preferably, the formula of the self-supervised training and feedback mechanism in step S5 is:

[0079] Adjust(model_performance)=sum from i=1to I of(eta_i*adjust_i(model_performance,Lambda_i))

[0080] in:

[0081] i represents the i-th adjustment operation,

[0082] I represents the total number of adjustment operations,

[0083] Model_performance represents the current performance index of the model, which can be loss function value, accuracy or other related indicators.

[0084] adjust_i is an adjustment function that calculates how to adjust the model's training strategy or parameters based on the model's performance and a set of parameters Lambda_i.

[0085] eta_i is the weight of the i-th adjustment operation, indicating the relative importance of different adjustment operations in the overall adjustment strategy.

[0086] Lambda_i is the set of parameters passed to the i-th adjustment function, which controls the specific behavior and direction of the adjustment operation.

[0087] By implementing self-supervised training and feedback mechanisms, the model training process becomes more dynamic and adaptive. The above formula allows the model to automatically adjust its training strategy according to the current learning status and performance, such as changing the intensity of data enhancement, adjusting the learning rate, and selecting more effective samples for training. This can not only speed up the convergence of the model, but also improve the accuracy and robustness of the final model, especially in the face of complex and changing data environments.

[0088] Preferably, the formula for dynamic adjustment and optimization in step S6 is:

[0089] Optimize(D_current)=D_current+xi*Delta D

[0090] in:

[0091] xi represents the learning rate, which controls the magnitude of the adjustment step, that is, the degree to which the data is changed in one adjustment.

[0092] D_current represents the currently used data set.

[0093] Delta D represents the direction of data set adjustment based on model performance evaluation, including adding new data, modifying existing data, or removing data that is not conducive to model training.

[0094] The above formula indicates that the new data set is obtained by updating the current data set according to the adjustment direction and step size.

[0095] Beneficial Effects

[0096] The beneficial effects of the present invention are as follows:

[0097] 1) The present invention can generate more realistic and diverse training data through dynamic background generation, attribute adjustment and semantic fusion technology, thereby effectively improving the generalization ability of the model;

[0098] 2) This paper proposes a method based on dynamic semantic fusion, which can not only process and integrate multimodal data from different sources, but also ensure semantic consistency in the data fusion process, providing richer and more consistent training data for deep learning models;

[0099] 3) The present invention realizes dynamic adjustment and optimization of data enhancement strategy by introducing self-supervised training and feedback mechanism. According to the real-time performance and training requirements of the model, the parameters and strategies of data generation are automatically adjusted to ensure that the training data can always most effectively support model learning and improve the efficiency and effect of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1It is a flow chart of the present invention. DETAILED DESCRIPTION

[0101] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0102] Embodiment 1

[0103] A multimodal data enhancement method based on dynamic semantic fusion includes the following steps:

[0104] S1. Multimodal data collection and preprocessing: Collect multimodal data of objects from various data sources of objects, and extract key information and features through preprocessing technology;

[0105] S2, dynamic background, attribute and action generation: based on the data obtained after preprocessing in step S1, dynamically generate a background for the object according to given environmental conditions and dynamic factors, and adjust the attributes and actions of the object;

[0106] S3, semantic fusion and enhancement: placing the preprocessed object in step S1 into the background generated in step S2 to obtain a fused image, and adjusting and fusion according to the attributes or actions of the object and the characteristics of the background;

[0107] S4, multi-view and multi-scene synthesis: performing multi-view rendering and multi-scene synthesis on the fused image in step S3 to obtain diversified data;

[0108] S5, self-supervised training and feedback mechanism: train the deep learning model of the object according to the data set generated in steps S1 to S4, and dynamically adjust the training strategy and parameters according to the performance of the model in combination with the feedback mechanism;

[0109] S6, dynamic adjustment and optimization: According to the model performance and the requirements of specific tasks, the dynamic background, attribute and action generation in step S2 and the semantic fusion and enhancement in step S3 are continuously optimized to improve the generalization ability and performance of the model;

[0110] Wherein, the multimodal object in step S1 includes images, videos, 3D models, and texts;

[0111] The preprocessing in step S1 includes semantic segmentation of images, semantic segmentation of videos, simplification of 3D models, and semantic analysis of texts.

[0112] The formula for semantic segmentation of the image in step S1 is:

[0113] S(I)=sum from n=1to N of(w_n*f_n(I))

[0114] Where I represents the image,

[0115] n represents the nth feature,

[0116] N represents the total number of features,

[0117] f_n(I) represents the extraction function of the nth feature,

[0118] w_n represents the weight corresponding to each function.

[0119] The formula for semantic segmentation of the video in step S1 is:

[0120] S(V)=sum from t=1to T of sum from n=1to N of(w_n,t*f_n,t(V))

[0121] Among them, V represents video,

[0122] n represents the nth feature,

[0123] N represents the feature,

[0124] t represents the time point,

[0125] f_n,t(V) represents the nth feature extraction function applied at time point t,

[0126] w_n,t represents the weight corresponding to each function.

[0127] The formula for simplifying the 3D model in step S1 is:

[0128] P(M)=minimize over M'of(L(M,M')+lambda*R(M'))

[0129] M represents the original model,

[0130] M' represents a simplified model,

[0131] L(M,M') represents the difference between the original model M and the simplified model M'.

[0132] R(M') represents the model complexity, which is adjusted by the regularization parameter lambda.

[0133] The formula for semantic analysis of text in step S1 is:

[0134] A(T)=union from k=1to K of(phi_k(T))

[0135] T stands for text,

[0136] K represents the total number of information types that need to be extracted.

[0137] phi_k(T) represents the extraction function.

[0138] The formula for generating the dynamic background in step S2 is:

[0139] G_b(C,D)=beta*g(C)+delta*h(D)

[0140] Where C represents the environmental conditions,

[0141] g(C) represents the function that generates the background according to the environmental condition C,

[0142] D represents the dynamic factor,

[0143] h(D) represents the function that adjusts the background according to the dynamic factor D.

[0144] Beta and delta represent the corresponding weight parameters, which are used to adjust the influence of environmental conditions and dynamic factors on the background generation results.

[0145] The formula for generating attributes and actions in step S2 is:

[0146] G_p(K)=sum from k=1to K of(alpha_k*psi_k(K))

[0147] k represents the kth attribute or action,

[0148] K represents the total number of attributes or actions that need to be adjusted or generated.

[0149] psi_k(K) represents the function that adjusts or generates the kth attribute or action

[0150] alpha_k is the corresponding weight parameter.

[0151] The formula for semantic fusion and enhancement in step S3 is:

[0152] F(P,B)=integral of(p(P)*q(B))over P and B

[0153] in,

[0154] P represents an object,

[0155] p(P) represents the probability distribution function of the object attribute P. This distribution defines the influence weight of different attributes on how the object blends with the background.

[0156] B stands for background,

[0157] q(B) represents the probability distribution function of the background B.

[0158] The formula for multi-view and multi-scene synthesis in step S4 is:

[0159] V(I',theta,phi)=gamma*rot(I',theta,phi)

[0160] in:

[0161] I' represents the image after semantic fusion and enhancement, that is, the image that integrates object attributes and dynamic background.

[0162] Theta pitch angle defines the vertical offset of the viewing angle.

[0163] phi: yaw angle, defines the horizontal deviation of the viewing angle.

[0164] rot(I',theta,phi) represents the rotation operation of image I' according to the given pitch angle and yaw angle.

[0165] Gamma is the scaling factor.

[0166] The formula of the self-supervised training and feedback mechanism in step S5 is:

[0167] Adjust(model_performance)=sum from i=1to I of(eta_i*adjust_i(model_performance,Lambda_i))

[0168] in:

[0169] i represents the i-th adjustment operation,

[0170] I represents the total number of adjustment operations,

[0171] model_performance represents the current performance index of the model.

[0172] adjust_i is an adjustment function.

[0173] eta_i is the weight of the ith adjustment operation,

[0174] Lambda_i is the set of parameters passed to the i-th adjustment function.

[0175] The formula for dynamic adjustment and optimization in step S6 is:

[0176] Optimize(D_current)=D_current+xi*Delta D

[0177] in:

[0178] xi represents the learning rate,

[0179] D_current represents the currently used data set.

[0180] Delta D represents the direction of dataset adjustment based on model performance evaluation.

[0181] Embodiment 2

[0182] Embodiment 2 is a practical application example based on embodiment 1.

[0183] A deep learning model for an autonomous driving system needs to be able to accurately identify and understand urban traffic scenes under different weather and lighting conditions, including vehicles, pedestrians, traffic signs, etc. Due to the complexity and variability of the real world, data augmentation methods are required to generate sufficiently diverse and realistic training data to improve the generalization ability of the model.

[0184] The specific steps are as follows:

[0185] S1. Multimodal data collection and preprocessing: Collect a large number of images and videos of urban traffic scenes, including data at different times (daytime, nighttime) and different weather conditions (sunny, rainy, foggy), and then use preprocessing technology to extract information and features of related objects such as vehicles and pedestrians; construct 3D models of related objects such as vehicles and pedestrians, and simplify them through preprocessing; collect relevant text descriptions, such as traffic rules, typical behavior descriptions of vehicles and pedestrians, and perform preprocessing;

[0186] S2, Dynamic Background, Attribute and Action Generation: Use conditional generation technology to dynamically generate background images based on actual weather and lighting conditions; adjust object attributes, such as the color of vehicles and pedestrians’ clothing; simulate the actions of vehicles driving and pedestrians walking, as well as simulate raindrops, fog and other effects;

[0187] S3, semantic fusion and enhancement: Fuse the pre-processed objects (vehicles, pedestrians) with the dynamically generated background image to ensure the natural placement and visual consistency of the objects in the new background; and use text information to further enhance the semantic information of the scene, such as adding the correct location and meaning of traffic signs through text descriptions;

[0188] S4, multi-view and multi-scene synthesis: Perform multi-view rendering on the fused image to simulate observing the scene from different driving perspectives and heights; and place the same object in a variety of different backgrounds to generate scene images under various environmental conditions;

[0189] S5, self-supervised training and feedback mechanism: Use the data generated in steps S1 to S4 to train the autonomous driving model, monitor the performance of the model on specific tasks (such as object recognition and scene understanding), and adjust the data generation strategy based on the performance feedback of the model, such as adjusting the conditions for background generation and optimizing the adjustment range of object attributes;

[0190] 6. Dynamic adjustment and optimization: Continuously evaluate the performance of the model in various complex scenarios, and dynamically optimize the data enhancement strategy, such as introducing more dynamic elements and scene changes, exploring new data sources and enhancement technologies, and not expanding and optimizing the training data set.

[0191] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0192] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal data enhancement method based on dynamic semantic fusion, characterized in that: The following steps are involved: S1. Multimodal data collection and preprocessing: Collect multimodal data of objects from various data sources of objects, and extract key information and features through preprocessing technology; S2, dynamic background, attribute and action generation: based on the data obtained after preprocessing in step S1, dynamically generate a background for the object according to given environmental conditions and dynamic factors, and adjust the attributes and actions of the object; S3, semantic fusion and enhancement: placing the object preprocessed in step S1 into the background generated in step S2 to obtain a fused image, and adjusting and fusion according to the attributes or actions of the object and the characteristics of the background; S4, multi-view and multi-scene synthesis: performing multi-view rendering and multi-scene synthesis on the fused image in step S3 to obtain diversified data; S5, self-supervised training and feedback mechanism: train the deep learning model of the object according to the data set generated in steps S1 to S4, and dynamically adjust the training strategy and parameters according to the performance of the model in combination with the feedback mechanism; S6, dynamic adjustment and optimization: according to the model performance and the requirements of specific tasks, the dynamic background, attribute and action generation in step S2 and the semantic fusion and enhancement in step S3 are continuously optimized; Among them, the multimodality of the object in step S1 includes images, videos, 3D models, and texts; the preprocessing in step S1 includes semantic segmentation of images, semantic segmentation of videos, simplification of 3D models, and semantic analysis of texts.

2. The multimodal data enhancement method based on dynamic semantic fusion according to claim 1, characterized in that: The formula for semantic segmentation of the image in step S1 is: S(I)=sum from n=1to N of(w_n*f_n(I)) Where I represents the image, n represents the nth feature, N represents the total number of features, f_n(I) represents the extraction function of the nth feature, w_n represents the weight corresponding to each function.

3. The multimodal data enhancement method based on dynamic semantic fusion according to claim 1, characterized in that: The formula for semantic segmentation of the video in step S1 is: S(V)=sum from t=1to T of sum from n=1to N of(w_n,t*f_n,t(V))where, V stands for video, n represents the nth feature, N represents the feature, t represents the time point, f_n,t(V) represents the nth feature extraction function applied at time point t, w_n,t represents the weight corresponding to each function.

4. The multimodal data enhancement method based on dynamic semantic fusion according to claim 1, characterized in that: The formula for simplifying the 3D model in step S1 is: P(M)=minimize over M'of(L(M,M')+lambda*R(M')) M represents the original model, M' represents a simplified model, L(M,M') represents the difference between the original model M and the simplified model M'. R(M') represents the model complexity, which is adjusted by the regularization parameter lambda.

5. The multimodal data enhancement method based on dynamic semantic fusion according to claim 1, characterized in that: The formula for semantic analysis of text in step S1 is: A(T)=union from k=1to K of(phi_k(T)) T stands for text, K represents the total number of information types that need to be extracted. phi_k(T) represents the extraction function.

6. The multimodal data enhancement method based on dynamic semantic fusion according to claim 1, characterized in that: The formula for generating the dynamic background in step S2 is: G_b(C,D)=beta*g(C)+delta*h(D) Where C represents the environmental conditions, g(C) represents the function that generates the background according to the environmental condition C, D represents the dynamic factor, h(D) represents the function that adjusts the background according to the dynamic factor D. beta and delta represent the corresponding weight parameters, which are used to adjust the influence of environmental conditions and dynamic factors on the background generation results; The formula for generating attributes and actions in step S2 is: G_p(K)=sum from k=1to K of(alpha_k*psi_k(K)) k represents the kth attribute or action, K represents the total number of attributes or actions that need to be adjusted or generated. psi_k(K) represents the function that adjusts or generates the kth attribute or action alpha_k is the corresponding weight parameter.

7. The multimodal data enhancement method based on dynamic semantic fusion according to claim 1, characterized in that: The formula for semantic fusion and enhancement in step S3 is: F(P,B)=integral of(p(P)*q(B))over P and B in, P represents an object, p(P) represents the probability distribution function of the object attribute P, B stands for background, q(B) represents the probability distribution function of the background B.

8. The multimodal data enhancement method based on dynamic semantic fusion according to claim 1, characterized in that: The formula for multi-view and multi-scene synthesis in step S4 is: V(I',theta,phi)=gamma*rot(I',theta,phi) in: I' represents the image after semantic fusion and enhancement, theta and phi represent the pitch angle and yaw angle respectively, rot(I',theta,phi) represents the rotation operation of image I' according to the given pitch angle and yaw angle. Gamma is the scaling factor.

9. The multimodal data enhancement method based on dynamic semantic fusion according to claim 1, characterized in that: The formula of the self-supervised training and feedback mechanism in step S5 is: Adjust(model_performance)=sum from i=1 to I of(eta_i*adjust_i(model_performance,Lambda_i)) in: i represents the i-th adjustment operation, I represents the total number of adjustment operations, model_performance represents the current performance index of the model. adjust_i is an adjustment function. eta_i is the weight of the ith adjustment operation, Lambda_i is the set of parameters passed to the i-th adjustment function.

10. The multimodal data enhancement method based on dynamic semantic fusion according to claim 1, characterized in that: The formula for dynamic adjustment and optimization in step S6 is: Optimize(D_current)=D_current+xi*Delta D in: xi represents the learning rate, D_current indicates the currently used data set; Delta D represents the direction of dataset adjustment based on model performance evaluation.

Images