Semantic analysis model construction method and system based on multiple modes

Abstract

The invention relates to the technical field of semantic analysis, in particular to a multi-modal-based semantic analysis model construction method and system, and the method specifically comprises the steps: constructing cross-modal semantic similarity through analyzing the global semantic similarity condition and potential semantic impulse probability between each text data and corresponding image data; all text and image feature vectors are clustered, the clustering quality is evaluated according to contour coefficient distribution, and the semantic coverage completeness is constructed in combination with the feature vector direction consistency degree and the cross-modal semantic similarity so as to adjust temperature parameters in contrast learning, so that when incremental training is carried out, the accuracy of the temperature parameters is improved, and the accuracy of the temperature parameters is improved. The model can dynamically select difficult negative samples according to a whole test set, the representation learning ability of the negative samples is enhanced, and overfitting is prevented, so that the whole feature space distribution is more reasonable and ordered, the robustness of the multi-modal semantic analysis model in a cross-modal semantic alignment task is improved, and the accuracy of the cross-modal semantic alignment task is improved. And training and optimization of the multi-modal semantic analysis model are realized.

Description

Technical Field

[0001] This application relates to the field of semantic analysis technology, specifically to a method and system for constructing a multimodal semantic analysis model. Background Technology

[0002] As internet information evolves from plain text to multimedia formats such as images, audio, and video, many multimodal semantic analysis models have emerged to overcome the limitations of traditional unimodal semantic analysis models in understanding and expression. These models aim to simulate human cross-modal cognitive abilities and perform cross-modal semantic representation and analysis. Previously, most models focused on feature-level fusion, but deep learning-based models have evolved from simple alignment to deep-level interaction. For example, models incorporating Transformers and large-scale pre-training techniques (such as CLIP and GPT-4V) force representations from different modalities into a single space, achieving unified representation and alignment of cross-modal semantics and significantly improving the ability of multimodal information to characterize complex real-world problems.

[0003] The demonstration and analysis of the meta-properties of complex formal systems is currently the greatest challenge and difficulty. This challenge manifests as a verification dilemma regarding the "consistency" and "completeness" of models when constructing multimodal semantic analysis models. The root cause lies in the fact that semantic computation models of complex formal systems built using the fiber bundle method require corresponding consistency testing algorithms and completeness verification tools to support their transformation into semantic consistency and relative completeness verification. However, models built using the fiber bundle method are prone to internal contradictions when conducting reasoning and demonstration of the same semantic object across different modalities; that is, they lack corresponding supporting conditions and cannot ensure the completeness of the model's semantic coverage. In existing technologies, contrastive learning, by constructing positive and negative sample pairs to bring similar samples closer together and push away dissimilar samples, can improve the structure of the feature space to some extent. However, this method typically uses a fixed temperature coefficient, making it difficult for the model to adaptively adjust its attention to difficult samples according to the completeness state of the current semantic distribution. This results in either insufficient attention to difficult samples, failing to effectively distinguish fine-grained semantics, or excessive attention introducing noise and leading to overfitting. Summary of the Invention

[0004] To address the aforementioned technical problems, the purpose of this application is to provide a method and system for constructing a multimodal semantic analysis model, and the specific technical solution adopted is as follows: In a first aspect, embodiments of this application provide a method for constructing a multimodal semantic analysis model, the method comprising the following steps: Collect image data and its corresponding descriptive text within each cycle; The text feature vectors of each descriptive text and the image feature vectors of each image data are obtained through a semantic alignment model; the cross-modal semantic similarity of each period is constructed based on the global semantic similarity between each text feature vector and the corresponding image feature vector and the potential semantic collision probability. Cluster all text feature vectors and all image feature vectors for each period, and construct the semantic effective proportion of each period based on the distribution characteristics of the contour coefficients of all clusters; construct the semantic coverage completeness of each period based on the directional consistency between each text feature vector and the corresponding image feature vector, combined with the cross-modal semantic similarity and the semantic effective proportion. The temperature parameters in the CLIP model are adjusted based on semantic coverage completeness. Based on the image data of each period and its corresponding descriptive text, combined with the adjusted temperature parameters, the CLIP model is incrementally trained to optimize the semantic analysis model.

[0005] In one embodiment, the process of obtaining the cross-modal semantic similarity is as follows: Each image data and its corresponding descriptive text are treated as a text-image pair. The similarity score between each text feature vector and the corresponding image feature vector is calculated using a semantic alignment model. By using the text feature vectors and their corresponding image feature vectors as inputs to the logistic regression algorithm, the probability of contradiction for each text-image pair is obtained. The cross-modal semantic similarity of each period is determined based on the similarity score and the contradiction probability. The cross-modal semantic similarity is positively correlated with the similarity score and negatively correlated with the contradiction probability.

[0006] In one embodiment, the expression for the cross-modal semantic similarity is: In the formula, A is the cross-modal semantic similarity score of the current period, S is the mean of the cross-modal semantic similarity scores of all text-image pairs in the current period, and P is the mean of the contradiction probability of all text-image pairs in the current period. It is an exponential function with the natural constant e as the base. It is a preset minimum number greater than 1.

[0007] In one embodiment, the semantic effective proportion is the proportion of clusters whose mean contour coefficient is greater than a preset contour coefficient threshold.

[0008] In one embodiment, the process of obtaining the semantic coverage completeness is as follows: Calculate the cosine similarity between the text feature vector and the image feature vector corresponding to each text-image pair; determine the semantic coverage completeness of each period based on the effective semantic proportion, the cosine similarity, and the cross-modal semantic similarity. The semantic coverage completeness is positively correlated with the effective semantic proportion and the cross-modal semantic similarity, and negatively correlated with the cosine similarity.

[0009] In one embodiment, the expression for semantic coverage completeness is: In the formula, is the semantic coverage completeness of the current period; Scov is the effective proportion of the semantics in the current period; Sdiv is the mean of the cosine similarity of all text-image pairs in the current period; A is the cross-modal semantic similarity of the current period. It is an exponential function with the natural constant e as the base.

[0010] In one embodiment, adjusting the temperature parameters in the CLIP model based on semantic coverage completeness specifically involves: The semantic coverage completeness of each period is mapped to a positive number, and the fusion value of the positive number and the preset initial temperature parameter is used as the adjusted temperature parameter for each period.

[0011] In one embodiment, the expression for the adjusted temperature parameter is: In the formula, This refers to the adjusted temperature parameters for the current cycle. This represents the preset initial temperature parameter; Represents the adjustment coefficient; Represents the semantic coverage completeness of the current cycle; It is an exponential function with the natural constant e as the base.

[0012] In one embodiment, the incremental training of the CLIP model based on image data from each period and its corresponding descriptive text, combined with the corresponding adjusted temperature parameters, specifically involves: Obtain each positive text-image pair and each negative text-image pair for each period; wherein, each negative text-image pair is a text-image pair composed of one text data and one image data randomly selected from all the described text and image data; use all positive and negative sample pairs of the current period as the semantic alignment model trained in the previous period, and set the temperature parameter in the model as the adjusted temperature parameter of the current period, and perform incremental training on the model.

[0013] Secondly, embodiments of this application also provide a multimodal semantic analysis model construction system, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described above.

[0014] The embodiments of this application have at least the following beneficial effects: This application constructs cross-modal semantic similarity by analyzing the global semantic similarity between text data and corresponding image data, and by constructing cross-modal semantic similarity based on the latent semantic impulse probability of a discriminative model. This allows for a more detailed capture of the correspondence between the two modalities, overcoming the fine-grained errors that may arise from relying solely on overall similarity, and ensuring the consistency and reliability of the model's semantics. Clustering all text and image feature vectors and evaluating clustering quality based on silhouette coefficient distribution quantifies the breadth of feature space coverage. Combining the consistency of feature vector directions and cross-modal semantic similarity, a semantic coverage completeness index is constructed, which can uniformly measure the overall semantic performance of the current model within a given period. This index provides an objective basis for subsequent temperature parameter adjustment. The availability index adjusts the temperature parameter in contrastive learning, enabling the model to dynamically select difficult negative samples based on the entire test set during incremental training. This strengthens the model's representation learning ability for difficult negative samples, prevents overfitting, and makes the entire feature space distribution more reasonable and orderly. This achieves a joint improvement in feature semantic consistency and completeness. Through the aforementioned adaptive temperature parameter adjustment mechanism, contrastive learning can dynamically optimize the feature space distribution of the multimodal semantic analysis model in different training cycles, enhance the model's ability to learn from semantically similar but different-category negative samples, and thus improve the robustness of the multimodal semantic analysis model in cross-modal semantic alignment tasks, thereby realizing the training and optimization of the multimodal semantic analysis model. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating the steps of a multimodal semantic analysis model construction method provided in one embodiment of this application; Figure 2 This is a schematic diagram illustrating the process of obtaining cross-modal semantic similarity. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by this application to achieve the intended inventive purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the multimodal semantic analysis model construction method and system proposed in this application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0019] The following description, in conjunction with the accompanying drawings, details the specific scheme of the multimodal semantic analysis model construction method and system provided in this application.

[0020] Please see Figure 1 The diagram illustrates a flowchart of a method for constructing a multimodal semantic analysis model according to an embodiment of this application. The method includes the following steps: Step S1: Collect image data and its corresponding descriptive text in each cycle.

[0021] To accomplish the task of multimodal semantic analysis, this application obtains raw text data and image data from publicly available multimodal annotation datasets, specifically the COCO image dataset. The COCO image dataset itself provides text descriptions of images, treating each image data and its corresponding descriptive text as a positive text-image pair to ensure semantic relevance between the image and text data. A semantically mismatched text-image pair, randomly selected from all text and image data, is treated as a negative text-image pair, and a negative sample pool is constructed.

[0022] Based on periodic, timed data collection, the duration of a single period in this embodiment is one day, with a daily collection quantity of no less than 1000 text-image pairs to support incremental updates and timeliness verification. In this embodiment, the daily collection quantity is 1200 text-image pairs. Specifically, these 1200 text-image pairs include 600 semantically relevant positive text-image pairs directly collected from the COCO image dataset, and 600 negative text-image pairs randomly selected from the negative sample pool.

[0023] During data preprocessing, text data undergoes cleaning, word segmentation, and vectorization, and is uniformly truncated to 128 characters; image data undergoes scaling, normalization, and data augmentation. The cleaning, word segmentation, and vectorization of text data, as well as the scaling, normalization, and data augmentation of image data, are well-known processes and will not be elaborated upon here. This results in preprocessed text-image pairs.

[0024] Step S2: Obtain the text feature vectors of each descriptive text and the image feature vectors of each image data through the semantic alignment model; construct the cross-modal semantic similarity for each period based on the potential semantic collision probability of the global semantic similarity between each text feature vector and the corresponding image feature vector within each period.

[0025] Because multimodal semantic analysis models have different semantic representations of data in different modalities (e.g., text and images), semantic inconsistency will occur when the model performs cross-modal understanding. That is, the same concept has different representations in different modalities, which may lead to incorrect inference conclusions of the model, causing the model's inference results to contradict each other, reducing the model's credibility and application value.

[0026] Therefore, using preprocessed text and image data from each cycle as input, a cross-modal semantic alignment algorithm based on the CLIP model is used. The visual encoder is set to ViT-B / 32, the text encoder is set to Transformer, and the initial temperature parameter is set to 0.01 to control the sharpness of the similarity distribution. This results in the output of text feature vectors for each text data, image feature vectors for each image data, and cross-modal semantic similarity scores between the text and image feature vectors for each text-image pair. The output text and image feature vectors have the same length, and their numerical values ​​indicate the degree to which the output describes the unified semantic space constructed by CLIP for different modal data. The larger the value, the more consistent the understanding of the same semantics between the two modalities, indicating that the model can effectively complete the transformation of data from different modalities to the unified semantic representation space.

[0027] However, since global semantic similarity alone cannot completely solve the semantic contradiction problem, such as the phenomenon that the content of some images contradicts the keywords in the corresponding text, this paper takes the text feature vector and image feature vector corresponding to each text-image pair in each period as input, and a pre-trained logistic regression algorithm as the classifier. The regularization coefficient is set to 1.0 to maintain a certain generalization ability of the algorithm model. The liblinear optimizer is used to process high-dimensional features. The output of the linear decision function is mapped to a value of [0,1] through the sigmoid function in the logistic regression algorithm. This output value is recorded as the contradiction probability between the text feature vector and the image feature vector corresponding to the text-image pair. This output represents the possibility of mismatch between the overall style or implicit semantics of the text and image based on the statistical regularity of the data. The larger the value, the greater the possibility of semantic contradiction in the sample in the model, reflecting the contradiction that may exist in the fine-grained semantic understanding of the model.

[0028] Based on the above analysis, the cross-modal semantic similarity for each period is constructed. Taking any period as the current period, and using the current period as an example, preferably in this embodiment, the expression for the cross-modal semantic similarity A of the current period is: In the formula, A is the cross-modal semantic similarity score of the current period, S is the mean of the cross-modal semantic similarity scores of all text-image pairs in the current period, and P is the mean of the contradiction probability of all text-image pairs in the current period. It is an exponential function with the natural constant e as the base. This is a preset, extremely small number greater than 1. In this embodiment, The value is set to 1.01. Since the semantic similarity score ranges from -1 to 1, S is added to... To avoid negative calculation results.

[0029] In other embodiments of this application, the expression for the cross-modal semantic similarity may also be: ,in, The preset minimum positive number is used to prevent the denominator from being 0. In this embodiment, the minimum positive number is set as follows: The value is set to 0.001. In other embodiments of this application, the implementer may set the value according to the actual situation. The value of .

[0030] Cross-modal semantic similarity A, expressed by S, represents the degree of global semantic alignment between text and images in a unified semantic space under multimodal semantic analysis. It reflects the extent to which different heterogeneous modalities complete mapping within a common semantic space; a larger value indicates more consistent cross-modal understanding of the model. P is introduced to express the probability of potential semantic conflicts arising in fine-grained semantic understanding under multimodal semantic analysis. Then, using S and P, cross-modal semantic similarity A represents the overall consistency risk of the model in cross-modal semantic understanding. A larger value indicates better macroscopic alignment and a lower overall semantic mismatch risk, suggesting more reliable semantic expression from the model.

[0031] Step S3: Cluster all text feature vectors and all image feature vectors for each period, and construct the semantic effective proportion for each period based on the distribution characteristics of the contour coefficients of all clusters; construct the semantic coverage completeness for each period based on the directional consistency between each text feature vector and the corresponding image feature vector, combined with the cross-modal semantic similarity and the semantic effective proportion.

[0032] Multimodal semantic analysis models, when faced with the variability and complexity of the real world, suffer from imbalances in the distribution of semantic representations or the presence of blank areas in the semantic representation space. This results in the inability to effectively express semantics and causes some semantic representation gaps, leading to semantic coverage defects. Consequently, the model's generalization ability and robustness in open environments are affected. In such cases, the multimodal representation of the disease may be ignored by the model, resulting in missed diagnoses during intelligent diagnosis.

[0033] Therefore, in this case, using all text feature vectors and all image feature vectors within each period as input, a semantic coverage evaluation method based on K-means is used. The number of clusters is set to K=50 to achieve a balance between fine granularity and computation time. The K-means algorithm divides all multimodal fused feature vectors into K=50 semantic clusters. The silhouette coefficient of each element within each cluster is calculated to express the compactness of the data samples within each cluster and the separability of data samples between different clusters. The closer the value is to 1, the better the clustering effect and the clearer the cluster structure. Next, the number of clusters in which the mean silhouette coefficient of all elements within a cluster is greater than the silhouette coefficient threshold is counted. In this embodiment, the silhouette coefficient threshold is set to 0.25. In other embodiments of this application, the implementer can set the silhouette coefficient threshold according to the actual situation. Finally, the proportion of clusters with silhouette coefficients greater than the silhouette coefficient threshold to the total number of clusters K is calculated and used as the semantic effective proportion for each period. This semantic effective proportion expresses the effectiveness of the semantic clusters constructed in the model's feature space in covering the actual semantic categories. Essentially, it reflects whether the model can project different semantic concepts to different regions of the feature space, and also indirectly demonstrates the model's segmentation ability and coverage completeness in the semantic space. The K-means algorithm and silhouette coefficient are both well-known techniques, and their specific processes will not be elaborated upon.

[0034] It should be noted that this application provides only one clustering algorithm for clustering all feature vectors in each period. There are many existing clustering algorithms, and implementers may also use other clustering algorithms to cluster all feature vectors in each period. This application does not impose any specific restrictions.

[0035] A model with a wide semantic coverage does not necessarily have stronger robustness to complex semantic changes. If the model's features are distributed in one direction, and the distribution is too concentrated and lacks diversity, then even if the model can recognize more semantics, it may not be able to accurately distinguish semantically similar but actually different content, resulting in insufficient sensitivity to fine-grained semantic changes. Therefore, the text feature vector and image feature vector corresponding to each text-image pair in each period are used as inputs for cosine similarity to obtain the cosine similarity between the two feature vectors. The mean value Sdiv of the cosine similarity of all positive text-image pairs in each period is calculated. Sdiv represents the degree of consistency of the direction of text and image feature vectors in the same sample semantic space, that is, it represents the degree of alignment of the semantic content represented by the two in the direction of the vectors. It reflects the model's understanding level and robustness to fine-grained semantics. If Sdiv is high, it means that the text and image features are in the same direction in the semantic space, indicating that the model can map data from different modalities to the same semantic concept and achieve consistent cross-modal understanding.

[0036] Based on the above analysis, the semantic coverage completeness B for each period is constructed. Taking the current period as an example, the expression for the semantic coverage completeness is: In the formula, is the semantic coverage completeness of the current period; Scov is the effective proportion of the semantics in the current period; Sdiv is the mean of the cosine similarity of all text-image pairs in the current period; A is the cross-modal semantic similarity of the current period; It is an exponential function with the natural constant e as the base.

[0037] In other embodiments of this application, the expression for semantic coverage completeness may also be: ,in, The preset minimum positive number is used to prevent the denominator from being 0. In this embodiment, the minimum positive number is set as follows: The value is set to 0.001. In other embodiments of this application, the implementer may set the value according to the actual situation. The value of .

[0038] A higher Sdiv value indicates more accurate semantic alignment between the two modalities. In metric B, Scov measures the completeness of the model's feature space coverage of the true semantic categories. If the model can clearly distinguish different semantic concepts and smoothly map them to different regions of the feature space, the Scov value is relatively high. Sdiv, on the other hand, measures the directional consistency of different modalities in the semantic representation of the same concept by calculating the cosine similarity between text and image feature vectors. If the positive sample similarity Sdiv is too high, it may mean that the model is overfitting or in its comfort zone, lacking the ability to explore new samples. In this case, the value of B needs to be reduced to lower the value of the temperature parameter in subsequent calculations, forcing the model to be more sensitive to subtle differences and escape local optima. Conversely, a lower Sdiv means that the feature vectors have better discreteness in the semantic space. The model can effectively separate different or even similar semantic content and project it onto vectors with different directions, thus forming a semantic distribution with a clearer structure and more distinct boundaries, which can accommodate more complex semantic changes. A is the cross-modal semantic similarity. A higher A value indicates better overall consistency of the model in cross-modal semantic understanding and more reliable semantic expression. In summary, index B reflects the completeness of the model in terms of semantic coverage, which is reflected in the synergistic development between breadth and depth. The larger the value, the more complete the semantic coverage of the model, the stronger the ability to distinguish fine granularity, which means that the representation blind spots are reduced more and the model can better understand the diverse semantics in open environments.

[0039] Step S4: Adjust the temperature parameters in the CLIP model based on semantic coverage completeness. Based on the image data of each period and its corresponding descriptive text, combined with the adjusted temperature parameters, perform incremental training on the CLIP model to optimize the semantic analysis model.

[0040] After analyzing the various factors mentioned above, contrastive learning can be identified as the key optimization method. The feature vector pairs composed of text features and image features extracted from the original text data and image data are used as input, and positive sample pairs with the same semantics and negative sample pairs with different semantics are constructed to optimize the multimodal feature representation.

[0041] Because multimodal semantic analysis models operate in complex real-world scenarios, their semantic representation spaces are prone to uneven distribution and coverage blind spots. This results in the model learning fewer features from marginal semantics and low-density regions, leading to the fundamental problem of incomplete semantic coverage. Furthermore, this can cause poor generalization ability in open environments when faced with a wide variety of semantic expressions. For example, in intelligent healthcare, the inability to effectively learn the multimodal features of rare diseases can lead to missed diagnoses.

[0042] Therefore, a contrastive learning improvement strategy needs to be designed based on semantic coverage completeness to systematically optimize the feature space distribution. The specific contrastive learning improvement strategy is designed as follows: In contrastive learning, a temperature parameter is used to control the sharpness of the similarity distribution, thereby affecting the degree to which the loss function focuses on positive and negative samples. The temperature parameter in the CLIP semantic alignment model is adjusted based on semantic coverage completeness. Preferably, in this embodiment, the expression for adjusting the temperature parameter is: In the formula, This refers to the adjusted temperature parameters for the current cycle. The preset initial temperature parameter will be used in this embodiment. The value is 0.07; Represents the adjustment coefficient; Represents the semantic coverage completeness of the current cycle; This is an exponential function with the natural constant e as the base. In this embodiment, it will be... Setting it to 0.1 makes the negative exponential function smoother. In other embodiments of this application, the implementer can set it according to the actual situation. and The value of .

[0043] In order to avoid If the value is too large or too small, in this embodiment of the application, The value range is set to [0.01, 0.5]. In other embodiments of this application, the implementer may set the value according to the actual situation. The range of values ​​for .

[0044] exist In the calculation process, B represents the completeness of the model's semantic coverage, that is, the breadth and depth of the model's feature space's coverage of the real semantic categories. A higher value indicates a stronger semantic segmentation ability and fewer coverage blind spots; combined with the exponential function... The regulatory mechanism expressed, among which This reflects the degree of semantic coverage deficiency. The intensity of the control and regulation comprehensively reflects the temperature parameters. The completeness index B is dynamically adjusted. This illustrates the optimized setting of the temperature parameter during contrastive learning. A larger value indicates that the model uses a higher temperature parameter when the semantic coverage is more complete, resulting in a smoother similarity distribution and thus reducing the focus on difficult samples. This helps avoid overfitting when the feature space distribution is relatively uniform. Conversely, when B is small... Reducing the similarity value makes the distribution sharper, thereby enhancing the learning of difficult negative samples and improving the problem of insufficient semantic coverage. This design systematically optimizes the feature space distribution by adaptively adjusting the temperature parameter, improving the model's generalization ability in open environments.

[0045] Furthermore, all positive text-image pairs and all negative text-image pairs in the current period are incorporated into the semantic alignment model trained in the previous period. The temperature parameter in the model is set to the adjusted temperature parameter for the current period, and L2 regularization is applied to incrementally train the model. The incremental training process of the model is well-known and will not be elaborated further.

[0046] After applying the aforementioned contrastive learning improvement strategy, the CLIP semantic alignment model can automatically adjust the temperature parameter value during training based on the semantic coverage completeness B of the current period. A larger B indicates a more uniform distribution of the model's feature space and more complete semantic coverage; therefore, increasing the temperature parameter value is appropriate. To reduce the sensitivity of the loss function to difficult negative samples and avoid the model from overemphasizing similar but semantically different sample pairs, a smaller B indicates uneven distribution of the model's feature space and the existence of semantic coverage blind spots. In this case, adjusting the B value is appropriate. This approach increases the sensitivity of the loss function to difficult negative samples, forcing the model to focus more on these samples and thus learn fine-grained semantic distinctions. It dynamically optimizes the feature space to better learn difficult negative samples that are semantically similar but of different categories, avoiding the overfitting problem that easily occurs in uniformly distributed feature spaces. Through iterative optimization, a set of multimodal feature representations with higher semantic consistency and stronger generalization ability is obtained. This set of feature representations can map data from different modalities to a semantic space with better distribution and clustering, helping to improve the accuracy of subsequent downstream tasks such as cross-modal retrieval or semantic understanding.

[0047] Through the aforementioned adaptive temperature parameter adjustment mechanism, contrastive learning can dynamically optimize the feature space distribution of the multimodal semantic analysis model in different training cycles, enhance the learning ability of the multimodal semantic analysis model for semantically similar but different negative samples, and thus improve the robustness of the multimodal semantic analysis model in cross-modal semantic alignment tasks, thereby realizing the training and optimization of the multimodal semantic analysis model.

[0048] A schematic diagram of the process of obtaining cross-modal semantic similarity is shown below. Figure 2 As shown.

[0049] Based on the same inventive concept as the above methods, this application also provides a multimodal semantic analysis model construction system, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-described multimodal semantic analysis model construction methods.

[0050] In summary, this application provides a method for constructing a multimodal semantic analysis model. By analyzing the global semantic similarity and potential semantic concurrency probability between text data and corresponding image data, cross-modal semantic similarity is constructed. This method can capture the correspondence between two modalities more meticulously, overcome the fine-grained errors that may occur when relying solely on overall similarity, and ensure the consistency and reliability of the model's semantics. Clustering all text and image feature vectors and evaluating the clustering quality based on the silhouette coefficient distribution quantifies the breadth of feature space coverage. Combining the consistency of feature vector directions and cross-modal semantic similarity, a semantic coverage completeness is constructed, which can uniformly measure the overall semantic performance of the current model within a given period. This index serves as a basis for subsequent temperature parameter adjustments. This provides an objective basis for adjusting the temperature parameter in contrastive learning based on the completeness index. This allows the model to dynamically select difficult negative samples based on the entire test set during incremental training, enhancing its representation learning ability for difficult negative samples, preventing overfitting, and making the entire feature space distribution more reasonable and orderly. This achieves a joint improvement in feature semantic consistency and completeness. Through the above adaptive temperature parameter adjustment mechanism, contrastive learning can dynamically optimize the feature space distribution of the multimodal semantic analysis model in different training cycles, enhance the multimodal semantic analysis model's ability to learn from semantically similar but different negative samples, and thus improve the robustness of the multimodal semantic analysis model in cross-modal semantic alignment tasks, realizing the training and optimization of the multimodal semantic analysis model.

[0051] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0052] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0053] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1.A method for constructing a multi-modal based semantic analysis model, the method comprising: The method comprises the following steps: acquiring image data and corresponding description text in each cycle; obtaining text feature vectors of the description text and image feature vectors of the image data through a semantic alignment model; constructing cross-modal semantic similarity of each cycle based on global semantic similarity and latent semantic conflict probability between each text feature vector and the corresponding image feature vector in each cycle; performing clustering on all text feature vectors and all image feature vectors in each cycle, constructing semantic effective proportion of each cycle based on distribution characteristics of silhouette coefficients of all clustering clusters; constructing semantic coverage completeness of each cycle based on the direction consistency degree of each text feature vector and the corresponding image feature vector, combining the cross-modal semantic similarity and the semantic effective proportion. adjusting a temperature parameter in a CLIP model based on the semantic coverage completeness, and performing incremental training on the CLIP model based on the image data and the corresponding description text in each cycle and the corresponding adjusted temperature parameter to optimize the semantic analysis model. 2.The multi-modal based semantic analysis model construction method of claim 1, wherein, The process of obtaining the cross-modal semantic similarity is as follows: taking each image data and the corresponding description text as a positive text-image pair, calculating similarity scores between each text feature vector and the corresponding image feature vector through a semantic alignment model; taking each text feature vector and the corresponding image feature vector as input of a logistic regression algorithm to obtain a contradiction probability corresponding to each positive text-image pair; determining the cross-modal semantic similarity of each cycle based on the similarity scores and the contradiction probability, wherein the cross-modal semantic similarity is positively correlated with the similarity scores and negatively correlated with the contradiction probability. 3.The multi-modal based semantic analysis model construction method of claim 2, wherein, The expression of the cross-modal semantic similarity is as follows: wherein A is the cross-modal semantic similarity of the current cycle, S is the mean of the cross-modal semantic similarity scores of all positive text-image pairs of the current cycle, P is the mean of the contradiction probabilities of all positive text-image pairs of the current cycle, is an exponential function with the natural constant e as the base number, is a preset minimum number greater than 1. 4.The multi-modal based semantic analysis model construction method of claim 1, wherein, The semantic effective proportion is a proportion of the number of clustering clusters with an average silhouette coefficient greater than a preset silhouette coefficient threshold in all clustering clusters. 5.The multi-modal based semantic analysis model construction method of claim 2, wherein, The process of obtaining the semantic coverage completeness is as follows: calculating cosine similarity between the text feature vector and the image feature vector corresponding to each positive text-image pair; determining the semantic coverage completeness of each cycle based on the semantic effective proportion, the cosine similarity and the cross-modal semantic similarity, wherein the semantic coverage completeness is positively correlated with the semantic effective proportion and the cross-modal semantic similarity and negatively correlated with the cosine similarity. 6.The multi-modal based semantic analysis model construction method of claim 5, wherein, The expression of the semantic coverage completeness is as follows: In the formula, is the semantic coverage completeness of the current period; Scov is the semantic effective proportion of the current period; Sdiv is the mean value of the cosine similarity of all text-image pairs in the current period; A is the cross-modal semantic similarity of the current period; is an exponential function with the natural constant e as the base number. 7.The multi-modal based semantic analysis model construction method of claim 1, wherein, The temperature parameter in the CLIP model is adjusted based on the semantic coverage completeness, specifically as follows: mapping the semantic coverage completeness of each cycle to a positive number, and taking a fusion value of the positive number and a preset initial temperature parameter as an adjusted temperature parameter of each cycle. 8.The multi-modal based semantic analysis model construction method of claim 7, wherein, The expression of the adjusted temperature parameter is as follows: wherein, is the adjusted temperature parameter of the current cycle; represents a preset initial temperature parameter; represents an adjustment coefficient; represents the semantic coverage completeness of the current cycle; is an exponential function with the natural constant e as the base number. 9.The multi-modal based semantic analysis model construction method of claim 1, wherein, The CLIP model is incrementally trained based on the image data and the corresponding description text in each cycle and the corresponding adjusted temperature parameter, specifically as follows: Obtain each positive text-image pair and each negative text-image pair of each cycle, wherein each negative text-image pair is a text-image pair composed of one text data and one image data randomly extracted from all the description text and image data; and use all the positive and negative sample pairs of the current cycle as the semantic alignment model trained in the previous cycle, set the temperature parameter in the model as the adjusted temperature parameter of the current cycle, and perform incremental training on the model. 10.A multimodal based semantic analysis model construction system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1-9 when executing the computer program.

Images