A method for constructing a multimodal knowledge base of a domain-wide large model based on feature representation

Abstract

The present invention discloses a method for constructing a multimodal knowledge base of a domain large model based on feature representation, comprising the following steps: step one, collecting and processing information for constructing a knowledge base; step two, extracting multimodal information features based on the information of the knowledge base collected in step one; step three, constructing a multimodal knowledge base based on the information collected in step one and the multimodal information features extracted in step two; step four, extracting the demand information input by the user based on the multimodal knowledge base constructed in step three, constructing a domain intention recognition model, and determining the specific demand intention of the user; step five, extracting the demand information input by the user, querying the knowledge base, inputting the queried knowledge block into the domain large model, and outputting the query result required by the user with the help of the large model generation capability. The method for constructing a multimodal knowledge base of a domain large model based on feature representation of the present invention makes up for the deficiencies of the existing large language model in processing multimodal data and enhances the user experience.

Description

Technical Field

[0001] The present invention relates to a knowledge base construction method, and more specifically to a method for constructing a domain large model multimodal knowledge base based on feature representation. Background Art

[0002] Multimodal large models process multiple types of information such as text, images, audio, and video at the same time, and are highly integrated with the real world. They are expected to become intelligent assistants for humans and promote AI towards AGI; multimodality is more in line with the way humans receive, process, and express information. Humans can perceive multiple types of information, each type of information is a modality, and this information is often interrelated. Multimodal information makes large models smarter. Multimodality is more convenient for users to interact with. Thanks to the support of multimodal input, users can interact and communicate with intelligent assistants in a more flexible way, improving their task-solving capabilities.

[0003] With the advent of the Internet, information has exploded rapidly. It has become a major challenge to effectively find answers from a large amount of information based on questions. Open multimodal question answering can help alleviate the problem of information overload by retrieving external knowledge bases based on questions and generating correct answers. In recent years, people have conducted in-depth explorations on some advanced multimodal large language models, such as BLIP-2 and GPT-4, to improve their performance by implicitly storing a large amount of external knowledge in model parameters, which now have expanded to hundreds of billions. Although these models have achieved exciting results on various tasks, their storage methods are not only poorly interpretable and computationally expensive, but also have a fixed amount of information stored. If the knowledge dependency of downstream task reasoning exceeds the current knowledge scope, it is difficult for large models to give correct answers. Despite the success of the above methods, the following problems still exist: (1) Existing methods are insufficient in information fusion and interaction between multimodalities, existing methods do not check the relevance of retrieved knowledge in the generation stage, current large language models are limited in processing multimodal information in vertical fields, and multimodal information requires manual annotation. In order to solve the technical problems existing in the above-mentioned background technologies, the patent (CN117648429B) provides a question-answering method and system based on a multimodal adaptive retrieval-enhanced large model, which improves the accuracy of retrieval and answer generation of modal question-answering. The patent (CN118035416A) proposes a streaming question-answering illustration method and system. The patent (CN117093744A) proposes an intelligent audio-visual interaction system based on a large language model, which converts audio into corresponding text descriptions to form a local knowledge base, and then connects the local knowledge base to the large language model, effectively reducing the waste of information and cost consumption of manual annotation.

[0004] Most of the existing language models can only accept text input and output modes for question answering, which has the following defects and affects the user experience:

[0005] (1) Limitations of input form:

[0006] Existing large models mainly rely on text input, which limits their ability to process non-text data such as images, audio, video, etc. For scenarios with rich visual or auditory information, such as image recognition and speech recognition, large models that only rely on text input will not be able to directly utilize this information, thus reducing their effectiveness in these scenarios.

[0007] (2) Lack of multimodal interaction:

[0008] In multimodal interaction, users can interact with the system through different senses (such as vision, hearing, touch, etc.), providing a richer and more natural interactive experience. However, large models that only support text input cannot achieve such multimodal interaction, limiting their applicability and user experience in practical applications.

[0009] (3) Limitations of information expression:

[0010] When describing complex spatial relationships, dynamic processes, or emotional states, text may not be able to fully capture the details and complexity of this information. In contrast, multimedia forms such as images and videos can express this information more intuitively. Large models that only support text input cannot directly process this multimedia information, thus limiting their ability to express and understand complex information.

[0011] (4) Limitations of scalability and adaptability:

[0012] As technology continues to develop and application scenarios continue to expand, new data types and interaction methods will continue to emerge. Language models that only support text input may not be able to adapt to these new changes, thus limiting their scalability and adaptability.

[0013] To meet ever-changing needs, large models need to have stronger cross-modal learning and fusion capabilities so that they can handle different types of data and interaction methods.

[0014] Currently, although there are some AI systems that attempt to integrate information from multiple modalities, such as multimodal dialogue systems based on deep learning, they tend to focus on specific tasks and have limited modal fusion, failing to fully explore and utilize the complementary advantages of data from different modalities, resulting in limited efficiency and accuracy when processing complex and comprehensive tasks. Moreover, compared with single modality, multimodal large model algorithms and engineering are more difficult, and face more difficulties in representation, alignment, reasoning, generation, migration, quantification and other aspects. Multimodal large models need to process larger amounts of data, and coupled with more difficulties in training engineering, the corresponding computing power requirements are higher, which cannot be supported by ordinary businesses. Summary of the invention

[0015] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a method for constructing a domain large model multimodal knowledge base based on collaborative feature representation, which can effectively process users' multimodal query information, make up for the deficiencies of existing large language models in processing multimodal data, and enhance user experience.

[0016] To achieve the above object, the present invention provides the following technical solution: a method for constructing a domain large model multimodal knowledge base based on feature representation, characterized in that it comprises the following steps:

[0017] Step 1: Collect and process information for building the knowledge base;

[0018] Step 2: extract multimodal information features based on the information of the knowledge base collected in step 1;

[0019] Step 3: construct a multimodal knowledge base by vectorizing the information collected in step 1 and the multimodal information extracted in step 2;

[0020] Step 4: Based on the multimodal knowledge base constructed in step 3, extract the demand information input by the user, build a domain intention recognition model, and determine the user's specific demand intention;

[0021] Step 5: Extract the demand information input by the user, query the knowledge base, embed the query results into the prompt word input domain big model, and use the big model generation capability to output the query results required by the user.

[0022] As a further improvement of the present invention, the specific steps of collecting and processing the information for building the knowledge base in step 1 are as follows:

[0023] Step 1. Collect data from various channels, such as books, journals, the Internet, databases, social media, and video platforms. Collect data for each modality separately, including text documents, pictures, audio files, and video clips;

[0024] Steps 1 and 2: Clean the data, remove noise, duplicates, and correct errors. Perform word segmentation, annotation, and standardization on text; convert, scale, and crop images and videos; and reduce noise and transcode audio.

[0025] As a further improvement of the present invention, the specific steps of extracting multimodal information features based on the information of the knowledge base collected in step 1 in step 2 are as follows:

[0026] Step 21: Use different feature extraction methods for different modalities based on the modality of the collected information. For text data, use word embedding technology (such as Word2Vec, BERT, etc.) to extract text features; for image data, use convolutional neural network (CNN) to extract visual features; for audio data, use short-time Fourier transform (STFT) and continuous wavelet transform (CWT) to extract audio features;

[0027] Step 22: for various external data in text data form, select different loading methods to load information without destroying the knowledge structure, and select appropriate segmentation tools to segment the external data into blocks according to the knowledge structure;

[0028] Steps 2 and 3: For each text information block, use word embedding and syntactic and semantic analysis to extract features.

[0029] As a further improvement of the present invention, the specific steps of constructing a multimodal knowledge base in step 3 are as follows:

[0030] Step 31: Design a correlation model for multimodal data and determine the linking method between different modal data, such as through common topics, timestamps, and geographic locations. Apply multimodal information fusion technology to integrate different modal data to form a unified knowledge representation and complete multimodal data fusion.

[0031] Step 32: Define the entities, attributes and relationships of the knowledge graph, build an ontology model, use the annotated and associated data to fill the knowledge graph, form knowledge in the form of triples, and use a graph database or knowledge graph platform to store and manage the knowledge graph;

[0032] Step 33: Based on the application scenario, train the machine learning or deep learning model to enhance the query and reasoning capabilities of the knowledge base, conduct model evaluation and adjustment, and ensure high accuracy and robustness in model training and optimization.

[0033] As a further improvement of the present invention, the specific steps of constructing a domain intention recognition model in step 4 and determining the user's specific demand intention are as follows:

[0034] Step 41: for the demand information input by the user, use the multimodal feature extraction in step 2 to obtain the effective expression of the feature vector of each mode in the respective feature space;

[0035] Step 42: Design or select a mechanism to establish the relationship between modalities, such as a shared encoder, a cross-modal mapping function, or a joint embedding space, so that the feature vectors of different modalities can be compared or aligned to a certain extent, ensuring that cross-modal features with the same semantic content are close in high-dimensional space, adding an attention mechanism during feature fusion or mapping to allow the model to automatically learn the importance of different modalities and dynamically adjust the weights of each modal feature for better matching and coordination to complete cross-modal alignment;

[0036] Step 43: By designing a multi-objective loss function, correlation constraints are introduced by simultaneously considering the representation ability of a single modality and the consistency between modalities during the training process.

[0037] As a further improvement of the present invention, the target loss function in step 43 is as follows:

[0038]

[0039] in, It is a knowledge tag. is the number of samples, Non-text and text The encoding, is the softmax function, It is a feature collaboration module. Given a training set .

[0040] As a further improvement of the present invention, the specific steps of extracting the demand information input by the user and querying the knowledge base in step 5 are as follows:

[0041] Step 51: extract the user demand information and the features corresponding to each piece of knowledge in the multimodal knowledge base, and map them into a unified multimodal semantic space. Then, evaluate the matching degree between each piece of knowledge in the knowledge base and the query question, and select the top 100 most relevant pieces of knowledge as the query result.

[0042] The similarity is calculated as:

[0043]

[0044] in, and doc Respectively represent user demand information and multimodal knowledge base, and Indicates the corresponding encoder;

[0045] Step 52: according to the user's needs, the similarity between the query results obtained by the spatial distance calculation and the needs is calculated, and the query results are re-ranked;

[0046] Step 53: Select appropriate temperature and top_10 parameters, integrate the optimized query results into the prompt as context, input them into the large language model, and get the correct answer.

[0047] As a further improvement of the present invention, the similarity between the query result and the requirement in step 52 is calculated using a twin network, the main input of which is , then the loss function can be defined as follows:

[0048]

[0049]

[0050] Among them, n represents the training sample, is the loss function; are model parameters; and are the two inputs whose similarity needs to be evaluated, represent and Are there similarities? If the two are similar is 0, otherwise is 1; A loss function that represents a high degree of similarity between two input data; A loss function representing the lack of similarity between two input data; Used to describe the distance between two vectors after two input data are converted into vectors; Indicates the data subscript currently being input.

[0051] As a further improvement of the present invention, the loss function Designed to increase monotonically, Monotonically decreasing, needs to satisfy:

[0052] ;

[0053] Then we get an accurate loss function for a single sample:

[0054] ;

[0055] in, is an additive exponential function, ,because is bounded, so It is also bounded. For setting The upper bound of .

[0056] Beneficial effects of the present invention:

[0057] Constructed a multimodal knowledge base

[0058] Multimodal knowledge base integrates information from different modalities (such as text, images, audio, etc.) into a unified knowledge representation framework. It mainly includes data collection, data preprocessing, modality fusion, knowledge representation and knowledge base construction.

[0059] Multimodal feature extraction

[0060] For each modality of the questioner's input, select an appropriate feature extraction method. For example, for text data, you can use word embedding technology (such as Word2Vec, BERT, etc.) to extract text features; for image data, you can use convolutional neural networks (CNN) to extract visual features.

[0061] Feature Collaborative Representation

[0062] By using collaborative representation learning, each modality in the multimodality is mapped to its own representation space, and at the same time, the vectors satisfy certain correlation constraints, which preserves the specificity of each modality while capturing the association between the modalities.

[0063] Feature Understanding

[0064] Build an Attention semantic understanding model to automatically segment knowledge features, ensure the semantic integrity of each knowledge block, and improve the retrieval recall rate.

[0065] Domain Intent Recognition

[0066] Build a domain intent recognition model to determine the user's specific query intent; search the multimodal knowledge base; integrate the search results and the user's specific query intent into the prompt, and hand it over to the large model for processing and answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The present invention is a flowchart of the method for constructing a domain large model multimodal knowledge base based on feature representation. DETAILED DESCRIPTION

[0068] The present invention will be further described below in conjunction with the embodiments given in the accompanying drawings.

[0069] Reference Figure 1 As shown, a method for constructing a domain large model multimodal knowledge base based on feature representation in this embodiment includes the following steps:

[0070] S1. Collect and process information to build a knowledge base

[0071] S1.1. Collect data from various channels, such as books, journals, the Internet, databases, social media, video platforms, etc. Collect data of each modality separately, including text documents, pictures, audio files, video clips, etc.

[0072] S1.2. Clean the data, remove noise, duplicates, and correct errors. Perform word segmentation, annotation, and standardization on text; perform format conversion, scaling, and cropping on images and videos; perform noise reduction and transcoding on audio.

[0073] S2. Multimodal information feature extraction

[0074] S2.1. According to the modality of the collected information, different feature extraction methods are used for different modalities. For image and audio data, the corresponding image understanding and speech description models are used respectively to output textual interpretation data of the image and audio.

[0075] S2.2. For various text data forms of external data, select different loading methods to load information without destroying the knowledge structure, and select appropriate segmentation tools to segment the external data into chunks according to the knowledge structure.

[0076] S2.3. For each block of text information, use word embedding and syntactic and semantic analysis to extract features. Common word embedding methods include Word2Vec, GloVe or BERT, among which pre-trained models such as BERT can provide richer contextual information. Syntactic and semantic analysis uses natural language processing technology to extract sentence structure features (such as dependency relationships, syntactic trees) and deep semantic features (such as concept role labeling, entity relationships).

[0077] S3. Construction of multimodal knowledge base

[0078] S3.1. Multimodal data fusion. Design a correlation model for multimodal data and determine how to link data of different modalities, such as through common topics, timestamps, geographic locations, etc. Apply multimodal information fusion technology to integrate data of different modalities to form a unified knowledge representation.

[0079] S3.2, knowledge graph construction. Define the entities, attributes and relationships of the knowledge graph and build an ontology model. Use the labeled and related data to fill the knowledge graph to form knowledge in the form of triples (entity-relationship-entity). Use graph databases (such as Neo4j) or knowledge graph platforms (such as Apache Jena, Alibaba Cloud Graph Database) to store and manage knowledge graphs.

[0080] S3.3, Model training and optimization: Based on the application scenario, train machine learning or deep learning models to enhance the query and reasoning capabilities of the knowledge base. Perform model evaluation and adjustment to ensure high accuracy and robustness.

[0081] S4: Extract the demand information input by the user, build a domain intention recognition model, and determine the user's specific demand intention. It specifically includes the following sub-steps:

[0082] S4.1. For the demand information input by the user, use the multimodal feature extraction in S2 to obtain the effective expression of the feature vector of each mode in the respective feature space.

[0083] S4.2, cross-modal alignment. Design or select a mechanism to establish a relationship between modalities, such as a shared encoder, cross-modal mapping function, or joint embedding space, so that the feature vectors of different modalities can be compared or aligned to a certain extent, ensuring that cross-modal features with the same semantic content are close in high-dimensional space. Add an attention mechanism during the feature fusion or mapping process to allow the model to automatically learn the importance of different modalities and dynamically adjust the weights of each modal feature for better matching and coordination.

[0084] S4.3. Introduce correlation constraints. By designing a multi-objective loss function, both the representation ability of a single modality and the consistency between modalities are considered during the training process.

[0085] Non-text and text pair matching modeling is used to fuse multimodal feature representations. Given a training set , the loss function is as follows:

[0086]

[0087] in, It is a knowledge tag. is the number of samples, Non-text and text The encoding, is the softmax function, It is a feature collaboration module.

[0088] S5: Extract the user input requirement information and query the knowledge base. This includes the following sub-steps:

[0089] S5.1. Extract user demand information and the features corresponding to each piece of knowledge in the multimodal knowledge base, and map them into a unified multimodal semantic space. Then, evaluate the matching degree between each piece of knowledge in the knowledge base and the query question, and select the top 100 most relevant pieces of knowledge as the query results.

[0090] The similarity is calculated as:

[0091]

[0092] in, and doc Respectively represent user demand information and multimodal knowledge base, and Indicates the corresponding encoder.

[0093] S5.2. According to user needs, calculate the similarity between the query results obtained by the above spatial distance calculation and the needs, and re-rank the query results. The twin network is used to calculate the similarity between the needs and the query results. The main idea of ​​the twin network is to map the input to the target space through a function, and use a simple distance (Euclidean distance, etc.) in the target space to compare the similarity. This process has been implemented when multimodal features are collaboratively represented. In the training phase, minimize the loss function value of a pair of samples from the same category and maximize the loss function value of a bunch of samples from different categories.

[0094] The main input of the Siamese network is , then the loss function can be defined as follows:

[0095]

[0096]

[0097] in, A loss function that represents a high degree of similarity between two input data; Represents the loss function when two input data are not similar. n represents the training sample.

[0098] By separating the two, we can achieve the goal of reducing the energy of pairs of the same category and increasing the energy of pairs of different categories when minimizing the loss function. A very simple and intuitive way to achieve this is to just change Designed to increase monotonically, Monotonically decreasing is fine, but we must ensure that the distance between different pairs of images must be greater than the distance between the same pairs of images, so we must satisfy:

[0099]

[0100] This is a loss function for an exact single sample:

[0101]

[0102] S5.3. Select appropriate temperature (adjusts the "randomness" or "uncertainty" of the model's generated output) and top_10 (controls how much information about the probability distribution is retained by the model when generating the output). Integrate the optimized query results into the prompt as context, input it into the large language model, and get the correct answer.

[0103] In summary, the method for constructing a domain large model multimodal knowledge base based on feature representation in this embodiment has the following characteristics:

[0104] (1) Collaborative representation of multimodal features

[0105] Multimodal feature collaborative representation is a representation learning method that involves mapping data information from different modalities (such as text, images, audio, etc.) into their respective representation spaces, but at the same time ensuring that these mapped vectors satisfy certain correlation constraints (such as linear correlation).

[0106] The core idea of ​​collaborative feature representation is to learn a separate feature extraction model for each modality and coordinate different modalities through a constraint. This constraint can be based on a similarity model, that is, minimizing the distance between different modalities in the collaborative representation space to ensure that the content with the same semantics is closer in the representation space. In addition, there are structured collaborative representation models, which enforce additional constraints on the representation of different modalities. The specific structured constraints vary depending on the application.

[0107] The advantage of collaborative representation methods is that they can maintain the unique features and exclusivity of each single modality, while using the complementarity between multiple modalities to eliminate the redundancy between modalities, thereby learning better feature representation. In cross-modal learning, collaborative architectures have been widely used. The mainstream collaborative method is based on the cross-modal similarity method, which aims to learn a common subspace by directly measuring the distance between vectors and different modalities.

[0108] (2) Building a multimodal knowledge base

[0109] Building a multimodal knowledge base is a complex process involving multiple steps and technologies, mainly including data collection, data preprocessing, modality fusion, knowledge representation, knowledge reasoning, knowledge base construction and maintenance, etc.

[0110] (3) User multimodal intent recognition

[0111] Match the fused multimodal feature vectors with the entities and relations in the multimodal knowledge base. Use the structured information in the knowledge base, combined with machine learning or deep learning models, to infer and judge the user's intent. If there is no direct matching intent in the knowledge base, similarity-based methods or transfer learning techniques can be used to classify intent.

[0112] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for constructing a domain large model multimodal knowledge base based on feature representation, characterized in that: The steps include: Step 1: Collect and process information for building the knowledge base; Step 2: extract multimodal information features based on the information of the knowledge base collected in step 1; Step 3: construct a multimodal knowledge base by vectorizing the information collected in step 1 and the multimodal information extracted in step 2; Step 4: Based on the multimodal knowledge base constructed in step 3, extract the demand information input by the user, build a domain intention recognition model, and determine the user's specific demand intention; Step 5: Extract the user's input demand information, query the knowledge base, embed the query results into the prompt word input domain big model, and generate the query results required by the user with the help of the big model; The specific steps of constructing a multimodal knowledge base in step 3 are as follows: Step 31: Design a correlation model for multimodal data, determine the linking method between different modal data, and apply multimodal information fusion technology to integrate different modal data through common topics, timestamps, and geographic locations to form a unified knowledge representation and complete multimodal data fusion; Step 32: Define the entities, attributes and relationships of the knowledge graph, build an ontology model, use the annotated and associated data to fill the knowledge graph, form knowledge in the form of triples, and use a graph database or knowledge graph platform to store and manage the knowledge graph; Step 33: Based on the application scenario, train machine learning or deep learning models to enhance the query and reasoning capabilities of the knowledge base, conduct model evaluation and adjustment, and ensure high accuracy and robustness to complete model training and optimization; The specific steps of constructing a domain intent recognition model in step 4 and determining the user's specific demand intent are as follows: Step 41: for the demand information input by the user, use the multimodal feature extraction in step 2 to obtain the effective expression of the feature vector of each mode in the respective feature space; Step 42: Design or select a mechanism to establish the relationship between modalities, including shared encoders, cross-modal mapping functions or joint embedding spaces, so that the feature vectors of different modalities can be compared or aligned to a certain extent, ensuring that cross-modal features with the same semantic content are close in high-dimensional space, adding an attention mechanism during feature fusion or mapping to allow the model to automatically learn the importance of different modalities and dynamically adjust the weights of each modal feature for better matching and coordination to complete cross-modal alignment; Step 43: By designing a multi-objective loss function, correlation constraints are introduced by simultaneously considering the representation ability of a single modality and the consistency between modalities during the training process.

2. The method for constructing a domain large model multimodal knowledge base based on feature representation according to claim 1 is characterized in that: The specific steps of collecting and processing the information used to build the knowledge base in step 1 are as follows: Step 1. Collect data from various channels, including books, journals, the Internet, databases, social media, and video platforms. Collect data for each modality separately, including text documents, pictures, audio files, and video clips; Steps 1 and 2: clean the data, remove noise and duplicates, correct errors, and segment, annotate, and standardize the text; Convert, scale, and crop images and videos; reduce noise and transcode audio.

3. The method for constructing a domain large model multimodal knowledge base based on feature representation according to claim 2 is characterized in that: The specific steps of extracting multimodal information features based on the information of the knowledge base collected in step 1 in step 2 are as follows: Step 21: According to the modality of the collected information, different feature extraction methods are used for different modalities. For text data, word embedding technology is used to extract text features; for image data, convolutional neural networks are used to extract visual features; for audio data, short-time Fourier transform and continuous wavelet transform are used to extract audio features; Step 22: for various external data in text data form, select different loading methods to load information without destroying the knowledge structure, and select appropriate segmentation tools to segment the external data into blocks according to the knowledge structure; Step 2 and 3: For each text information block, use word embedding and syntactic and semantic analysis to extract features.

4. The method for constructing a domain large model multimodal knowledge base based on feature representation according to claim 3 is characterized in that: The objective loss function in step 43 is as follows: ； in, It is a knowledge tag. is the number of samples, Non-text and text The encoding, is the softmax function, It is a feature collaboration module. Given a training set .

5. The method for constructing a domain large model multimodal knowledge base based on feature representation according to claim 4 is characterized in that: The specific steps of extracting the user input requirement information in step 5 and querying the knowledge base are as follows: Step 51: extract the user demand information and the features corresponding to each piece of knowledge in the multimodal knowledge base, and map them into a unified multimodal semantic space. Then, evaluate the matching degree between each piece of knowledge in the knowledge base and the query question, and select the top 100 most relevant pieces of knowledge as the query result. The similarity is calculated as: ； in, and doc Respectively represent user demand information and multimodal knowledge base, and Indicates the corresponding encoder; Step 52: according to user needs, calculate the similarity between the query results obtained by spatial distance calculation and the needs, and re-rank the query results; Step 53: Select appropriate temperature and top_10 parameters, integrate the optimized query results into the prompt as context, input them into the large language model, and get the correct answer.

6. The method for constructing a domain large model multimodal knowledge base based on feature representation according to claim 5 is characterized in that: The similarity between the query result and the requirement in step 52 is calculated using a twin network, the main input of which is , then the loss function is defined as follows: ； ； Among them, n represents the training sample, is the loss function; are model parameters; and are the two inputs whose similarity needs to be evaluated, represent and Are there similarities? If the two are similar is 0, otherwise is 1; A loss function that represents a high degree of similarity between two input data; A loss function representing the lack of similarity between two input data; Used to describe the distance between two vectors after two input data are converted into vectors; Indicates the data subscript currently being input.

7. The method for constructing a domain large model multimodal knowledge base based on feature representation according to claim 6 is characterized in that: The loss function Designed to increase monotonically, Monotonically decreasing, needs to satisfy: ； Then we get an accurate loss function for a single sample: ； in, is an additive exponential function, ,because is bounded, so It is also bounded. For setting The upper bound of .

Images