Knowledge base construction method and device based on multi-modal large language model

Abstract

The application relates to the technical field of knowledge base construction, and provides a knowledge base construction method and device based on a multi-modal large language model, which comprises the following steps: acquiring multi-modal data and corresponding category labels, wherein the multi-modal data comprises at least one of an image, a video, audio, point cloud, thermal imaging, events and text; generating a text description corresponding to the multi-modal data by a multi-modal large language model based on a preset prompt word template; calculating the cosine similarity between the generated text description and the corresponding category label by a pre-trained text encoder; and constructing a knowledge base of the corresponding category according to the text description whose cosine similarity is higher than a preset threshold. In this way, semantic alignment of heterogeneous modalities such as images, audio and point cloud with text is achieved, and the problem of semantic fragmentation between modalities in traditional methods is solved.

Description

Technical Field

[0001] This application relates to the field of knowledge base construction technology, and in particular to a method and apparatus for constructing a knowledge base based on a multimodal large language model. Background Technology

[0002] As a key component of multimodal representation learning, the knowledge base provides semantic priors for multimodal embedding and center localization. Traditional methods rely on manual annotation or structured databases, but their scalability and semantic coverage are limited. With the rise of LLMs, their generative capabilities are being leveraged to automatically construct knowledge bases. For example, prompting engineering allows LLMs to generate diverse text descriptions for each category, breaking through the limitations of traditional methods that rely on a single high-quality text source. GPT-4 is used to generate category descriptions, and combined with data-related text generated by multimodal LLMs, a hybrid knowledge base is constructed to enhance the semantic richness of the embedding centers. However, descriptions generated solely by LLMs may contain irrelevant semantics (such as background details), causing the embedding centers to deviate from the true category distribution. For example, for the "helicopter" category, the generated text may overemphasize the "blue sky" background while ignoring key features such as the rotor structure.

[0003] In summary, most existing methods treat multimodal data and multimodal text descriptions separately, failing to fully activate the generative potential of multimodal large language models (MLLMs), resulting in sparse cross-modal semantic associations and insufficient embedding space alignment accuracy. Summary of the Invention

[0004] This application provides a knowledge base construction method based on multimodal large language models, which aims to build a knowledge base based on categories and MLLMs, and improve the discriminativeness of embedded data and localization centers by enhancing irrelevant descriptions.

[0005] According to a first aspect of this application, a method for constructing a knowledge base based on a multimodal large language model is provided, including:

[0006] Acquire multimodal data and its corresponding category labels, wherein the multimodal data includes at least one of images, videos, audio, point clouds, thermal imaging, events, and text;

[0007] Based on preset prompt word templates, a text description corresponding to the multimodal data is generated using a multimodal large language model. The multimodal large language model includes the InstructBLIP model and the LLaMA-adapter model. The text description corresponding to the multimodal data is generated using the following formula:

[0008]

[0009] in, Represents multimodal data input. It is a text description corresponding to multimodal data. This represents the generator function of a multimodal large language model. The training is performed based on a pre-trained multimodal encoder, which includes the frozen visual encoder of the InstructBLIP model and the base encoder corresponding to the cross-modal adaptation layer of the LLaMA-adapter model.

[0010] The training process of the generator function of the multimodal large language model includes:

[0011] Cross-modal embedding space alignment: Based on the pre-trained multimodal encoder, the multimodal data and its corresponding text description are input into the contrastive learning framework, and the embedding space of the multimodal data is aligned by optimizing the contrastive learning loss function;

[0012] Parameter freezing and adaptation layer training: Freeze the main parameters of the pre-trained visual encoder and text encoder, and train only the lightweight adaptation layer consisting of cross-modal attention modules or query transformers to minimize the cross-entropy loss between text descriptions and real category labels;

[0013] Joint optimization: The total loss function is determined by weighting the contrastive learning loss and cross-entropy loss, and the adaptation layer parameters are jointly optimized through the backpropagation algorithm until the model converges.

[0014] The step of optimizing the contrastive learning loss function to align the embedding space of the multimodal data includes:

[0015] The input multimodal data is mapped to intermediate feature representations based on the visual encoder and text encoder;

[0016] By projecting intermediate features of each modality onto a shared semantic space through a parameterized mapping function, a semantic representation that can be directly compared is formed.

[0017] Positive and negative sample pairs are constructed based on data association relationships. Positive sample pairs include semantically consistent cross-modal data, while negative sample pairs contain semantically inconsistent cross-modal data.

[0018] By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, the contrastive learning loss function is optimized to achieve multimodal embedding space alignment.

[0019] The cosine similarity between the generated text description and the corresponding category label is calculated using a pre-trained text encoder.

[0020] A knowledge base for the corresponding category is constructed based on text descriptions with a cosine similarity higher than a preset threshold.

[0021] In a preferred embodiment, generating a text description corresponding to the multimodal data using a multimodal large language model includes:

[0022] For image, event, and thermal imaging modal data, text descriptions are generated using the visual-language alignment module of the InstructBLIP model, through a frozen visual encoder and a lightweight Querying Transformer.

[0023] For audio, video, and point cloud modal data, non-textual modal features are mapped to the language model embedding space through the cross-modal adaptation layer of the LLaMA-adapter model to generate text descriptions.

[0024] In a preferred embodiment, before generating the text description corresponding to the multimodal data using a multimodal large language model based on a preset prompt word template, the multimodal data needs to be preprocessed, including:

[0025] Image, video, and thermal imaging data are normalized and spatially aligned to ensure that the input resolution is consistent with the requirements of a multimodal large language model;

[0026] The audio data is converted into a Mel spectrogram and time-sliced ​​to generate fixed-length spectrum segments, ensuring consistency of the input dimensions;

[0027] The point cloud data is subjected to dimensionality reduction and feature extraction using voxelization or farthest point sampling (FPS) algorithms.

[0028] In a preferred embodiment, the method further includes:

[0029] Based on the preset prompt word template, the multimodal large language model generates negative sample descriptions that are semantically unrelated to the category label.

[0030] The negative sample descriptions are compared and learned with the multimodal data to enhance the discriminative power between different categories in the embedding space;

[0031] The negative sample descriptions and their corresponding multimodal data features are stored in the knowledge base to enhance the class boundary discriminability of the embedding space.

[0032] According to a second aspect of this application, a knowledge base construction apparatus based on a multimodal large language model is provided, comprising:

[0033] A multimodal data acquisition module is used to acquire multimodal data and its corresponding category labels, wherein the multimodal data includes at least one of images, videos, audio, point clouds, thermal imaging, events, and text;

[0034] The text description generation module is used to generate a text description corresponding to the multimodal data based on a preset prompt word template and a multimodal large language model. The multimodal large language model includes the InstructBLIP model and the LLaMA-adapter model. The text description corresponding to the multimodal data is generated using the following formula:

[0035]

[0036] in, Represents multimodal data input. It is a text description corresponding to multimodal data. This represents the generator function of a multimodal large language model. The training is performed based on a pre-trained multimodal encoder, which includes the frozen visual encoder of the InstructBLIP model and the base encoder corresponding to the cross-modal adaptation layer of the LLaMA-adapter model.

[0037] The training process of the generator function of the multimodal large language model includes:

[0038] Cross-modal embedding space alignment: Based on the pre-trained multimodal encoder, the multimodal data and its corresponding text description are input into the contrastive learning framework, and the embedding space of the multimodal data is aligned by optimizing the contrastive learning loss function;

[0039] Parameter freezing and adaptation layer training: Freeze the main parameters of the pre-trained visual encoder and text encoder, and train only the lightweight adaptation layer consisting of cross-modal attention modules or query transformers to minimize the cross-entropy loss between text descriptions and real category labels;

[0040] Joint optimization: The total loss function is determined by weighting the contrastive learning loss and cross-entropy loss, and the adaptation layer parameters are jointly optimized through the backpropagation algorithm until the model converges.

[0041] The step of optimizing the contrastive learning loss function to align the embedding space of the multimodal data includes:

[0042] The input multimodal data is mapped to intermediate feature representations based on the visual encoder and text encoder;

[0043] By projecting intermediate features of each modality onto a shared semantic space through a parameterized mapping function, a semantic representation that can be directly compared is formed.

[0044] Positive and negative sample pairs are constructed based on data association relationships. Positive sample pairs include semantically consistent cross-modal data, while negative sample pairs contain semantically inconsistent cross-modal data.

[0045] By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, the contrastive learning loss function is optimized to achieve multimodal embedding space alignment.

[0046] A similarity calculation unit is used to calculate the cosine similarity between the generated text description and the corresponding category label using a pre-trained text encoder;

[0047] The knowledge base construction unit is used to construct a knowledge base for the corresponding category based on text descriptions with a cosine similarity higher than a preset threshold.

[0048] According to a third aspect of this application, an electronic device is provided, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the knowledge base construction method based on a multimodal large language model as described above.

[0049] Compared with the prior art, this application achieves the following beneficial effects:

[0050] (1) This application uses a contrastive learning framework to map multimodal data and its text description to a shared semantic space, thereby achieving semantic alignment between heterogeneous modalities such as images, audio, and point clouds and text, and solving the problem of semantic fragmentation between modalities in traditional methods.

[0051] (2) This application introduces a positive and negative sample pair optimization mechanism, which significantly improves the discriminativeness of multimodal embedding by maximizing the similarity of positive samples and minimizing the similarity of negative samples, making similar samples more compact in the embedding space and dissimilar samples more separated.

[0052] (3) In this application, the main parameters of the pre-trained encoder are frozen during model training, and only lightweight adaptation layers such as cross-modal attention modules or query transformers are trained, which greatly reduces the amount of computation while retaining the generalization ability of the trained model.

[0053] It should be understood that the description in the Summary Section is not intended to limit the key or essential features of the embodiments of this application, nor is it intended to restrict the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0054] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of the application. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0055] Figure 1A flowchart illustrating the knowledge base construction method based on a multimodal large language model according to an embodiment of this application is shown.

[0056] Figure 2 A flowchart illustrating the knowledge base construction process of an embodiment of this application is shown;

[0057] Figure 3 A schematic diagram illustrating description generation based on the InstructBLIP model of this application is shown;

[0058] Figure 4 A schematic diagram of description generation based on LLaMA-adapter according to an embodiment of this application is shown;

[0059] Figure 5 A block diagram of a knowledge base construction apparatus based on a multimodal large language model according to an embodiment of this application is shown;

[0060] Figure 6 A schematic diagram of an exemplary electronic device according to an embodiment of this application is shown. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0063] like Figure 1 The diagram shows a flowchart of the knowledge base construction method based on a multimodal large language model according to this application. The method 100 includes:

[0064] S110: Obtain multimodal data and its corresponding category labels, wherein the multimodal data includes at least one of images, videos, audio, point clouds, thermal imaging, events, and text.

[0065] In some embodiments, multimodal data generally originates from public datasets or custom acquisition devices (such as cameras, LiDAR, etc.). Public datasets can directly provide standardized category labels, while data acquired by custom acquisition devices can generate labels through manual annotation, automated pre-annotation, or cross-modal association, ensuring a strict correspondence between labels and data content. For example:

[0066] Image / Video: Acquired through optical imaging devices (such as RGB cameras), and extracted as image frames or continuous video streams at a preset resolution (such as 1080p);

[0067] Point cloud: The scene is scanned using a 3D sensing device (such as LiDAR) at a set sampling frequency (such as 10Hz) to obtain 3D spatial coordinates and reflection intensity information;

[0068] Thermal imaging: Acquiring thermal radiation distribution maps in specific wavelength bands (e.g., 8-14μm) using infrared sensing devices (such as thermal imagers);

[0069] Event data: A stream of pixel-level brightness change events is recorded using a dynamic vision sensor (DVS).

[0070] Furthermore, to ensure compatibility of the input format with subsequent models, the acquired multimodal data also needs to be preprocessed, including:

[0071] Image, video, and thermal imaging data are normalized and spatially aligned to ensure that the input resolution is consistent with the requirements of a multimodal large language model;

[0072] The audio data is converted into a Mel spectrogram and time-sliced ​​to generate fixed-length spectrum segments, ensuring consistency of the input dimensions;

[0073] The point cloud data is subjected to dimensionality reduction and feature extraction using voxelization or farthest point sampling (FPS) algorithms.

[0074] The impulse noise in the event data or the sensor noise in the image is smoothed using Gaussian filtering or median filtering.

[0075] S120: Based on a preset prompt word template, generate a text description corresponding to the multimodal data through a multimodal large language model.

[0076] In some embodiments, traditional methods directly align multimodal features through text embedding of category names, but they still face two major bottlenecks:

[0077] Semantic uniformity: A single category name is difficult to cover the diversity within a category (e.g., "laptop" may be associated with fine-grained attributes such as "keyboard" and "display").

[0078] Modal bias: Text-driven alignment relies too heavily on language priors, resulting in the loss of fine-grained semantics in visual, auditory, and other modalities.

[0079] To solve this problem, such as Figure 2 As shown, this application first establishes a prompt word template ("Generate adescription of this {modal type} that is related to the {Category}."), then uses a multimodal large language model to generate a text description corresponding to the multimodal data, and finally constructs a knowledge base for the corresponding category based on the cosine similarity between the text description and the category label.

[0080] The multimodal large language model includes the InstructBLIP model and the LLaMA-adapter model. The text descriptions corresponding to the multimodal data are generated using the following formula:

[0081]

[0082] in, Represents multimodal data input. It is a text description corresponding to multimodal data. This represents the generation function of a multimodal large language model.

[0083] In some embodiments, generating a text description corresponding to the multimodal data using a multimodal large language model includes:

[0084] For image, event, and thermal imaging modal data, the InstructBLIP model is used to generate multimodal data descriptions. Specifically, for event and thermal imaging data, key visual features (such as edges and contours) of their paired RGB images can be extracted. These visual features and their corresponding category labels are then input into the InstructBLIP model. A cross-modal attention mechanism aligns the image features with linguistic semantics to generate fine-grained text descriptions. An exemplary description of the generation process is as follows: Figure 3 As shown.

[0085] For audio, video, and point cloud modal data, the LLaMA-adapter model is used to generate descriptions for multimodal data. Specifically, feature encoding is performed on the audio, video, or point cloud modal data to extract key modal features (such as point cloud geometry). These key modal features, along with their corresponding category labels, are then input into the LLaMA-adapter model to generate context-sensitive semantic descriptions. An exemplary description generation process is as follows: Figure 4 As shown.

[0086] In some embodiments, the training process of the above-mentioned multimodal large language models (MLLMs) includes:

[0087] Cross-modal embedding space alignment: Based on a pre-trained multimodal encoder, the multimodal data and its corresponding text description are input into a contrastive learning framework, and the embedding space of the multimodal data is aligned by optimizing the contrastive learning loss function;

[0088] Parameter freezing and adaptation layer training: Freeze the main parameters of the pre-trained visual encoder and text encoder, and train only the lightweight adaptation layer consisting of cross-modal attention modules or query transformers to minimize the cross-entropy loss between text descriptions and real category labels;

[0089] Joint optimization: The total loss function is determined by weighting the contrastive learning loss and cross-entropy loss, and the adaptation layer parameters are jointly optimized through the backpropagation algorithm until the model converges.

[0090] In some embodiments, optimizing the contrastive learning loss function to align the embedding space of the multimodal data includes:

[0091] The input multimodal data is mapped to intermediate feature representations based on the visual encoder and text encoder;

[0092] By projecting intermediate features of each modality onto a shared semantic space through a parameterized mapping function, a semantic representation that can be directly compared is formed.

[0093] Positive and negative sample pairs are constructed based on data association relationships. Positive sample pairs include semantically consistent cross-modal data, while negative sample pairs contain semantically inconsistent cross-modal data.

[0094] By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, the contrastive learning loss function is optimized to achieve multimodal embedding space alignment.

[0095] S130: Calculate the cosine similarity between the generated text description and the corresponding category label using a pre-trained text encoder.

[0096] In some embodiments, the generated text description (such as “a white car turns left at an intersection”) and the corresponding category label (such as “car”) are input into a pre-trained text encoder (such as CLIP’s text encoder or BERT) to generate embedding vectors for the text description and category labels, respectively.

[0097] Further, the cosine similarity between the embedding vector of the text description and the embedding vector of the category label is calculated. :

[0098] ;

[0099] in, , These represent the embedding vectors for the text description and the category label, respectively.

[0100] S140: Construct a knowledge base for the corresponding category based on the text descriptions with a cosine similarity higher than a preset threshold.

[0101] In some embodiments, a similarity threshold (e.g., 0.7) is set, and text descriptions with similarity values ​​above the threshold are filtered and stored in a knowledge base according to category.

[0102] In some embodiments, to ensure that the text description is semantically consistent with the original multimodal data, the method further includes:

[0103] Feature vectors are extracted from multimodal data, and image-text similarity scores are calculated using CLIP's visual-text matching module. Set a threshold for image-text similarity (e.g., 0.6), and remove descriptions with scores below this threshold to ensure that the text descriptions are semantically consistent with the original data.

[0104] In some embodiments, to enhance the class boundary discriminativeness of the embedding space, the method further includes:

[0105] Based on a preset prompt word template, a negative sample description that is semantically independent of the category label is generated through a multimodal large language model; the negative sample description is compared and learned with the multimodal data to enhance the distinguishability between different categories in the embedding space; and the negative sample description and its corresponding multimodal data features are stored in the knowledge base to enhance the category boundary discriminability of the embedding space.

[0106] According to the above embodiments of this application, the following technical effects are achieved:

[0107] (1) This application uses a contrastive learning framework to map multimodal data and its text description to a shared semantic space, thereby achieving semantic alignment between heterogeneous modalities such as images, audio, and point clouds and text, and solving the problem of semantic fragmentation between modalities in traditional methods.

[0108] (2) This application introduces a positive and negative sample pair optimization mechanism, which significantly improves the discriminativeness of multimodal embedding by maximizing the similarity of positive samples and minimizing the similarity of negative samples, making similar samples more compact in the embedding space and dissimilar samples more separated.

[0109] (3) In this application, the main parameters of the pre-trained encoder are frozen during model training, and only lightweight adaptation layers such as cross-modal attention modules or query transformers are trained, which greatly reduces the amount of computation while retaining the generalization ability of the trained model.

[0110] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0111] The above is an introduction to the method embodiments. The following describes the solution described in this application through device embodiments.

[0112] Figure 5 A block diagram of a knowledge base construction apparatus based on a multimodal large language model according to an embodiment of this application is shown. Figure 5 As shown, device 50 includes:

[0113] The multimodal data acquisition module 510 is used to acquire multimodal data and its corresponding category labels, wherein the multimodal data includes at least one of images, videos, audio, point clouds, thermal imaging, events, and text;

[0114] The text description generation module 520 is used to generate a text description corresponding to the multimodal data based on a preset prompt word template and a multimodal large language model.

[0115] The similarity calculation unit 530 is used to calculate the cosine similarity between the generated text description and the corresponding category label through a pre-trained text encoder;

[0116] The knowledge base construction unit 540 is used to construct a knowledge base for a corresponding category based on text descriptions with a cosine similarity higher than a preset threshold.

[0117] The acquisition, storage, and application of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0118] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0119] Figure 6A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0120] Electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in ROM 602 or a computer program loaded into RAM 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. I / O interface 605 is also connected to bus 604.

[0121] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0122] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0123] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0125] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0127] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0128] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0129] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0130] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for constructing a knowledge base based on a multi-modal large language model, characterized in that, The method comprises the steps of: acquiring multi-modal data and its corresponding category label, wherein the multi-modal data comprises at least one of image, video, audio, point cloud, thermal imaging, event and text, wherein the event refers to a pixel-level brightness change event collected by a dynamic visual sensor; generating a text description corresponding to the multi-modal data based on a preset prompt word template through a multi-modal large language model, wherein the multi-modal large language model comprises an InstructBLIP model and an LLaMA-adapter model, and the text description corresponding to the multi-modal data is generated through the following formula: ； wherein, represents a multi-modal data input, is a text description corresponding to the multi-modal data, denotes a generation function of a multi-modal large language model, the generation function is trained according to a pre-trained multi-modal encoder, the pre-trained multi-modal encoder comprising a frozen visual encoder of the InstructBLIP model and a base encoder corresponding to a cross-modal adaptation layer of the LLaMA-adapter model; the training process of the generation function of the multi-modal large language model comprises: inputting the multi-modal data and its corresponding text description into a contrast learning framework based on the pre-trained multi-modal encoder, and aligning the embedding space of the multi-modal data by optimizing the contrast learning loss function; freezing the main parameters of the pre-trained visual encoder and text encoder, only training the lightweight adapter layer composed of cross-modal attention module or query transformer, and minimizing the cross-entropy loss between the text description and the real category label; determining the total loss function by weighting the contrast learning loss and the cross-entropy loss, and jointly optimizing the adapter layer parameters through the back propagation algorithm until the model converges; the contrast learning loss function comprises: mapping the input multi-modal data to intermediate feature representation; projecting the intermediate features of each modality to a shared semantic space through a parameterized mapping function to form a direct comparison semantic representation; constructing positive and negative sample pairs based on data association, wherein the positive sample pair comprises cross-modal data with consistent semantics, and the negative sample pair comprises cross-modal data with inconsistent semantics; optimizing the contrast learning loss function by maximizing the similarity of the positive sample pair and minimizing the similarity of the negative sample pair to realize the alignment of the multi-modal embedding space; calculating the cosine similarity between the generated text description and the corresponding category label through the text encoder of CLIP or BERT; constructing a knowledge base of the corresponding category according to the text description with the cosine similarity higher than a preset threshold; the multi-modal large language model comprises: for image, event and thermal imaging modal data, generating a text description through the frozen visual encoder of the InstructBLIP model and the lightweight Querying Transformer; for audio, video and point cloud modal data, mapping non-text modal features to the language model embedding space through the cross-modal adapter layer of the LLaMA-adapter model to generate a text description.

2. The method of claim 1, wherein, before generating the text description corresponding to the multi-modal data based on the preset prompt word template through the multi-modal large language model, the multi-modal data needs to be preprocessed, which comprises: standardizing and spatially aligning the image, video and thermal imaging data to ensure that the input resolution is consistent with the requirements of the multi-modal large language model; converting the audio data into a mel spectrum graph and performing time slicing to generate fixed-length spectral segments, ensuring input dimension consistency; The point cloud data is reduced in dimension and features are extracted by using a voxelization or farthest point sampling algorithm.

3. The method of claim 1, wherein, The method further comprises: Based on the preset prompt word template, the multi-modal large language model generates negative sample descriptions irrelevant to the category label semantics; The negative sample descriptions are compared with the multi-modal data for contrast learning to enhance the distinction between different categories in the embedding space; The negative sample descriptions and their corresponding multi-modal data features are stored in the knowledge base to enhance the category boundary discriminability of the embedding space.

4. The knowledge base construction apparatus based on a multi-modal large language model, characterized by, It comprises: A multi-modal data acquisition module is configured to acquire multi-modal data and its corresponding category labels, wherein the multi-modal data comprises at least one of images, videos, audio, point clouds, thermal images, events, and text, wherein events refer to pixel-level brightness change events collected by dynamic visual sensors; A text description generation module is configured to generate text descriptions corresponding to the multi-modal data based on a preset prompt word template through a multi-modal large language model; The multi-modal large language model comprises an InstructBLIP model and an LLaMA-adapter model, and the generation of text descriptions corresponding to the multi-modal data through the multi-modal large language model comprises: For image, event, and thermal image modal data, the InstructBLIP model's frozen visual encoder and lightweight Querying Transformer are used to generate text descriptions; For audio, video, and point cloud modal data, the LLaMA-adapter model's cross-modal adaptation layer is used to map non-text modal features to the language model embedding space to generate text descriptions; The text description corresponding to the multi-modal data is generated by the following formula: ； wherein, represents a multi-modal data input, is a text description corresponding to the multi-modal data, denotes a generation function of a multi-modal large language model, the generation function is trained according to a pre-trained multi-modal encoder, the pre-trained multi-modal encoder comprising a frozen visual encoder of the InstructBLIP model and a base encoder corresponding to a cross-modal adaptation layer of the LLaMA-adapter model; The training process of the generation function of the multi-modal large language model comprises: Based on the pre-trained multi-modal encoder, the multi-modal data and its corresponding text description are input into a contrast learning framework to align the embedding space of the multi-modal data by optimizing the contrast learning loss function; The main parameters of the pre-trained visual encoder and text encoder are frozen, and only the lightweight adaptation layer composed of the cross-modal attention module or the Querying Transformer is trained to minimize the cross-entropy loss between the text description and the true category label; The contrast learning loss and the cross-entropy loss are weighted to determine the total loss function, and the adaptation layer parameters are jointly optimized by the backpropagation algorithm until the model converges; The contrast learning loss function is optimized to align the embedding space of the multi-modal data, comprising: Map the input multi-modal data to intermediate feature representations; Project each modal intermediate feature to a shared semantic space through a parameterized mapping function to form a direct comparison of semantic representations; Based on the data association relationship, construct positive and negative sample pairs, wherein the positive sample pairs include semantically consistent cross-modal data, and the negative sample pairs include semantically inconsistent cross-modal data; Optimize the contrast learning loss function by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs to achieve multi-modal embedding space alignment. The similarity calculation module is configured to calculate a cosine similarity between the generated text description and the corresponding category label by using a text encoder of CLIP or BERT. The knowledge base construction module is configured to construct a knowledge base of the corresponding category according to the text description whose cosine similarity is higher than a preset threshold.

5. An electronic device, comprising: The electronic device comprises: at least one processor, and a memory connected to the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method in any one of claims 1-3.

Images