Multi-modal entity recognition method based on large language model

Abstract

The invention discloses a multi-modal entity recognition method based on a large language model. The multi-modal entity recognition method comprises the following steps: S1, constructing a multi-modal input data set; s2, determining and applying preprocessing methods of different modal input data through a self-adaptive modal interaction framework; s3, inputting the preprocessed multi-modal input data set into a large language model with a modal perception mechanism; s4, generating an optimized fusion feature representation; s5, performing cross-modal entity recognition based on the fusion feature representation; s6, automatically matching the identified entity with an existing knowledge graph through a dynamic mapping algorithm, and if the entity is a new entity, automatically generating a new node and updating an existing knowledge graph node relationship according to a self-adaptive modal interaction result; and S7, by continuously inputting new multi-modal data, the system automatically adjusts and optimizes the modal perception mechanism and the adaptive modal interaction framework of the large language model, and dynamically expands the knowledge graph. According to the invention, the accuracy of complex entity identification is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal entity recognition, and in particular to a multimodal entity recognition method based on a large language model. Background Art

[0002] In the prior art, multimodal entity recognition mainly relies on the independent processing of single modal data, and performs entity recognition in each modality separately through traditional machine learning or deep learning methods. The single modal data method can achieve certain recognition effects under a single modality. However, when faced with complex scenarios containing information from multiple modalities, the prior art has significant limitations in the comprehensive processing capabilities of data fusion and cross-modal information. Since data from different modalities are usually processed independently in their own ways, this fragmented approach leads to the dispersion of information and the inability to fully utilize the complementarity between multiple modalities, ultimately resulting in low entity recognition accuracy.

[0003] In addition, existing multimodal processing technologies also show shortcomings when dealing with complex contexts and ambiguous information. When processing the fusion of multimodal data such as text, images, and audio, they are unable to effectively integrate data from different modalities. When there is information missing or misalignment between modalities, the identified entities are not accurate enough and the relationship reasoning is not precise enough, which seriously affects the actual application effect.

[0004] Existing technologies also have certain bottlenecks in the updating and maintenance of knowledge graphs. Traditional knowledge graph construction and updating mostly rely on manual or semi-automatic methods, which are not only inefficient but also unable to timely reflect the latest entity information and its relationships. In the scenario of growing multimodal data, the complexity of manually updating knowledge graphs has increased significantly, making it difficult to meet the needs of real-time expansion and dynamic updating.

[0005] To sum up, the existing technologies have the following main defects: first, the accuracy of cross-modal entity recognition is not high, and the complementarity of multimodal data cannot be fully utilized; second, the existing multimodal data fusion methods are insufficient when processing complex contexts and fuzzy information, resulting in poor entity recognition and relationship reasoning effects; finally, the existing knowledge graph update method is inefficient and cannot meet the real-time dynamic update requirements in multimodal data scenarios. Summary of the invention

[0006] One object of the present invention is to propose a multimodal entity recognition method based on a large language model, which improves the accuracy of complex entity recognition.

[0007] A multimodal entity recognition method based on a large language model according to an embodiment of the present invention comprises the following steps:

[0008] S1, receiving text data, image data and audio data, and constructing a multimodal input data set;

[0009] S2, determining and applying preprocessing methods for input data of different modalities through an adaptive modal interaction framework to preprocess the multimodal input data set;

[0010] S3, input the preprocessed multimodal input dataset into a large language model with a modality-aware mechanism, and the large language model is used to generate a comprehensive semantic representation containing multimodal context understanding, combining the semantic information of text, image and audio;

[0011] S4. Adaptively fuse the comprehensive semantic representation generated by the large language model with other modal features using an adaptive modal interaction framework, and generate an optimized fused feature representation by dynamically adjusting the interaction weights of different modalities.

[0012] S5. Perform cross-modal entity recognition based on fused feature representation. Cross-modal entity recognition includes using modality-aware mechanisms to identify complex entities and relationships in text and other modal information, and generating corresponding classification labels and relationship mappings for entities.

[0013] S6. Automatically match the identified entity with the existing knowledge graph through a dynamic mapping algorithm. If the entity is a new entity, automatically generate a new node and update the existing knowledge graph node relationship based on the result of the adaptive modal interaction;

[0014] S7. By continuously inputting new multimodal data, the system automatically adjusts and optimizes the modal perception mechanism and adaptive modal interaction framework of the large language model, continuously learns new knowledge, and dynamically expands the knowledge graph.

[0015] Optionally, the S1 step includes:

[0016] S11, receiving a text dataset, an image dataset, and an audio dataset;

[0017] S12. For the text dataset D t , each text sample x t ∈D t A set of sentences {s1,s2,…,s n}, where s n is the nth sentence in the text sample;

[0018] S13. For image dataset D i , each image sample x i ∈D i A pixel matrix M = {p 11 ,p 12 ,…,p mn} means that pmn Represents the pixel value of the mth row and nth column in the image;

[0019] S14. For audio dataset D a , each audio sample x a ∈D a A set of frequency domain feature vectors F = {f1, f2, ..., f k} means that f k is the kth frequency component of the audio signal;

[0020] S15, the text data set D t , image dataset D i And audio dataset D a Combine into a multimodal input dataset:

[0021] D = {x t ,x i ,x a}.

[0022] Optionally, the S2 step includes:

[0023] S21, through the adaptive modal interaction framework, the multimodal input data set D = {x t ,x i ,x a}Perform modality identification and determine the type and data structure of each modality;

[0024] S22. According to the characteristics of each modality, different preprocessing methods are applied, including the text dataset D t Perform word segmentation and denoising on the image dataset D i After standardization and size normalization, the audio dataset D a Perform denoising and frequency domain conversion processing;

[0025] S23, the preprocessed text dataset P t (x t ), image dataset P i (x i ) and audio dataset P a (x a ) are combined into a preprocessed multimodal input dataset:

[0026] P D = {P t (x t ),P i (x i ),P a (x a )}.

[0027] Optionally, the S3 step includes:

[0028] S31, the preprocessed multimodal input data set P D Input into a large language model with modality awareness mechanism;

[0029] S32. For text data P t (x t ) Generate text semantic representation through the text embedding layer of the large language model:

[0030]

[0031] Among them, E t (x t ) represents text data x t The semantic representation vector of j is the jth word in the text; φ(w j ) is the word embedding obtained by the pre-trained embedding matrix, α j is the attention weight, γ j is the context vector;

[0032] S33, for the image data P i (x i ), generate image semantic representation through image embedding layer:

[0033]

[0034] Among them, E i (x i ) represents the image data x i The semantic representation vector of mn is the pixel point of the image, f(p mn ) represents the image features extracted by convolutional neural network; β mn is the weight of the image feature, indicating the visual importance of a specific area, δ mn Encode for position;

[0035] S34. For audio data P a (x a ), generate audio semantic representation through the audio embedding layer:

[0036]

[0037] Among them, E a (x a ) represents audio data x a The semantic representation vector of k is the kth frequency component in the audio signal, ψ(f k ) is the frequency domain feature extracted by Fourier transform; ηk is the feature weight of the audio signal, σ k Encode the time step;

[0038] S35, the generated text semantic representation E t (x t ), image semantic representation E i (x i ) and audio semantic representation E a (x a ) is fused through a modality-aware mechanism to generate a comprehensive semantic representation that includes multimodal context understanding:

[0039] E multi =λ1·g(E t (x t ))+λ2·g(E i (x i ))+λ3·g(E a (x a ))+θ(C);

[0040] Among them, E multi is the multimodal comprehensive semantic representation, g(E) represents the nonlinear transformation function of each modal representation, λ1, λ2, λ3 are the importance weights of each modal feature, and θ(C) is the multimodal context fusion function.

[0041] Optionally, the S4 step includes:

[0042] S41. Comprehensive semantic representations generated by large language models using an adaptive modal interaction framework multi Adaptively fuse the features of each modality, and the adaptive modal interaction framework dynamically adjusts the interaction weights of different modalities;

[0043] S42, through the interaction function h(E multi ,E m ) represents the text semantics E t (x t ), image semantic representation E i (x i ) and audio semantic representation E a (x a ) for feature fusion:

[0044]

[0045] in, is the feature interaction function of the pth row and qth column in the fusion process, combined with the comprehensive semantic representation E multi With modal characteristics E m The local interaction of pqis the local attention weight, β pq is the weight of the feature position, combined with the position encoding of the modal features to optimize the fused spatial feature representation;

[0046] S43. Through the interaction mechanism between modalities, the weights of different modalities are dynamically adjusted according to the importance of each modality. m :

[0047]

[0048] Among them, μ m is the basic weight of mode m, is the multiple interaction function between modes, ζ i is the dynamic adjustment parameter related to the i-th feature dimension;

[0049] S44. Generate optimized fusion feature representation F opt , the fusion feature representation optimization process is achieved through the nonlinear transformation function g(F fusion )conduct:

[0050]

[0051] Among them, F opt represents the final optimized fusion feature representation, γ r is the weight of the rth nonlinear transformation channel, is the nonlinear activation function of the fusion feature representation, δ r is the dynamic weight of each channel.

[0052] Optionally, the step S5 includes:

[0053] S51, based on fusion feature representation F opt Recognize complex entities and relationships in text, image, and audio data through modality-aware mechanisms;

[0054] S52. The explicit and implicit entities in the text modality are identified through the modality perception mechanism. The identification of the implicit entity is expressed as:

[0055]

[0056] Among them, E implicit is an implicit entity representation; w j is the jth word in the text; δ j represents the implicit weight of the current word; c j is the context information of the word; ξ(w j ,∑λ k ·g(φ k (c j ,h j))) is the comprehensive relationship function between words and context, where λ k is the implicit weight adjustment parameter, φ k (c j ,h j ) indicates context c j and the hidden layer state h j interaction function; g is the activation function for nonlinear transformation, ∈ k is random noise;

[0057] S53. For image and audio modalities, identify the relationship between entities through modality perception mechanism:

[0058]

[0059] Among them, R relation is the relationship mapping between entities; pq is the relationship weight between modalities; is the mutual influence between entities of mode p and mode q; α pq is the local interaction weight of different modal features; is the local feature interaction function in mode p and mode q, z∈Ω is the integration domain, representing the feature space of modes p and q;

[0060] S54, generating corresponding classification labels C for the recognized implicit entities in the text modality and the entities in the image and audio modalities entity :

[0061]

[0062] Among them, C entity is the classification label of the entity; W c and b c are the weight matrix and bias term of the classifier respectively; γ l and r are the weight coefficients of entity features and relationship mapping respectively; tanh represents the activation function, which is used to generate the final classification result after fusing entity features and relationship information; the softmax function is used to map the result into probability distribution.

[0063] Optionally, the step S6 includes:

[0064] S61, the identified entity E entity Through the dynamic mapping algorithm and the existing knowledge graph G = (V, R) automatic matching, entity E entity The multimodal features of are compared with the knowledge graph node v through modal similarity calculation. i ∈V for matching;

[0065] S62: If the identified entity Eentity Failed to match existing node v i , then according to the result of adaptive modal interaction, a new node v is automatically generated new , the new node is represented as:

[0066]

[0067] Among them, v new is the new node generated, γ l is the weight of the feature dimension, φ l Represents the nonlinear activation function for generating new nodes combined with entity features E entity and classification label C entity The interaction, α k is the weight of the classification label, is the transformation function of the multimodal feature, y∈Θ is the offset in the feature space;

[0068] S63. Update the relationship R between entities and generate a new entity v through relationship mapping new and the existing entity node v i The relationship between:

[0069]

[0070] Among them, r(v new ,v i ) is the generated new relation mapping, β j is the weight of the relationship mapping, ρ j represents the feature interaction function between entities and nodes, γ j (z) represents the position shift function of the local feature, and z∈Ω is the feature space.

[0071] S64. Automatically update the knowledge graph to generate an expanded knowledge graph G'=(V',R'):

[0072]

[0073] Among them, G' is the expanded knowledge graph, η n is the weight of the new relationship, ζ n The nonlinear transformation function representing the relationship updates the nodes and relationships of the knowledge graph by combining the relationship mapping between new nodes and existing nodes. dz is the integral operation in the feature space.

[0074] The beneficial effects of the present invention are:

[0075] (1) The present invention dynamically adjusts the information interaction weights between different modalities and fully utilizes the complementarity of text, image and audio multimodal data to achieve more accurate feature fusion. Under the adaptive modal interaction framework, the weights between modalities are adaptively adjusted according to different tasks and data characteristics, thereby improving the understanding and recognition capabilities of cross-modal data. Compared with traditional single modality or simple fusion methods, the present invention significantly improves the accuracy and robustness of entity recognition in complex scenarios, especially showing stronger adaptability when processing multimodal data containing noise or incomplete information.

[0076] (2) The present invention can simultaneously process and understand data from different modalities by introducing the modality perception mechanism of the large language model, thereby realizing the semantic fusion of multimodal contexts. While processing text data, the large language model can combine the information in images and audio to generate a cross-modal comprehensive semantic representation. It can not only recognize explicit entity information, but also infer implicit entities and their relationships through context, thereby improving the accuracy of complex entity recognition.

[0077] (3) The present invention combines the multimodal entity recognition results with the dynamic knowledge graph, automatically generates new entity nodes and updates the relationships between entities. New entities identified by the dynamic mapping algorithm can be added to the existing knowledge graph in real time, and the relationship mapping between nodes is automatically generated based on the relationship information between the modalities, effectively coping with the ever-changing entities and relationships in multimodal data, ensuring that the knowledge graph is always in the latest state, and significantly improving the dynamic adaptability of the system and the integrity of the knowledge graph. Compared with traditional manual or semi-automatic update methods, the dynamic update mechanism of the present invention greatly improves efficiency and can timely reflect the latest entity and relationship structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0079] Figure 1 This is a flowchart of a multimodal entity recognition method based on a large language model proposed by the present invention. DETAILED DESCRIPTION

[0080] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0081] refer to Figure 1 , a multimodal entity recognition method based on a large language model, comprising the following steps:

[0082] S1, receiving text data, image data and audio data, and constructing a multimodal input data set;

[0083] S2, determining and applying preprocessing methods for input data of different modalities through an adaptive modal interaction framework to preprocess the multimodal input data set;

[0084] S3, input the preprocessed multimodal input dataset into a large language model with a modality-aware mechanism, and the large language model is used to generate a comprehensive semantic representation containing multimodal context understanding, combining the semantic information of text, image and audio;

[0085] S4. Adaptively fuse the comprehensive semantic representation generated by the large language model with other modal features using an adaptive modal interaction framework, and generate an optimized fused feature representation by dynamically adjusting the interaction weights of different modalities.

[0086] S5. Perform cross-modal entity recognition based on fused feature representation. Cross-modal entity recognition includes using modality-aware mechanisms to identify complex entities and relationships in text and other modal information, and generating corresponding classification labels and relationship mappings for entities.

[0087] S6. Automatically match the identified entity with the existing knowledge graph through a dynamic mapping algorithm. If the entity is a new entity, automatically generate a new node and update the existing knowledge graph node relationship based on the result of the adaptive modal interaction;

[0088] S7. By continuously inputting new multimodal data, the system automatically adjusts and optimizes the modal perception mechanism and adaptive modal interaction framework of the large language model, continuously learns new knowledge, and dynamically expands the knowledge graph.

[0089] In this implementation, step S1 includes:

[0090] S11, receiving a text dataset, an image dataset, and an audio dataset;

[0091] S12. For the text dataset D t , each text sample x t ∈D t A set of sentences {s1,s2,…,s n}, where s n is the nth sentence in the text sample;

[0092] S13. For image dataset D i , each image sample x i ∈D i A pixel matrix M = {p 11 ,p 12 ,…,p mn} means that pmn Represents the pixel value of the mth row and nth column in the image;

[0093] S14. For audio dataset D a , each audio sample x a ∈D a A set of frequency domain feature vectors F = {f1, f2, ..., f k} means that f k is the kth frequency component of the audio signal;

[0094] S15. The text dataset D t , image dataset D i And audio dataset D a Combine into a multimodal input dataset:

[0095] D = {x t ,x i ,x a}.

[0096] In this embodiment, step S2 includes:

[0097] S21, through the adaptive modal interaction framework, the multimodal input data set D = {x t ,x i ,x a}Perform modality identification and determine the type and data structure of each modality;

[0098] S22. According to the characteristics of each modality, different preprocessing methods are applied, including the text dataset D t Perform word segmentation and denoising on the image dataset D i After standardization and size normalization, the audio dataset D a Perform denoising and frequency domain conversion processing;

[0099] S23, the preprocessed text dataset P t (x t ), image dataset P i (x i ) and audio dataset P a (x a ) are combined into a preprocessed multimodal input dataset:

[0100] P D = {P t (x t ),P i (x i ),P a (x a )}.

[0101] In this implementation, step S3 includes:

[0102] S31, the preprocessed multimodal input data set P D Input into a large language model with modality awareness mechanism;

[0103] S32. For text data P t (x t ) Generate text semantic representation through the text embedding layer of the large language model:

[0104]

[0105] Among them, E t (x t ) represents text data x t The semantic representation vector of j is the jth word in the text; φ(w j ) is the word embedding obtained by the pre-trained embedding matrix, α j is the attention weight, γ j is the context vector;

[0106] S33, for the image data P i (x i ), generate image semantic representation through image embedding layer:

[0107]

[0108] Among them, E i (x i ) represents the image data x i The semantic representation vector of mn is the pixel of the image, f(p mn ) represents the image features extracted by convolutional neural network; β mn is the weight of the image feature, indicating the visual importance of a specific area, δ mn Encode for position;

[0109] S34. For audio data P a (x a ), generate audio semantic representation through the audio embedding layer:

[0110]

[0111] Among them, E a (x a ) represents audio data x a The semantic representation vector of k is the kth frequency component in the audio signal, ψ(f k ) is the frequency domain feature extracted by Fourier transform; ηk is the feature weight of the audio signal, σ k Encode the time step;

[0112] S35, the generated text semantic representation E t (x t ), image semantic representation E i (x i ) and audio semantic representation E a (x a ) is fused through a modality-aware mechanism to generate a comprehensive semantic representation that includes multimodal context understanding:

[0113] E multi =λ1·g(E t (x t ))+λ2·g(E i (x i ))+λ3·g(E a (x a ))+θ(C);

[0114] Among them, E multi is the multimodal comprehensive semantic representation, g(E) represents the nonlinear transformation function of each modal representation, λ1, λ2, λ3 are the importance weights of each modal feature, and θ(C) is the multimodal context fusion function.

[0115] In this embodiment, step S4 includes:

[0116] S41. Comprehensive semantic representations generated by large language models using an adaptive modal interaction framework multi Adaptively fuse the features of each modality, and the adaptive modal interaction framework dynamically adjusts the interaction weights of different modalities;

[0117] S42, through the interaction function h(E multi ,E m ) represents the text semantics E t (x t ), image semantic representation E i (x i ) and audio semantic representation E a (x a ) for feature fusion:

[0118]

[0119] in, is the feature interaction function of the pth row and qth column in the fusion process, combined with the comprehensive semantic representation E multi With modal characteristics E m The local interaction of pqis the local attention weight, β pq is the weight of the feature position, combined with the position encoding of the modal features to optimize the fused spatial feature representation;

[0120] S43. Through the interaction mechanism between modalities, the weights of different modalities are dynamically adjusted according to the importance of each modality. m :

[0121]

[0122] Among them, μ m is the basic weight of mode m, is the multiple interaction function between modes, ζ i is the dynamic adjustment parameter related to the i-th feature dimension;

[0123] S44. Generate optimized fusion feature representation F opt , the fusion feature representation optimization process is achieved through the nonlinear transformation function g(F fusion )conduct:

[0124]

[0125] Among them, F opt represents the final optimized fusion feature representation, γ r is the weight of the rth nonlinear transformation channel, is the nonlinear activation function of the fusion feature representation, δ r is the dynamic weight of each channel.

[0126] In this implementation, step S5 includes:

[0127] S51, based on fusion feature representation F opt Recognize complex entities and relationships in text, image, and audio data through modality-aware mechanisms;

[0128] S52. The explicit and implicit entities in the text modality are identified through the modality perception mechanism. The identification of the implicit entity is expressed as:

[0129]

[0130] Among them, E implicit is an implicit entity representation; w j is the jth word in the text; δ j represents the implicit weight of the current word; c j is the context information of the word; ξ(w j ,∑λ k ·g(φ k (c j ,h j))) is the comprehensive relationship function between words and context, where λ k is the implicit weight adjustment parameter, φ k (c j ,h j ) indicates context c j and the hidden layer state h j interaction function; g is the activation function for nonlinear transformation, ∈ k is random noise;

[0131] S53. For image and audio modalities, identify the relationship between entities through modality perception mechanism:

[0132]

[0133] Among them, R relation is the relationship mapping between entities; pq is the relationship weight between modalities; is the mutual influence between entities of mode p and mode q; α pq is the local interaction weight of different modal features; is the local feature interaction function in mode p and mode q, z∈Ω is the integration domain, representing the feature space of modes p and q;

[0134] S54, generating corresponding classification labels C for the recognized implicit entities in the text modality and the entities in the image and audio modalities entity :

[0135]

[0136] Among them, C entity is the classification label of the entity; W c and b c are the weight matrix and bias term of the classifier respectively; γ l and r are the weight coefficients of entity features and relationship mapping respectively; tanh represents the activation function, which is used to generate the final classification result after fusing entity features and relationship information; the softmax function is used to map the result into probability distribution.

[0137] In this implementation, step S6 includes:

[0138] S61, the identified entity E entity Through the dynamic mapping algorithm and the existing knowledge graph G = (V, R) automatic matching, entity E entity The multimodal features of are calculated by modality similarity and compared with the knowledge graph node v i ∈V for matching;

[0139] S62: If the identified entity Eentity Failed to match existing node v i , then according to the result of adaptive modal interaction, a new node v is automatically generated new , the new node is represented as:

[0140]

[0141] Among them, v new is the generated new node, γ l is the weight of the feature dimension, φ l Represents the nonlinear activation function for generating new nodes combined with entity features E entity and classification label C entity The interaction, α k is the weight of the classification label, is the transformation function of the multimodal feature, y∈Θ is the offset in the feature space;

[0142] S63. Update the relationship R between entities and generate a new entity v through relationship mapping new and the existing entity node v i The relationship between:

[0143]

[0144] Among them, r(v new ,v i ) is the generated new relation mapping, β j is the weight of the relationship mapping, ρ j represents the feature interaction function between entities and nodes, γ j (z) represents the position shift function of the local feature, and z∈Ω is the feature space.

[0145] S64. Automatically update the knowledge graph to generate an expanded knowledge graph G'=(V',R'):

[0146]

[0147] Among them, G' is the expanded knowledge graph, η n is the weight of the new relationship, ζ n The nonlinear transformation function representing the relationship updates the nodes and relationships of the knowledge graph by combining the relationship mapping between new nodes and existing nodes. dz is the integral operation in the feature space.

[0148] Embodiment 1:

[0149] In this embodiment 1, a traffic management system of a smart city is used as an application scenario to demonstrate how the present invention effectively identifies and processes entity information in traffic conditions, and automatically updates related entities and their relationships through a knowledge graph, thereby proving the feasibility and effectiveness of the present invention. The traffic management system of a smart city needs to process data from different modalities in real time, including text reports, images captured by traffic cameras, and audio information collected in the monitoring system. The above data together constitute the multimodal input of the system, which aims to timely identify traffic events and related entities (vehicles, pedestrians, and traffic lights), and dynamically update road conditions and traffic flow conditions through a knowledge graph.

[0150] On August 15, 2024, in City A, a bus rear-ended a private car, causing traffic congestion on the road. After the accident, the traffic management system received three types of data through smart devices:

[0151] The first category is audio data sent by the police officer via voice, describing the occurrence of the accident;

[0152] The second category is image data captured by traffic cameras, including live photos of buses and private cars;

[0153] The third category is the road condition monitoring text report, which describes the impact area of ​​the accident and the traffic flow situation.

[0154] The system processes data through the method of the present invention to automatically identify key entities in the accident (vehicles, road sections and accident types), and updates relevant knowledge graphs to optimize road control measures.

[0155] First, the system receives data from three modalities: text, image, and audio. The text data includes a road condition report, which specifically describes the location of the accident (the second ring road of City A, east to west, about 500 meters from the intersection) and the degree of congestion (the length of the congestion is about 3 kilometers, and vehicles are moving slowly); the image data comes from the traffic camera, which captures the specific scene of the collision between the bus and the private car; the audio data comes from the description of the police, which is "the bus and the private car rear-ended each other, and the traffic was blocked on the second ring road of City A, which needs to be dealt with urgently."

[0156] The data are respectively used as multimodal inputs in the method of the present invention and are preprocessed through an adaptive modal interaction framework. In text data, the system first performs word segmentation and denoising on the accident report, extracts key location, time, and event information, and identifies the entities of "bus", "private car", "rear-end collision", and "Second Ring Road of City A". In image data processing, the system uses a convolutional neural network to extract visual features in the image, identifies the collision position and shape features of the bus and the private car, and matches the vehicles in the image with the "bus" and "private car" information in the text through a modal perception mechanism. For audio data, the system converts the descriptive information in the audio into text through frequency domain feature extraction and speech recognition, and associates it with entities in the text and image.

[0157] Next, the system uses the modal perception mechanism of the large language model to combine semantic information from different modalities to generate a comprehensive semantic representation. For the "bus" entity in the text, the system combines the visual features in the image to determine that its model is a "city bus" and recognizes that the accident occurred on the Second Ring Road of City A. At the same time, the "rear-end collision" event in the audio data is also matched with the collision scene in the image through comprehensive semantic analysis. Through the adaptive interaction mechanism between modalities, the system dynamically adjusts the weights of different modalities to ensure that data information from different sources is fully utilized.

[0158] The system then matches the identified "bus", "private car", and "rear-end collision" entities with the existing traffic management knowledge graph through the dynamic mapping algorithm of the present invention. Since the vehicles and events involved in this accident are new entities, the system automatically generates a new node "bus rear-end collision" and establishes a new relationship between it and the "Second Ring Road of City A", "bus", and "private car" entities. The knowledge graph is automatically expanded to generate new node and relationship mappings. The updated knowledge graph can promptly reflect the latest road traffic conditions.

[0159] In order to demonstrate the superiority of the method of the present invention, a large number of experiments were conducted to compare the performance of the method of the present invention with the existing single-modal entity recognition method in different traffic scenarios. The following Table 1 shows the experimental results based on a real traffic dataset, which contains multimodal data of text, image and audio modalities, and evaluates the performance of the two methods in terms of entity recognition accuracy and knowledge graph update speed:

[0160] Table 1 Experimental results based on real traffic dataset

[0161]

[0162] As can be seen from Table 1, the entity recognition accuracy of the method of the present invention reaches 95.8%, which is significantly higher than the existing single-modal methods (78.5%, 80.2% and 76.4% for text, image and audio respectively). In particular, in terms of the fusion and utilization of multimodal data, the method of the present invention reaches 92%, far exceeding the utilization of existing single-modal methods. In addition, the method of the present invention can complete the automatic update of the knowledge graph within 1.2 seconds, which is significantly faster than the 3.5 to 4.0 seconds of the prior art. The data in Table 1 proves the significant advantages of the present invention in improving recognition accuracy, reducing latency and improving the efficiency of knowledge graph update.

[0163] Through the above-mentioned application scenario of smart city traffic management, this Example 1 demonstrates in detail the feasibility and effectiveness of the multimodal entity recognition method based on the large language model of the present invention in practice. By processing multimodal data and combining the semantic understanding ability of the large language model and the adaptive modal interaction framework, the system can achieve efficient and accurate entity recognition and automatically update the knowledge graph through a dynamic mapping algorithm. Through the comparison of experimental data, the method of the present invention is superior to the existing technology in terms of entity recognition accuracy, knowledge graph update speed and multimodal data utilization, and can effectively solve the problems of insufficient cross-modal data fusion and lagging knowledge graph update in the existing technology, and has broad application prospects.

[0164] The present invention achieves more accurate feature fusion by dynamically adjusting the information interaction weights between different modalities and making full use of the complementarity of text, image and audio multimodal data. Under the adaptive modal interaction framework, the weights between modalities are adaptively adjusted according to different tasks and data characteristics, thereby improving the understanding and recognition capabilities of cross-modal data. Compared with traditional single modality or simple fusion methods, the present invention significantly improves the accuracy and robustness of entity recognition in complex scenarios, especially showing stronger adaptability when processing multimodal data containing noise or incomplete information.

[0165] The present invention can simultaneously process and understand data from different modalities by introducing the modality perception mechanism of the large language model, thereby realizing the semantic fusion of multimodal contexts. While processing text data, the large language model can combine the information in images and audio to generate a cross-modal comprehensive semantic representation. It can not only identify explicit entity information, but also infer implicit entities and their relationships through context, thereby improving the accuracy of complex entity recognition.

[0166] The present invention combines the results of multimodal entity recognition with dynamic knowledge graphs, automatically generates new entity nodes and updates the relationships between entities. New entities identified by the dynamic mapping algorithm can be added to the existing knowledge graph in real time, and automatically generate relationship mappings between nodes based on the relationship information between modalities, effectively coping with the ever-changing entities and relationships in multimodal data, ensuring that the knowledge graph is always up to date, significantly improving the system's dynamic adaptability and the integrity of the knowledge graph. Compared with traditional manual or semi-automatic update methods, the dynamic update mechanism of the present invention greatly improves efficiency and can timely reflect the latest entity and relationship structures.

[0167] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multimodal entity recognition method based on a large language model, It is characterized in that The steps include: S1, receiving text data, image data and audio data, and constructing a multimodal input data set; S2, determining and applying preprocessing methods for input data of different modalities through an adaptive modal interaction framework to preprocess the multimodal input data set; S3, input the preprocessed multimodal input dataset into a large language model with a modality-aware mechanism, and the large language model is used to generate a comprehensive semantic representation containing multimodal context understanding, combining the semantic information of text, image and audio; S4. Adaptively fuse the comprehensive semantic representation generated by the large language model with other modal features using an adaptive modal interaction framework, and generate an optimized fused feature representation by dynamically adjusting the interaction weights of different modalities. S5. Perform cross-modal entity recognition based on fused feature representation. Cross-modal entity recognition includes using modality-aware mechanisms to identify complex entities and relationships in text and other modal information, and generating corresponding classification labels and relationship mappings for entities. S6. Automatically match the identified entity with the existing knowledge graph through a dynamic mapping algorithm. If the entity is a new entity, automatically generate a new node and update the existing knowledge graph node relationship based on the result of the adaptive modal interaction; S7. By continuously inputting new multimodal data, the system automatically adjusts and optimizes the modal perception mechanism and adaptive modal interaction framework of the large language model, continuously learns new knowledge, and dynamically expands the knowledge graph.

2. According to the multimodal entity recognition method based on a large language model according to claim 1, It is characterized in that The S1 step includes: S11, receiving a text dataset, an image dataset, and an audio dataset; S12. For the text dataset D t , each text sample x t ∈D t A set of sentences {s 1 ,s 2 ,…,s n }, where s n is the nth sentence in the text sample; S13. For image dataset D i , each image sample x i ∈D i A pixel matrix M = {p 11 ,p 12 ,…,p mn } means that p mn Represents the pixel value of the mth row and nth column in the image; S14. For audio dataset D a , each audio sample x a ∈D a A set of frequency domain feature vectors F = {f 1 ,f 2 ,…,f k } means that f k is the kth frequency component of the audio signal; S15, the text data set D t , image dataset D i And audio dataset D a Combine into a multimodal input dataset: D={x t ,x i ,x a }。 3. According to the multimodal entity recognition method based on a large language model according to claim 1, It is characterized in that The S2 step includes: S21, through the adaptive modal interaction framework, the multimodal input data set D = {x t ,x i ,x a }Perform modality identification and determine the type and data structure of each modality; S22. According to the characteristics of each modality, different preprocessing methods are applied, including the text dataset D t Perform word segmentation and denoising on the image dataset D i After standardization and size normalization, the audio dataset D a Perform denoising and frequency domain conversion processing; S23, the preprocessed text dataset P t (x t ), image dataset P i (x i ) and audio dataset P a (x a ) are combined into a preprocessed multimodal input dataset: P D ={P t (x t ),P i (x i ),P a (x a )}。 4. According to the multimodal entity recognition method based on a large language model according to claim 1, It is characterized in that The S3 step includes: S31, the preprocessed multimodal input data set P D Input into a large language model with modality awareness mechanism; S32. For text data P t (x t ) Generate text semantic representation through the text embedding layer of the large language model: Among them, E t (x t ) represents text data x t The semantic representation vector of j is the jth word in the text; φ(w j ) is the word embedding obtained by the pre-trained embedding matrix, α j is the attention weight, γ j is the context vector; S33, for the image data P i (x i ), generate image semantic representation through image embedding layer: Among them, E i (x i ) represents the image data x i The semantic representation vector of mn is the pixel point of the image, f(p mn ) represents the image features extracted by convolutional neural network; β mn is the weight of the image feature, indicating the visual importance of a specific area, δ mn Encode for position; S34. For audio data P a (x a ), generate audio semantic representation through the audio embedding layer: Among them, E a (x a ) represents audio data x a The semantic representation vector of k is the kth frequency component in the audio signal, ψ(f k ) is the frequency domain feature extracted by Fourier transform; η k is the feature weight of the audio signal, σ k Encode the time step; S35, the generated text semantic representation E t (x t ), image semantic representation E i (x i ) and audio semantic representation E a (x a ) is fused through a modality-aware mechanism to generate a comprehensive semantic representation that includes multimodal context understanding: E multi =λ 1 ·g(E t (x t ))+λ 2 ·g(E i (x i ))+λ 3 ·g(E a (x a ))+θ(C); Among them, E multi is a multimodal comprehensive semantic representation, g(E) represents the nonlinear transformation function of each modality, and λ 1 , 2 , 3 is the importance weight of each modal feature, and θ(C) is the multimodal context fusion function.

5. According to the multimodal entity recognition method based on a large language model according to claim 1, It is characterized in that The S4 step includes: S41. Comprehensive semantic representations generated by large language models using an adaptive modal interaction framework multi Adaptively fuse the features of each modality, and the adaptive modal interaction framework dynamically adjusts the interaction weights of different modalities; S42, through the interaction function h(E multi ,E m ) represents the text semantics E t (x t ), image semantic representation E i (x i ) and audio semantic representation E a (x a ) for feature fusion: in, is the feature interaction function of the pth row and qth column in the fusion process, combined with the comprehensive semantic representation E multi With modal characteristics E m The local interaction of pq is the local attention weight, β pq is the weight of the feature position, combined with the position encoding of the modal features to optimize the fused spatial feature representation; S43. Through the interaction mechanism between modalities, the weights of different modalities are dynamically adjusted according to the importance of each modality. m : Among them, μ m is the basic weight of mode m, is the multiple interaction function between modes, ζ i is the dynamic adjustment parameter related to the i-th feature dimension; S44. Generate optimized fusion feature representation F opt , the fusion feature representation optimization process is achieved through the nonlinear transformation function g(F fusion )conduct: Among them, F opt represents the final optimized fusion feature representation, γ r is the weight of the rth nonlinear transformation channel, is the nonlinear activation function of the fusion feature representation, δ r is the dynamic weight of each channel.

6. According to the multimodal entity recognition method based on a large language model according to claim 1, It is characterized in that The S5 step includes: S51, based on fusion feature representation F opt Recognize complex entities and relationships in text, image, and audio data through modality-aware mechanisms; S52. The explicit and implicit entities in the text modality are identified through the modality perception mechanism. The identification of the implicit entity is expressed as: Among them, E implicit is an implicit entity representation; w j is the jth word in the text; δ j represents the implicit weight of the current word; c j is the context information of the word; ξ(w j ,∑λ k ·g(φ k (c j ,h j ))) is the comprehensive relationship function between words and context, where λ k is the implicit weight adjustment parameter, φ k (c j ,h j ) indicates context c j and the hidden layer state h j interaction function; g is the activation function for nonlinear transformation, ∈ k is random noise; S53. For image and audio modalities, the relationship between entities is identified through modality perception mechanism: Among them, R relation is the relationship mapping between entities; pq is the relationship weight between modalities; is the mutual influence between entities of mode p and mode q; α pq is the local interaction weight of different modal features; is the local feature interaction function in mode p and mode q, z∈Ω is the integration domain, representing the feature space of modes p and q; S54, generating corresponding classification labels C for the recognized implicit entities in the text modality and the entities in the image and audio modalities entity : Among them, C entity is the classification label of the entity; W c and b c are the weight matrix and bias term of the classifier respectively; γ l and r are the weight coefficients of entity features and relationship mapping respectively; tanh represents the activation function, which is used to generate the final classification result after fusing entity features and relationship information; the softmax function is used to map the result into probability distribution.

7. The multimodal entity recognition method based on a large language model according to claim 1, It is characterized in that The S6 step includes: S61, the identified entity E entity Through the dynamic mapping algorithm and the existing knowledge graph G = (V, R) automatic matching, entity E entity The multimodal features of are calculated by modality similarity and compared with the knowledge graph node v i ∈V for matching; S62: If the identified entity E entity Failed to match existing node v i , then according to the result of adaptive modal interaction, a new node v is automatically generated new , the new node is represented as: Among them, v new is the generated new node, γ l is the weight of the feature dimension, φ l Represents the nonlinear activation function for generating new nodes combined with entity features E entity and classification label C entity The interaction, α k is the weight of the classification label, is the transformation function of the multimodal feature, y∈Θ is the offset in the feature space; S63. Update the relationship R between entities and generate a new entity v through relationship mapping new and the existing entity node v i The relationship between: Among them, r(v new ,v i ) is the generated new relation mapping, β j is the weight of the relationship mapping, ρ j represents the feature interaction function between entities and nodes, γ j (z) represents the position shift function of the local feature, and z∈Ω is the feature space. S64, automatically update the knowledge graph to generate an extended knowledge graph G'=(V',R'): Among them, G' is the expanded knowledge graph, η n is the weight of the new relationship, ζ n The nonlinear transformation function representing the relationship updates the nodes and relationships of the knowledge graph by combining the relationship mapping between new nodes and existing nodes. dz is the integral operation in the feature space.

Images