Multi-modal semantic alignment method for emergency scene

Abstract

The invention relates to the technical field of natural language processing, and discloses an emergency scene-oriented multi-modal semantic alignment method, which comprises the following steps of: acquiring multi-modal data, and mapping dialect vocabularies in the multi-modal data to a standard language family; using a multi-language pre-training model to generate an initial semantic vector in the text / voice; identifying feature elements in the image / video based on the target detection model, generating a structured visual semantic tag and embedding a vector; language features in the initial semantic vector in the text / voice are eliminated based on cross-language adversarial training, and a language-independent semantic vector is generated; calculating the similarity between language-independent semantic vectors and visual vectors based on cross-modal contrast learning, and optimizing the inter-modal distance through a triple loss function; meanwhile, parameters of cross-language adversarial training and cross-modal comparative learning are updated, and a total loss function is minimized. The problem of multi-modal information processing in a multi-dialect environment in the prior art is solved, and the efficiency and accuracy of emergency response are improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically to a multimodal semantic alignment method for emergency scenarios. Background Technology

[0002] With the continuous advancement of artificial intelligence technology, multimodal semantic alignment is playing an increasingly important role in key areas such as intelligent emergency response and disaster information processing. In emergency scenarios, information sources are diverse, involving multimodal data such as voice, text, images, and videos from people in different regions.

[0003] Currently, most multimodal semantic alignment methods are primarily based on standard Mandarin for modeling and training, lacking effective mechanisms for handling non-standard language variants (such as dialects). In practical applications, dialects differ significantly from standard Mandarin in terms of pronunciation, vocabulary usage, and sentence structure, leading to a substantial decrease in the semantic understanding ability of existing models when faced with dialect input, thereby affecting the matching accuracy of cross-modal information such as images and videos.

[0004] In modal modeling, existing methods typically couple features from text, speech, and visual modalities together for joint training, lacking fine-grained decoupling of the semantic structure within each modality. This leads to modal and linguistic biases in the semantic space, affecting the accuracy of semantic alignment. This bias is particularly pronounced in multi-dialect environments, severely impacting the system's generalization ability and practical application effectiveness.

[0005] To address the aforementioned issues, this invention proposes a multimodal semantic alignment method for emergency scenarios. By introducing a dialect-to-standard language semantic mapping mechanism, a cross-linguistic adversarial training strategy, and a cross-modal contrastive learning framework, it effectively eliminates semantic shifts caused by dialect differences, constructs a language-independent unified semantic space, and achieves efficient alignment between text / speech and images / videos. This method significantly improves the system's adaptability and robustness in multi-dialect environments while ensuring semantic consistency, demonstrating broad application prospects and practical value. Summary of the Invention

[0006] In view of this, the present invention provides a multimodal semantic alignment method for emergency scenarios, which aims to solve the problem of multimodal information processing in multi-dialect environments in the prior art and improve the efficiency and accuracy of emergency response.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A multimodal semantic alignment method for emergency scenarios includes:

[0009] Step 1: Obtain multimodal data and map dialect words in the multimodal data to standard language families;

[0010] Step 2: Extract the features of each modality separately through multimodal feature decoupling:

[0011] Use a multilingual pre-trained model to generate initial semantic vectors in text / speech;

[0012] Based on the object detection model, feature elements in images / videos are identified, structured visual semantic labels are generated and embedded into vectors;

[0013] Step 3: Eliminate linguistic features from the initial semantic vectors in text / speech based on cross-linguistic adversarial training to generate language-independent semantic vectors;

[0014] Based on cross-modal contrastive learning, the similarity between language-independent semantic vectors and visual vectors is calculated, and the intermodal distance is optimized through a triple loss function;

[0015] Simultaneously update the parameters for cross-language adversarial training and cross-modal contrastive learning to minimize the total loss function.

[0016] Preferably, cross-linguistic adversarial training is used to eliminate linguistic features from the initial semantic vectors of text / speech, generating language-independent semantic vectors, specifically including:

[0017] The initial semantic vector in the text / speech is sent as input to the gradient inversion layer. During the forward and backward propagation, the gradient inversion layer remains unchanged and directly outputs the semantic vector corresponding to the input.

[0018] Semantic vectors are identified by a language discriminator to determine the language category of the semantic vectors, and the language classification loss is calculated.

[0019] The language classification loss is backpropagated to the gradient reversal layer, where the gradient is reversed, and the adversarial training loss is calculated.

[0020]

[0021] Where θ represents the generator parameters, and λ is a regularization coefficient used to balance the relationship between semantic loss and language classification loss. Indicates semantic loss, Indicates language classification loss, This represents the gradient of the semantic loss with respect to the parameter θ. This represents the gradient of the language classification loss with respect to the parameter θ;

[0022] Based on the adversarial training loss, the generator's parameters θ are updated by optimizing the algorithm to generate language-independent semantic vectors.

[0023] The preferred objective function for cross-language adversarial training is:

[0024]

[0025] Where G represents the generator, D represents the discriminator, V represents the initial semantic vector, E represents the expected value, E[logD(V|2)] represents the expected value of the probability that the discriminator D correctly identifies the sample as real for V, and E[log(1-D(G(V)))] represents the expected value of the probability that the discriminator D misclassifies the fake sample G(V) generated by the generator G as real.

[0026] Preferably, the triple loss function is:

[0027] C align =max(0,δ+φ(V,V′)-φ(V,V"))

[0028] Where δ is the boundary hyperparameter used to control the minimum difference between positive and negative sample pairs, V represents the language-independent semantic vector, V′ is a positive sample related to V, i.e., a visual vector from the same event; V″ is a negative sample unrelated to V, i.e., a visual vector from different events, and φ represents the cosine similarity.

[0029] Preferably, the total loss function is:

[0030] L total =L align +λ1C lang+ λ2C minzu

[0031] Among them, L align It is semantic alignment loss. C represents the language classification loss. minzu λ1 and λ2 are dialect enhancement loss terms, and λ1 and λ2 are balance coefficients.

[0032] Preferably, mapping dialect words in the input to the standard language family specifically includes:

[0033] Dialect recognition models can be used to identify dialect words in text; or speech can be converted into text using a speech recognition system and then dialect words can be identified.

[0034] Based on mapping rules in a predefined dictionary, dialect words are converted into standard language words.

[0035] A computer device is characterized by comprising: a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the processor executes the computer program, it implements a multimodal semantic alignment method for emergency scenarios.

[0036] A computer-readable storage medium is characterized in that a computer program is stored on the storage medium, which, when executed by a processor, implements a multimodal semantic alignment method for emergency scenarios.

[0037] As can be seen from the above technical solution, compared with the prior art, this invention discloses a multimodal semantic alignment method for emergency scenarios. By introducing a dialect-to-standard language semantic mapping mechanism, a cross-language adversarial training strategy, and a cross-modal contrastive learning framework, it effectively eliminates semantic shifts caused by dialect differences, constructs a language-independent unified semantic space, and achieves efficient alignment between text / speech and images / videos. While ensuring semantic consistency, this method significantly improves the system's adaptability and robustness in multi-dialect environments, enhances emergency response efficiency, and provides more accurate and comprehensive information support for command and decision-making. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 The flowchart of a multimodal semantic alignment method for emergency scenarios provided by the present invention is shown.

[0040] Figure 2 This is a flowchart of the cross-language adversarial training process of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] This invention discloses a multimodal semantic alignment method for emergency scenarios, such as... Figure 1 As shown, it includes:

[0043] Step 1: Acquire multimodal data and map dialect words in the multimodal data to standard language systems. The multimodal data includes speech, text, and video / images.

[0044] Step 2: Extract the features of each modality separately through multimodal feature decoupling:

[0045] Use a multilingual pre-trained model to generate initial semantic vectors in text / speech;

[0046] Based on the object detection model, feature elements in images / videos are identified, structured visual semantic labels are generated and embedded into vectors;

[0047] Step 3: Eliminate linguistic features from the initial semantic vectors in text / speech based on cross-linguistic adversarial training to generate language-independent semantic vectors;

[0048] Based on cross-modal contrastive learning, the similarity between language-independent semantic vectors and visual vectors is calculated, and the intermodal distance is optimized through a triple loss function;

[0049] Simultaneously update the parameters for cross-language adversarial training and cross-modal contrastive learning to minimize the total loss function.

[0050] In this embodiment, mapping dialect words in the input to the standard language family specifically includes:

[0051] Dialect recognition models can be used to identify dialect words in text; or speech can be converted into text using a speech recognition system and then dialect words can be identified.

[0052] Establish mapping rules from dialects to standard language families. These rules can be based on a predefined dictionary containing dialect vocabulary and its corresponding standard language family vocabulary. Based on the mapping rules in the predefined dictionary, dialect vocabulary is converted into standard language family vocabulary.

[0053] In this embodiment, a multilingual pre-trained model (such as mBERT, XLM-R) is used to generate the initial semantic vector.

[0054] Structured visual semantic labels are extracted using an object detection model and embedded into vectors.

[0055] In this embodiment, the present invention introduces adversarial training to further eliminate linguistic features, making the generated semantic vectors independent of language. Specifically, this includes:

[0056] The initial semantic vector in the text / speech is sent as input to the gradient inversion layer. During the forward and backward propagation, the gradient inversion layer remains unchanged and directly outputs the semantic vector corresponding to the input.

[0057] Semantic vectors are identified by a language discriminator to determine the language category of the semantic vectors and the language classification loss is calculated. The goal of the discriminator is to predict the language label as accurately as possible.

[0058] The language classification loss is backpropagated to the Gradient Reversal Layer (GRL). The Gradient Reversal Layer multiplies the gradient by a negative constant -λ, thereby reversing the gradient. This causes the generator to receive an update signal in the opposite direction to the language discriminator, prompting the generator to learn how to generate semantic vectors that are difficult to be correctly classified by the language discriminator, thereby gradually eliminating the language features in the semantic vectors.

[0059] Calculate adversarial training loss

[0060]

[0061] Where θ represents the generator parameters, and λ is a regularization coefficient used to balance the relationship between semantic loss and language classification loss. Indicates semantic loss, Indicates language classification loss, This represents the gradient of the semantic loss with respect to the parameter θ. This represents the gradient of the language classification loss with respect to the parameter θ;

[0062] Based on the adversarial training loss, the generator's parameters θ are updated using an optimization algorithm (such as gradient descent) to generate language-independent semantic vectors.

[0063] The objective function for cross-language adversarial training is:

[0064]

[0065] Where G represents the generator, D represents the discriminator, V represents the initial semantic vector, E represents the expected value, E[logD(V|2)] represents the expected value of the probability that the discriminator D correctly identifies the sample as real for V, and E[log(1-D(G(V)))] represents the expected value of the probability that the discriminator D misclassifies the fake sample G(V) generated by the generator G as real.

[0066] Cross-linguistic adversarial training aims to learn a more general, language-independent semantic representation, particularly suitable for scenarios involving multiple language variants, such as Chinese and minority languages, ensuring accurate understanding and response even when faced with different language inputs. This invention achieves the goal of generating language-independent semantic vectors through the synergistic effect of gradient inversion layers and language discriminators, providing strong support for cross-linguistic and cross-modal data processing.

[0067] In this embodiment, the similarity between language-independent semantic vectors and visual vectors is calculated based on cross-modal contrastive learning, and the intermodal distance is optimized through a triple loss function; at the same time, the parameters of cross-language adversarial training and cross-modal contrastive learning are updated to minimize the total loss function.

[0068] To optimize the distance between language-independent semantic vectors and visual vectors, a triplet loss method is employed:

[0069] C align =max(0,δ+φ(V,V′)-φ(V,V"))

[0070]

[0071] Here, δ is a boundary hyperparameter used to control the minimum difference between positive and negative sample pairs. V represents a language-independent semantic vector. V′ is a positive sample related to V, i.e., a visual vector from the same event (semantically related). V″ is a negative sample unrelated to V, i.e., a visual vector from a different event (semantically unrelated). In the shared semantic space, for each pair (language-independent semantic vector and visual vector), cosine similarity φ is used to measure their similarity. Cosine similarity is used to evaluate the directional difference between two vectors, with a value range of [-1, 1], where 1 indicates exactly the same direction, 0 indicates no linear correlation, and -1 indicates completely opposite directions. ||V|| and ||V′|| represent the norms of vectors V and V′, respectively.

[0072] This invention utilizes a triple loss function to optimize the distance between language-independent semantic vectors and visual vectors through cross-modal contrastive learning, thereby enhancing the correlation and consistency between different modal data and improving the response efficiency and accuracy of multimodal systems in emergency scenarios.

[0073] The total loss function is:

[0074] L total =L align +λ1C lang +λ2C minzu

[0075] Among them, L align It is semantic alignment loss. C represents the language classification loss. minzu λ1 and λ2 are dialect enhancement loss terms, and λ1 and λ2 are balance coefficients. Typically, λ1 = 0.3 and λ2 = 0.8.

[0076]

[0077] Among them, V ang (i) and V han (i) represent the dialect vector and the corresponding Chinese vector of the i-th sample, respectively, and N represents the total number of samples, i.e., the number of all pairwise comparisons involved in calculating the loss. 2It represents the square of the Euclidean distance between two vectors.

[0078] This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements a multimodal semantic alignment method for emergency scenarios.

[0079] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a multimodal semantic alignment method for emergency scenarios.

[0080] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0082] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal semantic alignment method for emergency scenarios, characterized in that, include: Step 1: Obtain multimodal data and map dialect words in the multimodal data to standard language families; Step 2: Extract the features of each modality separately through multimodal feature decoupling: Use a multilingual pre-trained model to generate initial semantic vectors in text / speech; Based on the object detection model, feature elements in images / videos are identified, structured visual semantic labels are generated and embedded into vectors; Step 3: Eliminate linguistic features from the initial semantic vectors in text / speech based on cross-linguistic adversarial training to generate language-independent semantic vectors; Based on cross-modal contrastive learning, the similarity between language-independent semantic vectors and visual vectors is calculated, and the intermodal distance is optimized through a triple loss function; Simultaneously update the parameters for cross-language adversarial training and cross-modal contrastive learning to minimize the total loss function.

2. The multimodal semantic alignment method for emergency scenarios according to claim 1, characterized in that, Cross-linguistic adversarial training is used to remove linguistic features from the initial semantic vectors of text / speech, generating language-independent semantic vectors, specifically including: The initial semantic vector in the text / speech is sent as input to the gradient inversion layer. During the forward and backward propagation, the gradient inversion layer remains unchanged and directly outputs the semantic vector corresponding to the input. Semantic vectors are identified by a language discriminator to determine the language category of the semantic vectors, and the language classification loss is calculated. The language classification loss is backpropagated to the gradient reversal layer, where the gradient is reversed, and the adversarial training loss is calculated. Where θ represents the generator parameters, and λ is a regularization coefficient used to balance the relationship between semantic loss and language classification loss. Indicates semantic loss, Indicates language classification loss, This represents the gradient of the semantic loss with respect to the parameter θ. This represents the gradient of the language classification loss with respect to the parameter θ; Based on the adversarial training loss, the generator's parameters θ are updated by optimizing the algorithm to generate language-independent semantic vectors.

3. The multimodal semantic alignment method for emergency scenarios according to claim 2, characterized in that, The objective function for cross-language adversarial training is: Where G represents the generator, D represents the discriminator, V represents the initial semantic vector, E represents the expected value, E[logD(V|2)] represents the expected value of the probability that the discriminator D correctly identifies the sample as real for V, and E[log(1-D(G(V)))} represents the expected value of the probability that the discriminator D misclassifies the fake sample G(V) generated by the generator G as real.

4. The multimodal semantic alignment method for emergency scenarios according to claim 2, characterized in that, The triple loss function is: C align =max(0,δ+φ(V,V′)-φ(V,V")) Where δ is the boundary hyperparameter used to control the minimum difference between positive and negative sample pairs, V represents the language-independent semantic vector, V′ is a positive sample related to V, i.e., a visual vector from the same event; V″ is a negative sample unrelated to V, i.e., a visual vector from different events, and φ represents the cosine similarity.

5. A multimodal semantic alignment method for emergency scenarios according to claim 1, characterized in that, The total loss function is: L total =L align +λ1C lang +λ2C minzu Among them, L align It is semantic alignment loss. C represents the language classification loss. minzu λ1 and λ2 are dialect enhancement loss terms, and λ1 and λ2 are balance coefficients.

6. A multimodal semantic alignment method for emergency scenarios according to claim 1, characterized in that, Mapping dialect words in the input to the standard language family specifically includes: Dialect recognition models can be used to identify dialect words in text; or speech can be converted into text using a speech recognition system and then dialect words can be identified. Based on mapping rules in a predefined dictionary, dialect words are converted into standard language words.

7. A computer device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor, when executing the computer program, implements the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 6.

Images