Semantic recognition method and device, equipment, medium and product

Abstract

The embodiment of the invention discloses a semantic recognition method and device, equipment, a medium and a product. The semantic recognition method comprises the steps of obtaining multi-modal input data corresponding to data to be recognized; the multi-modal input data comprises text data and image data; performing alignment processing on the context sensing vector, the text data and the image data through a dynamic encoder and a cross-modal alignment mechanism to determine text dynamic features, image dynamic features and context features; the context feature is determined based on the to-be-identified data; performing weighted fusion on the text dynamic features, the image dynamic features and the context features through a cross attention mechanism and a gating weighted fusion mechanism to determine a cross-modal joint feature vector; and determining a semantic recognition result of the to-be-recognized data based on the cross-modal joint feature vector. According to the technical scheme, the problem that a traditional alignment method cannot adapt to data distribution changes and semantic hierarchy differences is solved, and cross-modal semantic matching with fine granularity, context sensitivity and dynamic evolution is achieved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a semantic recognition method, apparatus, device, medium, and product. Background Technology

[0002] In the information age, the volume of data is growing exponentially, and the semantic representation of different modalities of data, such as text, images, videos, and sensor data, varies greatly. Cross-modal data retrieval, enabling efficient and accurate searching across different modalities (such as images, text, and videos), is becoming increasingly important in fields such as smart cities, healthcare, and finance. With the development of artificial intelligence and big data technologies, large models and deep learning mechanisms have been widely applied in cross-modal data retrieval. Common retrieval algorithms include large models, contrastive learning, generative adversarial networks, and attention mechanisms.

[0003] Among these approaches, large-scale model-based methods focus on "feature mapping" between modalities but lack sufficient modeling of the essence of semantics (such as conceptual levels and common-sense knowledge), making it difficult to resolve semantic ambiguities implicit in cross-modal data. Furthermore, existing methods rely on large-scale labeled data for training, leading to a significant drop in cross-modal retrieval performance in domain transfer scenarios (such as from natural images to medical images) or small-sample scenarios, and a lack of generalization ability to unseen semantics. Contrastive learning-based methods rely on randomly sampling negative samples, making it difficult to effectively mine "hard samples" (i.e., samples that are semantically similar but have large modal differences) in the cross-modal space, resulting in insufficient ability to distinguish subtle semantic differences. Generative adversarial networks (GANs) are typically designed for unidirectional generation, lacking explicit semantic constraints in the generation process, which may cause cross-modal transformation results to deviate from the original semantics, leading to unbalanced retrieval performance. Attention mechanisms lack cross-modal interactive guidance, causing attention weight allocation to deviate from the core of semantic association. In complex scenarios, attention mechanisms are susceptible to interference from noisy features. While the three methods have promoted the development of cross-modal retrieval technology for large models, they still have the following problems in general: first, insufficient semantic modeling makes it impossible to resolve implicit semantic ambiguities; second, they are limited by labeled data, and their generalization ability is limited by domain bias. Summary of the Invention

[0004] This disclosure provides a semantic recognition method, apparatus, device, medium, and product that solves the problem that traditional alignment methods cannot adapt to changes in data distribution and differences in semantic levels, and achieves fine-grained, context-sensitive, and dynamically evolving cross-modal semantic matching.

[0005] Firstly, a semantic recognition method is provided, including:

[0006] Acquire the multimodal input data corresponding to the data to be identified; the multimodal input data includes text data and image data;

[0007] The context-aware vector, the text data, and the image data are aligned using a dynamic encoder and a cross-modal alignment mechanism to determine text dynamic features, image dynamic features, and context features; the context features are determined based on the data to be identified.

[0008] By employing a cross-attention mechanism and a gated weighted fusion mechanism, the text dynamic features, the image dynamic features, and the context features are weighted and fused to determine a cross-modal joint feature vector.

[0009] The semantic recognition result of the data to be identified is determined based on the cross-modal joint feature vector.

[0010] Secondly, a semantic recognition device is provided, comprising:

[0011] The data acquisition module is used to acquire multimodal input data corresponding to the data to be identified; the multimodal input data includes text data and image data;

[0012] The feature determination module is used to align the context-aware vector, the text data, and the image data through a dynamic encoder and a cross-modal alignment mechanism to determine text dynamic features, image dynamic features, and context features; the context features are determined based on the data to be identified.

[0013] The cross-modal feature vector determination module is used to perform weighted fusion of the text dynamic features, the image dynamic features and the context features through a cross-attention mechanism and a gated weighted fusion mechanism to determine the cross-modal joint feature vector;

[0014] The recognition result determination module is used to determine the semantic recognition result of the data to be recognized based on the cross-modal joint feature vector.

[0015] Thirdly, an electronic device is provided, comprising:

[0016] At least one processor; and,

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the semantic recognition method as described in the first aspect above.

[0019] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the semantic recognition method as described in the first aspect above.

[0020] Fifthly, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the semantic recognition method as described in the first aspect above.

[0021] This disclosure provides a semantic recognition method, apparatus, device, medium, and product. The method includes: acquiring multimodal input data corresponding to data to be recognized; the multimodal input data includes text data and image data; aligning the context-aware vector, the text data, and the image data using a dynamic encoder and a cross-modal alignment mechanism to determine text dynamic features, image dynamic features, and context features; the context features are determined based on the data to be recognized; weightedly fusing the text dynamic features, image dynamic features, and context features using a cross-attention mechanism and a gated weighted fusion mechanism to determine a cross-modal joint feature vector; and determining the semantic recognition result of the data to be recognized based on the cross-modal joint feature vector. This technical solution uses a dynamic encoder and a cross-modal alignment mechanism to fuse text data, image data, and context-aware vectors, extracting text dynamic features, image dynamic features, and context features, and then uses a cross-attention mechanism and a gated weighted fusion mechanism to weightedly fuse these features to generate a cross-modal joint feature vector. Based on this cross-modal joint feature vector, the semantic recognition result of the data to be identified is determined, which solves the problem that traditional alignment methods cannot adapt to changes in data distribution and differences in semantic level, and realizes fine-grained, context-sensitive, and dynamically evolving cross-modal semantic matching.

[0022] It should be understood that the description in this section is not intended to identify key or important features of the embodiments of this disclosure, nor is it intended to limit the scope of the embodiments of this disclosure. Other features of the embodiments of this disclosure will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a semantic recognition method provided in Embodiment 1 of this disclosure;

[0025] Figure 2 This is a schematic diagram illustrating the implementation process of a cross-modal alignment mechanism provided in Embodiment 1 of this disclosure;

[0026] Figure 3This is a schematic diagram of the structure of a text dynamic encoder provided in Embodiment 1 of this disclosure;

[0027] Figure 4 This is a schematic diagram of the structure of an image dynamic encoder provided in Embodiment 1 of this disclosure;

[0028] Figure 5 This is a schematic diagram illustrating the execution process of another semantic recognition method provided in Embodiment 1 of this disclosure;

[0029] Figure 6 This is a schematic diagram of the structure of a semantic recognition device provided in Embodiment 2 of this disclosure;

[0030] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of this disclosure. Detailed Implementation

[0031] To enable those skilled in the art to better understand the solutions of the embodiments of this disclosure, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the protection scope of the embodiments of this disclosure.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1

[0034] Figure 1 This is a flowchart of a semantic recognition method provided in Embodiment 1 of this disclosure. This embodiment is applicable to situations requiring semantic recognition. The method can be executed by a semantic recognition device, which can be implemented in hardware and / or software. This semantic recognition device can be configured in an electronic device, including but not limited to computers, PCs, electronic devices, and servers, which are devices with data processing capabilities. Figure 1 As shown, the method includes:

[0035] S110. Obtain the multimodal input data corresponding to the data to be identified; the multimodal input data includes text data and image data.

[0036] In this embodiment, the data to be identified can be user-input data that requires semantic recognition. The data to be identified can be modal input data, such as text, images, audio, and video. Multimodal input data corresponding to the data to be identified can be obtained. Multimodal input data refers to an input set containing multiple different types of data. Multimodal input data can include text data and image data.

[0037] S120. By using a dynamic encoder and a cross-modal alignment mechanism, the context-aware vector, text data, and image data are aligned to determine the text dynamic features, image dynamic features, and context features; the context features are determined based on the data to be identified.

[0038] In this embodiment, a dynamic encoder can be an encoder structure capable of dynamically processing input data and generating feature representations. It typically adjusts its encoding method dynamically based on the contextual information of the input data, thereby better capturing the dynamic features of the data. Dynamic encoders can include text dynamic encoders and image dynamic encoders; specifically, a text dynamic encoder can dynamically extract features from text data, capturing dynamic changes and contextual information within the text. A text dynamic encoder can dynamically adjust its encoding method based on the context of the input text, generating richer feature representations. Similarly, an image dynamic encoder can dynamically extract features from image data, capturing dynamic changes and contextual information within the image. An image dynamic encoder can dynamically adjust its encoding method based on the content of the input image, generating richer feature representations.

[0039] As described above, a cross-modal alignment mechanism can be used to align data of different modalities (such as text and images) so that they are comparable in feature space, enabling data of different modalities to be effectively fused.

[0040] Specifically, contextual features can be determined from the data to be identified. For example, temporal information such as user search history and time intervals can be captured from the data to be identified to enhance semantic dynamism. After determining the contextual features, the context-aware vector, the text data, and the image data can be aligned to determine text dynamic features, image dynamic features, and contextual features.

[0041] S130. By using a cross-attention mechanism and a gated weighted fusion mechanism, the dynamic features of text, the dynamic features of images, and the context features are weighted and fused to determine the cross-modal joint feature vector.

[0042] Specifically, after obtaining the text dynamic features, image dynamic features, and contextual features, a cross-attention mechanism and a gated weighted fusion mechanism can be used to fuse these features to obtain a cross-modal joint feature vector. This cross-modal joint feature vector is a comprehensive feature vector generated by fusing features from different modalities (such as text, image, and context). This vector contains information from multiple modalities, enabling a more comprehensive representation of the semantic content of the data to be identified.

[0043] Cross-attention mechanisms can extract information through the interaction between two different input sequences. In multimodal tasks, this allows one modality to receive information from another, enabling information exchange between modalities. Gated weighted fusion mechanisms can be a technique for multimodal fusion, dynamically adjusting the contribution values ​​of features from different modalities through gating functions to achieve more effective information fusion.

[0044] S140. Determine the semantic recognition result of the data to be recognized based on the cross-modal joint feature vector.

[0045] Specifically, after obtaining the cross-modal joint feature vector, the cross-modal joint feature vector can be input into the classification model or the recognition model to obtain the semantic recognition result of the data to be recognized.

[0046] It should be noted that after obtaining the semantic recognition results of the data to be recognized, real-time user feedback data can be acquired based on these results. The direction for optimization can then be determined based on this feedback data. According to the optimization direction, the parameters in the process of determining the semantic recognition results are scored, and parameters with high importance are selected for fine-tuning. Specifically, the gradient change magnitude of the parameters in the process of determining the semantic recognition results during historical training can be calculated, and parameters that significantly affect the retrieval results (such as classifier weights) can be selected. Based on a sparse update mechanism, only the top 10% of parameters in terms of importance score are fine-tuned, while the remaining parameters are frozen to reduce computational load.

[0047] This embodiment provides a semantic recognition method, including: acquiring multimodal input data corresponding to data to be recognized; the multimodal input data includes text data and image data; aligning the context-aware vector, the text data, and the image data using a dynamic encoder and a cross-modal alignment mechanism to determine text dynamic features, image dynamic features, and context features; the context features are determined based on the data to be recognized; weighting and fusing the text dynamic features, the image dynamic features, and the context features using a cross-attention mechanism and a gated weighted fusion mechanism to determine a cross-modal joint feature vector; and determining the semantic recognition result of the data to be recognized based on the cross-modal joint feature vector. This technical solution solves the problem that traditional alignment methods cannot adapt to changes in data distribution and differences in semantic hierarchy, achieving fine-grained, context-sensitive, and dynamically evolving cross-modal semantic matching.

[0048] As an optional implementation of this embodiment, the step of obtaining the multimodal input data corresponding to the data to be identified includes:

[0049] 1) Obtain the initial multimodal input data corresponding to the data to be identified; the initial multimodal input data includes first text data, first image data, initial audio data and initial video data.

[0050] In this embodiment, initial multimodal input data corresponding to the data to be identified can be obtained. The initial multimodal input data includes first text data, first image data, initial audio data, and initial video data. For example, the initial multimodal data may include text data, image data, audio data, and video data.

[0051] 2) Slice the initial audio data and the initial video data to determine the second text data and the second image data.

[0052] Specifically, the initial audio data and the initial video data can be sliced ​​to determine the second text data and the second image data. Slicing can be the process of dividing continuous audio or video data into multiple small segments or fragments. This process is typically used to extract data within a specific time period for further analysis or processing. For the initial audio data, it can be segmented into second text data, which includes multiple text data items; for the initial video data, it can be segmented into second image data, which can include multiple image data items.

[0053] 3) Determine the text data based on the first text data and the second text data.

[0054] Specifically, the first text data and the second text data can be defined as text data.

[0055] 4) Determine the image data based on the first image data and the second image data.

[0056] Specifically, the first image data and the second image data can be defined as image data.

[0057] As an optional implementation of this embodiment, the step of aligning the context-aware vector, the text data, and the image data using a dynamic encoder and a cross-modal alignment mechanism to determine text dynamic features, image dynamic features, and context features includes:

[0058] 1) The initial text feature vector is dynamically weighted and weighted summed using a text dynamic encoder to obtain a dynamic text vector; the text dynamic encoder includes a first input layer, a time-aware long short-term memory (LSTM) layer, and a dynamic weight allocation layer; the initial text feature vector is determined by encoding the text data.

[0059] Specifically, after determining the text data, the initial text data can be encoded using Bidirectional Encoder Representations from Transformers (BERT) technology to obtain initial text feature vectors. BERT word vector technology uses a bidirectional Transformer encoder to generate high-quality word vectors and contextual representations.

[0060] As described above, after determining the initial text feature vector, a dynamic text encoder can be used to dynamically assign weights and perform weighted summation on the initial text feature vector to obtain a dynamic text vector. The dynamic text encoder includes a first input layer, a time-aware long short-term memory (LSTM) layer, and a dynamic weight assignment layer.

[0061] 2) Local features are extracted from the initial image feature vector using an image dynamic encoder to obtain a dynamic image vector; the dynamic image encoder includes a second input layer, a local attention layer, and a global semantic extraction layer; the initial image feature vector is determined by encoding the image data.

[0062] Specifically, after determining the image data, the initial image data can be encoded using image segmentation technology to obtain the initial image feature vector. Image segmentation technology divides an image into multiple smaller blocks.

[0063] As described above, after determining the initial image feature vector, a dynamic image encoder can be used to extract local features from the initial image feature vector to obtain a dynamic image vector. The dynamic image encoder can include a second input layer, a local attention layer, and a global semantic extraction layer.

[0064] 3) Based on the cross-modal alignment mechanism, the context-aware vector, the dynamic text vector, and the dynamic image vector are cross-modal temporally aligned to obtain the text dynamic features, the image dynamic features, and the context features.

[0065] Specifically, after obtaining the context-aware vector, dynamic text vector, and dynamic image vector, a cross-modal alignment mechanism can be used to perform cross-modal temporal alignment of the context-aware vector, dynamic text vector, and dynamic image vector, thereby obtaining text dynamic features, image dynamic features, and context features.

[0066] Figure 2 This is a schematic diagram illustrating the implementation process of a cross-modal alignment mechanism provided in this embodiment, as shown below. Figure 2 As shown, the user context behavior pool can be used to store the semantic vectors of a user's most recent k searches, weighted by time decay. The user context behavior pool can be used to obtain the semantic bias of historical users based on the semantic vectors of their historical searches, which can be used to enhance semantic bias during dynamic semantic fusion. Specifically, dynamic semantic fusion can be used to concatenate the dynamic text vector and dynamic image vector of the current query with the context pool vector, and generate the final semantic representation (i.e., dynamic text features, dynamic image features, and context features) through a fully connected layer.

[0067] As an optional implementation of this embodiment, the step of weightedly fusing the text dynamic features, the image dynamic features, and the context features through a cross-attention mechanism and a gated weighted fusion mechanism to determine a cross-modal joint feature vector includes:

[0068] 1) Preprocess the text dynamic features and the image dynamic features to obtain the first text dynamic features and the first image dynamic features.

[0069] Specifically, after obtaining the text dynamic features and image dynamic features, preprocessing can be performed on the text dynamic features and image dynamic features to obtain the first text dynamic features and the first image dynamic features. The first text dynamic features can be preprocessed text dynamic features, and the first image dynamic features can be preprocessed image dynamic features. The preprocessing operations can include linear projection and normalization. Linear projection ensures that all feature dimensions are consistent, and layer normalization is performed on the projected features to map them to the same dimensional space.

[0070] 2) Perform bidirectional cross-attention calculation on the first text dynamic features and the first image dynamic features through a cross-attention mechanism to obtain the second text dynamic features and the second image dynamic features.

[0071] In this embodiment, the cross-attention mechanism (which can be an extended form of an attention mechanism) is mainly used to handle the interaction between two different sequences or modalities. It allows one sequence (or modality) to obtain information from another sequence (or modality), thereby achieving richer feature fusion and information interaction.

[0072] Specifically, a cross-attention mechanism is used to perform bidirectional cross-attention calculation on the first text dynamic features and the first image dynamic features to obtain second text dynamic features and second image dynamic features. The second text dynamic features can be text features enhanced by the cross-attention mechanism, and the second image dynamic features can be image features enhanced by the cross-attention mechanism.

[0073] For example, a second dynamic image feature (enhanced image feature vector) can be calculated based on the first dynamic text feature: the first dynamic text feature can serve as a query vector (Query), and the first dynamic image feature provides a key vector (Key) and a value vector (Value); the similarity between each text word (Query) and each image region (Key) is calculated. For example, a dot product or scaled dot product can be used to calculate the similarity; the similarity is normalized using a softmax function to obtain the association weight between each text word and each image region; the features (Value) of the image regions are weighted and summed using the normalized association weights, enhancing the features of the image regions most relevant to the text words to obtain the second dynamic image feature (enhanced image feature vector).

[0074] Following the above description, a second text dynamic feature (enhanced text feature vector) can also be calculated based on the first image dynamic feature: the first image dynamic feature can be used as a query vector Query, and the first text dynamic feature provides a key vector Key and a value vector Value; calculate the similarity between each image region (Query) and each text word (Key); normalize the similarity using the softmax function to obtain the association weight between each image region and each text word; use the normalized association weight to perform a weighted summation of the text word features (Value) to enhance the text features most relevant to the image region, so as to obtain the second text dynamic feature (enhanced text feature vector).

[0075] 3) Based on the fusion weight, the second text dynamic features and the second image dynamic features are weighted and fused to obtain a fused feature vector; the fusion weight is determined based on the gated weighted fusion mechanism.

[0076] In this embodiment, the gated weighted fusion mechanism can be a technique for multimodal data fusion, which dynamically adjusts the contribution of different modal features through a gating function. The core of the gated weighted fusion mechanism is to normalize the weights using a Sigmoid or Tanh function, thereby adjusting the weights.

[0077] It is known that fusion weights can be determined based on a gated weighted fusion mechanism. These fusion weights can include the weights corresponding to the second text dynamic features and the second image dynamic features. Based on these fusion weights, the second text dynamic features and the second image dynamic features can be weighted and fused to obtain a fused feature vector. For example, the fused feature vector can be represented as:

[0078]

[0079] in, It can represent a fused feature vector. and The fusion weights can be represented separately. It can represent the dynamic features of the second text. It can represent the dynamic features of the second image. It can represent the first One characteristic, It can represent the total number of features.

[0080] 4) Determine the cross-modal joint feature vector based on the fused feature vector and the context feature vector.

[0081] Specifically, after obtaining the fused feature vector, it can be fused with the context feature vector to determine the cross-modal joint feature vector. In the cross-modal fusion process, local attention and bidirectional attention mechanisms are comprehensively used to construct a multi-level semantic interaction network. For image semantic vector processing, a local attention mechanism is introduced, dividing the image into multiple feature levels. By calculating the attention weights of different regions within each level, the importance of each region in semantic representation is dynamically adjusted, achieving global semantic constraints on the image and effectively capturing semantic information of different granularities. In cross-modal fusion, a bidirectional attention computation mechanism is introduced to establish a bidirectional dynamic association between text and image features. During forward computation, by calculating the association weights between text words and image regions, the model dynamically focuses on key visual elements in the image, enhancing the semantic expression of visual features. During backward computation, image features provide semantic feedback to the text description, enhancing the semantic accuracy of text features, forming a bidirectional enhancement of text and image features, thereby improving the semantic accuracy, richness, and robustness of cross-modal retrieval.

[0082] As an optional implementation of this embodiment, the step of dynamically assigning weights and performing weighted summation on the initial text feature vector using a text dynamic encoder to obtain a dynamic text vector includes:

[0083] 1) The initial text feature vector is transmitted to the time-aware LSTM layer using the first input layer.

[0084] Specifically, Figure 3 This is a schematic diagram of the structure of a text dynamic encoder provided in this embodiment, as shown below. Figure 3 As shown, the text dynamic encoder includes a first input layer, a time-aware Long Short-Term Memory (LSTM) layer, and a dynamic weight allocation layer. The initial text feature vector can be transferred to the time-aware LSTM layer using the first input layer.

[0085] 2) The time-aware LSTM layer is used to process the time-series information of the initial text feature vector to obtain a time-aware word vector set; the time-aware word vector set includes at least one word vector.

[0086] Specifically, upon receiving the initial text feature vector, the time-aware LSTM layer can process the time-series information of the initial text feature vector to obtain a time-aware word vector set; the time-aware word vector set includes at least one word vector.

[0087] For example, the input layer: after the original text is segmented, an initial text feature vector is generated through the BERT-base model: A time-aware LSTM layer can calculate the time interval between the user's current query and the previous search. (Unit: seconds), mapped to a 32-dimensional vector : Initial text feature vector Embedded with time interval Concatenate and input into the time-aware LSTM unit:

[0088]

[0089] in, It can represent the set of time-aware word vectors at the current moment. It can represent the time-aware word vector set of the previous time step in the current time step.

[0090] 3) The dynamic weight allocation layer is used to weight each word vector in the time-aware word vector set to obtain the dynamic text vector.

[0091] It is known that, using a dynamic weight allocation layer, the importance weight of each word vector can be calculated through a self-attention mechanism. The dynamic text vector is obtained by weighting each word vector in the time-aware word vector set using the importance weights corresponding to each word vector.

[0092] For text semantic vector processing, this technical solution proposes an innovative approach based on the fusion of gating units and a global semantic graph. In the model structure, gating units such as the LSTM forget gate are introduced as semantic information filters to process the input text sequence time-by-time. During processing, the forget gate dynamically decides whether to retain or discard local semantic information based on the current input and historical states. Simultaneously, prior knowledge from the global semantic graph is integrated into the model. The global semantic graph pre-constructs a semantic network covering a large number of entities, relationships, and concepts, which can effectively suppress local noise interference and provide richer and more accurate feature information for cross-modal retrieval.

[0093] As an optional implementation of this embodiment, the step of extracting local features from the initial image feature vector using an image dynamic encoder to obtain a dynamic image vector includes:

[0094] 1) The initial image feature vector is segmented using the second input layer to obtain a set of image blocks.

[0095] Specifically, Figure 4 This is a schematic diagram of the structure of an image dynamic encoder provided in this embodiment, as shown below. Figure 4 As shown, the image motion encoder may include a second input layer, a local attention layer, and a global semantic extraction layer. The second input layer can be used to segment the initial image feature vector to obtain a set of image patches. For example, the second input layer segments the initial image feature vector into different patches, which are linearly mapped to 768-dimensional vectors (patches), and thus linearly mapped to 768-dimensional vectors (the set of image patches).

[0096] 2) Through a local attention layer, the feature vector of each image block in the image block set is weighted using an attention mechanism to obtain a weighted set of image block feature vectors.

[0097] It is known that a local attention layer can be used to weight the feature vectors of each image patch in the image patch set using an attention mechanism, thereby obtaining a weighted set of image patch feature vectors. The local attention layer is an implementation of an attention mechanism used to calculate the weights of feature vectors within a local scope (i.e., within each image patch).

[0098] For example, a feature aggregation vector is obtained by calculating spatial attention weights: for each image patch, its relationship with neighboring patches can be calculated. The associated weights of the window are used to perform weighted fusion on the features of each image patch to obtain a set of weighted image patch feature vectors.

[0099] 3) The dynamic image vector is generated based on the weighted set of image patch feature vectors through the global semantic extraction layer.

[0100] It is known that the weighted set of image patch feature vectors is input into the Transformer Encoder layer through the global semantic extraction layer to generate dynamic image vectors.

[0101] Figure 5 This is a schematic diagram illustrating the execution process of another semantic recognition method provided in this embodiment, as shown below. Figure 5 As shown, text, image, audio, and video data can be sliced ​​and processed. Dynamic text features, image features, and contextual features are determined through dynamic text encoding, dynamic image encoding, and cross-modal alignment mechanisms. These dynamic features are then used to determine the final cross-modal feature vector through data preprocessing, a bidirectional cross-attention mechanism, weight fusion, and up-down perception recalibration. Semantic recognition is then performed based on this cross-modal feature vector to obtain the semantic recognition result. Furthermore, the model parameters in data preprocessing, bidirectional cross-attention, weight fusion, and up-down perception recalibration can be fine-tuned through importance scoring and coefficient update mechanisms.

[0102] The aforementioned technical solution overcomes the limitations of traditional static semantic alignment by updating semantic representations in real time based on user context. It dynamically adjusts the multimodal feature fusion weights and intermodal mapping weights using mechanisms such as attention gating and reinforcement learning. This enables the model to more keenly capture semantic changes, deeply understand semantic details, and achieve accurate semantic mapping. Especially when dealing with complex and dynamic semantic scenarios, it can significantly improve retrieval accuracy and robustness.

[0103] Following the above description, prior knowledge from the global semantic graph is integrated into the text semantic vector through a gating unit, effectively suppressing local noise interference. During multimodal retrieval, this mechanism ensures the model is not misled by noise in the data, making judgments based on reliable semantic knowledge, thus providing richer and more accurate feature information for retrieval and improving the reliability of retrieval results. Simultaneously, local attention and bidirectional attention mechanisms are comprehensively utilized. In image semantic vector processing, local attention can dynamically adjust weights at different feature levels to achieve global semantic constraints on the image, avoiding interference from local features. During cross-modal fusion, the bidirectional attention mechanism establishes a dynamic association between text and image features, focusing on key visual elements to enhance visual features while also enhancing text features through inverse constraints. This optimizes semantic alignment from multiple dimensions, greatly improving the semantic accuracy, richness, and robustness of cross-modal retrieval, meeting the needs of complex retrieval tasks.

[0104] Example 2

[0105] Figure 6 This is a schematic diagram of the structure of a semantic recognition device provided in Embodiment 2 of this disclosure; as shown Figure 6 As shown, the device includes: a data acquisition module 210, a feature determination module 220, a cross-modal feature vector determination module 230, and a recognition result determination module 240.

[0106] The data acquisition module 210 is used to acquire multimodal input data corresponding to the data to be identified; the multimodal input data includes text data and image data.

[0107] The feature determination module 220 is used to perform alignment processing on the context-aware vector, the text data, and the image data through a dynamic encoder and a cross-modal alignment mechanism to determine text dynamic features, image dynamic features, and context features; the context features are determined based on the data to be identified.

[0108] The cross-modal feature vector determination module 230 is used to perform weighted fusion of the text dynamic features, the image dynamic features and the context features through a cross-attention mechanism and a gated weighted fusion mechanism to determine the cross-modal joint feature vector;

[0109] The recognition result determination module 240 is used to determine the semantic recognition result of the data to be recognized based on the cross-modal joint feature vector.

[0110] Embodiment 2 of this disclosure provides a semantic recognition device that solves the problem that traditional alignment methods cannot adapt to changes in data distribution and differences in semantic levels, and achieves fine-grained, context-sensitive, and dynamically evolving cross-modal semantic matching.

[0111] Furthermore, the dynamic encoder includes a text dynamic encoder and an image dynamic encoder; the device feature determination module 220 further includes:

[0112] The dynamic text vector determination unit is used to dynamically assign weights and perform weighted summation on the initial text feature vector using a dynamic text encoder to obtain a dynamic text vector; the dynamic text encoder includes a first input layer, a time-aware long short-term memory (LSTM) layer, and a dynamic weight assignment layer; the initial text feature vector is determined by encoding the text data;

[0113] A dynamic image vector determination unit is used to extract local features from an initial image feature vector using an image dynamic encoder to obtain a dynamic image vector; the dynamic image encoder includes a second input layer, a local attention layer, and a global semantic extraction layer; the initial image feature vector is determined by encoding the image data;

[0114] The cross-modal temporal alignment unit is used to perform cross-modal temporal alignment of the context-aware vector, the dynamic text vector, and the dynamic image vector based on the cross-modal alignment mechanism to obtain the text dynamic features, the image dynamic features, and the context features.

[0115] Furthermore, the dynamic text vector determination unit is also used for:

[0116] The initial text feature vector is transmitted to the time-aware LSTM layer using the first input layer.

[0117] A time-aware LSTM layer is used to process the time-series information of the initial text feature vector to obtain a time-aware word vector set; the time-aware word vector set includes at least one word vector.

[0118] The dynamic text vector is obtained by weighting each word vector in the time-aware word vector set using the dynamic weight allocation layer.

[0119] Furthermore, the dynamic image vector determination unit is also used for:

[0120] The initial image feature vector is segmented using the second input layer to obtain a set of image patches;

[0121] By using a local attention layer, the feature vector of each image block in the image block set is weighted using an attention mechanism to obtain a weighted set of image block feature vectors.

[0122] The dynamic image vector is generated based on the weighted set of image patch feature vectors through a global semantic extraction layer.

[0123] Furthermore, the cross-modal feature vector determination module 230 is also used for:

[0124] The text dynamic features and the image dynamic features are preprocessed to obtain the first text dynamic features and the first image dynamic features;

[0125] The first text dynamic features and the first image dynamic features are calculated by bidirectional cross-attention mechanism to obtain the second text dynamic features and the second image dynamic features.

[0126] According to the fusion weight, the second text dynamic features and the second image dynamic features are weighted and fused to obtain a fused feature vector; the fusion weight is determined based on the gated weighted fusion mechanism.

[0127] The cross-modal joint feature vector is determined based on the fused feature vector and the context feature vector.

[0128] Furthermore, the data acquisition module 210 is also used for:

[0129] Obtain the initial multimodal input data corresponding to the data to be identified; the initial multimodal input data includes first text data, first image data, initial audio data, and initial video data;

[0130] The initial audio data and the initial video data are sliced ​​to determine the second text data and the second image data;

[0131] The text data is determined based on the first text data and the second text data;

[0132] The image data is determined based on the first image data and the second image data.

[0133] The semantic recognition device provided in this disclosure can execute the semantic recognition method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method execution.

[0134] Example 3

[0135] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the embodiments of the present disclosure described and / or claimed herein.

[0136] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0137] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microprocessor, etc. Processor 11 performs the various methods and processes described above, such as semantic recognition methods.

[0139] In some embodiments, the semantic recognition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the semantic recognition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the semantic recognition method by any other suitable means (e.g., by means of firmware).

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] Computer programs for implementing the methods of embodiments of this disclosure may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of embodiments of this disclosure, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0145] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0146] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the embodiments of this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of the embodiments of this disclosure can be achieved, and this document does not impose any limitations.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of the embodiments disclosed herein. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the embodiments disclosed herein should be included within the scope of protection of the embodiments disclosed herein.

[0148] This disclosure also provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implements the semantic recognition method provided in any embodiment of this application.

[0149] In implementing a computer program product, computer program code for performing the operations of the embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0150] Note that the above are merely preferred embodiments and the technical principles applied in this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the protection scope of this disclosure. Therefore, although the embodiments of this disclosure have been described in detail above, this disclosure is not limited to the above embodiments. More other equivalent embodiments may be included without departing from the concept of this disclosure, and the scope of this disclosure is determined by the scope of the appended claims.

Claims

1. A semantic recognition method, characterized in that, include: Acquire the multimodal input data corresponding to the data to be identified; the multimodal input data includes text data and image data; By using a dynamic encoder and a cross-modal alignment mechanism, the context-aware vector, the text data, and the image data are aligned to determine the text dynamic features, image dynamic features, and context features. The contextual features are determined based on the data to be identified; By employing a cross-attention mechanism and a gated weighted fusion mechanism, the text dynamic features, the image dynamic features, and the context features are weighted and fused to determine a cross-modal joint feature vector. The semantic recognition result of the data to be identified is determined based on the cross-modal joint feature vector.

2. The method according to claim 1, characterized in that, The dynamic encoder includes a text dynamic encoder and an image dynamic encoder; The process of aligning the context-aware vector, the text data, and the image data using a dynamic encoder and a cross-modal alignment mechanism to determine text dynamic features, image dynamic features, and context features includes: A dynamic text vector is obtained by dynamically assigning weights and performing weighted summation on the initial text feature vector using a dynamic text encoder. The dynamic text encoder includes a first input layer, a time-aware long short-term memory (LSTM) layer, and a dynamic weight assignment layer. The initial text feature vector is determined by encoding the text data. A dynamic image vector is obtained by extracting local features from the initial image feature vector using an image dynamic encoder; the dynamic image encoder includes a second input layer, a local attention layer, and a global semantic extraction layer; the initial image feature vector is determined by encoding the image data. Based on the cross-modal alignment mechanism, the context-aware vector, the dynamic text vector, and the dynamic image vector are aligned across modal time sequences to obtain the text dynamic features, the image dynamic features, and the context features.

3. The method according to claim 2, characterized in that, The process of dynamically assigning weights and performing weighted summation on the initial text feature vector using a text dynamic encoder to obtain a dynamic text vector includes: The initial text feature vector is transmitted to the time-aware LSTM layer using the first input layer. A time-aware LSTM layer is used to process the time-series information of the initial text feature vector to obtain a time-aware word vector set; the time-aware word vector set includes at least one word vector. The dynamic text vector is obtained by weighting each word vector in the time-aware word vector set using the dynamic weight allocation layer.

4. The method according to claim 2, characterized in that, The step of extracting local features from the initial image feature vector using an image dynamic encoder to obtain a dynamic image vector includes: The initial image feature vector is segmented using the second input layer to obtain a set of image patches; By using a local attention layer, the feature vector of each image block in the image block set is weighted using an attention mechanism to obtain a weighted set of image block feature vectors. The dynamic image vector is generated based on the weighted set of image patch feature vectors through a global semantic extraction layer.

5. The method according to claim 1, characterized in that, The method of using a cross-attention mechanism and a gated weighted fusion mechanism to weightedly fuse the text dynamic features, the image dynamic features, and the context features to determine a cross-modal joint feature vector includes: The text dynamic features and the image dynamic features are preprocessed to obtain the first text dynamic features and the first image dynamic features; The first text dynamic features and the first image dynamic features are calculated by performing bidirectional cross-attention mechanism to obtain the second text dynamic features and the second image dynamic features. According to the fusion weight, the second text dynamic features and the second image dynamic features are weighted and fused to obtain a fused feature vector; the fusion weight is determined based on the gated weighted fusion mechanism. The cross-modal joint feature vector is determined based on the fused feature vector and the context feature vector.

6. The method according to claim 1, characterized in that, The acquisition of multimodal input data corresponding to the data to be identified includes: Obtain the initial multimodal input data corresponding to the data to be identified; the initial multimodal input data includes first text data, first image data, initial audio data, and initial video data; The initial audio data and the initial video data are sliced ​​to determine the second text data and the second image data; The text data is determined based on the first text data and the second text data; The image data is determined based on the first image data and the second image data.

7. A semantic recognition device, characterized in that, include: The data acquisition module is used to acquire multimodal input data corresponding to the data to be identified; the multimodal input data includes text data and image data; The feature determination module is used to align the context-aware vector, the text data, and the image data through a dynamic encoder and a cross-modal alignment mechanism to determine the text dynamic features, image dynamic features, and context features. The contextual features are determined based on the data to be identified; The cross-modal feature vector determination module is used to perform weighted fusion of the text dynamic features, the image dynamic features and the context features through a cross-attention mechanism and a gated weighted fusion mechanism to determine the cross-modal joint feature vector; The recognition result determination module is used to determine the semantic recognition result of the data to be recognized based on the cross-modal joint feature vector.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the semantic recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the semantic recognition method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the semantic recognition method as described in any one of claims 1-6.

Images