Deep semantic embedding method and device based on cross-environment separated architecture

Abstract

The invention relates to a deep semantic embedding method and device based on a cross-environment separated architecture, and the method comprises the steps: a cloud side generates global semantic embedding through a pre-training language model and a Hadamard matrix based on a class label text, mines a label hierarchical structure through a double-block attention mechanism, and obtains a classification label; generating a high-level semantic hash code containing intra-layer similarity and cross-layer correlation, and constructing an anchor point library; the edge device receives an anchor point library and global semantic embedding, learns a low-level semantic hash code through a discrete cyclic coordinate descent algorithm, and introduces an instance-level fine-grained weight to dynamically distribute a multi-modal data fusion weight; a cloud and an edge device record a corresponding relation of high-layer and low-layer hash codes through a dynamic semantic mapping table, a bit-by-bit calibration algorithm corrects cross-environment deviation, and the edge device only transmits hash code bits different from an anchor point library and compresses semantic information in combination with category-level high-level codes, so that deep semantic embedding with consistent cross-environment semantics is realized. Cloud precision and edge real-time performance are balanced, and efficient retrieval of multi-modal data is supported.

Description

Technical Field

[0001] This application relates to the field of semantic embedding technology, and in particular to a deep semantic embedding method and apparatus based on a cross-environmental separation architecture. Background Technology

[0002] With the rapid development of internet technology, multimodal data such as images, text, and audio are experiencing explosive growth, driving an increasingly urgent demand for deep semantic embedding technology in application scenarios such as intelligent retrieval and cross-modal interaction. Deep semantic embedding technology maps unstructured data into low-dimensional dense vectors (hash codes) to achieve efficient semantic matching, serving as a core support for large-scale media retrieval. However, existing technologies mostly rely on single-environment deployment, facing problems such as resource mismatch and poor semantic consistency in cross-environment scenarios—cloud models, while highly accurate, suffer from high latency, and edge models, while lightweight, have limited semantic representation capabilities, making it difficult to meet the dual requirements of real-time performance and accuracy.

[0003] In existing technologies, solutions represented by pure cloud-based deep semantic embedding models are widely used. These models generate high-quality semantic hash codes by deploying large-parameter encoders in the cloud, combining large-scale corpus pre-training with supervised fine-tuning, and achieving excellent performance on authoritative leaderboards such as MTEB. Their core principle is to map inputs such as text and images to a unified semantic space, and achieve cross-modal matching through cosine similarity calculation, making them suitable for centralized cloud-based retrieval scenarios such as search engines and intelligent recommendations.

[0004] The aforementioned pure cloud-based deep semantic embedding model lacks a unified semantic benchmark for the embedding vectors between the cloud and the edge. The hash codes generated by the same data in different environments exhibit semantic shifts, which seriously affects the consistency of cross-environment retrieval.

[0005] Based on this, this application provides a deep semantic embedding method and apparatus based on a cross-environmental decoupled architecture. Summary of the Invention

[0006] To address the issues of a lack of unified semantic benchmarks for embedding vectors between the cloud and edge in pure cloud-based deep semantic embedding models, and semantic offsets in hash codes generated from the same data in different environments, which seriously affect the consistency of cross-environment retrieval, this application provides a deep semantic embedding method and apparatus based on a cross-environment decoupled architecture.

[0007] Firstly, this application provides a deep semantic embedding method based on a cross-environment decoupled architecture, employing the following technical solution: including: Based on category-labeled text data, a pre-trained language model and Hadamard matrix are used to generate global semantic embeddings of category labels, mapping the label text to fixed-dimensional real-valued vectors as a semantic benchmark shared across environments. A dual-block attention mechanism is used to mine the label hierarchy structure and generate high-level semantic hash codes that include intra-layer similarity and cross-layer correlation. Randomly select samples from historical data as anchor points, and store the high-level semantic hash codes corresponding to the anchor points as an anchor point library; The anchor point library and the global semantic embedding are transmitted to the edge device, and a low-level semantic hash code is learned based on the discrete cyclic coordinate descent algorithm. The low-level semantic hash code achieves semantic alignment by minimizing the Hamming distance with the hash code of the anchor point library, and instance-level fine-grained weights are introduced to dynamically allocate fusion weights to the multimodal data of the edge device. A dynamic semantic mapping table is maintained by the cloud and edge devices to record the correspondence between the high-level semantic hash code and the low-level semantic hash code. Cross-environment hash code deviation is corrected by a bit-by-bit calibration algorithm. At the same time, the edge devices are controlled to transmit only the hash code bits that differ from the anchor library to the cloud. Semantic information is compressed by combining category-level high-level code.

[0008] Preferably, the method further includes: filtering out low-confidence tags from noisy tags collected by the edge device based on a threshold to obtain the remaining noise; The remaining noise is corrected based on the global semantic embedding in the cloud to obtain the corrected remaining label vector. The cosine similarity between the remaining label vector and the global semantic embedding is calculated, and the labels with a cosine similarity greater than a preset value are retained for hash code learning. When an edge device detects new category data, it sends the name of the new category data to the cloud. The cloud uses a language model to generate new tag embeddings and updates the global semantic library. The control edge device learns the new category low-level hash code based on the new label embedding and local samples through a discrete latent factor model.

[0009] Preferably, the step of generating global semantic embeddings of category labels based on category label text data using a pre-trained language model and a Hadamard matrix, mapping the label text to fixed-dimensional real-valued vectors as a semantic benchmark shared across environments, and mining the label hierarchy structure through a dual-block attention mechanism to generate high-level semantic hash codes containing intra-layer similarity and cross-layer correlation, including: Based on the category label text data, the category label text data is input into a pre-trained language model to generate initial label embeddings; The initial label embedding is orthogonally transformed based on the Hadamard matrix, mapping the label text to a fixed-dimensional real-valued vector, generating the global semantic embedding of the category label, and using the global semantic embedding as a semantic benchmark shared across environments. For the label hierarchy contained in the category label text data, a dual-block attention mechanism is used to mine the instance-category similarity within the same label layer and the cross-layer correlation between different label layers. The instance-category similarity within the same layer is determined by measuring the semantic distance between the sample and the corresponding category center within the same label layer, and the cross-layer correlation is determined by calculating the cosine similarity of category nodes between different label layers. By integrating the intra-layer similarity and the cross-layer correlation, and using the global semantic embedding as the initial value, the high-level semantic hash code containing the intra-layer similarity and the cross-layer correlation is generated through iterative optimization using a discrete cyclic coordinate descent algorithm.

[0010] Preferably, the step of performing an orthogonal transformation on the initial label embedding based on the Hadamard matrix to map the label text into a fixed-dimensional real-valued vector, generating the global semantic embedding of the category label, and using the global semantic embedding as a semantic benchmark shared across environments, includes: Obtain a Hadamard matrix of a preset order, wherein the order of the Hadamard matrix is ​​determined according to the dimension of the target global semantic embedding, and satisfies that the row vectors and column vectors are pairwise orthogonal; The initial label embedding is multiplied by the Hadamard matrix, and the initial label embedding is mapped from the original dimension to the target fixed dimension through orthogonal transformation to obtain an intermediate semantic vector, wherein the target fixed dimension matches the length of the high-level semantic hash code. The intermediate semantic vector is normalized, and the vector elements are scaled to a preset range to generate a real-valued vector with a fixed dimension, thereby obtaining the global semantic embedding of the category label. The global semantics are embedded and stored in a cloud semantic library, serving as a semantic alignment benchmark when edge devices generate the low-level semantic hash code, and as a reference standard for cross-environment semantic consistency verification.

[0011] Preferably, the step of transmitting the anchor library and the global semantic embedding to the edge device, and learning a low-level semantic hash code based on the discrete cyclic coordinate descent algorithm, wherein the low-level semantic hash code achieves semantic alignment by minimizing the Hamming distance with the hash code of the anchor library, and introducing instance-level fine-grained weights to dynamically allocate fusion weights to the multimodal data of the edge device, includes: The edge device receives and stores the global semantic embedding of the high-level semantic hash code and category label corresponding to the anchor point in the anchor point library; Local multimodal data is acquired through edge devices. The multimodal data includes image feature data and text feature data, wherein the image feature data is extracted through a lightweight convolutional neural network, and the text feature data is extracted through a bag-of-words model or a pre-trained language model. The low-level semantic hash code is initialized based on the discrete cyclic coordinate descent algorithm. The high-level semantic hash code of the anchor library is used as a reference. The hash code bits are iteratively optimized by minimizing the Hamming distance between the low-level semantic hash code and the hash code of the anchor library. The low-level semantic hash code is updated bit by bit until convergence. The fusion weights are dynamically allocated by calculating the discriminative index of each modality feature in the multimodal data: if the discriminative index of the image feature is higher than that of the text feature, the image modality weight is set as the first weight and the text modality weight is set as the second weight, and the first weight is greater than the second weight; otherwise, the weight ratio is adjusted. Multimodal features are fused based on the assigned instance-level fine-grained fusion weights. The fused features are then input into the optimized low-level semantic hash code, and the low-level semantic hash code, which is aligned with the semantics of the cloud, is output.

[0012] Preferably, the step of initializing the low-level semantic hash code based on the discrete cyclic coordinate descent algorithm, using the high-level semantic hash code of the anchor library as a benchmark, iteratively optimizing the hash code bits by minimizing the Hamming distance between the low-level semantic hash code and the hash code of the anchor library, and updating the low-level semantic hash code bit by bit until convergence, includes: By using the feature dimensions of local multimodal data from the edge device, the low-level semantic hash code is randomly initialized as an r-bit binary vector, with the initial hash code bit values ​​being {-1, 1}. Extract the high-level semantic hash code corresponding to the anchor point from the anchor point library as the optimization benchmark, denoted as the benchmark hash code B_anchor∈{-1,1}^{e×r}, where e is the number of anchor points and r is the length of the low-level semantic hash code. Calculate the Hamming distance D between the currently initialized low-level semantic hash code B_current and the base hash code B_anchor, which is D = ||B_current⊕B_anchor||_0, where ⊕ is the XOR operation and ||·||_0 is the L0 norm. The discrete cyclic coordinate descent algorithm is used to iteratively optimize the hash code bits: all hash code bits except the k-th bit are fixed, where k is not greater than r, and the optimal value of the k-th bit is determined by maximizing the objective function, and each bit of B_current is updated bit by bit; Repeat the bit-by-bit update step, recalculate the Hamming distance after each iteration, and stop iterating when the change in Hamming distance between two consecutive iterations is less than a preset threshold or the number of iterations reaches a preset upper limit. At this time, the low-level semantic hash code converges to the state with the minimum Hamming distance with the base hash code, thereby realizing the semantic structure alignment between the edge device and the cloud.

[0013] Preferably, the fusion of multimodal features based on the allocated instance-level fine-grained fusion weights, inputting the fused features into the optimized low-level semantic hash code, and outputting the low-level semantic hash code aligned with cloud semantics, includes: The multimodal feature vectors are weighted and fused based on the instance-level fine-grained weights: the image feature vector is scaled according to the image modality weights, the text feature vector is scaled according to the text modality weights, and the fused feature vector is generated by adding or concatenating the vectors element by element, wherein the weight allocation satisfies that the sum of the image modality weights and the text modality weights is 1; The fused feature vector is input into the optimized generation model of the low-level semantic hash code. The generation model is a binary neural network that has been iteratively optimized by the discrete cyclic coordinate descent algorithm. The model parameters have been initially optimized by minimizing the Hamming distance with the anchor library hash code. The fused feature vector is non-linearly mapped using the generative model to output an initial low-level semantic hash code. Using the global semantic embedding in the cloud as a reference, the cosine similarity between the semantic vector corresponding to the initial low-level semantic hash code and the global semantic embedding of the category label is calculated. If the cosine similarity is not less than a preset threshold, the initial low-level semantic hash code is directly output. If the similarity is less than the preset threshold, the low-level semantic hash code is regenerated by fine-tuning the model parameters until the cosine similarity is not less than the preset threshold, thus obtaining the low-level semantic hash code that is aligned with the semantics in the cloud.

[0014] Preferably, the step of maintaining a dynamic semantic mapping table through cloud and edge devices to record the correspondence between the high-level semantic hash code and the low-level semantic hash code, correcting cross-environment hash code deviations through a bit-by-bit calibration algorithm, and controlling the edge devices to only transmit hash code bits that differ from the anchor library to the cloud, combined with category-level high-level code compression of semantic information, includes: The dynamic semantic mapping table is initialized jointly by the cloud and edge devices. The dynamic semantic mapping table includes fields: category label ID, the high-level semantic hash code, the low-level semantic hash code, and the corresponding relationship update timestamp, which are used to record the mapping relationship of hash codes across environments. The cloud and edge devices periodically synchronize the dynamic semantic mapping table, compare the high-level semantic hash code and the low-level semantic hash code of each record in the table bit by bit, calculate the difference value of the corresponding bit, and when the cumulative difference value exceeds the preset threshold, trigger the calibration mechanism to adjust the corresponding bit of the low-level semantic hash code so that the cross-environment hash code deviation is controlled within the preset range. When the edge device feeds back the local low-level semantic hash code to the cloud, it compares the hash code to be transmitted with the high-level semantic hash code of the corresponding anchor in the anchor library through an XOR operation, identifies the difference bits, extracts and transmits only the position and value information of the difference bits, and shields the redundant transmission of the same bits. The category-level high-level code is predefined by cloud and edge devices. The category-level high-level code is a fixed-length code based on category semantics. When transmitting differential hash code bits, the differential bit information is bound to the high-level code of the corresponding category. The semantic context of the differential bit is compressed by the high-level code to ensure that the transmitted semantic information is compact and undistorted. By receiving differential hash code bits and compressed category-level high-level code transmitted from edge devices in the cloud, the correspondence between the high-level semantic hash code and the low-level semantic hash code is updated based on the dynamic semantic mapping table to complete cross-environment semantic collaboration.

[0015] Preferably, the method further includes: Receive cross-modal retrieval requests, which include image queries or text queries; If the cross-modal retrieval request is initiated by the cloud, the high-level semantic hash code stored in the cloud is called for retrieval, and the retrieval result is initially obtained by calculating the Hamming distance between the hash code to be queried and the high-level semantic hash code in the database; If the cross-modal retrieval request is initiated by an edge device, the low-level semantic hash code generated by the edge device is invoked for local retrieval. By calculating the Hamming distance between the hash code to be queried and the low-level semantic hash code in the local database, preliminary retrieval results are obtained. When the local retrieval results of the edge device are insufficient or the retrieval results are lower than a preset threshold, the edge device transmits the hash code to be queried and the corresponding category-level high-level code to the cloud. The cloud maps the low-level semantic hash code to the high-level semantic hash code based on the dynamic semantic mapping table, and performs a global retrieval in conjunction with the cloud database to supplement the retrieval results. The cloud and edge devices are controlled to perform semantic consistency verification on the search results through the global semantic embedding: the cosine similarity between the semantic vector corresponding to the hash code of the search result and the global semantic embedding of the query category is calculated, and search results with cosine similarity lower than a preset threshold are filtered out, while semantically consistent search results are retained. It aggregates local search results and cloud-based supplementary search results, sorts them by similarity, and returns the final search results.

[0016] Secondly, this application provides a deep semantic embedding device based on a cross-environment decoupled architecture, which adopts the following technical solution, including: The global semantic module is used to generate global semantic embeddings of category labels based on category label text data using a pre-trained language model and a Hadamard matrix. It maps the label text into a fixed-dimensional real-valued vector as a semantic benchmark shared across environments. It also mines the label hierarchy structure through a dual-block attention mechanism to generate a high-level semantic hash code that includes intra-layer similarity and cross-layer correlation. An anchor point generation module is used to randomly select samples from historical data as anchor points and store the high-level semantic hash codes corresponding to the anchor points as an anchor point library; The edge retrieval module is used to transmit the anchor point library and the global semantic embedding to the edge device, and learn the low-level semantic hash code based on the discrete cyclic coordinate descent algorithm. The low-level semantic hash code achieves semantic alignment by minimizing the Hamming distance with the hash code of the anchor point library, and introduces instance-level fine-grained weights to dynamically allocate fusion weights to the multimodal data of the edge device. The data transmission module is used to maintain a dynamic semantic mapping table between the cloud and edge devices, record the correspondence between the high-level semantic hash code and the low-level semantic hash code, correct cross-environment hash code deviations through a bit-by-bit calibration algorithm, and control the edge devices to only transmit hash code bits that differ from the anchor library to the cloud, and compress semantic information in combination with category-level high-level code.

[0017] Thirdly, this application also provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded and executed by the processor, such as the deep semantic embedding method based on the cross-environment decoupled architecture described above.

[0018] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above regarding the deep semantic embedding method based on a cross-environmental decoupled architecture.

[0019] In summary, this application utilizes a cloud-based approach based on category-labeled text. A pre-trained language model and Hadamard matrix are used to generate global semantic embeddings. A dual-block attention mechanism is then employed to mine the hierarchical structure of the labels, generating high-level semantic hash codes that include intra-layer similarity and cross-layer correlation. An anchor point library is constructed to store typical sample hash codes. Edge devices receive the anchor point library and global semantic embeddings, learn low-level semantic hash codes using a discrete cyclic coordinate descent algorithm, and dynamically allocate multimodal data fusion weights using instance-level fine-grained weights. The cloud and edge devices record the correspondence between high- and low-level hash codes through a dynamic semantic mapping table. A bit-by-bit calibration algorithm corrects cross-environmental deviations. The edge device only transmits hash code bits that differ from the anchor point library and combines them with category-level high-level code to compress semantic information. Simultaneously, a two-stage denoising process improves label quality. When a new category appears, the cloud updates the global semantic library, and the edge device learns the low-level hash code for the new category based on the new label embedding and local samples. During cross-environment collaborative retrieval, the cloud and edge devices respectively call the high- and low-level hash codes for initial retrieval, and output the results after semantic consistency verification. This enables deep semantic embedding with consistent semantics across environments, balancing cloud accuracy and edge real-time performance, supporting efficient retrieval of large-scale multimodal data, improving retrieval accuracy and resource utilization efficiency in cross-environment collaborative scenarios, and reducing transmission latency and privacy leakage risks when applied to edge devices without uploading the original data to the cloud for embedding computation. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a deep semantic embedding method based on a cross-environment decoupled architecture.

[0021] Figure 2 This is a structural block diagram of a deep semantic embedding device based on a cross-environment decoupled architecture. Detailed Implementation

[0022] The following combination Figure 1 - Figure 2 This application will be described in further detail.

[0023] Pure cloud-based deep semantic embedding models rely on cloud computing power. When applied to edge devices, raw data must be uploaded to the cloud for embedding computation, leading to transmission delays and privacy risks. Deploying lightweight models directly on edge devices requires compressing model parameters, resulting in decreased semantic representation accuracy. Furthermore, the embedding vectors between the cloud and edge lack a unified semantic benchmark; hash codes generated from the same data in different environments exhibit semantic shifts, severely impacting the consistency of cross-environment retrieval. Therefore, existing pure cloud models cannot meet the cross-environment collaborative retrieval requirements of real-time edge response and global cloud optimization. Based on this, this application proposes a deep semantic embedding method based on a cross-environment decoupled architecture.

[0024] Reference Figure 1 The embodiments of this application include at least steps S10 to S40.

[0025] S10, based on category label text data, uses a pre-trained language model and Hadamard matrix to generate global semantic embeddings of category labels, maps the label text into fixed-dimensional real-valued vectors as a semantic benchmark shared across environments, and mines the label hierarchical structure through a dual-block attention mechanism to generate high-level semantic hash codes that include intra-layer similarity and cross-layer correlation.

[0026] S20: Randomly select samples from historical data as anchor points, and store the high-level semantic hash codes corresponding to the anchor points as an anchor point library.

[0027] S30 transmits the anchor library and global semantic embedding to the edge device, and learns the low-level semantic hash code based on the discrete cyclic coordinate descent algorithm. The low-level semantic hash code achieves semantic alignment by minimizing the Hamming distance with the hash code of the anchor library, and introduces instance-level fine-grained weights to dynamically allocate fusion weights to the multimodal data of the edge device.

[0028] S40 maintains a dynamic semantic mapping table through cloud and edge devices, records the correspondence between high-level semantic hash codes and low-level semantic hash codes, corrects cross-environment hash code deviations through a bit-by-bit calibration algorithm, and controls edge devices to only transmit hash code bits that differ from the anchor library to the cloud, and combines category-level high-level code compression semantic information.

[0029] Specifically, deep semantic embedding is achieved based on a cross-environmental separation architecture. In the cloud, global semantic embeddings are generated using a pre-trained language model and a Hadamard matrix based on category-labeled text (sharing a semantic baseline across environments). A dual-block attention mechanism is used to mine the label hierarchy, generating high-level semantic hash codes containing intra-layer similarity and cross-layer correlation, and constructing an anchor point library. Edge devices receive the anchor point library and the global semantic embeddings, learning low-level semantic hash codes through a discrete cyclic coordinate descent algorithm (minimizing the Hamming distance with the anchor point library to achieve semantic alignment). Instance-level fine-grained weights are introduced to dynamically allocate multimodal data fusion weights. The cloud and edge devices record the correspondence between high and low-level hash codes through a dynamic semantic mapping table, and a bit-by-bit calibration algorithm corrects cross-environmental deviations. Edge devices only transmit hash code bits that differ from the anchor point library and combine them with category-level high-level code to compress semantic information, achieving deep semantic embedding with consistent semantics across environments. This balances cloud accuracy and edge real-time performance, supporting efficient retrieval of large-scale multimodal data.

[0030] In some embodiments, considering the emergence of new category data, the corresponding processing steps for updating the global semantics are as follows: For noisy labels collected by the edge device, low-confidence labels are filtered based on a threshold to obtain residual noise; the residual noise is corrected based on the global semantic embedding in the cloud to obtain the corrected residual label vector; the cosine similarity between the residual label vector and the global semantic embedding is calculated, and labels with a cosine similarity greater than a preset value are retained for hash code learning; when the edge device detects new category data, the new category data name is sent to the cloud, and the cloud uses a language model to generate a new label embedding and update the global semantic database; the edge device is controlled to learn the low-level hash code of the new category based on the new label embedding and local samples through a discrete latent factor model.

[0031] In practice, when new category data appears, to achieve global semantic updates, the noisy labels collected by edge devices are first denoised in two stages: low-confidence labels (e.g., confidence < 0.5) are filtered out by thresholding to obtain the remaining noise, and then the remaining noise is corrected based on the global semantic embedding in the cloud. The cosine similarity between the label vector and the global semantic embedding is calculated, and labels with similarity > preset value are retained for hash code learning. Subsequently, the edge device sends the new category name to the cloud, and the cloud uses a language model to generate a new label embedding and update the global semantic library. Finally, the edge device learns the low-level hash code of the new category based on the new label embedding and local samples through a discrete latent factor model. This eliminates the need to retrain historical data, ensuring cross-environment semantic consistency when the new category is added and maintaining the long-term semantic stability of the hash code.

[0032] In some embodiments, step S10 specifically includes the following steps: based on the category label text data, inputting the category label text data into a pre-trained language model to generate an initial label embedding; performing an orthogonal transformation on the initial label embedding based on the Hadamard matrix to map the label text into a fixed-dimensional real-valued vector, generating a global semantic embedding of the category labels, and using the global semantic embedding as a semantic benchmark shared across environments; for the label hierarchy structure contained in the category label text data, mining instance-category similarity within the same label layer and cross-layer correlation between different label layers through a dual-block attention mechanism, wherein instance-category similarity within the same layer is determined by measuring the semantic distance between samples within the same label layer and the corresponding category center, and cross-layer correlation is determined by calculating the cosine similarity of category nodes between different label layers; integrating intra-layer similarity and cross-layer correlation, using the global semantic embedding as the initial value, iteratively optimizing through a discrete cyclic coordinate descent algorithm to generate a high-level semantic hash code containing intra-layer similarity and cross-layer correlation.

[0033] The formula for positive valence transformation is as follows: Where H is the Hadamard matrix. Embed the initial tag. For global semantic embedding; Cross-layer correlation is represented as follows: Embedding of adjacent layer categories; The discrete optimization objective is: For intra-layer similarity, B is the high-level semantic hash code.

[0034] To uncover intra-layer similarity and cross-layer correlation within the tag hierarchy, this scheme employs a hierarchical semantic fusion objective function, the specific formula of which is: ; Where B is the sample hash code matrix, Let k be the category hash code matrix of the k-th layer. Let k be the label matrix. This is the cross-layer category similarity matrix between the k-th layer and the (k+1)-th layer (calculated using cosine similarity). To balance the parameters, this formula addresses the problem of semantic information loss caused by traditional methods that only use single-level labels. This is achieved by constraining instance-category similarity within a layer in the first term (ensuring that the hash codes of samples of the same type in the same layer are similar) and constraining cross-layer category correlation in the second term (ensuring semantic association between adjacent layers). This allows the generated high-level semantic hash codes to simultaneously contain both fine-grained and coarse-grained semantics in the hierarchical structure.

[0035] Specifically, the class label text is first input into a pre-trained language model to generate initial label embeddings. Then, the initial embeddings are orthogonally transformed based on the Hadamard matrix, mapping them into fixed-dimensional real-valued vectors as global semantic embeddings shared across environments. Next, a dual-block attention mechanism is used to mine the label hierarchy structure. Intra-layer instance-class similarity is determined by the semantic distance between the sample and the class center, and cross-layer correlation is calculated by the cosine similarity of class nodes. Finally, the intra-layer and cross-layer semantic information are integrated, and the global semantic embedding is used as the initial value. Through iterative optimization using a discrete cyclic coordinate descent algorithm, a high-level semantic hash code containing hierarchical semantics is generated. Furthermore, step S10 also includes the following steps: obtaining a Hadamard matrix of a preset order, wherein the order of the Hadamard matrix is ​​determined according to the dimension of the target global semantic embedding and satisfies that the row vectors and column vectors are pairwise orthogonal; performing matrix multiplication on the initial label embedding and the Hadamard matrix, and mapping the initial label embedding from the original dimension to the target fixed dimension through orthogonal transformation to obtain an intermediate semantic vector, wherein the target fixed dimension matches the length of the high-level semantic hash code; normalizing the intermediate semantic vector, scaling the vector elements to a preset range, generating a real-valued vector with a fixed dimension, and obtaining the global semantic embedding of the category label; storing the global semantic embedding in the cloud semantic library as a semantic alignment benchmark when edge devices generate low-level semantic hash codes, and as a reference standard for cross-environment semantic consistency verification.

[0036] Specifically, firstly, a Hadamard matrix of a preset order is obtained based on the target global semantic embedding dimension (matching the length of the high-level semantic hash code), which satisfies the condition that the row and column vectors are pairwise orthogonal: Next, the initial label embedding is multiplied by the Hadamard matrix, and then orthogonal transformation is performed: The original dimensions are mapped to the target fixed dimensions to obtain the intermediate semantic vector. ; then Normalization is performed: ; Scale to a preset range ([0,1][0,1]) to generate a global semantic embedding. Finally, the semantics are stored in the cloud library as a benchmark for edge semantic alignment and a standard for cross-environment consistency verification. The semantic structure is preserved through orthogonal transformation and the scale is normalized to ensure the stability and portability of the cross-environment semantic benchmark.

[0037] In some embodiments, step S30 specifically includes the following steps: receiving and storing the global semantic embedding of the high-level semantic hash code and category label corresponding to the anchor in the anchor library through an edge device; acquiring local multimodal data through the edge device, the multimodal data including image feature data and text feature data, wherein the image feature data is extracted through a lightweight convolutional neural network, and the text feature data is extracted through a bag-of-words model or a pre-trained language model; initializing the low-level semantic hash code based on the discrete cyclic coordinate descent algorithm, and using the high-level semantic hash code of the anchor library as a benchmark, minimizing the low-level semantic hash code and the anchor... The Hamming distance between the point-based hash codes is iteratively optimized to improve the hash code bits, updating the low-level semantic hash code bit by bit until convergence. The fusion weights are dynamically allocated by calculating the discriminative index of each modality feature in the multimodal data: if the discriminative index of the image feature is higher than that of the text feature, the image modality weight is set as the first weight and the text modality weight is set as the second weight, and the first weight is greater than the second weight; otherwise, the weight ratio is adjusted. Based on the allocated instance-level fine-grained fusion weights, the multimodal features are fused, and the fused features are input into the optimized low-level semantic hash code, outputting a low-level semantic hash code that is aligned with the semantics of the cloud.

[0038] Furthermore, step S30 also includes the following steps: randomly initializing the low-level semantic hash code as an r-bit binary vector based on the feature dimension of the local multimodal data by the edge device, with the initial hash code bit value being {-1, 1}; Extract the high-level semantic hash code corresponding to the anchor point from the anchor point library as the optimization benchmark, denoted as the benchmark hash code B_anchor∈{-1,1}^{e×r}, where e is the number of anchor points and r is the length of the low-level semantic hash code. Calculate the Hamming distance D between the currently initialized low-level semantic hash code B_current and the baseline hash code B_anchor, which is D = ||B_current⊕B_anchor||_0, where ⊕ is the XOR operation and ||·||_0 is the L0 norm. The discrete cyclic coordinate descent algorithm is used to iteratively optimize the hash code bits: all hash code bits except the k-th bit are fixed, where k is not greater than r, and the optimal value of the k-th bit is determined by maximizing the objective function, and each bit of B_current is updated bit by bit; Repeat the bit-by-bit update step, recalculate the Hamming distance after each iteration, and stop iterating when the change in Hamming distance between two consecutive iterations is less than a preset threshold or the number of iterations reaches a preset upper limit. At this time, the low-level semantic hash code converges to the state with the minimum Hamming distance with the base hash code, thereby achieving semantic structure alignment between edge devices and the cloud.

[0039] Furthermore, step S30 also includes the following steps: weighted fusion of multimodal feature vectors based on instance-level fine-grained weights: scaling the image feature vector according to the image modality weights, scaling the text feature vector according to the text modality weights, and generating a fused feature vector by adding or concatenating the vectors element by element, wherein the weight allocation satisfies that the sum of the image modality weights and the text modality weights is 1; The fused feature vector is input into the optimized low-level semantic hash code generation model. The generation model is a binary neural network that has been iteratively optimized using a discrete cyclic coordinate descent algorithm. The model parameters have been initially optimized by minimizing the Hamming distance with the anchor library hash code. The generation model performs a nonlinear mapping on the fused feature vector to output the initial low-level semantic hash code. Using the global semantic embedding in the cloud as a reference, the cosine similarity between the semantic vector corresponding to the initial low-level semantic hash code and the global semantic embedding of the category label is calculated. If the cosine similarity is not less than a preset threshold, the initial low-level semantic hash code is directly output. If the similarity is less than the preset threshold, the low-level semantic hash code is regenerated by fine-tuning the model parameters until the cosine similarity is not less than the preset threshold, thus obtaining a low-level semantic hash code that is aligned with the cloud semantics.

[0040] In this embodiment, a dynamic fine-grained weight calculation model is used to perform weighted fusion of multimodal features. The dynamic fine-grained weight calculation model is as follows: ; in, Let m be the weight of the m-th mode of the i-th sample. The entropy value of this modality feature measures discriminability. This represents the feature vector of the i-th sample in the m-th mode. Representing the eigenvector The j-th element, d feature vector Dimensions.

[0041] In practice, to ensure both "semantic alignment with cloud anchors" and "semantic accuracy of local multimodal data" during the generation of low-level semantic hash codes by edge devices, two core issues need to be addressed: first, how to define a discrete optimization objective that includes data similarity and label semantics; and second, how to efficiently solve this objective under binary constraints.

[0042] 1. After receiving the anchor library (containing a high-level semantic hash code E for e anchors) and global semantic embedding transmitted from the cloud, the edge device needs to generate an r-bit low-level semantic hash code B for n local multimodal samples. To ensure that B is semantically similar to the anchor hash code E (cross-environment alignment) and accurately encodes the true category label L of the samples, this embodiment uses a discrete latent factor optimization model, with the objective function defined as: ; Where S is the sample-anchor similarity matrix, P is the label embedding projection matrix, and γ=0.5 is the tradeoff parameter. The formula constrains the similarity of the hash codes of samples and anchors through the first term (ensuring cross-environment alignment), and constrains the semantic consistency of hash codes and category labels through the second term (ensuring local semantic accuracy), and forces B, EB, and E to be binary matrices (to avoid quantization errors).

[0043] 2. The above objective function is a high-dimensional discrete optimization problem (r-bit hash code has...). (Possible combinations). Therefore, a discrete cyclic coordinate descent algorithm is introduced to decompose the optimization of the r-bit hash code into r "single-bit optimizations". The specific steps are as follows: Initialization: Randomly generate initial hash codes ; Bit-by-bit optimization: For the k-th bit (k=1,2,...,r), fix the other r-1 bits of B, and optimize only the value of the k-th bit. It belongs to (0,1). At this point, the objective function can be simplified to about... For a single-variable function, the optimal value is determined by maximizing the increment of the objective function: ; in, For weight parameters, Take the k-th position The objective function value at that time; Iterative convergence: Repeat the step-by-step optimization until the change in the objective function is less than the threshold or the number of iterations reaches the preset number. At this point, B converges to a local optimum.

[0044] Specifically, it receives high-level hash codes from the anchor library and global semantic embeddings, extracts features from images (lightweight CNN) and text (bag-of-words / pre-trained models), and randomly initializes r-bit binary low-level hash codes. Using the anchor reference hash code (Banchor) as the optimization objective, the Hamming distance is calculated: ,in For XOR operation; a discrete cyclic coordinate descent algorithm is used for bit-by-bit optimization: all bits except the k-th bit are fixed, and the optimal bit is determined by maximizing the objective function. Iteration continues until the Hamming distance change is less than a threshold or the upper limit of iteration is reached, achieving semantic structure alignment; weights (image weights) are dynamically allocated based on modality discriminability. Text weight For image feature vectors, (Text feature vectors), fused feature vectors: Finally, the initial hash code is obtained by inputting the binarization generation model. The cosine similarity S with the global semantic embedding is calculated. If S ≥ the threshold, it is output; otherwise, the parameters are fine-tuned and regenerated to finally obtain the low-level semantic hash code that is aligned with the semantics in the cloud.

[0045] In some embodiments, step S40 specifically includes the following steps: Initializing a dynamic semantic mapping table jointly by the cloud and edge devices. The dynamic semantic mapping table includes fields: category label ID, high-level semantic hash code, low-level semantic hash code, and corresponding relationship update timestamp, used to record the mapping relationship of cross-environment hash codes; periodically synchronizing the dynamic semantic mapping table between the cloud and edge devices, comparing the high-level semantic hash code and low-level semantic hash code of each record in the table bit by bit, calculating the difference value of corresponding bits, and triggering a calibration mechanism when the accumulated difference value exceeds a preset threshold, adjusting the corresponding bits of the low-level semantic hash code to control the cross-environment hash code deviation within a preset range; when the edge device feeds back its local low-level semantic hash code to the cloud, it performs an XOR operation. The process involves comparing the hash code to be transmitted with the high-level semantic hash code of the corresponding anchor point in the anchor point library, identifying the difference bits, and extracting and transmitting only the position and value information of the difference bits, while shielding redundant transmission of identical bits. Predefined category-level high-level codes are used in the cloud and edge devices. These category-level high-level codes are fixed-length codes based on category semantics. When transmitting the difference hash code bits, the difference bit information is bound to the corresponding category's high-level code. The semantic context of the difference bits is compressed using the high-level code, ensuring that the transmitted semantic information is compact and distortion-free. The cloud receives the difference hash code bits and compressed category-level high-level code transmitted from the edge device, and updates the correspondence between the high-level and low-level semantic hash codes based on a dynamic semantic mapping table, completing cross-environment semantic collaboration.

[0046] Specifically, a dynamic semantic mapping table is jointly initialized to record category label IDs, high- and low-level semantic hash codes, and update timestamps. The mapping table is periodically synchronized, and the high- and low-level hash codes are compared bit by bit to calculate the difference value. When the accumulated difference exceeds a threshold, calibration is triggered, and the corresponding bits of the low-level hash code are adjusted to keep the deviation within a preset range. When edge devices feed back to the cloud, the hash code to be transmitted is compared with the high-level hash code in the anchor library through an XOR operation. Only the transmission difference position and value are extracted, and the difference bit information is bound together with the predefined category-level high-level code (fixed-length encoding based on category semantics) to compress the semantic context. After receiving the data, the cloud updates the correspondence between high- and low-level hash codes based on the mapping table to achieve cross-environment semantic collaboration and ensure dynamic alignment of hash codes and efficient communication.

[0047] In some embodiments, considering that the retrieval request is issued from the cloud or the edge, the corresponding processing steps for determining the retrieval results are as follows: A cross-modal retrieval request is received, which may include image or text queries; if the cross-modal retrieval request is initiated by the cloud, the high-level semantic hash code stored in the cloud is invoked for retrieval, and the retrieval result is initially obtained by calculating the Hamming distance between the hash code to be queried and the high-level semantic hash code in the database; if the cross-modal retrieval request is initiated by the edge device, the low-level semantic hash code generated by the edge device is invoked for local retrieval, and the retrieval result is initially obtained by calculating the Hamming distance between the hash code to be queried and the low-level semantic hash code in the local database; when the edge device's local retrieval result is obtained... If the search results are insufficient or fall below a preset threshold, the edge device transmits the hash code to be queried and the corresponding category-level high-level code to the cloud. The cloud uses a dynamic semantic mapping table to map the low-level semantic hash code to a high-level semantic hash code, and performs a global search in conjunction with the cloud database to supplement the search results. The cloud and edge devices are controlled to perform semantic consistency verification of the search results through global semantic embedding: the cosine similarity between the semantic vector corresponding to the hash code of the search result and the global semantic embedding of the query category is calculated, and search results with a cosine similarity below a preset threshold are filtered out, while semantically consistent search results are retained. The local search results and the cloud-supplemented search results are aggregated, sorted by similarity, and the final search results are returned.

[0048] Specifically, the system receives cross-modal retrieval requests (including image or text queries) and retrieves the corresponding hash code based on the initiator: when initiated from the cloud, it calls the high-level semantic hash code; when initiated from the edge, it calls the low-level semantic hash code. Initial retrieval results are obtained by calculating the Hamming distance between the hash code to be queried and the corresponding hash code in the database. When the local retrieval results at the edge are insufficient or below a preset threshold, the edge transmits the hash code to be queried and the category-level high-level code to the cloud. The cloud maps the low-level hash code to the high-level hash code based on a dynamic semantic mapping table and supplements the results with global retrieval from the cloud database. Subsequently, the retrieval results are verified through global semantic embedding, calculating the cosine similarity between the semantic vector of the result hash code and the global semantic embedding of the query category. Results below the threshold are filtered out, and semantically consistent retrieval results are retained. Finally, the local and cloud-supplemented results are aggregated, sorted by similarity, and the final retrieval results are returned.

[0049] The implementation principle of a deep semantic embedding method based on a cross-environmental separation architecture in this application is as follows: Global semantic embeddings are generated in the cloud based on category-labeled text using a pre-trained language model and a Hadamard matrix. A dual-block attention mechanism is used to mine the label hierarchy, generating high-level semantic hash codes containing intra-layer similarity and cross-layer correlation. An anchor point library is constructed to store typical sample hash codes. Edge devices receive the anchor point library and global semantic embeddings, learn low-level semantic hash codes using a discrete cyclic coordinate descent algorithm, and dynamically allocate multimodal data fusion weights using instance-level fine-grained weights. The cloud and edge devices record the correspondence between high- and low-level hash codes through a dynamic semantic mapping table. A bit-by-bit calibration algorithm corrects cross-environmental deviations. The edge device only transmits hash code bits that differ from the anchor point library and combines them with category-level high-level code to compress semantic information. Simultaneously, label quality is improved through two-stage denoising. When a new category appears, the cloud updates the global semantic library, and the edge device learns the low-level hash code for the new category based on the new label embedding and local samples. During cross-environmental collaborative retrieval, the cloud and edge devices respectively call the high- and low-level hash codes for preliminary retrieval, and output the results after semantic consistency verification. This enables deep semantic embedding with consistent semantics across environments, balancing cloud accuracy and edge real-time performance, supporting efficient retrieval of large-scale multimodal data, improving retrieval accuracy and resource utilization efficiency in cross-environment collaborative scenarios, and reducing transmission latency and privacy leakage risks when applied to edge devices without uploading the original data to the cloud for embedding computation.

[0050] Figure 1 This is a flowchart illustrating a deep semantic embedding method based on a cross-environment decoupled architecture in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated herein, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0051] Based on the same technical concept, referring to Figure 2 This application also provides a deep semantic embedding device based on a cross-environment decoupled architecture, which adopts the following technical solution: The device includes: The global semantic module is used to generate global semantic embeddings of category labels based on category label text data using a pre-trained language model and a Hadamard matrix. It maps the label text into a fixed-dimensional real-valued vector as a semantic benchmark shared across environments. It also mines the label hierarchy structure through a dual-block attention mechanism to generate a high-level semantic hash code that includes intra-layer similarity and cross-layer correlation. The anchor point generation module is used to randomly select samples from historical data as anchor points and store the high-level semantic hash codes corresponding to the anchor points as an anchor point library. The edge retrieval module is used to transmit the anchor library and global semantic embedding to the edge device, and learn the low-level semantic hash code based on the discrete cyclic coordinate descent algorithm. The low-level semantic hash code achieves semantic alignment by minimizing the Hamming distance with the hash code of the anchor library, and introduces instance-level fine-grained weights to dynamically allocate fusion weights to the multimodal data of the edge device. The data transmission module is used to maintain a dynamic semantic mapping table between the cloud and edge devices, record the correspondence between high-level semantic hash codes and low-level semantic hash codes, correct cross-environment hash code deviations through a bit-by-bit calibration algorithm, and control the edge devices to only transmit hash code bits that differ from the anchor library to the cloud, and compress semantic information in combination with category-level high-level code.

[0052] In some embodiments, the edge retrieval module is specifically used to filter out low-confidence tags based on a threshold from noisy tags collected by the edge device to obtain the remaining noise; The remaining noise is corrected by global semantic embedding based on the cloud, and the corrected remaining label vector is obtained. The cosine similarity between the remaining label vector and the global semantic embedding is calculated. Labels with a cosine similarity greater than a preset value are retained for hash code learning. When an edge device detects new category data, it sends the new category data name to the cloud. The cloud then uses a language model to generate new tags for embedding and updates the global semantic library. The control edge device learns new category low-level hash codes based on new label embeddings and local samples through a discrete latent factor model.

[0053] In some embodiments, the global semantic module is specifically used to input the category label text data into a pre-trained language model based on the category label text data to generate an initial label embedding; The initial label embedding is orthogonally transformed based on the Hadamard matrix, mapping the label text to a fixed-dimensional real-valued vector, generating a global semantic embedding of the category label, and using the global semantic embedding as a semantic benchmark shared across environments. For the label hierarchy contained in the category label text data, a dual-block attention mechanism is used to mine instance-category similarity within the same label layer and cross-layer correlation between different label layers. Instance-category similarity within the same layer is determined by measuring the semantic distance between the sample and the corresponding category center within the same label layer, and cross-layer correlation is determined by calculating the cosine similarity of category nodes between different label layers. By integrating intra-layer similarity and cross-layer correlation, and using global semantic embedding as the initial value, a high-level semantic hash code containing intra-layer similarity and cross-layer correlation is generated through iterative optimization using a discrete cyclic coordinate descent algorithm.

[0054] In some embodiments, the global semantic module is specifically used to obtain a Hadamard matrix of a preset order, wherein the order of the Hadamard matrix is ​​determined according to the dimension of the target global semantic embedding, and satisfies that the row vectors and column vectors are pairwise orthogonal; The initial label embedding is multiplied with the Hadamard matrix, and the initial label embedding is mapped from the original dimension to the target fixed dimension through orthogonal transformation to obtain the intermediate semantic vector, where the target fixed dimension matches the length of the high-level semantic hash code. The intermediate semantic vector is normalized and its elements are scaled to a preset range to generate a real-valued vector with a fixed dimension, thus obtaining the global semantic embedding of the category label. Global semantics are embedded and stored in a cloud semantic library, serving as a semantic alignment benchmark when edge devices generate low-level semantic hash codes, and as a reference standard for cross-environment semantic consistency verification.

[0055] In some embodiments, the edge retrieval module is specifically used to receive and store the global semantic embedding of the high-level semantic hash code and category label corresponding to the anchor in the anchor library through an edge device; Local multimodal data is acquired through edge devices. The multimodal data includes image feature data and text feature data. The image feature data is extracted through a lightweight convolutional neural network, and the text feature data is extracted through a bag-of-words model or a pre-trained language model. The low-level semantic hash code is initialized based on the discrete cyclic coordinate descent algorithm. The high-level semantic hash code of the anchor library is used as the benchmark. The hash code bits are iteratively optimized by minimizing the Hamming distance between the low-level semantic hash code and the hash code of the anchor library. The low-level semantic hash code is updated bit by bit until convergence. The fusion weights are dynamically allocated by calculating the discriminative index of each modality feature in the multimodal data: if the discriminative index of the image feature is higher than that of the text feature, the image modality weight is set as the first weight and the text modality weight is set as the second weight, and the first weight is greater than the second weight; otherwise, the weight ratio is adjusted. Multimodal features are fused based on assigned instance-level fine-grained fusion weights. The fused features are then input into an optimized low-level semantic hash code, and the output is a low-level semantic hash code that is aligned with the semantics in the cloud.

[0056] In some embodiments, the edge retrieval module is specifically used to randomly initialize the low-level semantic hash code as an r-bit binary vector based on the feature dimension of the local multimodal data by the edge device, with the initial hash code bit value being {-1, 1}. Extract the high-level semantic hash code corresponding to the anchor point from the anchor point library as the optimization benchmark, denoted as the benchmark hash code. , where e is the number of anchor points and r is the length of the low-level semantic hash code; Calculate the Hamming distance between the currently initialized low-level semantic hash code B_current and the baseline hash code B_anchor. , where ⊕ is the XOR operation and ||·||_0 is the L0 norm; The discrete cyclic coordinate descent algorithm is used to iteratively optimize the hash code bits: all hash code bits except the k-th bit are fixed, where k is not greater than r, and the optimal value of the k-th bit is determined by maximizing the objective function, and each bit of B_current is updated bit by bit; Repeat the bit-by-bit update step, recalculate the Hamming distance after each iteration, and stop iterating when the change in Hamming distance between two consecutive iterations is less than a preset threshold or the number of iterations reaches a preset upper limit. At this time, the low-level semantic hash code converges to the state with the minimum Hamming distance with the base hash code, thereby achieving semantic structure alignment between edge devices and the cloud.

[0057] In some embodiments, the edge retrieval module is specifically used to perform weighted fusion of multimodal feature vectors based on instance-level fine-grained weights: scaling the image feature vector according to the image modality weights, scaling the text feature vector according to the text modality weights, and generating a fused feature vector by adding or concatenating the vectors element by element, wherein the weight allocation satisfies that the sum of the image modality weights and the text modality weights is 1; The fused feature vector is input into the optimized low-level semantic hash code generation model. The generation model is a binary neural network that has been iteratively optimized by the discrete cyclic coordinate descent algorithm. The model parameters have been initially optimized by minimizing the Hamming distance with the anchor library hash code. The initial low-level semantic hash code is output by performing a non-linear mapping on the fused feature vector through a generative model. Using the global semantic embedding in the cloud as a reference, the cosine similarity between the semantic vector corresponding to the initial low-level semantic hash code and the global semantic embedding of the category label is calculated. If the cosine similarity is not less than a preset threshold, the initial low-level semantic hash code is directly output. If the similarity is less than the preset threshold, the low-level semantic hash code is regenerated by fine-tuning the model parameters until the cosine similarity is not less than the preset threshold, thus obtaining a low-level semantic hash code that is aligned with the semantics in the cloud.

[0058] In some embodiments, the data transmission module is specifically used to initialize a dynamic semantic mapping table jointly by the cloud and the edge device. The dynamic semantic mapping table includes fields: category label ID, high-level semantic hash code, low-level semantic hash code and corresponding relationship update timestamp, which are used to record the mapping relationship of hash codes across environments. The cloud and edge devices periodically synchronize the dynamic semantic mapping table. The high-level semantic hash code and low-level semantic hash code of each record in the table are compared bit by bit, and the difference value of the corresponding bit is calculated. When the cumulative difference value exceeds the preset threshold, the calibration mechanism is triggered. By adjusting the corresponding bit of the low-level semantic hash code, the cross-environment hash code deviation is controlled within the preset range. When the edge device feeds back the local low-level semantic hash code to the cloud, it compares the hash code to be transmitted with the high-level semantic hash code of the corresponding anchor in the anchor library through an XOR operation, identifies the difference bits, extracts and transmits only the position and value information of the difference bits, and shields the redundant transmission of the same bits. By predefining category-level high-level codes in the cloud and edge devices, the category-level high-level codes are fixed-length codes based on category semantics. When transmitting differential hash code bits, the differential bit information is bound to the corresponding category high-level code. The semantic context of the differential bits is compressed through the high-level code to ensure that the transmitted semantic information is compact and undistorted. By receiving differential hash code points and compressed category-level high-level code transmitted from edge devices in the cloud, and updating the correspondence between high-level semantic hash codes and low-level semantic hash codes based on a dynamic semantic mapping table, cross-environment semantic collaboration is achieved.

[0059] In some embodiments, the edge retrieval module is specifically used to receive cross-modal retrieval requests, which include image queries or text queries; If the cross-modal retrieval request is initiated by the cloud, the high-level semantic hash code stored in the cloud is called for retrieval, and the retrieval result is initially obtained by calculating the Hamming distance between the hash code to be queried and the high-level semantic hash code in the database; If a cross-modal retrieval request is initiated by an edge device, the low-level semantic hash code generated by the edge device is used for local retrieval. By calculating the Hamming distance between the hash code to be queried and the low-level semantic hash code in the local database, preliminary retrieval results are obtained. When the local retrieval results of the edge device are insufficient or the retrieval results are lower than a preset threshold, the edge device transmits the hash code to be queried and the corresponding category-level high-level code to the cloud. The cloud maps the low-level semantic hash code to a high-level semantic hash code based on a dynamic semantic mapping table, and performs a global retrieval in conjunction with the cloud database to supplement the retrieval results. Control cloud and edge devices to perform semantic consistency verification of search results through global semantic embedding: calculate the cosine similarity between the semantic vector corresponding to the hash code of the search result and the global semantic embedding of the query category, filter search results with cosine similarity lower than a preset threshold, and retain the semantically consistent search results; It aggregates local search results and cloud-based supplementary search results, sorts them by similarity, and returns the final search results.

[0060] This application also discloses a control device.

[0061] Specifically, the control device includes a memory and a processor, the memory storing a computer program that can be loaded and executed by the processor using the deep semantic embedding method based on a cross-environmental decoupling architecture.

[0062] This application also discloses a computer-readable storage medium.

[0063] Specifically, the computer-readable storage medium stores a computer program that can be loaded and executed by a processor, such as the deep semantic embedding method based on the cross-environmental decoupled architecture described above. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for deep semantic embedding based on cross-environment separated architecture, characterized in that, The method comprises the following steps: Based on the category label text data, a pre-trained language model and a Hadamard matrix are used to generate a global semantic embedding of the category label, map the label text into a fixed-dimensional real value vector, and serve as a semantic reference shared across environments. A double-block attention mechanism is used to mine the label hierarchy to generate a high-level semantic hash code containing intra-layer similarity and cross-layer relevance; Randomly select samples from historical data as anchor points, and store the high-level semantic hash code corresponding to the anchor points as an anchor point library; The anchor point library and the global semantic embedding are transmitted to the edge device, and a low-level semantic hash code is learned based on a discrete cyclic coordinate descent algorithm. The low-level semantic hash code realizes semantic alignment by minimizing the Hamming distance with the anchor point library hash code, and introduces an instance-level fine-grained weight to dynamically assign a fusion weight to the multi-modal data of the edge device; A dynamic semantic mapping table is maintained through the cloud and the edge device to record the correspondence between the high-level semantic hash code and the low-level semantic hash code. The cross-environment hash code bias is corrected through a bit-by-bit calibration algorithm, and the edge device is controlled to only transmit the hash code bits that differ from the anchor point library to the cloud. The category-level advanced code compression semantic information is combined. 2.The method of claim 1, wherein, The method further comprises the following steps: For the noisy labels collected by the edge device, low-confidence labels are filtered based on a threshold to obtain remaining noise; The remaining noise is corrected based on the global semantic embedding of the cloud to obtain a corrected remaining label vector. The cosine similarity between the remaining label vector and the global semantic embedding is calculated, and labels with a cosine similarity greater than a preset value are retained for hash code learning; When the edge device detects new category data, the name of the new category data is sent to the cloud. The cloud generates a new label embedding using a language model and updates the global semantic library; The edge device learns a new category low-level hash code based on the new label embedding and local samples through a discrete latent factor model. 3.The method of claim 1, wherein, The method for generating a global semantic embedding of a category label based on category label text data, mapping label text into a fixed-dimensional real value vector, and serving as a semantic reference shared across environments, comprises the following steps: Based on the category label text data, the category label text data is input into a pre-trained language model to generate an initial label embedding; Based on a Hadamard matrix, the initial label embedding is orthogonally transformed to map the label text into a fixed-dimensional real value vector, and the global semantic embedding of the category label is generated. The global semantic embedding serves as a semantic reference shared across environments; For the label hierarchy contained in the category label text data, a double-block attention mechanism is used to mine the instance-category similarity within the same label layer and the cross-layer relevance between different label layers, respectively. The instance-category similarity within the same layer is determined by measuring the semantic distance between the samples within the same label layer and the corresponding category center, and the cross-layer relevance is determined by calculating the cosine similarity between the category nodes in different label layers. The intra-layer similarity and the cross-layer relevance are integrated, the global semantic embedding is taken as an initial value, and a high-level semantic hash code containing the intra-layer similarity and the cross-layer relevance is generated by iterative optimization of a discrete cyclic coordinate descent algorithm.

4. The method of claim 3, wherein, The initial label embedding is orthogonally transformed based on the Hadamard matrix to map the label text into a fixed-dimensional real value vector, and the global semantic embedding of the category label is generated, which is taken as a semantic reference shared across environments, including: A Hadamard matrix of a preset order is obtained, wherein the order of the Hadamard matrix is determined according to the dimension of the target global semantic embedding, and the row vectors and the column vectors are orthogonal to each other; The initial label embedding is subjected to matrix multiplication operation with the Hadamard matrix, and the initial label embedding is mapped from the original dimension to the target fixed dimension through orthogonal transformation to obtain an intermediate semantic vector, wherein the target fixed dimension matches the length of the high-level semantic hash code; The intermediate semantic vector is normalized to scale the vector elements to a preset interval to generate a real value vector with a fixed dimension, and the global semantic embedding of the category label is obtained; The global semantic embedding is stored in a cloud semantic library as a semantic alignment reference when the edge device generates the low-level semantic hash code, and as a reference standard for cross-environment semantic consistency verification.

5. The method of claim 4, wherein, The anchor point library and the global semantic embedding are transmitted to the edge device, and a low-level semantic hash code is learned based on a discrete cyclic coordinate descent algorithm, the low-level semantic hash code is semantically aligned by minimizing the Hamming distance with the anchor point library hash code, and an instance-level fine-grained weight is introduced to dynamically assign a fusion weight to the multi-modal data of the edge device, including: The high-level semantic hash code corresponding to the anchor points in the anchor point library and the global semantic embedding of the category label are received and stored by the edge device; Local multi-modal data is obtained by the edge device, the multi-modal data includes image feature data and text feature data, wherein the image feature data is extracted by a lightweight convolutional neural network, and the text feature data is extracted by a bag-of-words model or a pre-trained language model; The low-level semantic hash code is initialized based on the discrete cyclic coordinate descent algorithm, and the high-level semantic hash code of the anchor point library is taken as a reference to iteratively optimize the hash code bits by minimizing the Hamming distance between the low-level semantic hash code and the anchor point library hash code, and the low-level semantic hash code is updated bit by bit until convergence; The discriminative index of each modality feature in the multi-modal data is calculated to dynamically assign a fusion weight: if the discriminative index of the image feature is higher than that of the text feature, the image modality weight is set to a first weight and the text modality weight is set to a second weight, and the first weight is greater than the second weight, otherwise the weight proportion is adjusted; Based on the assigned instance-level fine-grained fusion weight, the multi-modal features are fused, and the fused features are input into the optimized low-level semantic hash code to output the low-level semantic hash code aligned with the cloud semantic.

6. The method of claim 5, wherein, The low-level semantic hash code is initialized based on the discrete cyclic coordinate descent algorithm, the high-level semantic hash code of the anchor library is taken as a reference, the low-level semantic hash code is updated bit by bit until convergence by iteratively optimizing the hash code bit by minimizing the Hamming distance between the low-level semantic hash code and the anchor library hash code, and the low-level semantic hash code is updated bit by bit until convergence, including: The low-level semantic hash code is randomly initialized as an r-bit binary vector based on the feature dimension of the local multi-modal data by the edge device, and the initial hash code bit takes values {-1, 1}; The high-level semantic hash code corresponding to the anchor in the anchor library is extracted as an optimization reference, denoted as a reference hash code B_anchor ∈ {-1, 1}^{e×r}, where e is the number of anchors, and r is the length of the low-level semantic hash code; The Hamming distance D = ||B_current ⊕ B_anchor||_0 between the current initialized low-level semantic hash code B_current and the reference hash code B_anchor is calculated, where ⊕ is the exclusive or operation, and ||·||_0 is the L0 norm; The hash code bit is iteratively optimized by using the discrete cyclic coordinate descent algorithm: all hash code bits are fixed except the kth bit, where k is not greater than r, the optimal value of the kth bit is determined by maximizing the objective function, and each bit of B_current is updated bit by bit; The bit-by-bit updating step is repeatedly executed, the Hamming distance is recalculated after each iteration, and when the Hamming distance change of two consecutive iterations is less than a preset threshold or the number of iterations reaches a preset upper limit, the iteration is stopped, at this time, the low-level semantic hash code converges to the state of the minimum Hamming distance with the reference hash code, and the semantic structure alignment of the edge device and the cloud is realized.

7. The method of claim 5, wherein, The multi-modal features are fused based on the assigned instance-level fine-grained fusion weight, and the fused features are input into the optimized low-level semantic hash code to output the low-level semantic hash code aligned with the cloud semantics, including: The vector of the multi-modal feature is weighted and fused based on the instance-level fine-grained weight: the image feature vector is scaled according to the image modal weight, the text feature vector is scaled according to the text modal weight, and the fused feature vector is generated by element-wise addition or splicing of vectors, wherein the weight distribution satisfies that the sum of the image modal weight and the text modal weight is 1; The fused feature vector is input into the generation model of the optimized low-level semantic hash code, and the generation model is a binary neural network iteratively optimized by the discrete cyclic coordinate descent algorithm, and the model parameters have been preliminarily optimized by minimizing the Hamming distance with the anchor library hash code; The fused feature vector is nonlinearly mapped by the generation model to output the initial low-level semantic hash code; With the global semantic embedding in the cloud as a reference, the cosine similarity between the semantic vector corresponding to the initial low-level semantic hash code and the global semantic embedding of the class label is calculated. If the cosine similarity is not less than a preset threshold, the initial low-level semantic hash code is directly output. If the similarity is less than the preset threshold, the low-level semantic hash code is regenerated by fine-tuning the model parameters until the cosine similarity is not less than the preset threshold, to obtain the low-level semantic hash code aligned with the cloud semantics. 8.The method of claim 1, wherein, The dynamic semantic mapping table is maintained by the cloud and the edge device, and the correspondence between the high-level semantic hash code and the low-level semantic hash code is recorded. The cross-environment hash code bias is corrected through a bit-by-bit calibration algorithm, and the edge device is controlled to only transmit the hash code bits that are different from the anchor library to the cloud. The class-level advanced code compression semantic information is combined, including: The dynamic semantic mapping table is initialized by the cloud and the edge device. The dynamic semantic mapping table includes the fields of class label ID, high-level semantic hash code, low-level semantic hash code, and corresponding relationship update timestamp, which is used to record the mapping relationship of the cross-environment hash code. The cloud and the edge device periodically synchronize the dynamic semantic mapping table. The high-level semantic hash code and the low-level semantic hash code of each record in the table are compared bit by bit, and the difference value of the corresponding bit is calculated. When the cumulative difference value exceeds a preset threshold, the calibration mechanism is triggered. The corresponding bit of the low-level semantic hash code is adjusted to control the cross-environment hash code bias within a preset range. When the edge device feeds back the local low-level semantic hash code to the cloud, the hash code to be transmitted is compared with the high-level semantic hash code of the corresponding anchor in the anchor library through XOR operation, the difference bit is identified, and only the position and value information of the difference bit are extracted and transmitted. The redundant transmission of the same bit is shielded. The class-level advanced code is predefined by the cloud and the edge device. The class-level advanced code is a fixed-length coding based on class semantics. When transmitting the difference hash code bit, the difference bit information is bound with the high-level code of the corresponding class. The semantic context of the difference bit is compressed through the high-level code to ensure that the transmitted semantic information is compact and not distorted. The cloud receives the difference hash code bit and the compressed class-level advanced code transmitted by the edge device. Based on the dynamic semantic mapping table, the correspondence between the high-level semantic hash code and the low-level semantic hash code is updated, and the cross-environment semantic cooperation is completed.

9. The method of claim 1, wherein, The method further includes: receiving a cross-modal retrieval request, the cross-modal retrieval request including an image query or a text query; if the cross-modal retrieval request is initiated by the cloud, calling the high-level semantic hash code stored in the cloud for retrieval, and preliminarily obtaining the retrieval result by calculating the Hamming distance between the hash code to be queried and the high-level semantic hash code in the database; If the cross-modal retrieval request is initiated by an edge device, the low-level semantic hash code generated by the edge device is called for local retrieval, and the Hamming distance between the query hash code and the low-level semantic hash code in the local database is calculated to obtain preliminary retrieval results; when the local retrieval results of the edge device are insufficient or the retrieval results are below a preset threshold, the edge device transmits the query hash code and the corresponding category-level high-level code to the cloud, and the cloud maps the low-level semantic hash code to the high-level semantic hash code based on the dynamic semantic mapping table, and performs global retrieval on the cloud database to supplement the retrieval results; The cloud and the edge device control the semantic consistency of the retrieval results by the global semantic embedding: calculate the cosine similarity between the semantic vector corresponding to the retrieval result hash code and the global semantic embedding of the query category, filter the retrieval results with a cosine similarity below a preset threshold, and retain the retrieval results with semantic consistency; Aggregate the local retrieval results and the cloud-supplemented retrieval results, and return the final retrieval results after sorting by similarity.

10. A device for deep semantic embedding based on cross-environment separated architecture, characterized in that, The device comprises: A global semantic module for generating a global semantic embedding of a category label based on category label text data using a pre-trained language model and a Hadamard matrix, mapping the label text to a fixed-dimensional real-valued vector as a cross-environment shared semantic benchmark, and generating a high-level semantic hash code containing intra-layer similarity and cross-layer correlation through a double-block attention mechanism to excavate the label hierarchy; An anchor point generation module for randomly selecting samples from historical data as anchor points and storing the high-level semantic hash code corresponding to the anchor points as an anchor point library; An edge retrieval module for transmitting the anchor point library and the global semantic embedding to the edge device and learning a low-level semantic hash code based on a discrete cyclic coordinate descent algorithm, wherein the low-level semantic hash code is semantically aligned by minimizing the Hamming distance with the anchor point library hash code, and a fine-grained weight at the instance level is introduced to dynamically assign fusion weights to multi-modal data of the edge device; A data transmission module for maintaining a dynamic semantic mapping table through the cloud and the edge device to record the correspondence between the high-level semantic hash code and the low-level semantic hash code, correcting cross-environment hash code bias through a bit-by-bit calibration algorithm, and controlling the edge device to transmit only the hash code bits different from the anchor point library to the cloud, and combining the category-level high-level code to compress semantic information.

Images