A category prototype oriented enhanced cross-modal hashing retrieval method for copper smelting industry

Abstract

The application relates to a category prototype enhanced cross-modal hash retrieval method for the copper smelting industry, data sets are collected, and a multi-label matrix is constructed; a category co-occurrence matrix is calculated, the Jaccard coefficient is used to calculate the category adjacency strength, and a category semantic graph is obtained; based on the initial category prototype, multi-layer graph diffusion smoothing is adopted, and a graph-smoothed category prototype is output; an initial graph-text encoder with independent parameters and consistent output dimensions is built, a data set is input into the initial graph-text encoder to map out a unified continuous hash space, and a bimodal prototype response matrix is obtained; a two-modal prototype classification loss and a multi-positive sample bidirectional contrast loss are constructed, a progressive coefficient is introduced to superimpose a hash loss, the initial graph-text encoder is optimized with the minimum hash loss as the target, and a target graph-text encoder is generated. The semantic structure between the categories is constructed, the category prototype is taken as a graph-text shared reference, the heterogeneous modal semantic gap can be relieved, and the discriminability, stability and retrieval efficiency of the multi-label cross-modal hash code are improved.

Description

Technical Field

[0001] This invention relates to the field of copper smelting industry technology, and in particular to a category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry. Background Technology

[0002] Copper smelting production comprises six core processes: batching, smelting, blowing, refining, electrolysis, and flue gas desulfurization. The production process continuously generates massive amounts of multimodal data: on one hand, visual image data such as high-temperature furnace video frames, furnace mouth flame images, molten copper surface images, and electrode defect images; on the other hand, textual data such as process operation procedures, real-time DCS parameter texts, fault handling logs, quality inspection reports, and safety control texts. In industrial operation and maintenance and fault tracing scenarios, maintenance personnel often need to retrieve corresponding fault handling texts by searching for smelting anomaly images and retrieve images of similar operating conditions by searching for process parameter texts.

[0003] However, existing methods typically treat each category as an isolated supervisory signal, failing to fully utilize the naturally existing semantic prior structure between categories. This results in semantically similar categories being separated in the hash space, affecting the performance of semantic clustering and cross-modal retrieval.

[0004] The above problems urgently need to be addressed. Summary of the Invention

[0005] This invention discloses a category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry, aiming to solve the technical problems existing in the prior art.

[0006] This invention employs the following technical solution, comprising: acquiring copper smelting images and process text samples and dividing the dataset; preprocessing the dataset; constructing a multi-label matrix using process fault labels in the dataset; calculating a category co-occurrence matrix based on the multi-label matrix; calculating category adjacency strength using Jaccard coefficients; obtaining a category semantic graph through Top-k sparsity, symmetry, and degree normalization; generating initial category prototypes using smelting category unique heat vectors; applying multi-layer graph diffusion smoothing to the category semantic graph; and outputting a normalized graph smoothing category prototype; and constructing an initial graph encoder with independent parameters and consistent output dimensions, so that the preprocessed dataset is input into the initial graph encoder. A unified and continuous hash space is emitted, and the inner product of the hash space and the graph smoothing category prototype is used to obtain a bimodal prototype response matrix. A bimodal prototype classification loss is constructed using the multi-label matrix and the bimodal prototype response matrix, and the category responses of the image end and the text end in the category semantic graph are constrained to be consistent. A multi-positive sample bidirectional contrast loss is constructed by combining the bimodal prototype classification loss and the category semantic graph. An asymptotic coefficient superimposed hash loss is introduced into the multi-positive sample bidirectional contrast loss. The initial graph-text encoder is iteratively optimized with the goal of minimizing the hash loss to generate a target graph-text encoder. The output of the target graph-text encoder is mapped to a binary hash code to form a bidirectional retrieval.

[0007] Optionally, copper smelting images and process text samples are collected and the dataset is divided. The dataset is preprocessed, and a multi-label matrix is ​​constructed using process fault labels in the dataset. This includes: collecting paired samples of copper smelting furnace operating conditions and process fault texts, and dividing the dataset into a training set, a query set, and an industrial database set; performing L2 normalization on the image visual features and process text features respectively, wherein L2 normalization is used to indicate scaling the features to a feature vector of unit length; and constructing a multi-label matrix using the smelting process and fault multi-label vectors in the training set. The labels in the query set and the industrial database set are used to evaluate the retrieval accuracy of the target image encoder.

[0008] Optionally, a category co-occurrence matrix is ​​calculated based on the multi-label matrix, and the category adjacency strength is calculated using Jaccard coefficients, including: the category co-occurrence matrix is: Where M is the class co-occurrence matrix; Y is the training set multi-label matrix; This is the transpose of Y; The adjacency strength of the category is: in, Let this be the co-occurrence matrix of category a and category b in the training set. Let be the co-occurrence matrix of class a in the training set. Let b be the co-occurrence matrix of class b in the training set. Let ε be the Jaccard adjacency strength between category a and category b, where a is the index of category a, b is the index of category b, and ε is a very small positive number.

[0009] Optionally, an initial category prototype is generated using the unique heat vector of the smelting category. The category semantic graph is then smoothed using multi-layer graph diffusion, and the normalized result is used to output a graph-smoothed category prototype. This process includes: determining the smelting category prototype based on process fault labels in the dataset; determining semantically neighboring category prototypes strongly coupled with the smelting category prototype based on the category semantic graph, wherein the category semantic graph provides the weights for strong coupling; weighted fusion of the smelting category prototype and the semantically neighboring category prototypes to obtain the initial category prototype; and performing multi-layer propagation on the category semantic graph using the category semantic graph as weights and process fault categories in the dataset as nodes to obtain the graph-smoothed category prototype.

[0010] Optionally, an initial image-text encoder with independent parameters and consistent output dimensions is constructed, including: the initial image-text encoder includes an image encoder and a text encoder, the image encoder and the text encoder have the same number of layers and output dimensions, and their parameters are not shared; the final layer outputs of the image encoder and the text encoder retain continuous representations for calculating the bimodal prototype response matrix.

[0011] Optionally, the bimodal prototype response matrix is ​​obtained by multiplying the hash space with the graph smoothing category prototype inner product, including: the image-side category prototype response matrix is: in, For image-side category prototype response matrix, Let W be a matrix consisting of the consecutive hash representations of all images within a batch, and let W be the graph smoothing class prototype. Indicates the transpose of W; The text-based category prototype response matrix is ​​as follows: in, For text-based category prototype response matrices, Let W be a matrix consisting of the consecutive hash representations of all text within a batch, and let W be the prototype of the graph smoothing class. This represents the transpose of W.

[0012] Optionally, a two-modal prototype classification loss is constructed using the multi-label matrix and the two-modal prototype response matrix, including: the two-modal prototype classification loss is: in, Let BCE(·) be the two-modal prototype classification loss, BCE(·) be the binary cross-entropy loss, and Y be the training sample label matrix. For image-side category prototype response matrix, This is the text-based category prototype response matrix.

[0013] Optionally, before combining the two-modal prototype classification loss and the category semantic graph to construct the multi-positive sample bidirectional contrast loss, the method also includes: using similarity Top-k gating to retain positive pairs in the category semantic graph whose category adjacency strength is greater than a preset value; when there are no positive pairs in any row of the category semantic graph, the image and text of the same sample are retained as basic positive pairs.

[0014] Optionally, an asymptotic coefficient superimposed hash loss is introduced into the multi-positive-sample bidirectional contrast loss, including: the hash loss is: in, This represents the two-modal prototype classification loss. This represents the cross-modal prototype response consistency loss. This indicates the loss of graph structure preservation. This represents the bidirectional cross-modal contrast loss. Indicates quantified loss. Indicates bit equalization loss. This represents the decorrelation loss, where r is the asymptotic coefficient. , , All are modal loss coefficients. , , All are loss coefficients for multiple positive samples.

[0015] Optionally, an asymptotic coefficient superimposed hash loss is introduced into the multi-positive-sample bidirectional contrast loss, including: the asymptotic coefficient is: Where r is the asymptotic coefficient and e is the current training round; is the number of preheating cycles; min(·) is the minimum value operation.

[0016] The technical solution adopted in this invention can achieve at least one of the following beneficial effects: In this embodiment of the invention, a Jaccard industrial semantic graph is constructed based on real multi-label statistics of smelting production. The semantics of working conditions and faults caused by originally isolated process labels are associated. The category structure is extracted from the global multi-label distribution. The graph smooths the industrial category prototype fusion and associates process and fault semantics as a shared reference for smelting images and process texts. The bimodal approaches complete industrial semantic alignment in the prototype response space, achieving cross-modal semantic alignment. This significantly reduces the distribution differences between visual images and process texts, thereby alleviating the heterogeneous modal semantic gap and improving the semantic clustering and cross-modal retrieval effects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below, forming part of the present invention. The illustrative embodiments of the present invention and their descriptions explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings: Figure 1 This is a flowchart of a category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to Embodiment 1 of the present invention; Figure 2 This is a hash retrieval framework diagram of a category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry in Embodiment 1 of the present invention; Figure 3 This is a framework diagram for generating category semantic graphs to shared prototype anchors in a category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry, as described in Embodiment 1 of the present invention. Figure 4 This is a flowchart of an optional category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry, as described in Embodiment 2 of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. In the description of this invention, it should be noted that the term "or" is generally used to include the meaning of "and / or," unless otherwise expressly indicated.

[0019] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or a magnetic connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. Furthermore, in the description of this application, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. In the description of this invention, "a plurality of" means at least two, such as two, three, or more, unless otherwise explicitly specified.

[0020] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0021] First, to facilitate understanding of the embodiments of the present invention, some terms or nouns involved in the present invention will be explained below: Category prototype, used to indicate the central representative vector obtained by aggregating the features of all samples in the same category.

[0022] Cross-modal is used to indicate that data do not belong to the same carrier and need to be retrieved and matched between different types of data.

[0023] Hash retrieval is used to compress high-dimensional, complex raw data (tens of thousands of dimensions of image features, hundreds of dimensions of process parameters) into a short binary code (such as a 16-bit / 32-bit / 64-bit 01 string). This binary string is called a hash code, and matching and retrieval are performed using the hash code.

[0024] To address the problems existing in related technologies, this application provides a category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry.

[0025] Example 1 This embodiment provides a category-based prototype-enhanced cross-modal hash retrieval method for the copper smelting industry, such as... Figure 1 and Figure 2 As shown, Figure 1 This is a flowchart of a category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to Embodiment 1 of the present invention. Figure 2 This is a hash retrieval framework diagram of a category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to Embodiment 1 of the present invention. The method includes: Step S102: Collect copper smelting images and process text samples and divide the dataset. Preprocess the dataset and construct a multi-label matrix using the process fault labels in the dataset. Optionally, the collected images of copper smelting furnace operating conditions and text samples of process faults are paired and divided into a training set, a query set, and an industrial database set. Each sample in the dataset is then labeled. For each sample, the image features are denoted as... Text features are denoted as Multi-label vectors are denoted as Stack the multi-label vectors of the training set into a label matrix. Before inputting the data into the model, L2 normalization is performed on both image features and text features.

[0026] Where i represents the sample number; This represents the image feature vector of the i-th sample; This represents the text feature vector of the i-th sample; Indicates the dimension of image features; Indicates the dimension of text features; Let Y represent the multi-label vector of the i-th sample; C represents the total number of categories; Y represents the multi-label matrix composed of the multi-label vectors of all training samples; and N represents the total number of training samples.

[0027] Optionally, image samples are processed using a pre-trained convolutional network to extract image features, while text samples are processed using a bag-of-words model or a text encoder to extract text features. Training set labels are used to construct the category semantic graph, while query set and industry database set labels are used only during the evaluation phase to determine the semantic relevance of retrieval results. During the data loading phase, training samples, query samples, and industry database samples maintain the same feature organization, but training samples are used to update encoder parameters and construct the category semantic graph, query samples are used to generate retrieval requests, and industry database samples are used to form the retrieval codebase. This partitioning method ensures that the estimation of the category graph depends only on the training labels, preventing evaluation labels from leaking into the model training process.

[0028] Optionally, when the original image and text dimensions are inconsistent, it is not necessary to set them to have the same input dimension. Instead, the feature spaces of their respective modalities are preserved; the transformation to a unified dimension is completed by the subsequent dual-channel encoder. This approach can utilize existing pre-trained visual and text features while ensuring that the final hash representation is in the same semantic space.

[0029] In some preferred embodiments, copper smelting images and process text samples are collected and the dataset is divided. The dataset is preprocessed, and a multi-label matrix is ​​constructed using process fault labels in the dataset. This includes: collecting paired samples of copper smelting furnace operating conditions and process fault texts, and dividing the dataset into a training set, a query set, and an industrial database set; performing L2 normalization on the image visual features and process text features respectively, where L2 normalization is used to indicate scaling the features to a feature vector of unit length; constructing a multi-label matrix using the smelting process and fault multi-label vectors in the training set, and using the labels in the query set and the industrial database set for evaluating the retrieval accuracy of the target image encoder.

[0030] Step S104: Calculate the category co-occurrence matrix based on the multi-label matrix, calculate the category adjacency strength using Jaccard coefficients, and obtain the category semantic graph through Top-k sparsity, symmetry, and degree normalization; as shown... Figure 3 As shown, Figure 3 This is a framework diagram for generating category semantic graphs to shared prototype anchors in a category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry, as described in Embodiment 1 of the present invention.

[0031] Optional, based on the multi-label matrix Calculate the category co-occurrence matrix The class adjacency strength is calculated using the Jaccard coefficient. Where M represents the class co-occurrence matrix; Y represents the training set multi-label matrix; Indicates the transpose of Y; This represents the co-occurrence matrix of category a and category b in the training set, that is, the number of times category a and category b appear together in the training set. This indicates the number of times category a appears in the training set; This indicates the number of times category b appears in the training set; This represents the Jaccard adjacency strength between category a and category b; a and b represent category indices; ε represents a very small positive number to prevent the denominator from being zero. To prevent the denominator from being zero, a smoothing term can take the value of . or .

[0032] The above Jaccard adjacency strength As the edge weight between category a and category b in the category semantic graph. A higher Jaccard adjacency strength indicates a higher co-occurrence ratio between the two categories, and a greater weight in information transfer between them during subsequent category prototype propagation. This Jaccard adjacency strength determines the weight a category absorbs from related category prototypes, thus forming the basis for generating graph smoothing of category prototypes.

[0033] Optionally, during Top-k sparsity, the number of Top-k neighbors... Take values ​​from 5 to 10; when the number of categories is small, the number of Top-k neighbors. A value of 5 is acceptable, but 8 or 10 are preferable when the number of categories is large or the relationships between categories are more complex. The number of neighbors retained for each category in the category semantic graph. The maximum number of neighbors retained for each row. The neighbors are set to zero, and other elements are set to zero. Then, symmetry is performed. First, Top-k sparsity is performed on each row of the Jaccard adjacency matrix to retain the k neighbors with the strongest semantic relationship for each category. Then, the sparse matrix and its transpose are averaged to obtain a symmetric category semantic graph. Finally, degree normalization is performed. Top-k sparsity can reduce noise edges caused by accidental co-occurrence, and symmetric normalization makes the influence range of different frequency categories comparable in propagation.

[0034] Optionally, in the implementation process, class pairs with zero co-occurrence frequency can be excluded first, and then the most relevant neighbors for each class can be retained. For datasets with a small number of samples or a large number of long-tail classes, Top-k edge preservation can prevent rare classes from being completely overwhelmed by high-frequency classes; for datasets with dense class relationships, the number of neighbors can be appropriately increased to retain richer semantic associations.

[0035] Optionally, after completing Top-k sparsity and symmetry, a category semantic matrix is ​​generated based on Jaccard adjacency strength, which is the category semantic graph. Specifically, let... Given a degree matrix, the normalized category semantic graph is as follows: Where A represents the class adjacency matrix after Top-k sparsification and symmetry; D represents the degree matrix obtained from A, whose diagonal elements are the sum of the adjacency weights of the corresponding class and other classes; This represents the negative first power of the degree matrix; This represents the normalized category semantic graph and serves as the subsequent category prototype propagation matrix.

[0036] Optionally, the category semantic graph can be saved as a fixed structure before training once it is constructed. Since the category semantic graph comes from global label statistics rather than gradient updates from a single batch, it can provide a stable category prior and reduce the impact of class distribution fluctuations on prototype orientation during mini-batch training.

[0037] In some preferred embodiments, a category co-occurrence matrix is ​​calculated based on a multi-label matrix, and the category adjacency strength is calculated using Jaccard coefficients, including: the category co-occurrence matrix is: Where M is the class co-occurrence matrix; Y is the training set multi-label matrix; This is the transpose of Y; The adjacency strength of the category is: in, Let this be the co-occurrence matrix of category a and category b in the training set. Let be the co-occurrence matrix of class a in the training set. Let b be the co-occurrence matrix of class b in the training set. Let ε be the Jaccard adjacency strength between category a and category b, where a is the index of category a, b is the index of category b, and ε is a very small positive number.

[0038] Step S106: Generate an initial category prototype using the unique thermal vector of the smelting category, employ multi-layer graph diffusion smoothing of the category semantic graph, and output a graph smoothed category prototype after normalization. Optionally, set a unique heat vector for the smelting category. As a category-specific one-hot matrix, it is input into the category prototype generator. The initial category prototypes are obtained, where the category prototype generator adopts a multilayer perceptron structure. The input is a C-dimensional one-hot category matrix, which is sequentially passed through a linear layer, ReLU activation, another linear layer, ReLU activation, and a K-dimensional linear mapping to obtain a K-dimensional category prototype; then, after tanh activation and L2 normalization, a C×K initial category prototype matrix is ​​formed.

[0039] Specifically, the generated initial category prototype is as follows: Where E represents a C×C dimensional class-specific one-hot matrix, with each row corresponding to the identity vector of a class; Represents a category prototype generator; This represents the matrix corresponding to the initial category prototype obtained by the category prototype generator; K represents the hash code length; norm(·) represents the row-by-row L2 normalization operation.

[0040] Optionally, the input to the category prototype generator is category identity rather than sample features, thus yielding a category-level semantic reference. The initial category prototypes, after non-linear mapping, possess learnable expressive power, while graph propagation further injects information from adjacent categories into the prototype space, giving categories that "frequently co-occur" or "have close semantic relationships" a more consistent orientation.

[0041] The normalized category semantic graph is used as the category prototype propagation matrix to perform graph diffusion smoothing on the initial category prototypes. Specifically, the new prototype for each category is obtained by a weighted fusion of its own prototype and the prototypes of its semantic neighbors, where the weights are determined by the category adjacency strength after sparsification, symmetry, and degree normalization. This is performed on the category semantic graph. Layer propagation yields a graph smoothing category prototype: Where norm(·) represents the row-wise L2 normalization operation, W represents the final graph smoothing class prototype matrix, W(l) represents the class prototype matrix after propagation at the l-th layer, W(l-1) represents the class prototype matrix of the previous layer, l represents the propagation layer index, and L represents the total number of propagation layers. This represents the normalized category semantic graph. The formula shows that Jaccard adjacency strength is achieved through... The process directly proceeds to the category prototype propagation process. Specifically, the L-layer propagation adopts a layer-by-layer iterative update method. The 0th layer is the initial category prototype; the 1st layer prototype is obtained by multiplying the normalized category adjacency matrix with the (1-1)th layer prototype; after propagating L times, normalization is performed again to obtain the final graph-smoothed category prototype. The matrix exponentiation in the formula is a compact representation of the layer-by-layer propagation process.

[0042] Optionally, the graph-smoothed category prototype integrates the category's own representation and its semantic neighbor representation, ensuring that categories with stable co-occurrence relationships remain in similar orientations within the prototype space. Constrained by a global category graph, the graph-smoothed category prototype is less susceptible to fluctuations in individual batches of samples.

[0043] To avoid the prototype scale affecting subsequent inner product responses, row-wise normalization is performed after initial mapping and graph propagation. The normalized category prototypes primarily express semantic differences through direction, exhibiting a more stable numerical range when calculating similarity with image and text representations.

[0044] In some preferred embodiments, an initial category prototype is generated using the unique heat vector of the smelting category, and a multi-layer graph diffusion smoothing of the category semantic graph is employed. After normalization, a graph-smoothed category prototype is output. This includes: determining the smelting category prototype based on the process fault labels in the dataset; determining semantic neighbor category prototypes that are strongly coupled with the smelting category prototype based on the category semantic graph, wherein the category semantic graph is used to provide the weights for strong coupling; performing a weighted fusion of the smelting category prototype and the semantic neighbor category prototypes to obtain the initial category prototype; and performing multi-layer propagation on the category semantic graph with the process fault categories in the dataset as nodes, using the category semantic graph as weights, to obtain the graph-smoothed category prototype.

[0045] Step S108: Build an initial graph encoder with independent parameters and consistent output dimensions, so that the preprocessed dataset is input into the initial graph encoder to map out a unified and continuous hash space, and the hash space is multiplied by the graph smoothing class prototype inner product to obtain the bimodal prototype response matrix. Optionally, the initial image-text encoder includes an image encoder and a text encoder. The image encoder and text encoder can use the same number of layers and output dimensions, but their parameters are not shared to accommodate different statistical distributions of image features and text features. The final layer output of the image encoder and text encoder does not directly take the sign function, but first retains the continuous representation for calculating the class prototype response, cross-modal contrast relationship, and hash constraint.

[0046] The text contiguous hash representation and the image contiguous hash representation are as follows: in, Indicates an image encoder; Indicates a text encoder; This represents the continuous hash representation of the image of the i-th sample. This represents the continuous hash representation of the text of the i-th sample. This represents the image features of the i-th sample. Let K represent the text features of the i-th sample; K represents the dimension of the continuous hash representation.

[0047] Represent the text continuous hash matrix and the image continuous hash matrix , With graph smoothing category prototype Calculate the inner product to obtain the image-side and text-side category prototype responses, i.e., the bimodal prototype response matrix: in, A matrix representing the consecutive hash representations of all images within a batch; W represents the matrix consisting of the consecutive hash representations of all texts within a batch; W represents the graph smoothing class prototype matrix. Indicates the transpose of W; Represents the prototype response matrix of the image category; This represents the text-based category prototype response matrix.

[0048] Optionally, since both modalities share the same set of category prototypes, the model learns a consistent semantic distribution on a unified category anchor, thereby mitigating the heterogeneous representation differences between images and text. The category prototype response can be understood as the soft matching result of a sample to all category anchors. The category prototype response provides an intermediate semantic layer; even if image and text samples have significant differences in their original features, as long as their response distributions on the category prototypes are similar, they will be brought closer together by the training objective, thus mitigating the heterogeneous representation differences between images and text.

[0049] In some preferred embodiments, an initial image encoder with independent parameters and consistent output dimensions is constructed, including: the initial image encoder includes an image encoder and a text encoder, the image encoder and the text encoder have the same number of layers and output dimensions, and their parameters are not shared; the final layer outputs of the image encoder and the text encoder retain continuous representations for calculating the bimodal prototype response matrix.

[0050] In some preferred embodiments, the bimodal prototype response matrix is ​​obtained by multiplying the hash space with the graph smoothing class prototype inner product, including: the image-side class prototype response matrix is: in, For image-side category prototype response matrix, Let W be a matrix consisting of the consecutive hash representations of all images within a batch, and let W be the graph smoothing class prototype. Indicates the transpose of W; The text-based category prototype response matrix is ​​as follows: in, For text-based category prototype response matrices, Let W be a matrix consisting of the consecutive hash representations of all text within a batch, and let W be the prototype of the graph smoothing class. This represents the transpose of W.

[0051] Step S110: Construct a two-modal prototype classification loss using a multi-label matrix and a two-modal prototype response matrix, and constrain the category responses of the image end and the text end in the category semantic map to be consistent; Optionally, the two-modal prototype classification loss is responsible for enabling each modality to independently recognize multi-label semantics, while the response consistency loss is responsible for ensuring that the two modalities form similar distributions on the same set of class prototypes. When combined, the model neither relies solely on cross-modal pairwise supervision nor trains the image and text sides as independent classifiers.

[0052] Specifically, the two-modal prototype classification loss is as follows: in, BCE(·) represents the two-modal prototype classification loss; BCE(·) represents the binary cross-entropy loss; Y represents the multi-label matrix of the training samples. For image-side category prototype response matrix, This is the text-based category prototype response matrix.

[0053] The cross-modal prototype response consistency loss is as follows: in, B represents the cross-modal prototype response consistency loss; C represents the batch size; σ(·) represents the total number of categories; ||·||F represents the Frobenius norm. For image-side category prototype response matrix, This is the text-based category prototype response matrix.

[0054] The loss for preserving the graph structure is as follows: in, tr(·) represents the graph structure preservation loss; tr(·) represents the matrix trace operation; D represents the degree matrix; A represents the class adjacency matrix; DA represents the graph Laplacian matrix of the class semantic graph.

[0055] The aforementioned losses enable the sample representation to predict its own multi-labels and form a cross-modal consistent response on shared category prototypes; the graph structure term further constrains the proximity of adjacent category prototypes, thereby preserving the local structure in the category semantic graph.

[0056] Specifically, the graph structure term primarily constrains the local smoothness of the category prototypes themselves, ensuring that adjacent category prototypes remain appropriately close in space; meanwhile, non-adjacent categories can still maintain separability through classification loss and contrastive loss. Therefore, this constraint does not simply compress all categories, but rather strikes a balance between global structure and local discriminative power.

[0057] In some preferred embodiments, a two-modal prototype classification loss is constructed using a multi-label matrix and a two-modal prototype response matrix, including: the two-modal prototype classification loss is: in, Let BCE(·) be the two-modal prototype classification loss, BCE(·) be the binary cross-entropy loss, and Y be the training sample label matrix. For image-side category prototype response matrix, This is the text-based category prototype response matrix.

[0058] Step S112: Combine the two-modal prototype classification loss and the category semantic graph to construct a multi-positive sample bidirectional contrast loss. Introduce asymptotic coefficients and superimposed hash loss into the multi-positive sample bidirectional contrast loss. Iteratively optimize the initial graph-text encoder with the goal of minimizing the hash loss to generate the target graph-text encoder. The output of the target graph-text encoder is mapped to a binary hash code to form a bidirectional retrieval.

[0059] Optionally, in multi-label data, two different samples may share some labels, or correspond to adjacent categories in the semantic graph. Expanding the source of positive samples through label sharing and graph adjacency relationships better aligns with the relevance definition of multi-label retrieval.

[0060] in, This indicates the semantic sharing relationship between sample i and sample j; and Let i and j represent the multi-label vectors of sample i and sample j, respectively. and Let i and j represent the high-confidence category sets obtained from the prototype response, respectively; the symbol “~” indicates that the two category sets are adjacent in the category semantic graph; I[·] represents the indicator function, which takes the value 1 when the condition is true and 0 otherwise.

[0061] Where U represents the cross-modal similarity matrix; norm(·) represents the row normalization operation; Represents a continuous hash representation matrix of an image; Represents a continuous hash representation matrix of text; This represents the contrast learning temperature coefficient, used to adjust the smoothness of the similarity distribution.

[0062] in, B represents the multi-positive-sample contrast loss from image to text direction; B represents the batch size. This represents the gated positive sample weights from sample i to sample j; The similarity between the image representation of sample i and the text representation of sample j is represented by ; j represents the candidate sample index within the batch.

[0063] in, B represents the contrast loss in the text-to-image direction; B represents the batch size. This represents the gated positive sample weights from sample i to sample j; The similarity between the text representation of sample i and the image representation of sample j is represented; j represents the candidate sample index within the batch.

[0064] Optionally, bidirectional contrastive loss includes image-to-text and text-to-image directions. In the image-to-text direction, the image representation is used as the query and the text representation as the candidate; in the text-to-image direction, the text representation is used as the query and the image representation as the candidate. Multi-positive sample contrastive loss is calculated separately for each direction, and the average of the two is taken as the cross-modal contrastive loss. This approach not only utilizes image-text pairs of the same sample but also leverages cross-sample positive pairs with shared labels or adjacent graph structures, enabling multi-label semantics to form a more stable clustering structure in the hash space.

[0065] The bidirectional cross-modal contrast loss is: in, Indicates bidirectional cross-modal contrast loss; This represents the contrast loss in the image-to-text direction; The contrast loss represents the text-to-image direction; the average of the two is used to simultaneously constrain the two retrieval directions.

[0066] Optionally, to avoid too many weakly related samples entering the positive set, a similarity Top-k gating is further used to retain only positive pairs in the current batch that have clear semantic relationships and represent similarity. If a row has no additional positive pairs, then the image-text pairs of the same sample are retained as basic positive pairs, ensuring that each sample has stable supervision during training.

[0067] Similarity is based on the inner product similarity between the normalized image hash representation and the text hash representation, which is also equivalent to cosine similarity. The Top-k gating parameter k can be 5 to 10, with the optimal solution being k=6. For a candidate positive sample, if its similarity does not rank among the top k of the current sample, it is considered a weak positive pair and discarded; if a sample has no additional positive pairs, its original paired image and text are retained as a basic positive pair.

[0068] Continuous representations primarily handle semantic learning tasks in the early stages of training. Immediately imposing strong discretization can easily cause unseparated categories to be incorrectly pushed to binary endpoints. Therefore, asymptotic coefficients are used to adjust the hash constraint weights, allowing the model to first complete prototype alignment and cross-modal aggregation, and then gradually strengthen binary representation, balance, and decorrelation.

[0069] Specifically, the continuous outputs of the two modes are first processed through a hyperbolic tangent function to obtain a soft hash representation; training round e, warm-up round number and asymptotic coefficients It is uniformly represented by the following formula, which is used to control the process of introducing hash constraints from weak to strong.

[0070] Where r represents the asymptotic coefficient; e represents the current training round; This indicates the number of warm-up rounds; min(·) represents the minimum value operation. r gradually increases from 0 to 1 with each training round, used to control the introduction of hash constraints from weak to strong.

[0071] in, Indicates quantified loss; This represents the soft hash representation at the image end; represents the text-side soft hash representation; |·| represents element-wise absolute value operation; 1 represents an all-one matrix; B represents the batch size; K represents the hash code length; ||·||F represents the Frobenius norm.

[0072] in, Indicates bit equalization loss; This represents the transpose of a vector consisting entirely of 1s. and represents the soft hash representation of the image and text respectively; B represents the batch size, which is used to constrain the mean of each hash bit within the batch to be close to 0.

[0073] in, This represents the decorrelation loss; m represents the modality index, which can be either image modality I or text modality T; The centralized soft hash representation of mode m; This represents a K-dimensional identity matrix; K represents the hash code length.

[0074] Based on all the above loss functions, the superimposed hash loss is calculated as follows: in, This represents the two-modal prototype classification loss. This represents the cross-modal prototype response consistency loss. This indicates the loss of graph structure preservation. This represents the bidirectional cross-modal contrast loss. Indicates quantified loss. Indicates bit equalization loss. This represents the decorrelation loss, where r is the asymptotic coefficient. , , All are modal loss coefficients. , , All are loss coefficients for multiple positive samples.

[0075] Quantization terms improve binary representation, bit balance terms prevent certain hash bits from consistently biased towards the same value, and decorrelation terms reduce information redundancy between different bits. Asymptotic coefficients allow the model to first learn a stable semantic structure and then gradually strengthen the discrete constraints of the hash code. Bit balance constraints ensure that each bit takes on as many positive and negative values ​​as possible within a batch, avoiding invalid bits that are always positive or always negative; decorrelation constraints reduce repetitive representations between different bits, allowing hash codes of finite length to carry more complementary semantic information.

[0076] Optionally, the Adam optimization algorithm should be used to minimize the hash loss after the above stacking in batches to update the parameters of the image encoder and text encoder. The category semantic graph and graph smoothing category prototype can be pre-calculated based on the training set labels before training begins, or re-estimated based on global label statistics in predetermined rounds. The batch size can be set to 128, the learning rate to 10^-4, the weight decay to 10^-4, the number of training rounds to 100, and the hash length to 16, 32, 64, or 128. After iterative optimization, the image encoder and text encoder are obtained, ultimately forming the final target image-text encoder. The practical application of the target image-text encoder is as follows: a binary hash code can be obtained through the target image-text encoder: in, Represents the binary hash code of the image database; This represents the binary hash code of the text database; sign(·) represents the sign function; tanh(·) represents the hyperbolic tangent function; and These represent the image encoder and the text encoder, respectively.

[0077] in, (·) represents Hamming distance; This represents the K-bit binary hash code of the queried sample; This represents the K-bit binary hash code of the candidate samples in the database. The inner product similarity is converted into the Hamming distance between the binary codes using the method described above.

[0078] The deployment phase no longer requires category labels, co-occurrence matrices, or training batch relationships; only the trained image encoder, text encoder, and symbolization rules need to be retained. Database codes can be pre-computed and stored offline. When a user initiates a query, only the query code needs to be calculated and sorted by Hamming distance, thus reducing retrieval complexity.

[0079] During evaluation, if the query sample and the database sample share at least one true category, they are considered related. The average precision of image-to-text retrieval and text-to-image retrieval are calculated separately, and the two-way mean can be used as the model selection metric. This evaluation method is consistent with the actual needs of multi-label cross-modal retrieval tasks.

[0080] In some preferred embodiments, before constructing a bidirectional comparison loss for multiple positive samples by combining the two-modal prototype classification loss and the category semantic graph, the method further includes: using similarity Top-k gating to retain positive pairs in the category semantic graph whose category adjacency strength is greater than a preset value; when there are no positive pairs in any row of the category semantic graph, the image and text of the same sample are retained as basic positive pairs.

[0081] In some preferred embodiments, an asymptotic coefficient superimposed hash loss is introduced into the multi-positive-sample bidirectional contrast loss, including: the hash loss is: in, This represents the two-modal prototype classification loss. This represents the cross-modal prototype response consistency loss. This indicates the loss of graph structure preservation. This represents the bidirectional cross-modal contrast loss. Indicates quantified loss. Indicates bit equalization loss. This represents the decorrelation loss, where r is the asymptotic coefficient. , , All are modal loss coefficients. , , All are loss coefficients for multiple positive samples.

[0082] In some preferred embodiments, an asymptotic coefficient superimposed hash loss is introduced into the multi-positive-sample bidirectional contrast loss, including: the asymptotic coefficient is: Where r is the asymptotic coefficient and e is the current training round; is the number of preheating cycles; min(·) is the minimum value operation.

[0083] Through steps S102 to S112 above, with the category semantic graph and graph smoothing category prototype as the core, the global category relationship is transformed into cross-modal shared anchor points, and the structured alignment of images and text is completed in a unified hash space, ultimately achieving efficient and stable multi-label cross-modal retrieval.

[0084] Example 2 Based on the above embodiments and optional embodiments, the present invention also proposes an optional implementation method. Figure 4This is a flowchart of an optional category-based prototype-enhanced cross-modal hash retrieval method for the copper smelting industry, as described in Embodiment 2 of the present invention. Figure 4 As shown, the method includes: Step S1: Obtain image features, text features, and multi-label supervision information, and complete normalization preprocessing to form a multi-label matrix; S2. Statistically analyze the co-occurrence relationship of categories using the multi-label matrix, construct the category semantic adjacency matrix based on the Jaccard coefficient, and perform Top-k sparsification, symmetry, and degree normalization. S3. Generate initial category prototypes using category one-hot vectors as input, and obtain graph-smoothed category prototypes through semantic graph propagation; S4. Use an image encoder and a text encoder to map heterogeneous features to a unified hash space, and use graph smoothing class prototypes to calculate the bimodal prototype response matrix; S5. Using the bimodal prototype response matrix as a shared semantic anchor, construct prototype classification constraints, cross-modal response consistency constraints, and graph structure preservation constraints; S6. Construct a multi-positive sample comparison space based on the bimodal prototype response matrix and label sharing relationship to enhance cross-modal aggregation of samples of the same or related classes; S7. Jointly optimize semantic prototype alignment loss, cross-modal contrast loss, quantization loss, bit equalization loss, and decorrelation loss; S8. After training converges, the continuous representation is binarized, and the image retrieval text and text retrieval image are completed using Hamming distance.

[0085] By explicitly modeling the semantic structure between categories through steps S1 to S8, and using the category prototype as a reference for image-text sharing, the semantic gap between heterogeneous modalities can be alleviated, and the discriminativeness, stability and retrieval efficiency of multi-label cross-modal hash codes can be improved.

[0086] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A category-based prototype-enhanced cross-modal hash retrieval method for the copper smelting industry, characterized in that, include: Collect copper smelting images and process text samples and divide the dataset. Preprocess the dataset and construct a multi-label matrix using process fault labels in the dataset. The category co-occurrence matrix is ​​calculated based on the multi-label matrix, and the category adjacency strength is calculated using Jaccard coefficients. The category semantic graph is obtained by Top-k sparsity, symmetry, and degree normalization. An initial category prototype is generated using the unique heat vector of the smelting category. The category semantic graph is then smoothed using a multi-layer graph diffusion smoothing method. After normalization, the output graph smoothed category prototype is then created. An initial graph-text encoder with independent parameters and consistent output dimensions is constructed. The preprocessed dataset is input into the initial graph-text encoder to map out a unified and continuous hash space. The hash space is then multiplied by the graph smoothing category prototype inner product to obtain a bimodal prototype response matrix. A two-modal prototype classification loss is constructed using the multi-label matrix and the two-modal prototype response matrix, and the category responses of the image end and the text end in the category semantic map are constrained to be consistent. A multi-positive sample bidirectional contrast loss is constructed by combining a two-modal prototype classification loss and a category semantic graph. An asymptotic coefficient superimposed hash loss is introduced into the multi-positive sample bidirectional contrast loss. The initial image encoder is iteratively optimized with the goal of minimizing the hash loss to generate a target image encoder. The output of the target image encoder is mapped to a binary hash code to form a bidirectional retrieval.

2. The category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to claim 1, characterized in that, Collect copper smelting images and process text samples, divide the dataset, preprocess the dataset, and construct a multi-label matrix using the process fault labels in the dataset, including: Collect paired samples of copper smelting furnace and kiln operating condition images and process fault texts, and divide them into training set, query set, and industrial database set; L2 normalization is performed on the image visual features and the process text features respectively. L2 normalization is used to indicate that the features are scaled to a feature vector of unit length. A multi-label matrix is ​​constructed using smelting processes and fault multi-label vectors from the training set. The query set and the labels from the industrial database set are used to evaluate the retrieval accuracy of the target image encoder.

3. The category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to claim 1, characterized in that, The category co-occurrence matrix is ​​calculated based on the multi-label matrix, and the category adjacency strength is calculated using Jaccard coefficients, including: The category co-occurrence matrix is ​​as follows: Where M is the class co-occurrence matrix; Y is the training set multi-label matrix; This is the transpose of Y; The adjacency strength of the category is: in, Let this be the co-occurrence matrix of category a and category b in the training set. Let be the co-occurrence matrix of class a in the training set. Let b be the co-occurrence matrix of class b in the training set. Let ε be the Jaccard adjacency strength between category a and category b, where a is the index of category a, b is the index of category b, and ε is a very small positive number.

4. The category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to claim 1, characterized in that, An initial category prototype is generated using the unique heat vector of the smelting category. The category semantic graph is then smoothed using multi-layer graph diffusion smoothing. After normalization, the output graph-smoothed category prototype includes: Determine smelting category prototypes based on process fault labels in the dataset; Based on the category semantic graph, semantic neighbor category prototypes that are strongly coupled with the smelting category prototype are determined, wherein the category semantic graph is used to provide the weights for strong coupling; The initial category prototype is obtained by weighted fusion of the smelting category prototype and the semantic neighbor category prototype; Using the category semantic graph as weights and the process failure categories in the dataset as nodes, multi-layer propagation is performed on the category semantic graph to obtain a graph-smoothed category prototype.

5. The category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to claim 1, characterized in that, Build an initial image-text encoder with independent parameters and consistent output dimensions, including: The initial image encoder includes an image encoder and a text encoder. The image encoder and the text encoder have the same number of layers and output dimensions, and their parameters are not shared. The final layer outputs of the image encoder and the text encoder retain a continuous representation, which is used to calculate the bimodal prototype response matrix.

6. The category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to claim 5, characterized in that, The bimodal prototype response matrix is ​​obtained by multiplying the hash space with the graph smoothing category prototype inner product, including: The image-side category prototype response matrix is ​​as follows: in, For image-side category prototype response matrix, Let W be a matrix consisting of the consecutive hash representations of all images within a batch, and let W be the graph smoothing class prototype. Indicates the transpose of W; The text-based category prototype response matrix is ​​as follows: in, For text-based category prototype response matrices, Let W be a matrix consisting of the consecutive hash representations of all texts within a batch, and let W be the prototype of the graph smoothing class. This represents the transpose of W.

7. The category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to claim 5, characterized in that, The two-modal prototype classification loss is constructed using the multi-label matrix and the two-modal prototype response matrix, including: The two-modal prototype classification loss is: in, Let BCE(·) be the two-modal prototype classification loss, BCE(·) be the binary cross-entropy loss, and Y be the training sample label matrix. For image-side category prototype response matrix, This is the text-based category prototype response matrix.

8. The category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to claim 1, characterized in that, Before combining the two-modal prototype classification loss with the category semantic map to construct a multi-positive sample bidirectional contrastive loss, the method also includes: Using Top-k similarity gating, pairs of categories with a class adjacency strength greater than a preset value are retained in the category semantic graph; If no row in the category semantic graph is directly aligned, the image and text of the same sample are retained as the basic alignment.

9. The category prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to claim 1, characterized in that, Introducing asymptotic coefficient superimposed hash loss into the multi-positive-sample bidirectional contrast loss includes: The hash loss is: in, This represents the two-modal prototype classification loss. This represents the cross-modal prototype response consistency loss. This indicates the loss of graph structure preservation. This represents the bidirectional cross-modal contrast loss. Indicates quantified loss. Indicates bit equalization loss. This represents the decorrelation loss, where r is the asymptotic coefficient. , , All are modal loss coefficients. , , All are loss coefficients for multiple positive samples.

10. A category-based prototype-enhanced cross-modal hash retrieval method for the copper smelting industry according to claim 9, characterized in that, Introducing asymptotic coefficient superimposed hash loss into the multi-positive-sample bidirectional contrast loss includes: The asymptotic coefficient is: Where r is the asymptotic coefficient and e is the current training round; is the number of preheating cycles; min(·) is the minimum value operation.

Images