Multi-label zero sample feature synthesis method, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to the field of machine learning, in particular to a multi-label zero sample feature synthesis method, electronic equipment and a storage medium, and the method comprises the steps: obtaining vector data obtained by converting target sample data source data, and converting the vector data into time sequence data through a preset time sequence scaling model; based on the time sequence data, class-specific embedding vectors corresponding to a plurality of preset tags are extracted; in combination with a time sequence scaling algorithm provided by the time sequence scaling model, generating a global embedding vector and a first synthesis feature based on various specific embedding vectors; and based on each class-specific embedded vector, correspondingly generating a class-specific potential feature and a second synthetic feature, and based on the first synthetic feature and the second synthetic feature, generating a final fusion feature. The fusion features comprise information from different sources, the recognition capability of the model for unseen categories is enhanced, and the generalization performance of the model in a multi-label zero sample classification task is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of machine learning, and particularly to a method for synthesizing multi-label zero-shot features, an electronic device, and a storage medium. Background Art

[0002] In real-world tasks, most involve multi-label recognition, where an image can contain multiple and diverse class labels. In the zero-shot learning setting, multi-label classification becomes even more challenging, with the test set containing either only unseen classes or both seen and unseen classes.

[0003] Existing multi-label zero-shot learning methods mainly utilize global image representations, structured knowledge graphs, and attention-based mechanisms, but these methods have limitations in synthesizing multi-label features. Single-label zero-shot learning has received significant attention, and generative methods dominate in this area, but these methods are not applicable to multi-label scenarios. Summary of the Invention

[0004] The purpose of the present invention is to provide at least a method for synthesizing multi-label zero-shot features, an electronic device, and a storage medium, which can at least solve the technical problem that single-label zero-shot learning methods are not applicable to multi-label scenarios, and can at least achieve the effect of improving the accuracy of large-scale multi-label zero-shot classification and generalization zero-shot classification.

[0005] To solve the above technical problems, at least one embodiment of the present application provides a method for synthesizing multi-label zero-shot features, including: Obtain vector data obtained by transforming target sample source data, and use a preset time series scaling model to scale the vector data to generate time series data; Based on the time series data, extract class-specific embedding vectors corresponding to multiple preset labels; Combine the time series scaling algorithm provided by the time series scaling model, and based on each of the class-specific embedding vectors, generate a global embedding vector. According to the global embedding vector, generate a first synthetic feature, where the global embedding vector is a global description of the positive label of the target to be recognized; Combine the time series scaling algorithm provided by the time series scaling model, and based on each of the class-specific embedding vectors, generate a class-specific latent feature correspondingly. Synthesize a preset number of the class-specific latent features to generate a second synthetic feature; Based on the first synthetic feature and the second synthetic feature, generate a fusion feature, which is used for multi-label zero-shot classification.

[0006] In an alternative embodiment, the temporal scaling algorithm provided in combination with the temporal scaling model generates a global embedding vector based on each of the class-specific embedding vectors, including: Calculating the average of each of the class-specific embedding vectors according to the following calculation formula to obtain the global embedding vector:

[0007] Wherein, Is the global embedding vector, Represents a multi-hot label with Positive classes, Is the positive class The corresponding class-specific embedding vector, Is the temporal scaling algorithm.

[0008] In an alternative embodiment, generating a first synthetic feature according to the global embedding vector includes: Calculating the first synthetic feature according to the following calculation formula:

[0009] Wherein, Is the first synthetic feature, Is a preset noise, Is a preset first generator for capturing the correlation between labels.

[0010] In an alternative embodiment, the temporal scaling algorithm provided in combination with the temporal scaling model generates a class-specific latent feature corresponding to each of the class-specific embedding vectors, including: Calculating the class-specific latent feature according to the following calculation formula:

[0011] Wherein, Represents a multi-hot label with Positive classes, Is the positive class The corresponding class-specific embedding vector, Is a preset noise, Is a preset second generator, Is Is the positive class The synthesized class-specific latent feature, Is the temporal scaling algorithm.

[0012] In an alternative embodiment, synthesizing a preset number of the class-specific latent features to generate a second synthetic feature includes: The second synthetic feature is calculated according to the following calculation formula:

[0013] wherein, is the second synthetic feature.

[0014] In an alternative embodiment, generating the fused feature based on the first synthetic feature and the second synthetic feature includes: Stacking the first synthetic feature and the second synthetic feature to generate a feature matrix; Based on the multi-head self-attention mechanism, using a projection head to project each feature in the feature matrix into a low-dimensional space to generate a triple composed of a query vector, a key vector, and a value vector, wherein the state of each feature in the current form in the feature matrix is retained in the value vector, and the query vector is used to find the correlation between the query vector and the key obtained from two features in the feature matrix; For each of the triples given the projection head, performing internal head processing and cross-head processing in sequence to obtain the fused feature, wherein the internal head processing is used to associate each query vector with the key vector from the two features to generate an attention feature based on the correlation between different features during the feature fusion process, and the cross-head processing is used to integrate the attention features from different projection heads to generate the fused feature.

[0015] In an alternative embodiment, in the internal head processing and cross-head processing performed in sequence for each of the triples given the projection head, the internal head processing includes: Using the following function to obtain the normalized relationship score:

[0016] wherein, is function, is the normalized relationship score, is the query vector, is the key vector, is the dimension of the low-dimensional space, is the time series scaling method; Using the following calculation formula to obtain the attention feature: , wherein, is the attention feature, is the value vector.

[0017] In an alternative embodiment, in the internal head processing and cross-head processing performed successively for each of the triples for the given projection head, the cross-head processing includes: Concatenating the low-dimensional attention features from each head to generate an output vector of the original dimension; Performing mean pooling on the output vector along the row dimension to obtain a single fused feature.

[0018] In an alternative embodiment, the performing mean pooling on the output vector along the row dimension to obtain a single fused feature includes: Processing the output vector according to the following calculation formula where, is the output vector, is the feature matrix, is a preset residual sub-network, is the output vector after adding a residual branch; Performing mean pooling on the output vector after adding a residual branch along the row dimension to obtain a single fused feature.

[0019] In an alternative embodiment, the calculation formula of the time series scaling algorithm is as follows:

[0020] where, is the time series scaling algorithm, is the model proximity point threshold of the time series scaling model, is the expansion factor in the training of the time series scaling model, is the normalization coefficient to be obtained,<{ is the exponent, is the number of parameters of the time series scaling model.

[0021] At least one embodiment of the present application further provides a multi-label zero-shot feature synthesis device, including: A time series processing module, configured to obtain vector data obtained by transforming target sample source data, and perform scaling processing on the vector data by using a preset time series scaling model to generate time series data; A class-specific embedding vector generation module, configured to extract class-specific embedding vectors corresponding to a plurality of preset labels based on the time series data; A first synthetic feature generation module, configured to combine the time series scaling algorithm provided by the time series scaling model, generate a global embedding vector based on each of the class-specific embedding vectors, and generate a first synthetic feature according to the global embedding vector, where the global embedding vector is a global description of the positive label of the target to be recognized;​ A second synthetic feature generation module, configured to combine the temporal scaling algorithm provided by the temporal scaling model, and correspondingly generate a class-specific latent feature based on each of the class-specific embedding vectors, and synthesize a preset number of the class-specific latent features to generate a second synthetic feature; A fused feature generation module, configured to generate a fused feature based on the first synthetic feature and the second synthetic feature, where the fused feature is used for multi-label zero-shot classification.

[0022] At least one embodiment of the present application further provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above multi-label zero-shot feature synthesis method.

[0023] At least one embodiment of the present application further provides a computer-readable storage medium storing a computer program, where the computer program implements the above multi-label zero-shot feature synthesis method when executed by a processor.

[0024] The multi-label zero-shot feature synthesis method, electronic device, and storage medium provided by the embodiments of the present application perform scaling processing on vector data through a temporal scaling model, so that different features or variables are within a similar range or dimension numerically, which helps to eliminate the dimensional difference between different features and is conducive to extracting more accurate and meaningful class-specific embedding vectors, thereby being able to better reflect the correlation between the target sample and the preset label, and thus improving the accuracy of feature extraction. At the same time, the temporal scaling processing not only helps with the feature extraction of the current data, but also can simulate the feature distribution of unseen categories to a certain extent, which helps to enhance the model's recognition ability for unseen categories and improve the generalization performance of the model in the multi-label zero-shot classification task. After the temporal scaling processing and the extraction of class-specific embedding vectors, generating a global embedding vector based on the extracted class-specific embedding vectors helps the model to describe the overall features of the recognized target. At the same time, by generating class-specific latent features and synthetic features, the model can capture more refined local features. By combining the global embedding vector and the synthetic feature to generate a fused feature, the fused feature contains information from different sources, which enables the model to better handle unseen categories, comprehensively consider global and local information, and thus improve the classification accuracy. Description of the Drawings

[0025] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings, and these exemplary illustrations do not limit the embodiments.

[0026] Figure 1It is a flowchart of a method for synthesizing multi-label zero-shot features provided by an embodiment of the present application; Figure 2 It is a schematic diagram of a multi-label zero-shot feature synthesis device provided by another embodiment of the present application. Detailed implementation manners

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will elaborate on each embodiment of the present application in conjunction with the accompanying drawings. However, those of ordinary skill in the art can understand that in each embodiment of the present application, many technical details are presented for the better understanding of the readers. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented. The division of the following embodiments is for convenience of description and should not constitute any limitation on the specific implementation manner of the present application. The various embodiments can be combined and cross-referenced with each other on the premise of not being contradictory.

[0028] In real-world tasks, most involve multi-label recognition, where an image can contain multiple and diverse class labels. In the zero-shot learning setting, multi-label classification becomes even more challenging, with the test set either containing only unseen classes or both seen and unseen classes.

[0029] Multi-label classification is a challenging problem, especially in the context of zero-shot learning. Existing multi-label zero-shot learning methods mainly utilize global image representations, structured knowledge graphs, and attention-based mechanisms, but these methods have limitations in synthesizing multi-label features. Single-label zero-shot learning has received significant attention, and generative methods dominate in this area, but these methods are not applicable to multi-label scenarios.

[0030] To solve the technical problem that the above single-label zero-shot learning methods are not applicable to multi-label scenarios, this embodiment proposes a method for synthesizing multi-label zero-shot features, which uses the generative model capabilities of large models to learn the latent data distribution of seen classes and mimics the fully supervised setting by synthesizing (fake) features of unseen classes. Different from the single-label setting, multi-label feature synthesis requires accurate integration of multiple category information that co-occurs in an image. Therefore, in multi-label feature synthesis, it becomes crucial to accurately fuse the category information of each label embedding.

[0031] The following specifically describes the implementation details of the multi-label zero-shot feature synthesis method of this embodiment. The following content is only implementation details provided for convenience of understanding and is not necessary for implementing this solution.

[0032] Embodiment 1: The synthesis method of multi-label zero-sample features of this embodiment can be applied to electronic devices with communication, computing and data storage capabilities. The specific process can be as follows: Figure 1 As shown, including: Step 110: obtaining vector data obtained by transforming the target sample source data, and scaling the vector data using a preset time series scaling model to generate time series data; In this embodiment, the target sample data source includes at least one of documents, databases, unstructured data, semi-structured data, and knowledge graphs. Before conversion to vector data, the target sample data source may be preprocessed and cleaned using methods such as smoothing, padding, correction, conversion, normalization, and discretization to improve data quality and usability.

[0033] The preset time series scaling model is used to scale time series data. Scaling refers to adjusting the numerical range or distribution of data through some mathematical transformation to ensure a consistent scale or distribution of the data during subsequent analysis or modeling, thereby preventing certain features from being ignored or overemphasized due to excessively large or small numerical ranges. The time series scaling model in this embodiment can achieve this goal using methods such as linear scaling, nonlinear scaling, normalization, and standardization.

[0034] After scaling, the data becomes time series data. Although the original target sample source data may not have clear time series properties, after scaling, the data exhibits an ordered or consistent distribution in terms of values. This data state facilitates subsequent feature extraction, model training, and analysis.

[0035] Step 120: extracting class-specific embedding vectors corresponding to a plurality of preset labels based on the time series data; In this embodiment, the preset labels are a series of categories or labels defined in advance, which are used to represent or classify data. In multi-label classification tasks, a target sample may belong to multiple label categories at the same time. Embedding vectors are a technology that maps high-dimensional data to low-dimensional space while preserving the original structure and information of the data as much as possible. Here, the class-specific embedding vector refers to a low-dimensional feature vector with specific meaning that is extracted from the time series data and corresponds to each preset label. These vectors can reflect the characteristic performance of the target sample in a specific label category, providing high-quality feature input for subsequent classification, clustering or other analysis tasks.

[0036] Step 130: Combine the temporal scaling algorithm provided by the temporal scaling model, generate a global embedding vector based on each of the class-specific embedding vectors, and generate a first synthetic feature according to the global embedding vector, where the global embedding vector is a global description of the positive label of the target to be recognized; In this embodiment, the global embedding vector is a global description of the positive label of the target to be recognized (i.e., the correct class to which the target belongs). The global embedding vector can be obtained by aggregating multiple class-specific embedding vectors in a certain form, such as averaging, weighted averaging, maximum value, minimum value, etc.

[0037] The first synthetic feature is generated based on the global embedding vector. The first synthetic feature can be some statistical attributes (such as mean, variance, entropy, etc.) containing the global embedding vector, or a new feature representation obtained through a certain transformation (such as linear transformation, non-linear transformation, etc.). The first synthetic feature is used in subsequent classification, regression, clustering, or other machine learning tasks to improve the performance or accuracy of the model.

[0038] By combining with the temporal scaling algorithm, the first synthetic feature may be able to capture more dynamic changes and trend information about the time series data. This makes the first synthetic feature not only contain the information of the global embedding vector but also incorporate the temporal characteristics of the time series data, thereby enhancing the expressive ability of the feature.

[0039] In this embodiment, the temporal scaling algorithm can be used to reduce the dimension or complexity of the data for more efficient subsequent processing. By combining with the temporal scaling algorithm, the process of generating the first synthetic feature may become more efficient, which helps to reduce the use of computing resources, improve the speed of data processing, and reduce the time cost of model training and inference.

[0040] Step 140: Combine the temporal scaling algorithm provided by the temporal scaling model, generate a class-specific latent feature corresponding to each of the class-specific embedding vectors, and generate a second synthetic feature by synthesizing a preset number of the class-specific latent features; In this embodiment, the generation method of the class-specific latent feature can include calculating the correlation between the class-specific embedding vector and the candidate feature, and selecting the feature highly correlated with the class-specific embedding vector as the class-specific latent feature.

[0041] The implementation method of feature importance (correlation) evaluation can be to use machine learning algorithms (such as random forest, gradient boosting tree, etc.) to evaluate the importance of each feature. Then, select the features with higher importance as the class-specific latent features.

[0042] Before synthesizing specific latent features of different classes to generate the second synthetic feature, a preset quantity needs to be determined. This quantity can be determined based on the requirements of the application scenario, the characteristics of the data, and the limitations of computing resources. According to the preset quantity, a certain number of features are selected from the filtered class-specific latent features for synthesis. The synthesis methods may include concatenation, linear combination, non-linear transformation, etc. Among them, concatenation is to simply place multiple features side by side to form a new vector; linear combination is to perform weighted summation of multiple features according to certain weights; non-linear transformation may include using models such as neural networks and support vector machines to further transform the features.

[0043] The feature after the synthesis process is the second synthetic feature. The second synthetic feature contains the information of multiple class-specific latent features and has undergone further transformation and fusion, thus more comprehensively and effectively reflecting the characteristics of the original data.

[0044] In this embodiment, combining with the time series scaling algorithm to generate the second synthetic feature enables the second synthetic feature to potentially capture more dynamic changes and trend information of the time series data, while reducing the dimension or complexity of the data, which helps to improve the speed of data processing.

[0045] Step 150, generate a fusion feature based on the first synthetic feature and the second synthetic feature, where the fusion feature is used for multi-label zero-shot classification.

[0046] In this embodiment, the first synthetic feature emphasizes label dependence (the correlation between labels) and does not explicitly encode the discrimination information about each category. Therefore, the first synthetic feature generated based on the global embedding vector has low category-specific discrimination. During the process of generating the second synthetic feature, the generated latent features better retain the category-specific discrimination information of the positive categories existing in the recognized target, but synthesize them independently of each other. Therefore, the second synthetic feature does not explicitly encode the label dependence in the image.

[0047] It can be seen that although the first synthetic feature emphasizes label dependence, it performs weakly in terms of category discrimination. The second synthetic feature, by generating latent features, better retains the category-specific discrimination information of the positive categories. Combining the two, the fusion feature can significantly improve category discrimination while maintaining label dependence.

[0048] In the multi-label classification task, there are often complex dependencies and correlations between labels. The first synthetic feature provides a global perspective on the relationship between labels for the fusion feature by capturing these label dependencies. The second synthetic feature, on the other hand, provides a fine-grained description of the local characteristics of individual categories for the fusion feature by independently synthesizing latent features. The combination of the two helps to capture the complex relationship between labels and categories.

[0049] In a zero-shot classification task, the model needs to make predictions on unseen classes. By combining the advantages of the first synthetic feature and the second synthetic feature, the fused feature can more effectively utilize the information of known classes to infer the features of unseen classes. This helps to improve the generalization ability and accuracy of the model in the zero-shot classification task.

[0050] In summary, for the method for synthesizing multi-label zero-shot features provided in this embodiment, the vector data is scaled by the time series scaling model, so that different features or variables are within a similar range or dimension numerically, which helps to eliminate the dimensional differences between different features and is conducive to extracting more accurate and meaningful class-specific embedding vectors, thereby being able to better reflect the correlation between the target sample and the preset label, and thus improving the accuracy of feature extraction. At the same time, the time series scaling process not only helps with the feature extraction of the current data, but also can, to a certain extent, simulate the feature distribution of unseen classes, which helps to enhance the model's recognition ability for unseen classes and improve the generalization performance of the model in the multi-label zero-shot classification task. After the time series scaling process and the extraction of class-specific embedding vectors, a global embedding vector is generated based on the extracted class-specific embedding vectors, which helps the model to describe the overall features of the recognized target. At the same time, by generating class-specific latent features and synthetic features, the model can capture more refined local features. By combining the global embedding vector and the synthetic feature to generate a fused feature, the fused feature contains information from different sources, which enables the model to better handle unseen classes, be able to comprehensively consider global and local information, and thus improve the classification accuracy.

[0051] In an optional embodiment, the step of generating a global embedding vector based on each of the class-specific embedding vectors by combining the time series scaling algorithm provided by the time series scaling model includes: calculating the average of each of the class-specific embedding vectors according to the following calculation formula to obtain the global embedding vector:

[0052] where is the global embedding vector, represents a multi-hot label with positive classes, is the class-specific embedding vector corresponding to the positive class , is the time series scaling algorithm, and the attribute set of the image can be expressed as , .

[0053] In this embodiment, the global embedding vector is obtained by averaging the class-specific embedding vectors, which means that this global embedding vector synthesizes the information of all positive labels, thus providing a global and comprehensive perspective. This global description ability helps the model better understand the overall content of the image, rather than being limited to a single label or feature.

[0054] In an alternative embodiment, the calculation formula of the time series scaling algorithm is as follows:

[0055] Wherein, is the time series scaling algorithm, is the model proximity point threshold of the time series scaling model, is the expansion factor in the training of the time series scaling model, is the normalization coefficient to be obtained, is the exponent, is the number of parameters of the time series scaling model.

[0056] In this embodiment, the normalization coefficient helps to eliminate the differences between different data magnitudes, enabling the model to more stably learn the features in the data. By adjusting the parameters of the algorithm (such as the number of parameters N), the complexity of the model can be controlled, thereby avoiding overfitting or underfitting problems. The model proximity point threshold, as a regularization means, helps the model be more robust when dealing with extreme or abnormal data. When a data point approaches or exceeds this threshold, the algorithm can take corresponding measures (such as ignoring these data points or performing special processing) to avoid their excessive impact on the model. By introducing the threshold, the algorithm can better distinguish the noise and real signals in the training data, thereby enhancing the generalization ability of the model to new data and helping the model perform better on unseen data.

[0057] In an alternative embodiment, generating the first synthetic feature according to the global embedding vector includes: calculating the first synthetic feature according to the following calculation formula:

[0058] Wherein, is the first synthetic feature, is the preset noise, is the preset first generator, and the first generator is used to capture the correlation between labels, The calculation formula of is the same as the time series scaling algorithm in the above embodiment.

[0059] In this embodiment, by inputting the global embedding vector and the preset noise into the first generator the first synthetic feature can be generated . By introducing the preset noise , to a certain extent, can simulate the data uncertainty in the real world, enabling the model to learn how to handle this uncertainty during training, which helps improve the robustness of the model in practical applications.

[0060] In an optional embodiment, combining the temporal scaling algorithm provided by the temporal scaling model, based on each of the class-specific embedding vectors, a class-specific latent feature is correspondingly generated, including: calculating the class-specific latent feature according to the following calculation formula:

[0061] where, represents a multi-hot label with positive classes, is the positive class corresponding class-specific embedding vector, is a preset noise, is a preset second generator, is is the positive class synthesized class-specific latent feature, is the temporal scaling algorithm.

[0062] In this embodiment, inputting n class-specific embedding vectors and the preset noise together into the second generator , n class-specific latent features can be obtained. For a fixed noise , the second generator synthesizes a fixed latent feature for the class , regardless of whether there are other classes in the recognized target. The latent feature better retains the class-specific discrimination information of the positive class existing in the recognized target.

[0063] Introducing the preset noise can simulate the data uncertainty in the real world, helps improve the robustness of the model in practical applications, and enables it to better handle various complex and changing data situations.

[0064] In an optional embodiment, synthesizing a preset number of the class-specific latent features to generate a second synthetic feature, including: calculating the second synthetic feature according to the following calculation formula:

[0065] where, is the second synthetic feature, It is a class-specific latent feature.

[0066] In this embodiment, the information of multiple features is combined by taking the mean of multiple class-specific latent features to generate a second synthetic feature, thereby generating a more representative new feature. Among them, It is a time series scaling algorithm.

[0067] In an alternative embodiment, generating a fused feature based on the first synthetic feature and the second synthetic feature includes: stacking the first synthetic feature and the second synthetic feature to generate a feature matrix; based on the multi-head self-attention mechanism, using a projection head to project each feature in the feature matrix into a low-dimensional space to generate a triple composed of a query vector, a key vector, and a value vector, where the state of each feature in the current form in the feature matrix is retained in the value vector, and the query vector is used to find the correlation between the query vector and the key obtained from two features in the feature matrix; for each triple given the projection head, performing internal head processing and cross-head processing in sequence to obtain the fused feature, where the internal head processing is used to associate each query vector with the key vectors from the two features to generate attention features based on the correlation between different features during the feature fusion process, and the cross-head processing is used to integrate the attention features from different projection heads to generate the fused feature.

[0068] In this embodiment, stack individual second synthetic features and the first synthetic feature to create a matrix . Then, use a total of H projection heads to linearly project the features in this matrix into a low-dimensional space ( ) to create the following query-key-value triples: , ,

[0069] Among them, is the query vector, is the key vector, is the value vector, , and , , is the time series scaling algorithm.

[0070] For each feature, its state in its current form is kept in the “value” embedding, while the query vector derived from each input feature is used to find its correlation with the key obtained from the two features. Given these triplets for each head, the features undergo two levels of processing: i) intra-head processing on the triples, and ii) cross-head processing.

[0071] In an optional embodiment, for each triplet of a given projection head, the inner head processing and the cross-head processing are sequentially performed, and the inner head processing includes: using the following The normalized relationship score obtained by the function:

[0072] in, for function, is the normalized relationship score, is the query vector, is the key vector, is the dimension of the low-dimensional space, The timing scaling method; The attention feature is obtained using the following calculation formula: , in, is the attention feature, is the value vector.

[0073] In this embodiment, from Normalized relationship score obtained by the function Used to reweight the corresponding value vector to obtain the attention feature. , .

[0074] In an optional embodiment, for each triplet of a given projection head, internal head processing and cross-head processing are performed in sequence, and the cross-head processing includes: splicing the low-dimensional attention features from each head to generate an output vector of the original dimension; and performing mean pooling on the output vector along the row dimension to obtain a single fused feature.

[0075] In this embodiment, after obtaining the attention features, in order to aggregate information among all heads, these low-dimensional attention features from each head are concatenated and processed through an output layer to generate the original dimensional output vector . As shown below:

[0076] Among them, is a learnable weight matrix.

[0077] In an optional embodiment, the mean pooling of the output vector along the row dimension to obtain a single fusion feature includes: processing the output vector according to the following calculation formula:

[0078] Among them, is the output vector, is the feature matrix, is a preset residual sub-network, is the output vector after adding a residual branch; Perform mean pooling on the output vector after adding a residual branch along the row dimension to obtain a single fusion feature.

[0079] In this embodiment, after obtaining the output vector a residual branch is added from the input to the attention feature and further processed through a small residual sub-network to help the network first focus on the local neighborhood and then gradually focus on other levels of features,

[0080] such that , which encourages the network to selectively add supplementary information to the source vector (feature matrix) Add supplementary information.

[0081] By adding supplementary information to the source vector (i.e., the result of the attention feature processed by the small residual sub-network), the model can capture more and richer feature information. This information helps the model to more accurately understand the input data, thereby improving the overall performance of the model. The introduction of the residual branch enables the network to first focus on the features of the local neighborhood and then gradually focus on other levels of features. This step-by-step fusion process helps the model to capture local details while not losing the global perspective, thereby improving the generalization ability of the model.

[0082] Finally, perform mean pooling on the matrix along the row dimension to obtain a single fusion feature, specifically as follows: ,

[0083] Fusion feature Explicitly encodes the correlations between the tags of the identified targets and the category - specific discrimination information of the existing positive classes.

[0084] Next, the synthesis method of multi - label zero - shot features is integrated into the large - model generation framework for multi - label zero - shot classification. Briefly, the condition is the class - specific embedding vectors , and a classifier that has seen the classes, for multi - label (generalized) zero - shot classification. That is, the first synthetic feature is synthesized by integrating the class - specific embedding vectors at the generator input. The second synthetic feature separately synthesizes features from the class - specific embedding vectors and integrates them in the feature space. The fused feature combines the advantages of the first synthetic feature and the second synthetic feature, utilizes each individual - level feature, and focuses on the two - level context. Therefore, the individual - level features self - adjust to generate rich synthetic features, which are then aggregated to obtain the final output. After testing, the synthesis method of multi - label zero - shot features in this embodiment also performs well on standard multi - label classification tasks and has good reliability in (generalized) zero - shot detection.

[0085] Embodiment 2: Based on the content of the above - mentioned embodiment, this embodiment provides a synthesis system for multi - label zero - shot features, which applies the synthesis method of multi - label zero - shot features in any of the above - mentioned embodiments. In this embodiment, by building the system based on the Mamba architecture, it can achieve powerful zero - shot performance on real - world data sets, and at the same time has a lower inference time than the time - series - based model based on the transformer architecture, and incorporates cross - level feature fusion technology. The model is only trained on synthetic data and can generate predictions for the entire prediction range at one time, exceeding traditional autoregressive methods. The present invention realizes the synthesis of semantically consistent multi - label visual features from the corresponding class attributes (embeddings), improving the accuracy of multi - label classification. The implementation steps are as follows: I. Basic data preparation Prepare the source data of the target sample materials, including data such as documents, databases, unstructured data, semi - structured data, knowledge graphs, etc., and perform pre - processing and cleaning such as data smoothing, data filling, data correction, data conversion, normalization, discretization, etc. to improve the data quality and usability. Then convert the relevant source data into vector data Embedding.

[0086] II. Model architecture 2.1. Dynamic decomposition module: The dynamic decomposition module deeply models the source data of the sample materials and its time stamps through a data encoder and a time encoder. Subsequently, stacked decomposition modules are used to extract stable components. Finally, offset subtraction technology is used to separate the trend components. The dynamic decomposition module specifically includes the following functional modules: (1) Data Encoder: Composed of a spatio-temporal Transformer module, which is specifically designed to model the temporal dependencies and variable correlations in multi-dimensional time series. The spatio-temporal Transformer module integrates a temporal attention mechanism and a spatial attention mechanism that operate in parallel.

[0087] (2) Time Encoder: Composed of a single time Transformer module, which models the temporal dependencies of timestamps.

[0088] (3) Decomposition Module: Introduces a data-time hybrid attention mechanism. By integrating the dot product similarity between timestamps and the dot product similarity between observations, it realizes the effective fusion of multi-source information and generates stable components accordingly.

[0089] (4) Offset Subtraction: Although the same moment in different cycles often exhibits a certain degree of similar fluctuation patterns, they are not always strictly one-to-one corresponding. Therefore, in this embodiment, an offset subtraction strategy is adopted to effectively solve the potential horizontal displacement problem.

[0090] 2.2. Diffusion Reconstruction Module: The diffusion reconstruction module uses noise diffusion technology to create an external information bottleneck and directly reconstructs the contaminated data with the backbone network. The reconstruction error is the anomaly score. The diffusion reconstruction module specifically includes the following functional modules: (1) Noise Diffusion: Different from the traditional DDPM method, while adding noise to the input, in this embodiment, attention is paid to keeping the original trend unchanged, so that the algorithm can focus more on those more important stable components.

[0091] (2) Backbone Network: Also constructed based on the spatio-temporal Transformer module, it takes into account the changes in both stable components and trend components while paying attention to stable components.

[0092] III. Scaling of the Time Series Model: 1. Cross-step Scaling Law: Specifically, the expression of the time series scaling algorithm is as follows:

[0093] Among them, is the time series scaling algorithm, is the model proximity point threshold of the time series scaling model, is the expansion factor in the training of the time series scaling model, is the normalization coefficient to be obtained, is the exponent, is the number of parameters of the time series scaling model.

[0094] 2. Cross-architecture Scaling Law: The basic models of time series are mainly based on the Transformer architecture. The expansion behavior of encoder-only and decoder-only Transformer architectures is increased.

[0095] IV. Generative Multi-label Feature Synthesis: The large model generates multi-labels through the following methods, where m in the following text is the time series scaling algorithm. In this embodiment, it is used to generate fused features based on a preset image: 1. Attribute-level Fusion: In the attribute-level fusion method, a global image-level embedding vector is obtained from a set of class-specific embedding vectors corresponding to multiple labels in the preset image. The image-level embedding represents the global description of the positive labels in the image. The global embedding is obtained by averaging the individual class embeddings. Then, the global embedding is input into the first generator together with noise to synthesize the first synthetic feature. The attribute-level fusion is given by the following formula:

[0096] where, is the global embedding vector, represents a multi-hot label with positive classes, is the positive class corresponding to the class-specific embedding vector;

[0097] where, is the first synthetic feature, is the preset noise, is the preset first generator, and the first generator is used to capture the correlation between labels.

[0098] The first generator in the attribute-level fusion method performs global image-level feature generation, thus capturing the label dependencies (correlations between labels) in the image. However, since the distinguishing information about each category is not explicitly encoded, the features generated from the global embedding have low category-specific distinguishability.

[0099] 2. Feature-level Fusion: The second synthetic feature is synthesized separately from the class-specific embedding vectors. This enables the feature-level fusion method to better retain the category-specific distinguishability in the synthetic features. Different from the attribute-level fusion method that integrates class embeddings, the feature-level fusion method first inputs n class-specific embedding vectors into the second generator and generates n class-specific latent features. Then, the n class-specific latent features are integrated through an averaging operation to obtain the final second synthetic feature. The feature-level fusion is represented by the following formula:

[0100] where, For positive classes the synthesized class-specific latent features is the preset noise is the preset second generator;

[0101] wherein is the second synthesized feature.

[0102] It can be observed that for a fixed noise , the second generator synthesizes fixed latent features for the classes regardless of the presence of other classes in the image. Therefore, although the generated latent features better preserve the class-specific discriminative information of the positive classes present in the image, they are synthesized independently of each other. As a result, the synthesized features do not explicitly encode the label dependencies in the image. Therefore, the above two fusion methods (the attribute-level fusion method and the feature-level fusion method) both have drawbacks when synthesizing multi-label features. Next, a fusion method is introduced that combines the advantages of the label dependency of the attribute-level fusion method and the class-specific discriminability of the feature-level fusion method.

[0103] 3. Relational Fusion: The proposed relational fusion aims to combine the advantages of the attribute-level fusion method and the feature-level fusion method. The relational feature fusion method combines the label dependency of the attribute-level fusion method and the class-specific discriminability of the feature-level fusion method during the feature generation stage. To this end, and are fed forward into the feature fusion block. Inspired by multi-head self-attention, the feature fusion block enriches each corresponding feature by obtaining guidance from the other branch.

[0104] Specifically, a feature matrix and is created by stacking individual . Then, the features in these feature matrices are linearly projected into a low-dimensional space ( ) using a total of H projection heads to create query-key-value triples, , ,

[0105] wherein is the query vector, is the key vector, is the value vector, , and , For each feature, the state of its current form is retained in the "value" embedding, while the query vectors derived from each input feature are used to find its correlation with the keys obtained from two features.

[0106] Given these triples for each head, the features undergo two levels of processing: internal head processing on the triples, and cross-head processing. For the first case, each query vector is associated with the "keys" from two features using the following formula: Utilize the following function to obtain the normalized relationship score:

[0107] where is the function, is the said normalized relationship score, is the said query vector, is the said key vector, is the dimension of the said low-dimensional space, is the said temporal scaling method; Utilize the following calculation formula to obtain the said attention feature: , where is the said attention feature, is the said value vector.

[0108] In this embodiment, the normalized relationship score obtained from the function is used to re-weight the corresponding value vectors, thereby obtaining the attention feature. , .

[0109] To aggregate information among all heads, these low-dimensional attention features from each head are concatenated and processed through an output layer to generate the original -dimensional output vector,

[0110] where is a learnable weight matrix.

[0111] After obtaining the self-attention features, a residual branch is added from the input to the attention features and further processed through a small residual sub-network to help the network first focus on the local neighborhood and then gradually focus on other levels of features:

[0112] Among them, is the said output vector, is the said feature matrix, is the residual sub-network, is the said output vector after adding the residual branch; This encourages the network to selectively add supplementary information to the feature matrix . Finally, perform mean pooling on along the row dimension to obtain a single cross-level fusion feature : ,

[0113] The cross-level fusion feature is obtained by effectively fusing the features generated by attribute-level fusion and feature-level fusion. Therefore, the correlations between the labels in the image, as well as the class-specific discrimination information of the positive classes present in the image, are explicitly encoded.

[0114] V. Through multi-label zero-shot classification: Integrate the above-mentioned synthesis method of multi-label zero-shot features into two representative generative large model architectures for multi-label zero-shot classification. Since the relational feature fusion integrates attribute fusion and feature fusion, we only describe here the integration of relational feature fusion into two classification frameworks. Next, we describe the integration of relational feature fusion into the large model generative framework. Briefly speaking, the condition is the class-specific embedding vector , and a classifier that has seen the classes, for multi-label (generalized) zero-shot classification.

[0115] To sum up, this embodiment uses a new architecture and a series of fusion technologies to propose a synthesis method for multi-label zero-shot features, solving the challenging problems of current large-scale multi-label zero-shot classification and generalized zero-shot classification. By introducing a zero-shot prediction model based on the Mamba architecture, it is able to achieve powerful zero-shot performance on real-world datasets, while having a lower inference time than the time series-based model based on the transformer architecture, and adding cross-level feature fusion technology. The model is only trained on synthetic data and can generate predictions for the entire prediction range at once, surpassing traditional autoregressive methods. The present invention realizes the synthesis of semantically consistent multi-label visual features from the corresponding class attributes (embeddings), improving the accuracy of multi-label classification.

[0116] Embodiment III: Another embodiment of the present application relates to a multi-label zero-shot feature synthesis device. The implementation details of the multi-label zero-shot feature synthesis device in this embodiment will be specifically described below. The following content is only the implementation details provided for convenient understanding and is not necessary for implementing this solution. The schematic diagram of the multi-label zero-shot feature synthesis device in this embodiment can be as shown in Figure 2 shown, including a timing processing module 310, a class-specific embedding vector generation module 320, a first synthetic feature generation module 330, a second synthetic feature generation module 340, and a fusion feature generation module 350.

[0117] The timing processing module 310 is configured to obtain vector data obtained by converting the target sample source data, and perform scaling processing on the vector data by using a preset timing scaling model to generate timing data; The class-specific embedding vector generation module 320 is configured to extract class-specific embedding vectors corresponding to a plurality of preset labels based on the timing data; The first synthetic feature generation module 330 is configured to combine the timing scaling algorithm provided by the timing scaling model, generate a global embedding vector based on each of the class-specific embedding vectors, and generate a first synthetic feature according to the global embedding vector, where the global embedding vector is a global description of the positive label of the target to be recognized; The second synthetic feature generation module 340 is configured to combine the timing scaling algorithm provided by the timing scaling model, generate a class-specific latent feature corresponding to each of the class-specific embedding vectors, and synthesize a preset number of the class-specific latent features to generate a second synthetic feature; The fusion feature generation module 350 is configured to generate a fusion feature based on the first synthetic feature and the second synthetic feature, and the fusion feature is used for multi-label zero-shot classification.

[0118] It is worth mentioning that each module involved in this embodiment is a logical module. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or implemented by a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, units that are not closely related to solving the technical problems proposed in the present application are not introduced in this embodiment, but this does not mean that there are no other units in this embodiment.

[0119] Embodiment 4: Another embodiment of the present application relates to an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-label zero-shot feature synthesis method in the above embodiments.

[0120] Among them, the memory and the processor are connected in a bus manner. The bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus may also connect various other circuits together, such as peripheral devices, voltage regulators, and power management circuits, etc., which are well known in the art, and thus will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be an element or multiple elements, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices on the transmission medium. The data processed by the processor is transmitted on the wireless medium through the antenna. Further, the antenna also receives data and transmits the data to the processor.

[0121] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store the data used by the processor when executing operations.

[0122] Embodiment Five: Another embodiment of the present application relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-mentioned embodiment of the method for synthesizing multi-label zero-shot features.

[0123] That is, those skilled in the art can understand that all or part of the steps of implementing the above-mentioned embodiment methods can be completed by instructing relevant hardware through a program. This program is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM for short), random access memories (RAM for short), magnetic disks, or optical discs, etc., which are various media that can store program codes.

[0124] Those of ordinary skill in the art can understand that the above-mentioned embodiments are specific embodiments for implementing the present application, and in practical applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A synthesis method for multi-label zero-shot features, characterized in that, Including: Obtain the vector data obtained after transformation from the target sample source data, and use a preset time series scaling model to perform scaling processing on the vector data to generate time series data; Based on the time series data, extract class-specific embedding vectors corresponding to a plurality of preset labels; Combined with the time series scaling algorithm provided by the time series scaling model, based on each of the class-specific embedding vectors, generate a global embedding vector, and based on the global embedding vector, generate a first composite feature, wherein the global embedding vector is a global description of the positive label of the target to be recognized; Combined with the time series scaling algorithm provided by the time series scaling model, based on each of the class-specific embedding vectors, correspondingly generate a class-specific latent feature, and synthesize a preset number of the class-specific latent features to generate a second composite feature; Based on the first composite feature and the second composite feature, generate a fusion feature, and the fusion feature is used for multi-label zero-shot classification.

2. The synthesis method of multi-label zero-shot features according to claim 1, wherein The combining the time series scaling algorithm provided by the time series scaling model, and generating a global embedding vector based on each of the class-specific embedding vectors includes: Average each of the class-specific embedding vectors according to the following calculation formula to obtain the global embedding vector: Among them, e μ is the global embedding vector, y represents a multi-hot label with n positive classes, and e(y j ) is the class-specific embedding vector corresponding to the positive class j, and m is the temporal scaling algorithm.

3. The synthesis method of multi-label zero-shot features according to claim 2, wherein The generating a first composite feature based on the global embedding vector includes: Calculate the first composite feature according to the following calculation formula: Among them, is the first synthetic feature, z is the preset noise, and G a is the preset first generator, and the first generator is used to capture the correlation between tags.

4. The synthesis method of multi-label zero-shot features according to claim 1, characterized in that The combining the time series scaling algorithm provided by the time series scaling model, and correspondingly generating a class-specific latent feature based on each of the class-specific embedding vectors includes: Calculate the class-specific latent feature according to the following calculation formula: x j = G f (z, e(y j ))m Among them, y represents a multi-hot label with n positive classes, and e(y j ) is the class-specific embedding vector corresponding to the positive class j, z is the preset noise, G f is the preset second generator, x j is the class-specific latent feature synthesized by G f for the positive class j, and m is the time series scaling algorithm.

5. The synthesis method of multi-label zero-shot features according to claim 4, wherein The synthesizing a preset number of the class-specific latent features to generate a second composite feature includes: Calculate the second composite feature according to the following calculation formula: Among them, x f is the second synthetic feature.

6. The synthesis method of multi-label zero-shot features according to claim 1, wherein The generating a fusion feature based on the first composite feature and the second composite feature includes: Stack the first composite feature and the second composite feature to generate a feature matrix; Based on the multi-head self-attention mechanism, use a projection head to project each feature in the feature matrix into a low-dimensional space to generate a triple composed of a query vector, a key vector, and a value vector, wherein the state of each feature in the current form in the feature matrix is retained in the value vector, and the query vector is used to find the correlation between the query vector and the key obtained from two features in the feature matrix; For each triple given the projection head, perform internal head processing and cross-head processing in sequence to obtain the fusion feature, wherein the internal head processing is used to associate each query vector with the key vector from the two features to generate an attention feature based on the correlation between different features during the feature fusion process, and the cross-head processing is used to integrate the attention features from different projection heads to generate the fusion feature.

7. The synthesis method of multi-label zero-shot features according to claim 6, characterized in that In the performing internal head processing and cross-head processing in sequence for each triple given the projection head, the internal head processing includes: Obtain the normalized relationship score using the following softmax function: where, σ is the softmax function, r h is the normalized relationship score, q h is the query vector, k h is the key vector, d , is the dimension of the low-dimensional space, and m is the time series scaling method; Obtain the attention feature using the following calculation formula: α h = r h v h + m, Among them, α h is the attention feature, and v h is the value vector.

8. The synthesis method of multi-label zero-shot features according to claim 6, wherein For each of the triples for the given projection head, during the internal head processing and cross-head processing in sequence, the cross-head processing includes: Concatenating the low-dimensional attention features from each head to generate an output vector of the original dimension; Performing mean pooling on the output vector along the row dimension to obtain a single fused feature.

9. The synthesis method of multi-label zero-shot features according to claim 8, wherein The performing mean pooling on the output vector along the row dimension to obtain a single fused feature includes: Processing the output vector according to the following calculation formula: where, o is the output vector, is the feature matrix, and f() is a preset residual sub-network, is the output vector after adding the residual branch; Performing mean pooling on the output vector added with a residual branch along the row dimension to obtain a single fused feature.

10. The synthesis method of multi-label zero-shot features according to any one of claims 1-9, characterized in that The calculation formula of the temporal scaling algorithm is as follows: Among them, m is the timing scaling algorithm, M is the model proximity point threshold of the timing scaling model, X is the expansion factor in the training of the timing scaling model, and X c is the normalization coefficient to be obtained, τ is the exponent, and N is the number of parameters of the timing scaling model.

11. A synthesis device for multi-label zero-shot features, characterized in that, Including: A temporal processing module, configured to obtain vector data obtained by transforming target sample source data, and perform scaling processing on the vector data by using a preset temporal scaling model to generate temporal data; A class-specific embedding vector generation module, configured to extract class-specific embedding vectors corresponding to a plurality of preset labels based on the temporal data; A first composite feature generation module, configured to combine the temporal scaling algorithm provided by the temporal scaling model, generate a global embedding vector based on each of the class-specific embedding vectors, and generate a first composite feature according to the global embedding vector, wherein the global embedding vector is a global description of the positive label of the target to be recognized; A second composite feature generation module, configured to combine the temporal scaling algorithm provided by the temporal scaling model, generate a class-specific latent feature corresponding to each of the class-specific embedding vectors, and synthesize a second composite feature by integrating a preset number of the class-specific latent features; A fused feature generation module, configured to generate a fused feature based on the first composite feature and the second composite feature, and the fused feature is used for multi-label zero-shot classification.

12. An electronic device, characterized in that, Including: At least one processor; And, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-label zero-shot feature synthesis method as described in any one of claims 1 to 10.

13. A computer-readable storage medium stores a computer program, characterized in that, When the computer program is executed by a processor, it implements the multi-label zero-shot feature synthesis method as described in any one of claims 1 to 10.

Images