A zero-shot learning method and device based on semantic knowledge graph propagation

Abstract

The application belongs to the technical field of machine learning, and discloses a zero-shot learning method and device based on semantic knowledge graph propagation. Visible sample data and unseen sample data are obtained in an ImageNet dataset, a CNN model is used to modify a cost function and an aggregated loss function, the model is trained, and a trained CNN model is obtained; based on a GCN model, an optimization function is set, a feature constraint term is added to the optimization function, a CGCN model is obtained, the CGCN model is trained in a self-supervised mode, and a trained CGCN model is obtained; based on an AE model, a final loss function is set based on a matching loss function and a constrained loss function, the AE model is trained based on the final loss function, and a trained AE model is obtained. The distribution drift problem in the zero-shot learning process is alleviated, and the accuracy of the zero-shot learning model verification is improved.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a zero-trial learning method and apparatus based on semantic knowledge graph propagation. Background Technology

[0002] Zero-shot learning (ZSL), also known as zero-data learning, pursues generalized intelligence in machine learning, aiming to predict categories that have never appeared before. ZSL leverages semantic information such as attribute features, WordNet, and text keyword features to understand images beyond the visual level based on knowledge transfer between visible and unseen classes.

[0003] Amidst the surge in the development of Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs) and Graph Convolutional Neural Networks (GCNs) are advancing side-by-side in two different data structure spaces. CNNs excel at extracting visual features, while GCNs demonstrate powerful performance in processing graph-structured data, offering new research avenues for zero-trial learning. A common approach is to use CNNs and GCNs respectively to extract visual and semantic features and construct a mapping relationship for the zero-trial learning problem. However, current research still faces challenges. In zero-trial learning, the visible classes are used for model training, while the unseen classes are used for model testing. The distributions of visible and unseen classes are not identical, leading to distribution drift and consequently degrading model performance. Summary of the Invention

[0004] To address this, embodiments of this application provide a zero-trial learning method and apparatus based on semantic knowledge graph propagation, which solves the problems of distribution drift and information smoothing during model training and improves the performance of the model in zero-trial learning.

[0005] Firstly, this application provides a zero-trial learning method based on semantic knowledge graph propagation.

[0006] This application is achieved through the following technical solution:

[0007] A zero-trial learning method based on semantic knowledge graph propagation, the method comprising:

[0008] Obtain visible and unseen sample data from the ImageNet dataset;

[0009] Construct a visual feature extraction model: Based on the CNN model, modify the cost function of the CNN model, set the aggregation loss function of the CNN model based on the prediction output of the CNN model on unseen sample data, and train the CNN model using visible sample data and unseen sample data to obtain a trained CNN model;

[0010] Construct a semantic feature extraction model: Based on the GCN model, eliminate the nonlinear operation in the message transmission of the GCN model, set the optimization function of the GCN model, add the feature constraint term to the optimization function to obtain the CGCN model, and train the CGCN model in a self-supervised mode to obtain the trained CGCN model.

[0011] Constructing a feature mapping model: Based on the AE model, the matching loss function of the AE model is set according to the supervised training strategy based on the visible sample data, and the constrained loss function of the AE model is set according to the mapping restriction on the unseen sample data. The final loss function of the AE model is obtained based on the matching loss function and the constrained loss function. The AE model is trained based on the final loss function to obtain the trained AE model.

[0012] The trained CNN model, trained CGCN model, and trained AE model are used to perform class verification on the samples to be verified.

[0013] In a preferred embodiment of this application, the step of using the trained CNN model, the trained CGCN model, and the trained AE model to perform category verification on the samples to be verified includes:

[0014] The visual features of the sample to be verified are obtained by inputting the sample to be verified into the trained CNN model.

[0015] The visual features are input into the trained AE model to obtain the semantic features of the sample to be verified;

[0016] The semantic features of the unseen sample data are obtained by inputting the unseen sample data into the trained CGCN model, and the semantic features are then input into the AE model.

[0017] In the AE model, the distance between the semantic features of the sample to be verified and the semantic features of the unseen sample is calculated, and the category corresponding to the unseen sample with the smallest distance is selected as the category of the sample to be verified.

[0018] In a preferred example of this application, the cost function of the CNN model can be further configured as follows:

[0019]

[0020] Where γ is the trade-off parameter, It is the cross-entropy loss function. N is the deviation penalty term. s This represents the number of samples in the visible sample data. For the i-th image in the visible sample data, Represents the i-th image The corresponding label, N t This indicates the number of samples in the unseen sample data. It is the i-th image in the unseen sample data.

[0021] In a preferred example of this application, the aggregation loss function of the CNN model can be further defined as:

[0022]

[0023] Where, d bc d represents the distance between classes. wc Represents intra-class distance, c i c represents the distribution center of visual features belonging to the i-th category. j Let λ represent the distribution center of visual features belonging to the j-th category, λ be the weighting parameter, T represent the number of categories in the target dataset, and x represent the distribution center of visual features belonging to the j-th category. i Let i represent the i-th image among all images.

[0024] In a preferred example of this application, the optimization function of the GCN model can be further defined as follows:

[0025]

[0026] Among them, Z i Z represents the feature vector of node i. j This represents the feature vector of node j.

[0027] In a preferred example of this application, the final loss function of the AE model can be further configured as follows:

[0028]

[0029] Where φ is the trade-off parameter, L1 represents the matching loss function, and L2 represents the constrained loss function.

[0030] In a preferred example of this application, the matching loss function of the AE model can be further configured as follows:

[0031]

[0032] Where V represents visual features, S represents semantic features, and S gt S represents the output of the semantic knowledge graph CGCN. out V represents the intermediate embedding of the AE model. in V represents the visual feature input in the AE model. out This represents the visual feature output in the AE model.

[0033] In a preferred example of this application, the constrained loss function can be further configured as follows:

[0034]

[0035] in This represents the CGCN output of the i-th invisible class in the semantic KG. It is an intermediate embedding of invisible visual features in the AE model.

[0036] Based on the matching loss function and the constrained loss function, the final loss function of the AE model is as follows:

[0037]

[0038] in, These are the trade-off parameters; L1 represents the matching loss function, and L2 represents the constrained loss function.

[0039] Secondly, this application provides a zero-trial learning device based on semantic knowledge graph propagation.

[0040] This application is achieved through the following technical solution:

[0041] A zero-trial learning device based on semantic knowledge graph propagation, the device comprising:

[0042] The data acquisition module is used to acquire visible and unseen sample data from the ImageNet dataset.

[0043] The model building module is used to build visual feature extraction models, semantic feature extraction models, and feature mapping models.

[0044] The visual feature extraction model construction submodule is used to modify the cost function of the CNN model based on the CNN model, set the aggregation loss function of the CNN model based on the prediction output of the CNN model on unseen sample data, and train the CNN model using visible sample data and unseen sample data to obtain a trained CNN model.

[0045] The semantic feature extraction model construction submodule is based on the GCN model. It eliminates the nonlinear operation in the message transmission of the GCN model, sets the optimization function of the GCN model, adds feature constraint terms to the optimization function to obtain the CGCN model, and trains the CGCN model in a self-supervised mode to obtain the trained CGCN model.

[0046] The feature mapping model construction submodule is used to set the matching loss function of the AE model based on the supervised training strategy based on the visible sample data, set the constrained loss function of the AE model based on the mapping restriction of the unseen sample data, obtain the final loss function of the AE model based on the matching loss function and the constrained loss function, and train the AE model based on the final loss function to obtain the trained AE model.

[0047] The verification module is used to perform category verification on the samples to be verified using the trained CNN model, the trained CGCN model, and the trained AE model.

[0048] Thirdly, this application provides a computer device.

[0049] This application achieves its goal through the following technical solution:

[0050] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described zero-trial learning methods based on semantic knowledge graph propagation.

[0051] Fourthly, this application provides a computer-readable storage medium.

[0052] This application is achieved through the following technical solution:

[0053] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described zero-trial learning methods based on semantic knowledge graph propagation.

[0054] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:

[0055] Unseen sample data is added during CNN model training, and a modified loss function is used to enhance the distinguishability between seen and unseen sample data categories. The CGCN model is trained in a self-supervised manner, and an improved message transmission strategy is adopted to effectively alleviate the information smoothing problem caused by the increase in the number of layers during message transmission in the CGCN model. The dual mapping of visual features and semantic features is implemented in the AE model to further eliminate the distribution drift problem and improve the performance of the push-zero learning model. Attached Figure Description

[0056] Figure 1 A schematic diagram illustrating a zero-trial learning method based on semantic knowledge graph propagation, provided as an exemplary embodiment of this application;

[0057] Figure 2 This is a schematic diagram illustrating the process of performing category verification on a sample to be verified, which is another exemplary embodiment of this application. Detailed Implementation

[0058] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application.

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] Furthermore, the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this application, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0061] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0062] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0063] In one embodiment of this application, such as Figure 1 As shown, a zero-trial learning method based on semantic knowledge graph propagation is presented, and the main steps are described below:

[0064] S1: Obtain visible and unseen sample data from the ImageNet dataset.

[0065] Specifically, t-SNE technology is used to randomly select CNN feature distributions for 14 categories in the ImageNet dataset. The first 10 categories represent visible sample data, and are numbered sequentially from 0 to 9. The remaining 4 categories represent unseen sample data, and are numbered from 10 to 13 to distinguish between the different categories. Unlike traditional visual recognition problems, the target samples in Zero-Shot Learning (ZSL) are unlabeled, increasing the difficulty of feature extraction. When a CNN model trained only on visible sample data processes unseen sample data, the resulting feature distribution of the unseen sample data is relatively scattered and does not form obvious clusters, leading to unsatisfactory feature extraction results from the CNN model. In this embodiment, unseen sample data is added during model training to acquire prior knowledge and improve feature discrimination ability.

[0066] S2: Construct a visual feature extraction model: Based on the CNN model, modify the cost function of the CNN model, set the aggregation loss function of the CNN model based on the prediction output of the CNN model for unseen sample data, and train the CNN model using visible sample data and unseen sample data to obtain a trained CNN model.

[0067] It should be noted that this embodiment uses a Convolutional Neural Network (CNN) model to construct the visual feature extraction model. The CNN model mainly consists of three parts: convolutional layers, pooling layers, and fully connected layers. The CNN model extracts visual features by stacking multiple convolutional layers. Even though unseen sample data lacks supervised labels and its category information cannot be clearly defined, because the data has already been differentiated during acquisition, it is clear that unseen sample data does not belong to the visible sample data. Including unseen sample data during CNN model training can improve feature recognition capabilities.

[0068] Preferably, the cost function of the CNN model trained with visible sample data and its corresponding labels, as well as unseen sample data, can be modified as follows:

[0069]

[0070] Wherein, γ is a trade-off parameter, and in this embodiment, γ is 0.5. It is the cross-entropy loss function. N is the deviation penalty term. s This represents the number of samples in the visible sample data. For the i-th image in the visible sample data, Represents the i-th image The corresponding label, Nt This indicates the number of samples in the unseen sample data. It is the i-th image in the unseen sample data;

[0071] Deviation penalty is represented as follows:

[0072]

[0073] in, This represents the predicted probability of the class of the j-th unseen sample. (Bias penalty term) The settings can be used to mitigate the bias of unseen sample data on seen sample data. After modifying the cost function of the CNN model, the resulting feature distribution shows a significant difference in the feature distribution of seen sample data, while the feature distribution of unseen sample data is relatively concentrated and deviates from the distribution of seen sample data, thus solving the problem of the relatively scattered feature distribution of unseen sample data.

[0074] Furthermore, based on the CNN model's prediction output for unseen sample data, the aggregation loss function of the CNN model is modified to increase the distance between different categories and decrease the distance between the same category. This solves the problem of feature assimilation in unseen sample data, making the feature distribution of unseen sample data discriminative. The aggregation loss function of the CNN model is defined as follows:

[0075]

[0076] Where, d bc d represents the distance between classes. wc Represents intra-class distance, c i c represents the distribution center of visual features belonging to the i-th category. j Let λ represent the distribution center of visual features belonging to the j-th category, λ be the weighting parameter, T represent the number of categories in the target dataset, and x represent the distribution center of visual features belonging to the j-th category. i Let i represent the i-th image among all images.

[0077] Visible sample data and their corresponding labels, along with unseen sample data, are input into a CNN model for training. This process yields the visual features of the sample data. Once the model has trained to the point where its output reaches a preset accuracy, training is stopped, resulting in a trained CNN model. By adjusting the tradeoff parameter λ to regulate the distance between categories and utilizing the feature distribution center to calculate this distance, the distribution offset problem caused by the CNN model's prediction bias is significantly mitigated, improving the model's performance and making its output more accurate.

[0078] S3: Construct a semantic feature extraction model: Based on the GCN model, eliminate the nonlinear operation in the message transmission of the GCN model, set the optimization function of the GCN model, add feature constraint terms to the optimization function to obtain the CGCN model, and train the CGCN model in a self-supervised mode to obtain the trained CGCN model.

[0079] Specifically, this embodiment uses a Graph Convolutional Neural Network (GCN) model to construct a semantic feature extraction model, and uses the GCN model for semantic feature extraction. The categories in the visible and unseen sample data together constitute a huge graph structure network, where nodes represent category information and edges represent the relationships between categories. Message transmission in the GCN model includes aggregation and transformation processes, with network topology and nodes playing a major role in the message transmission process. In multi-layer GCN models, Laplace smoothing occurs during message transmission; when messages are transmitted to distant output layers, node features are assimilated. To improve the analysis speed of the GCN model, this embodiment eliminates non-linear operations in the GCN model's message transmission by setting an optimization function for the GCN model.

[0080] Preferably, the optimization function of the GCN model is expressed as:

[0081]

[0082] Among them, Z i Z represents the feature vector of node i. j This represents the feature vector of node j.

[0083] Specifically, information transfer in a GCN model with a spatial domain of k stacked layers can be represented as:

[0084]

[0085] Where F represents the node feature matrix and A represents the adjacency matrix. Let W represent the normalized adjacency matrix, where D is the degree matrix. (i) Let i = 1, 2, ..., K be a learnable parameter matrix, and Z be a parameter matrix. (i) ,i=1,2,…,K represents the output feature matrix of the i-th layer, and f(·) represents the nonlinear activation function, which can be the ReLU activation function.

[0086] After eliminating nonlinear operations, the following is achieved:

[0087]

[0088] Among them, W * =W (0) W (1) …W (K-1)Set Z (0) =FW (*) ,but:

[0089]

[0090] As K→∞, it can be further transformed into:

[0091] in To obtain the normalized symmetric positive semi-definition graph Laplace matrix, based on the partial derivative of the optimization function, we can obtain:

[0092]

[0093]

[0094] Based on the properties of the Tuples matrix, the optimization function is set as follows:

[0095]

[0096] Eliminating nonlinear operations in GCN model message transmission can simplify the analysis process during GCN model message transmission.

[0097] To further overcome the information smoothing problem in the message transmission process of the GCN model, a feature constraint term is added to the optimization function, such that:

[0098]

[0099] in, These are characteristic constraint terms, where H = FW represents the linear change of the characteristic matrix, and α represents the tradeoff coefficient, designed to control the impact of the constraint terms. More differentiated features are preserved by reducing the difference between H and Z. The partial derivative of the new optimization objective is then obtained as:

[0100]

[0101] Message transmission in the GCN model can be represented as:

[0102]

[0103] The model that adopts the above message propagation strategy is called the CGCN model.

[0104] Next, a self-supervised training model for the CGCN model is employed. Specifically, a similarity matrix SM is constructed based on the topological structure of the semantic knowledge graph (KG). The similarity matrix SM is a symmetric square matrix where each element represents the similarity of the semantic features of corresponding nodes. The values ​​of the diagonal elements in the similarity matrix SM are set to 1, while the values ​​of the non-diagonal elements are determined based on the connection distances between nodes. In this embodiment, the similarity of 2-hop neighborhoods is set to 0.5, the similarity of 3-hop neighborhoods is set to 0.2, and the similarity of other connections is ignored. By minimizing the similarity relationships between node features and the difference between the similarity matrix SM, the CGCN model is trained using a gradient forward propagation strategy. Training stops when the CGCN model reaches a preset accuracy, and the parameters of the CGCN model are fixed to obtain the trained CGCN model.

[0105] S4: Construct a feature mapping model: Based on the AE model, set the matching loss function of the AE model based on the supervised training strategy of the visible sample data, set the constrained loss function of the AE model based on the mapping restrictions of the unseen sample data, obtain the final loss function of the AE model based on the matching loss function and the constrained loss function, and train the AE model based on the final loss function to obtain the trained AE model.

[0106] Specifically, in the feature mapping and recognition stage, an AE model composed of multilayer perceptrons is used to construct a feature mapping model, and a transduction training strategy is applied to train the model. In the AE model, visual features are used as input, and semantic features are set as intermediate implicit representations.

[0107] Preferably, the matching loss function of the AE model is set as follows based on the supervised training strategy using visible sample data:

[0108] L1=min{||V in -V out || 2 +||S gt -S out || 2}

[0109] Where V represents visual features, S represents semantic features, and S gt S represents the output of the semantic knowledge graph CGCN. out V represents the intermediate embedding of the AE model. in V represents the visual feature input in the AE model. out This represents the visual feature output in the AE model. Using a matching loss function can improve the consistency between the input and output of the AE model and increase the accuracy of feature mapping.

[0110] The constrained loss function, derived from the mapping constraint on unseen sample data, is as follows:

[0111]

[0112] in This represents the CGCN output of the i-th invisible class in the semantic KG. It is an intermediate embedding of unseen visual features in the AE model. Adding additional mapping constraints to the unseen sample data can further reduce the problem of distribution shift.

[0113] Preferably, the final loss function of the AE model is:

[0114]

[0115] in, These are the tradeoff parameters. L1 represents the matching loss function, and L2 represents the constrained loss function. The AE model is trained using the final loss function as the objective function until it reaches the preset accuracy. Training then stops, and the trained AE model is obtained. It's important to note that the AE model is an autoencoder, a mapping function. Its input is visual features, its hidden layers contain semantic features, and its output is visual features. The goal is to make the output as consistent as possible with the input. The mapping process involves two steps: transforming the high-dimensional visual features into low-dimensional semantic features, and then restoring the low-dimensional semantic features back to high-dimensional visual features. This allows high-dimensional processing to be achieved through low-dimensional processing methods. The final loss function represents the difference between the input and output. Gradient descent is applied to the final loss function to update the AE model's network parameters. Once the AE model reaches the preset accuracy (i.e., the final loss function reaches the preset accuracy), the network parameters are fixed, resulting in the trained AE model.

[0116] S5: Use the trained CNN model, trained CGCN model, and trained AE model to perform class verification on the samples to be verified.

[0117] Preferred, such as Figure 2 As shown, step four consists of the following steps:

[0118] S51: Input the sample to be verified into the trained CNN model to obtain the visual features of the sample to be verified;

[0119] S52: Input the visual features into the trained AE model to obtain the semantic features of the sample to be verified;

[0120] S53: Input the unseen sample data into the trained CGCN model to obtain the semantic features of the unseen sample data, and input the semantic features into the AE model;

[0121] S54: In the AE model, the distance between the semantic features of the sample to be validated and the semantic features of unseen samples is calculated, and the category corresponding to the unseen sample with the smallest distance is selected as the category of the sample to be validated. Specifically, Euclidean distance can be used to calculate the distance between the semantic features of the sample to be validated and the semantic features of unseen samples. The formula for calculating Euclidean distance is: Where (x) i ,y i (x) represents the coordinates of the semantic features of the i-th category. j ,y j ) represents the coordinates of the semantic features of the j categories.

[0122] A zero-trial learning device based on semantic knowledge graph propagation, the device comprising:

[0123] The data acquisition module is used to acquire visible and unseen sample data from the ImageNet dataset.

[0124] The model building module is used to build visual feature extraction models, semantic feature extraction models, and feature mapping models.

[0125] The visual feature extraction model construction submodule is used to modify the cost function of the CNN model based on the CNN model, set the aggregation loss function of the CNN model based on the prediction output of the CNN model on unseen sample data, and train the CNN model using visible sample data and unseen sample data to obtain a trained CNN model.

[0126] The semantic feature extraction model construction submodule is based on the GCN model. It eliminates the non-linear operation in the message transmission of the GCN model, sets the optimization function of the GCN model, adds feature constraint terms to the optimization function to obtain the CGCN model, and trains the CGCN model in a self-supervised mode to obtain the trained CGCN model.

[0127] The feature mapping model construction submodule is used to set the matching loss function of the AE model based on the supervised training strategy based on the visible sample data, set the constrained loss function of the AE model based on the mapping constraints of the unseen sample data, obtain the final loss function of the AE model based on the matching loss function and the constrained loss function, and train the AE model based on the final loss function to obtain the trained AE model.

[0128] The validation module is used to perform class validation on the samples to be validated using the pre-trained CNN model, CGCN model, and AE model.

[0129] In one embodiment, a zero-trial learning device based on semantic knowledge graph propagation is provided, wherein the zero-trial learning device based on semantic knowledge graph propagation may be a server.

[0130] The zero-trial learning device for semantic knowledge graph propagation includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media contains an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs within the non-volatile storage media. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements any of the aforementioned zero-trial learning methods based on semantic knowledge graph propagation.

[0131] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described zero-trial learning methods based on semantic knowledge graph propagation.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink, DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system described in this application can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A zero-shot learning method based on semantic knowledge graph propagation, characterized in that, The method comprises: Obtain visible sample data and unseen sample data in an ImageNet dataset, wherein the visible sample data and the unseen sample data are image data; Construct a visual feature extraction model: based on a CNN model, modify the cost function of the CNN model, set the aggregation loss function of the CNN model based on the prediction output of the unseen sample data, input the visible sample data and the corresponding label and the unseen sample data into the CNN model for training and learning, and output the visual features of the visible sample data and the unseen sample data; when the CNN model is trained and learned to the output result reaching a preset accuracy, a trained CNN model is obtained; Construct a semantic feature extraction model: based on a GCN model, eliminate the nonlinear operation in the message transmission of the GCN model, set the optimization function of the GCN model, add a feature constraint term to the optimization function to obtain a CGCN model, train the CGCN model in a self-supervised mode to obtain a trained CGCN model, and the input data of the CGCN model is a graph structure network composed of categories of the visible sample data and the unseen sample data, wherein a node represents category information and an edge represents the mutual relationship between categories; the output data of the CGCN model is a semantic feature; Construct a feature mapping model: based on an AE model, set the matching loss function of the AE model based on a supervised training strategy of the visible sample data, set the constrained loss function of the AE model based on the mapping restriction of the unseen sample data, obtain the final loss function of the AE model based on the matching loss function and the constrained loss function, and train the AE model based on the final loss function to obtain a trained AE model; Use the trained CNN model, the trained CGCN model and the trained AE model to perform category verification on a to-be-verified sample, which comprises: Input the to-be-verified sample into the trained CNN model to obtain the visual feature of the to-be-verified sample; Input the visual feature into the trained AE model to obtain the semantic feature of the to-be-verified sample; Input the unseen sample data into the trained CGCN model to obtain the semantic feature of the unseen sample data, and input the semantic feature into the AE model; In the AE model, calculate the distance between the semantic feature of the to-be-verified sample and the semantic feature of the unseen sample, select the category corresponding to the unseen sample with the smallest distance as the category of the to-be-verified sample, the matching loss function L1 of the AE model is: L1 = min{||V in -V out || 2 +||S gt –S out || 2}, wherein V represents visual features, S represents semantic features, S gt CGCN output representing a semantic knowledge graph, S out intermediate embedding representing an AE model, V in visual feature input in an AE model, V out visual feature output in an AE model; The constrained loss function L2 is: wherein denotes the CGCN output for the i-th unseen class in the semantic KG, is the intermediate embedding of the unseen visual feature in the AE model. 2.The zero-shot learning method based on semantic knowledge graph propagation of claim 1, wherein, The cost function of the CNN model is: where γ is a trade-off parameter, is a cross-entropy loss function, is a bias penalty term, represents the predicted probability of the class of the i-th unseen sample, N s represents the number of samples in the seen sample data, is the i-th image in the seen sample data, represents the i-th image corresponding label, N t represents the number of samples in the unseen sample data, is the i-th image in the unseen sample data. 3.The zero-shot learning method based on semantic knowledge graph propagation of claim 1, wherein, The aggregation loss function of the CNN model is defined as: where d bc represents the inter-class distance, d wc represents the intra-class distance, c i represents the distribution center of the visual features belonging to the i-th class, λ is a trade-off parameter, T represents the number of classes in the target data set, x i represents the i-th image in all images. 4.The zero-shot learning method based on semantic knowledge graph propagation of claim 1, wherein, The optimization function of the GCN model is represented as: wherein Z i represents a feature vector of node i, Z j represents a feature vector of node j, the variable Z represents an output feature matrix, and the variable represents a normalized symmetric positive semi-definite graph Laplacian matrix, the variable N represents an output feature number, and the variable represents a normalized adjacency matrix. 5.The zero-shot learning method based on semantic knowledge graph propagation of claim 1, wherein, The final loss function of the AE model is: wherein, is a trade-off parameter, L1 denotes a matching loss function, and L2 denotes a constrained loss function.

6. A zero-shot learning device based on semantic knowledge graph propagation, characterized in that, The device is used for performing the semantic knowledge graph propagation-based zero-shot learning method of claim 1, and the device comprises: A data acquisition module is configured to obtain visible sample data and unseen sample data in an ImageNet dataset, wherein the visible sample data and the unseen sample data are image data; The model construction module is configured to construct a visual feature extraction model, construct a semantic feature extraction model, and construct a feature mapping model. The visual feature extraction model construction submodule is configured to modify a cost function of a CNN model based on the CNN model, set an aggregate loss function of the CNN model based on a prediction output of unseen sample data by the CNN model, input visible sample data and corresponding labels and the unseen sample data into the CNN model for training and learning, and output visual features of the visible sample data and the unseen sample data, until the CNN model is trained to a preset accuracy, and a trained CNN model is obtained. The semantic feature extraction model construction submodule is configured to eliminate nonlinear operations in message transmission of a GCN model based on the GCN model, set an optimization function of the GCN model, add a feature constraint term to the optimization function to obtain a CGCN model, train the CGCN model in a self-supervised mode, and obtain a trained CGCN model, wherein input data of the CGCN model is a graph structure network composed of categories of the visible sample data and the unseen sample data, a node represents category information, and an edge represents a mutual relationship between categories, and output data of the CGCN model is semantic features. The feature mapping model construction submodule is configured to set a matching loss function of an AE model based on the AE model based on a supervised training strategy of visible sample data, set a constrained loss function of the AE model based on a mapping constraint of unseen sample data, obtain a final loss function of the AE model based on the matching loss function and the constrained loss function, and train the AE model based on the final loss function to obtain a trained AE model. The verification module is configured to use the trained CNN model, the trained CGCN model, and the trained AE model to perform category verification on a to-be-verified sample, specifically configured to input the to-be-verified sample into the trained CNN model to obtain visual features of the to-be-verified sample, input the visual features into the trained AE model to obtain semantic features of the to-be-verified sample, input unseen sample data into the trained CGCN model to obtain semantic features of the unseen sample data, input the semantic features into the AE model, and calculate distances between the semantic features of the to-be-verified sample and the semantic features of the unseen sample in the AE model, and select a category corresponding to a minimum distance of the unseen sample as a category of the to-be-verified sample.

7. A computer device, characterized by The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images