Neural network model retrieval method based on knowledge consistency

Abstract

The invention discloses a neural network model retrieval method based on knowledge consistency, and the method comprises the steps: S1, generating a boundary sample for a neural network model through employing a black box adversarial sample generation method in an offline stage, and obtaining a semantic vector; s2, in the online stage, a semantic vector encoder of the query data set is used for generating semantic vectors for the query data set; and S3, in an online stage, performing knowledge consistency matching on the semantic vectors of the neural network models and the semantic vectors of the query data set, aligning knowledge implied in the respective semantic vectors, and screening out the most matched neural network model. According to the method, the accuracy of model retrieval is effectively improved, the risk of sensitive data leakage is reduced, and great benefits are brought to copyright protection of a neural network model, secure access of cloud services and the like.

Description

Technical Field

[0001] The invention belongs to the field of model retrieval, and in particular relates to a neural network model retrieval method based on knowledge consistency. Background Art

[0002] With the development of artificial intelligence technology, neural network models have been widely used in many fields. However, for ordinary users, training a neural network from scratch is usually laborious and may be limited by insufficient computing resources. Fortunately, online neural network model libraries allow users to directly use or fine-tune these models in a local environment.

[0003] Current model retrieval solutions attempt to convert neural network models into vectors to avoid the complex multiple inference processes required for the online model selection process. However, existing model vectorization processes are usually inaccurate because they do not accurately capture the correlation between the query dataset and the model. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a neural network model retrieval method based on knowledge consistency, applies the adversarial sample generation method to the retrieval task of the neural network model, and effectively improves the accuracy of model retrieval.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A neural network model retrieval method based on knowledge consistency, the method comprising the following steps:

[0007] Step S1: In the offline stage, a black-box adversarial sample generation method is used to generate boundary samples for the neural network model, and further obtain the semantic vector of the model;

[0008] Step S2: in the online stage, a semantic vector encoder of the query dataset is used to generate a semantic vector for the query dataset;

[0009] Step S3: In the online stage, the semantic vector of the neural network model and the semantic vector of the query data set are matched for knowledge consistency, and the best matching neural network model is further selected by aligning the knowledge implied by the two semantic vectors;

[0010] The beneficial effects of the present invention are:

[0011] The present invention applies the adversarial sample generation method to the retrieval task of the neural network model, uses the adversarial sample generation method to pre-acquire the semantic vector of the neural network model, obtains the semantic vector of the query data in the online stage, aligns the knowledge implicit in the vector for consistency, designs corresponding encoding and alignment frameworks, and uses the dot product calculation of the vector to perform model retrieval, thereby effectively improving the accuracy of model retrieval. From the perspective of data security, the present invention does not require any intermediate parameters of the neural network model, but only needs to access the interface of the neural network model, and can generate the semantic vector of the neural network model under black box conditions, which not only effectively reduces the risk of sensitive data leakage, but also is of great benefit to the copyright protection of the neural network model and the secure access to cloud services. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of boundary samples;

[0013] Figure 2 This is a schematic diagram of the neural network model retrieval method based on knowledge consistency of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0015] like Figure 2 As shown, a schematic diagram of a knowledge consistency neural network model retrieval method provided by the present invention includes the following steps:

[0016] Step S1: In the offline stage, a black-box adversarial sample generation method is used to generate boundary samples for the neural network model, and further obtain semantic vectors;

[0017] Step S2: In the online phase, the pre-trained query dataset encoder is used to generate a semantic vector for the query dataset;

[0018] Step S3: In the online stage, the semantic vector of the neural network model and the semantic vector of the query data set are matched for knowledge consistency, the knowledge implied in the two semantic vectors are aligned, and the best matching neural network model is selected.

[0019] In actual use, the neural network service platform usually stores multiple models with different semantics and structures in a unified server. The server needs to build a semantically rich query vector as a query index for the neural network model in the offline stage. In the online stage, it only needs to generate the corresponding semantic vector for the query data set through simple calculations. The server selects the most suitable neural network model through the matching relationship between the two vectors for subsequent direct use or fine-tuning the model after the user downloads it.

[0020] In one embodiment, the above step S1: in the offline stage, a black box adversarial sample generation method is used to generate boundary samples for the neural network model, and further obtain a semantic vector. Specifically includes:

[0021] Step S11: Use the existing public data set as the probe data set, access the neural network model through the black box interface, and generate a set of boundary samples for each neural network model:

[0022] ,

[0023] { },

[0024] in, Represents the neural network model library A neural network model, represents the probe dataset, is the boundary sample generation method, is the corresponding generated boundary sample set, which consists of subsets for each category. There are four categories: a, b, c, and d. are the boundary sample subsets for categories a, b, c, and d respectively. For example, for the boundary sample subset , which is defined as any class in the probe dataset The set of sample points and the boundary samples generated by them to other categories.

[0025] For neural network models , boundary samples are defined as sample points located near the decision boundary. They contain rich decision information of the neural network model. For example, the sample points near the decision boundary of category a and category b can be expressed as:

[0026] ,

[0027] For the 4-classification neural network model, Figure 1In the figure, the diagonal positions are sample points in each category, and the remaining positions are schematic diagrams of boundary samples from the category to which the horizontal axis belongs (the original category) to the category to which the vertical axis belongs (the target category). It can be clearly seen that the corresponding boundary samples already contain the most obvious features of the target category and the original category. According to an embodiment of the present invention, the method for generating boundary samples reuses the existing iteration-based boundary sample generation algorithm. The advantage of this is that the existing method can be directly used to generate when the server is idle, without any interaction with the query user.

[0028] Step S12: Boundary sample set The code is encoded into a semantic vector, and a well-designed two-layer attention mechanism neural network is used for knowledge extraction and encoding, as shown in the following formula:

[0029] ,

[0030] ,

[0031] ,

[0032] ,

[0033] ,

[0034] in, They are respectively the long short-term memory neural network model (LSTM) encoder based on the attention mechanism , , , The generated semantic vectors for categories a, b, c, d are Encoder for LSTM-based neural network model Further coding The semantic vector of the obtained neural network model, and The present invention models the category information as sequence information, and further introduces the LSTM mechanism as an encoder for encoding the semantic information of the neural network model contained in the boundary samples, thereby generating a semantic vector of the neural network model as a query index.

[0035] Similarly, the query image also needs to be converted into an index that can be used for semantic similarity calculation to support online retrieval. According to an embodiment of the present invention, the step S2: using the encoder of the query data set to generate a semantic vector for the query data set includes:

[0036] ,

[0037] ,

[0038] in, is a common semantic feature extractor in the prior art, x is a query sample, The estimated label is A sample of To query the data set, the category is The semantic vector of is an attention-based LSTM neural network model encoder for querying the dataset, which is used to further generate semantic vectors. is the semantic vector of the query dataset finally generated. The query user can use the public Pre-convert the query image set into a query semantic vector set { } to reduce the transmission delay in the online stage. In the online stage, the server further encodes to generate the semantic vector of the final query data set .

[0039] According to an embodiment of the present invention, step S3: using a carefully designed loss function to transform the semantic vector of the query data set Perform knowledge consistency matching with the semantic vectors of the neural network models stored in the neural network model library, align the knowledge implicit in the semantic vectors, calculate the dot product between the semantic vector of the query data set and the semantic vector of each neural network model, and select the result with the largest value as the most matching model:

[0040]

[0041] in, For the Neural network model The semantic vector of is the distance function, is the neural network model index with the best semantic matching. Used for calculation and In summary, the problem of calculating the semantic similarity between the neural network model and the query dataset can be directly converted into the problem of calculating the dot product of two vectors in the online stage.

[0042] In order to better verify the effect of the present invention on the model retrieval task, the present invention collected a total of 48 public pre-trained neural network model libraries and tested them on two commonly used data sets DTD and Aircraft. The experimental results are shown in Table 1. Obviously, whether it is the Pearson coefficient or the Sperman coefficient, the model transferability score predicted by the present invention on the two data sets and the actual test accuracy show a high correlation. This is because the present invention uses semantically rich knowledge vectors and precise matching algorithms, which fully proves the effectiveness of the method of the present invention.

[0043] Table 1

[0044]

[0045] In order to better verify the role of the LSTM-based model semantic vector generation method and the LSTM-based query data encoder designed by the present invention, the present invention designs corresponding ablation experiments, and the model retrieval accuracy (%) results are shown in Table 2. The LSTM encoding method is compared with the method of directly taking the average of all samples. It can be clearly seen that when the semantic vector encoder of the model and the semantic vector encoder of the query data are both LSTM, the data set retrieved by the model is the highest, which is 4.49% higher than the suboptimal result, which fully confirms the effectiveness and superiority of the LSTM-based knowledge vectorization method in the present invention.

[0046] Table 2

[0047]

[0048] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, and it should be clear that the present invention is not limited to the scope of the specific embodiments, for those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A knowledge consistency neural network model retrieval method, characterized in that: The method comprises the following steps: Step S1: In the offline stage, a black-box adversarial sample generation method is used to generate boundary samples for the neural network model and obtain semantic vectors; Step S2: in the online stage, a semantic vector encoder of the query dataset is used to generate a semantic vector for the query dataset; Step S3: In the online stage, the semantic vector of the neural network model and the semantic vector of the query data set are matched for knowledge consistency, the knowledge implicit in each semantic vector is aligned, and the best matching neural network model is selected.

2. A knowledge consistency neural network model retrieval method according to claim 1, characterized in that: The step S1 comprises: Step S11: Using the probe data set, access the neural network model library through the black box interface to generate a set of boundary samples for each neural network model: ， { }， in, Represents the neural network model library A neural network model, represents the probe dataset, is the boundary sample generation method, is the corresponding generated boundary sample set, There are four categories: a, b, c, and d. They are the boundary sample subsets for categories a, b, c, and d respectively; Step S12: Boundary sample set Encoded into semantic vectors, knowledge extraction and encoding are performed using a two-layer attention mechanism neural network: ， ， ， ， ， in, They are respectively the long short-term memory neural network model encoder based on the attention mechanism , , , The generated semantic vectors for categories a, b, c, d are Encoder for the long short-term memory neural network model based on the attention mechanism coding The resulting semantic vector of the neural network model, i.e. the query index, and are the neural network model parameters of all encoders.

3. The knowledge consistency neural network model retrieval method according to claim 1, characterized in that: The step S2 comprises: ， ， in, is a semantic feature extractor, x is a query sample, The estimated label is A sample of To query the data set, the category is The semantic vector of is an attention-based LSTM neural network model encoder for querying datasets. is the semantic vector of the query dataset finally generated.

4. The knowledge consistency neural network model retrieval method according to claim 1, characterized in that: The step S3 includes: performing dot product calculation on the semantic vector of the query data set and the semantic vector of each neural network model, and selecting the result with the largest value as the most matching model: ， in, For the Neural network model The semantic vector of is the distance function used to calculate and The dot product distance, is the neural network model index that best matches the semantics.

Images