Training method and device for neural network model
By structuring the class center vector of the neural network model and downsampling, a similarity matrix is generated, and used to retrain the model, the problem of high hardware requirements in large-scale training of neural network models is solved, and the hardware cost is reduced and the hardware cost is realized. Flexible online learning of models.
Patent Information
- Application Number
- CN202111430066.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-11-29
AI Technical Summary
In the large-scale training of neural network models, hardware requirements are too high, resulting in increased hardware costs.
By using input images to train the neural network model, extract class center vectors, perform structured calculations and downsampling, and generate a similarity matrix to retrain the neural network model to reduce hardware requirements.
It effectively reduces the hardware requirements for neural network model training, reduces hardware costs, and realizes flexible online learning of the model.
Smart Images

Figure CN114139704B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a training method and device for a neural network model. Background Art
[0002] The training of neural network models involves the calculation of class centers. Once the training scale is too large and there are too many class centers, the hardware requirements will also increase. For example, in the training of face recognition models, if there are 100 million categories, 100 million class centers are required. If the feature dimension of a face is 512, then the class center matrix dimension in this scenario is 100 million times 512. If it is trained directly on a GPU graphics card, the performance requirements for the graphics card are very high, and most graphics cards cannot meet this requirement.
[0003] In the process of implementing the concept of the present disclosure, the inventors discovered that there are at least the following technical problems in the related art: high requirements for hardware in large-scale training of neural network models. Summary of the invention
[0004] In view of this, the embodiments of the present disclosure provide a training method, device, electronic device and computer-readable storage medium for a neural network model to solve the problem of high hardware requirements in large-scale training of neural network models in the prior art.
[0005] According to a first aspect of an embodiment of the present disclosure, a training method for a neural network model is provided, including: training the neural network model using an input image; extracting a first-class center vector of each class in the input image using the trained neural network model; performing structured calculations on multiple first-class center vectors to obtain addressing matrices corresponding to multiple first-class center vectors; downsampling multiple first-class center vectors according to preset rules based on the addressing matrix to obtain multiple second-class center vectors; obtaining a batch of first samples, a number of first samples, from the input image, and extracting a first sample feature vector of each first sample using the trained neural network model; calculating the similarity between any two first sample feature vectors and the second-class center vectors to obtain a first similarity matrix corresponding to all first sample feature vectors and the second-class center vectors; and retraining the trained neural network model based on the first similarity matrix.
[0006] According to a second aspect of an embodiment of the present disclosure, a training device for a neural network model is provided, comprising: a first training module, configured to train the neural network model using an input image; a first extraction module, configured to extract a first-class center vector of each class in the input image using the trained neural network model; a first calculation module, configured to perform structured calculation on multiple first-class center vectors to obtain an addressing matrix corresponding to multiple first-class center vectors; a downsampling module, configured to downsample multiple first-class center vectors according to a preset rule based on the addressing matrix to obtain multiple second-class center vectors; a second extraction module, configured to obtain a batch of first samples from the input image, and extract a first sample feature vector of each first sample using the trained neural network model; a second calculation module, configured to calculate the similarity between any two first sample feature vectors and the second-class center vectors, and obtain a first similarity matrix corresponding to all first sample feature vectors and the second-class center vectors; and a second training module, configured to retrain the trained neural network model according to the first similarity matrix.
[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0009] Compared with the prior art, the disclosed embodiments have the following beneficial effects: training the neural network model by using the input image; extracting the first-class center vector of each class in the input image by using the trained neural network model; performing structured calculation on multiple first-class center vectors to obtain addressing matrices corresponding to multiple first-class center vectors; downsampling multiple first-class center vectors by preset rules according to the addressing matrix to obtain multiple second-class center vectors; obtaining batch samples of first samples from the input image, extracting the first sample feature vector of each first sample by using the trained neural network model; calculating the similarity between any two first sample feature vectors and the second-class center vectors, obtaining the first similarity matrix corresponding to all first sample feature vectors and the second-class center vectors; retraining the trained neural network model according to the first similarity matrix. The above-mentioned technical means can solve the problem of high hardware requirements in large-scale training of neural network models in the prior art, and thus the cost of model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0011] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present disclosure;
[0012] Figure 2 It is a flowchart of a training method of a neural network model provided by an embodiment of the present disclosure;
[0013] Figure 3 It is a structural schematic diagram of a training device for a neural network model provided by an embodiment of the present disclosure;
[0014] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0015] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.
[0016] A method and device for training a neural network model according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.
[0017] Figure 1 Schematic diagram of an application scenario of an embodiment of the present disclosure. The application scenario may include terminal devices 1, 2 and 3, a server 4 and a network 5.
[0018] Terminal devices 1, 2 and 3 can be hardware or software. When terminal devices 1, 2 and 3 are hardware, they can be various electronic devices with display screens and supporting communication with server 4, including but not limited to smart phones, tablet computers, laptop portable computers and desktop computers, etc.; when terminal devices 1, 2 and 3 are software, they can be installed in the above electronic devices. Terminal devices 1, 2 and 3 can be implemented as multiple software or software modules, or as a single software or software module, and the embodiments of the present disclosure are not limited to this. Furthermore, various applications can be installed on terminal devices 1, 2 and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.
[0019] The server 4 may be a server that provides various services, for example, a background server that receives a request sent by a terminal device that establishes a communication connection with the server, and the background server may receive and analyze the request sent by the terminal device, and generate a processing result. The server 4 may be a single server, or a server cluster composed of several servers, or a cloud computing service center, which is not limited in the embodiments of the present disclosure.
[0020] It should be noted that the server 4 can be hardware or software. When the server 4 is hardware, it can be various electronic devices that provide various services for the terminal devices 1, 2, and 3. When the server 4 is software, it can be multiple software or software modules that provide various services for the terminal devices 1, 2, and 3, or it can be a single software or software module that provides various services for the terminal devices 1, 2, and 3, and the embodiments of the present disclosure are not limited to this.
[0021] The network 5 can be a wired network connected by coaxial cable, twisted pair and optical fiber, or it can be a wireless network that can interconnect various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), infrared, etc., which is not limited in the embodiments of the present disclosure.
[0022] The user can establish a communication connection with the server 4 via the network 5 through the terminal devices 1, 2 and 3 to receive or send information, etc. It should be noted that the specific types, quantities and combinations of the terminal devices 1, 2 and 3, the server 4 and the network 5 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present disclosure are not limited to this.
[0023] Figure 2 It is a flowchart of a training method for a neural network model provided in an embodiment of the present disclosure. Figure 2 The training method of the neural network model can be Figure 1 The server executes. Figure 2 As shown, the training method of the neural network model includes:
[0024] S201, training a neural network model using an input image;
[0025] S202, extracting the first class center vector of each class in the input image using the trained neural network model;
[0026] S203, performing structured calculation on the multiple first-category center vectors to obtain addressing matrices corresponding to the multiple first-category center vectors;
[0027] S204, downsampling the plurality of first-category center vectors according to a preset rule according to the addressing matrix to obtain a plurality of second-category center vectors;
[0028] S205, obtaining a batch of first samples from the input image, and extracting a first sample feature vector of each first sample using a trained neural network model;
[0029] S206, calculating the similarity between any two first sample feature vectors and the second class center vectors, and obtaining a first similarity matrix corresponding to all first sample feature vectors and the second class center vectors;
[0030] S207: retrain the trained neural network model according to the first similarity matrix.
[0031] It should be noted that the neural network model can be trained using a part of the input image. For example, in face recognition, N face categories are defined, each face category has several pictures, and 100,000 face categories are randomly sampled from the N face categories to train the neural network model. A person's face photo is a class, a face category. The input image includes multiple classes, each class corresponds to a positive class center and multiple negative class centers, and each class has multiple samples. In face recognition, the class center corresponding to a class can be understood as the average value of the features of all pictures of a person, the positive class center corresponding to a class can be understood as the average value of the features of a person's pictures, and the negative class center corresponding to a class can be understood as the average value of the features of pictures of other people except this person. Calculate the similarity between any two first sample feature vectors and the second class center vector, that is, the similarity between any first sample feature vector and any second class center vector. The similarity can be cosine similarity, text similarity, etc.
[0032] According to the technical solution provided by the embodiment of the present disclosure, structured calculation is performed on multiple first-category center vectors to obtain addressing matrices corresponding to multiple first-category center vectors; based on the addressing matrix, multiple first-category center vectors are downsampled according to preset rules to obtain multiple second-category center vectors; a batch sample number of first samples is obtained from the input image, and a first sample feature vector of each first sample is extracted using a trained neural network model; the similarity between any two first sample feature vectors and the second-category center vectors is calculated to obtain a first similarity matrix corresponding to all first sample feature vectors and the second-category center vectors; based on the first similarity matrix, the trained neural network model is retrained. The above-mentioned technical means can solve the problem of high hardware requirements in large-scale training of neural network models in the prior art, and thus the cost of model training.
[0033] After executing step S204, according to the addressing matrix, multiple first-category center vectors are downsampled by preset rules to obtain multiple second-category center vectors, the method also includes: retraining the trained neural network model, including: determining the number of batch samples for retraining the neural network model each time, wherein the batch sample number is used to indicate the number of first samples obtained from the input image each time the neural network model is trained, and the retraining includes multiple batches of training; looping the following steps to retrain the neural network model: determining a batch sample number of first samples from the input image, and extracting a first sample feature vector of each first sample using the trained neural network model; calculating the similarity between any two first sample feature vectors and the second-category center vector, and obtaining a first similarity matrix corresponding to all first sample feature vectors and the second-category center vector; calculating the loss according to the first similarity matrix through the error function of the neural network model; retraining the neural network model according to the error, and updating the second-category center vector according to the error; in the retraining of the current batch, when the error is less than a preset threshold, ending the loop.
[0034] Batch processing is a technical term in model training and will not be explained in this disclosure. It should be noted that when determining the number of batch processing samples when training a neural network model, the maximum model dimension and hyperparameters of the neural network model can also be determined, such as the number of network layers, learning rate, learning rate decay, learning seed and other hyperparameters of the neural network model.
[0035] In an embodiment of the present disclosure, a batch of first samples is determined from an input image, wherein the batch of first samples may be a sample under one class, or may be multiple samples under one class, which is not limited here. The first similarity matrix contains the similarity between any first sample feature vector and any second class center vector. The error function may be a cross entropy loss function. Retraining the neural network model according to the error means that while updating the parameters of the neural network model, the second class center vector should also be updated according to the error. In the embodiment of the present disclosure, the second class center vector is updated in real time during the retraining of the neural network model, which is actually equivalent to realizing flexible online learning of the neural network model.
[0036] In step S207, the trained neural network model is retrained according to the first similarity matrix, including: according to the first similarity matrix, using the nearest neighbor algorithm to determine a first preset number of third-category center vectors for each first sample feature vector; according to the addressing matrix, using the nearest neighbor algorithm to determine a second preset number of fourth-category center vectors for each third-category center vector; calculating the similarity between any two first sample feature vectors and the second-category center vectors, and calculating the similarity between any two first sample feature vectors and the fourth-category center vectors to obtain a second similarity matrix, wherein the second similarity matrix includes the similarities between all first sample feature vectors and the second-category center vectors, and also includes the similarities between all first sample feature vectors and the fourth-category center vectors; according to the second similarity matrix, the trained neural network model is retrained.
[0037] Using the nearest neighbor algorithm to determine the first preset number of third-category center vectors for each first sample feature vector can be understood as selecting the first preset number of third-category center vectors that are most similar to the first sample feature vector from the second-category center vector according to the first similarity matrix. Using the nearest neighbor algorithm to determine the second preset number of fourth-category center vectors for each third-category center vector according to the addressing matrix can be understood as selecting the second preset number of third-category center vectors that are most similar to the third-category center vector from the first-category center vector according to the addressing matrix. For example: In the first similarity matrix (B, N1) similarity matrix, the 10 most similar negative-category centers (third-category center vectors) are found from the second-category center vector for each first sample feature vector, and then the top50 neighbors (fourth-category center vectors) of these 10 negative-category centers are found from the first-category center vector according to the addressing matrix. In this way, the number of our enlarged class centers is B*10*50. B is the number of the first sample feature vectors, N1 is the second class center vector, and the top50 neighbors of the 10 negative class centers represent the 50 most similar class centers to each negative class center.
[0038] According to the first similarity matrix, the trained neural network model is retrained, including: obtaining a preset multiple of second samples belonging to the same class as each first sample from the input image, and extracting a second sample feature vector of each second sample using the trained neural network model; weighted summing the preset multiple of second sample feature vectors corresponding to each first sample, averaging the weighted summing results, and using the averaged result as the local sample center vector; performing a replacement operation, including: using each local sample center vector to replace the class center vector corresponding to the first sample corresponding to each local sample center vector in a plurality of second class center vectors, and using each local sample center vector to replace the class center vector corresponding to the first sample corresponding to each local sample center vector in a plurality of fourth class center vectors; after performing the replacement operation, calculating the similarity between any two first sample feature vectors and the second class center vector, and calculating the similarity between any two first sample feature vectors and the fourth class center vector, to obtain a third similarity matrix, wherein the third similarity matrix includes the similarity between all first sample feature vectors and the second class center vector, and also includes the similarity between all first sample feature vectors and the fourth class center vector; according to the third similarity matrix, the trained neural network model is retrained.
[0039] In the mainstream methods of model training, class center learning and comparison learning are two systems. The disclosed embodiment integrates class center learning and comparison learning into model training. For example, when sampling B first samples, B*k second samples of corresponding categories are sampled in addition, and k (preset multiple) can be an integer greater than 0. For the convenience of understanding, the B first samples are regarded as sampling one sample from B different classes respectively, then sampling B*k second samples of corresponding categories is to sample k samples of corresponding categories from B different classes respectively. The k samples in each class are weighted and averaged, and the average result is used as the local sample center vector, so that the local sample center vectors corresponding to B different classes are obtained, because the B different classes belong to the B first samples respectively, therefore, the local sample center vectors corresponding to the B classes respectively can be regarded as the local sample center vectors corresponding to the B first samples respectively.
[0040] Use each local sample center vector to replace the class center vector corresponding to the first sample corresponding to each local sample center vector in multiple second class center vectors. For example, class A has a first sample B and k second samples C. The local sample center vector D corresponding to class A is calculated based on the k second samples C. The cluster of multiple second class center vectors of multiple classes is E. The second class center vector corresponding to class A in E is F. Use D to replace F in E.
[0041] In step S203, a structured calculation is performed on multiple first-category center vectors to obtain addressing matrices corresponding to multiple first-category center vectors, including: calculating the similarity between any two first-category center vectors to obtain a fourth similarity matrix; based on the fourth similarity matrix, using a nearest neighbor algorithm to determine a third preset number of fifth-category center vectors for each first-category center vector from all first-category center vectors; marking each fifth-category center vector to construct an addressing matrix corresponding to multiple first-category center vectors.
[0042] Marking each fifth-category center vector may be to note the position of each fifth-category center vector among all first-category center vectors. For example, the third preset number of fifth-category center vectors of a first-category center vector m are marked as (234, 4, 456, ...), where 245 represents that a fifth-category center vector of a first-category center vector m is the 245th class center vector among all first-category center vectors. By addressing the matrix, the neighboring vectors of each class center vector may be found, that is, the class center vectors similar to each class center vector. It should be noted that a fifth-category center vector n of a first-category center vector m is also a first-category center vector q, and the first-category center vector q belongs to the neighboring vectors of the first-category center vector m.
[0043] In step S204, according to the addressing matrix, multiple first-category center vectors are downsampled according to preset rules to obtain multiple second-category center vectors, including: calculating the similarity between any two first-category center vectors to obtain a fourth similarity matrix; according to the fourth similarity matrix, using a nearest neighbor algorithm to determine a third preset number of fifth-category center vectors for each first-category center vector from all first-category center vectors; according to preset rules, marking each first-category center vector and one of the fifth-category center vectors corresponding to each first-category center vector as true; and putting the class center vectors marked as true into a preset queue to obtain multiple second-category center vectors.
[0044] The preset rule includes: taking a preset position as a traversal starting point, and traversing the first type of central vector according to a preset interval.
[0045] Before traversal, the preset queue is empty. With the preset position as the traversal starting point, traverse the first-class center vector according to the preset interval, mark each first-class center vector and one of the fifth-class center vectors corresponding to each first-class center vector as true, mark the other class center vectors except the marked true class center vector in each first-class center vector and the fifth-class center vector corresponding to each first-class center vector as false, and put the marked true class center vector into the preset queue. For example, there are p fifth-class center vectors of the first-class center vector m (m is the sequence number of the class center vector), the preset position is m, and the preset interval is 3, then start traversal from the first-class center vector m, mark the first-class center vector m as true, add the first-class center vector m to the preset queue, and mark the p fifth-class center vectors similar to the first-class center vector m as false (vectors marked as false do not need to be added to the preset queue). After marking the first-class center vector m as true and marking the p fifth-class center vectors similar to the first-class center vector m as false, continue traversal until all the first-class center vectors are marked.
[0046] In an optional embodiment, a function PrototypeDownSample is constructed to complete the downsampling of multiple first-category center vectors:
[0047] For example, we can start from vector 20 and take 10,000 vectors at intervals of 3, which is: PrototypeDownSample(a=20, b=3, c=10000).
[0048] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.
[0049] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.
[0050] Figure 3 Schematic diagram of a training device for a neural network model provided by an embodiment of the present disclosure. Figure 3 As shown, the training device of the neural network model includes:
[0051] A first training module 301 is configured to train a neural network model using an input image;
[0052] A first extraction module 302 is configured to extract a first class center vector of each class in the input image using a trained neural network model;
[0053] A first calculation module 303 is configured to perform structured calculation on a plurality of first-type center vectors to obtain addressing matrices corresponding to the plurality of first-type center vectors;
[0054] A downsampling module 304 is configured to downsample the plurality of first-category center vectors according to a preset rule according to the addressing matrix to obtain a plurality of second-category center vectors;
[0055] The second extraction module 305 is configured to obtain a batch of first samples from the input image, and extract a first sample feature vector of each first sample using the trained neural network model;
[0056] The second calculation module 306 is configured to calculate the similarity between any two first sample feature vectors and the second type center vectors, and obtain a first similarity matrix corresponding to all first sample feature vectors and the second type center vectors;
[0057] The second training module 307 is configured to retrain the trained neural network model according to the first similarity matrix.
[0058] It should be noted that the neural network model can be trained using a part of the input image. For example, in face recognition, N face categories are defined, each face category has several pictures, and 100,000 face categories are randomly sampled from the N face categories to train the neural network model. A person's face photo is a class, a face category. The input image includes multiple classes, each class corresponds to a positive class center and multiple negative class centers, and each class has multiple samples. In face recognition, the class center corresponding to a class can be understood as the average value of the features of all pictures of a person, the positive class center corresponding to a class can be understood as the average value of the features of a person's pictures, and the negative class center corresponding to a class can be understood as the average value of the features of pictures of other people except this person. Calculate the similarity between any two first sample feature vectors and the second class center vector, that is, the similarity between any first sample feature vector and any second class center vector. The similarity can be cosine similarity, text similarity, etc.
[0059] According to the technical solution provided by the embodiment of the present disclosure, structured calculation is performed on multiple first-category center vectors to obtain addressing matrices corresponding to multiple first-category center vectors; based on the addressing matrix, multiple first-category center vectors are downsampled according to preset rules to obtain multiple second-category center vectors; a batch sample number of first samples is obtained from the input image, and a first sample feature vector of each first sample is extracted using a trained neural network model; the similarity between any two first sample feature vectors and the second-category center vectors is calculated to obtain a first similarity matrix corresponding to all first sample feature vectors and the second-category center vectors; based on the first similarity matrix, the trained neural network model is retrained. The above-mentioned technical means can solve the problem of high hardware requirements in large-scale training of neural network models in the prior art, and thus the cost of model training.
[0060] Optionally, the second training module 307 is also configured to retrain the trained neural network model, including: determining the number of batch samples for retraining the neural network model each time, wherein the number of batch samples is used to indicate the number of first samples obtained from the input image each time the neural network model is trained, and the retraining includes training of multiple batches; looping through the following steps to retrain the neural network model: determining a batch number of first samples from the input image, and extracting a first sample feature vector of each first sample using the trained neural network model; calculating the similarity between any two first sample feature vectors and the second type of center vector, and obtaining a first similarity matrix corresponding to all first sample feature vectors and the second type of center vector; calculating the loss according to the first similarity matrix through the error function of the neural network model; retraining the neural network model according to the error, and updating the second type of center vector according to the error; in the retraining of the current batch, when the error is less than a preset threshold, ending the loop.
[0061] Batch processing is a technical term in model training and will not be explained in this disclosure. It should be noted that when determining the number of batch processing samples when training a neural network model, the maximum model dimension and hyperparameters of the neural network model can also be determined, such as the number of network layers, learning rate, learning rate decay, learning seed and other hyperparameters of the neural network model.
[0062] In an embodiment of the present disclosure, a batch of first samples is determined from an input image, wherein the batch of first samples may be a sample under one class, or may be multiple samples under one class, which is not limited here. The first similarity matrix contains the similarity between any first sample feature vector and any second class center vector. The error function may be a cross entropy loss function. Retraining the neural network model according to the error means that while updating the parameters of the neural network model, the second class center vector should also be updated according to the error. In the embodiment of the present disclosure, the second class center vector is updated in real time during the retraining of the neural network model, which is actually equivalent to realizing flexible online learning of the neural network model.
[0063] Optionally, the second training module 307 is also configured to determine a first preset number of third-category center vectors for each first sample feature vector using a nearest neighbor algorithm based on the first similarity matrix; determine a second preset number of fourth-category center vectors for each third-category center vector using a nearest neighbor algorithm based on the addressing matrix; calculate the similarity between any two first sample feature vectors and the second-category center vectors, and calculate the similarity between any two first sample feature vectors and the fourth-category center vector to obtain a second similarity matrix, wherein the second similarity matrix includes the similarities between all first sample feature vectors and the second-category center vectors, and also includes the similarities between all first sample feature vectors and the fourth-category center vectors; and retrain the trained neural network model based on the second similarity matrix.
[0064] Using the nearest neighbor algorithm to determine the first preset number of third-category center vectors for each first sample feature vector can be understood as selecting the first preset number of third-category center vectors that are most similar to the first sample feature vector from the second-category center vector according to the first similarity matrix. Using the nearest neighbor algorithm to determine the second preset number of fourth-category center vectors for each third-category center vector according to the addressing matrix can be understood as selecting the second preset number of third-category center vectors that are most similar to the third-category center vector from the first-category center vector according to the addressing matrix. For example: In the first similarity matrix (B, N1) similarity matrix, the 10 most similar negative-category centers (third-category center vectors) are found from the second-category center vector for each first sample feature vector, and then the top50 neighbors (fourth-category center vectors) of these 10 negative-category centers are found from the first-category center vector according to the addressing matrix. In this way, the number of our enlarged class centers is B*10*50. B is the number of the first sample feature vectors, N1 is the second class center vector, and the top50 neighbors of the 10 negative class centers represent the 50 most similar class centers to each negative class center.
[0065] Optionally, the second training module 307 is further configured to obtain a preset multiple of second samples belonging to the same class as each first sample from the input image, and use the trained neural network model to extract the second sample feature vector of each second sample; weighted sum the preset multiple of second sample feature vectors corresponding to each first sample, average the weighted summation results, and use the averaged result as the local sample center vector; perform a replacement operation, including: using each local sample center vector to replace the class center vector corresponding to the first sample corresponding to each local sample center vector in multiple second-class center vectors, and using each local sample center vector to replace the class center vector corresponding to the first sample corresponding to each local sample center vector in multiple fourth-class center vectors; after performing the replacement operation, calculate the similarity between any two first sample feature vectors and the second-class center vectors, and calculate the similarity between any two first sample feature vectors and the fourth-class center vector, to obtain a third similarity matrix, wherein the third similarity matrix includes the similarity between all first sample feature vectors and the second-class center vectors, and also includes the similarity between all first sample feature vectors and the fourth-class center vectors; according to the third similarity matrix, retrain the trained neural network model.
[0066] In the mainstream methods of model training, class center learning and comparison learning are two systems. The disclosed embodiment integrates class center learning and comparison learning into model training. For example, when sampling B first samples, B*k second samples of corresponding categories are sampled in addition, and k (preset multiple) can be an integer greater than 0. For the convenience of understanding, the B first samples are regarded as sampling one sample from B different classes respectively, then sampling B*k second samples of corresponding categories is to sample k samples of corresponding categories from B different classes respectively. The k samples in each class are weighted and averaged, and the average result is used as the local sample center vector, so that the local sample center vectors corresponding to B different classes are obtained, because the B different classes belong to the B first samples respectively, therefore, the local sample center vectors corresponding to the B classes respectively can be regarded as the local sample center vectors corresponding to the B first samples respectively.
[0067] Use each local sample center vector to replace the class center vector corresponding to the first sample corresponding to each local sample center vector in multiple second class center vectors. For example, class A has a first sample B and k second samples C. The local sample center vector D corresponding to class A is calculated based on the k second samples C. The cluster of multiple second class center vectors of multiple classes is E. The second class center vector corresponding to class A in E is F. Use D to replace F in E.
[0068] Optionally, the first calculation module 303 is also configured to calculate the similarity between any two first-category center vectors to obtain a fourth similarity matrix; based on the fourth similarity matrix, use a nearest neighbor algorithm to determine a third preset number of fifth-category center vectors for each first-category center vector from all first-category center vectors; mark each fifth-category center vector to construct an addressing matrix corresponding to multiple first-category center vectors.
[0069] Marking each fifth-category center vector may be to note the position of each fifth-category center vector among all first-category center vectors. For example, the third preset number of fifth-category center vectors of a first-category center vector m are marked as (234, 4, 456, ...), where 245 represents that a fifth-category center vector of a first-category center vector m is the 245th class center vector among all first-category center vectors. By addressing the matrix, the neighboring vectors of each class center vector may be found, that is, the class center vectors similar to each class center vector. It should be noted that a fifth-category center vector n of a first-category center vector m is also a first-category center vector q, and the first-category center vector q belongs to the neighboring vectors of the first-category center vector m.
[0070] Optionally, the downsampling module 304 is also configured to calculate the similarity between any two first-class center vectors to obtain a fourth similarity matrix; based on the fourth similarity matrix, use a nearest neighbor algorithm to determine a third preset number of fifth-class center vectors for each first-class center vector from all first-class center vectors; through preset rules, mark each first-class center vector and one of the fifth-class center vectors corresponding to each first-class center vector as true; and put the class center vectors marked as true into a preset queue to obtain multiple second-class center vectors.
[0071] The preset rule includes: taking a preset position as a traversal starting point, and traversing the first type of central vector according to a preset interval.
[0072] Before traversal, the preset queue is empty. With the preset position as the traversal starting point, traverse the first-class center vector according to the preset interval, mark each first-class center vector and one of the fifth-class center vectors corresponding to each first-class center vector as true, mark the other class center vectors except the marked true class center vector in each first-class center vector and the fifth-class center vector corresponding to each first-class center vector as false, and put the marked true class center vector into the preset queue. For example, there are p fifth-class center vectors of the first-class center vector m (m is the sequence number of the class center vector), the preset position is m, and the preset interval is 3, then start traversal from the first-class center vector m, mark the first-class center vector m as true, add the first-class center vector m to the preset queue, and mark the p fifth-class center vectors similar to the first-class center vector m as false (vectors marked as false do not need to be added to the preset queue). After marking the first-class center vector m as true and marking the p fifth-class center vectors similar to the first-class center vector m as false, continue traversal until all the first-class center vectors are marked.
[0073] Optionally, the downsampling module 304 is further configured to construct a function PrototypeDownSample for completing downsampling of multiple first-category center vectors:
[0074] For example, we can start from vector 20 and take 10,000 vectors at intervals of 3, which is: PrototypeDownSample(a=20, b=3, c=10000).
[0075] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.
[0076] Figure 4 Schematic diagram of an electronic device 4 provided in an embodiment of the present disclosure. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 401 executes the computer program 403, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0077] Exemplarily, the computer program 403 may be divided into one or more modules / units, which are stored in the memory 402 and executed by the processor 401 to complete the present disclosure. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 403 in the electronic device 4.
[0078] The electronic device 4 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4 It is only an example of the electronic device 4 and does not constitute a limitation of the electronic device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.
[0079] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0080] The memory 402 may be an internal storage unit of the electronic device 4, for example, a hard disk or memory of the electronic device 4. The memory 402 may also be an external storage device of the electronic device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. Further, the memory 402 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device. The memory 402 may also be used to temporarily store data that has been output or is to be output.
[0081] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0082] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0083] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.
[0084] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.
[0085] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0086] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0087] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.
[0088] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.
Claims
1. A training method for a neural network model, characterized in that: include: Train the neural network model using the input image; Extracting the first class center vector of each class in the input image using the trained neural network model; Performing structured calculation on a plurality of the first-type center vectors to obtain addressing matrices corresponding to the plurality of the first-type center vectors; According to the addressing matrix, downsampling the plurality of first-category center vectors by a preset rule to obtain a plurality of second-category center vectors; Acquire a batch sample number of first samples from the input image, and extract a first sample feature vector of each first sample using the trained neural network model; Calculate the similarity between any two of the first sample feature vectors and the second class center vectors to obtain a first similarity matrix corresponding to all of the first sample feature vectors and the second class center vectors; Retraining the trained neural network model according to the first similarity matrix; Among them, the structured calculation is performed on multiple first-category center vectors to obtain addressing matrices corresponding to multiple first-category center vectors, including: calculating the similarity between any two first-category center vectors to obtain a fourth similarity matrix; according to the fourth similarity matrix, using a nearest neighbor algorithm to determine a third preset number of fifth-category center vectors for each first-category center vector from all the first-category center vectors; marking each of the fifth-category center vectors to construct the addressing matrix corresponding to multiple first-category center vectors.
2. The method according to claim 1, characterized in that: After downsampling the plurality of first-category center vectors according to the addressing matrix and obtaining the plurality of second-category center vectors according to a preset rule, the method further comprises: Retraining the trained neural network model includes: Determining the number of batch samples for retraining the neural network model each time, wherein the number of batch samples is used to indicate the number of first samples obtained from the input image each time the neural network model is trained, and the retraining includes training of multiple batches; The following steps are executed in a loop to retrain the neural network model: Determine the batch sample number of first samples from the input image, and extract a first sample feature vector of each of the first samples using the trained neural network model; Calculate the similarity between any two of the first sample feature vectors and the second class center vectors to obtain a first similarity matrix corresponding to all of the first sample feature vectors and the second class center vectors; Calculating the loss through the error function of the neural network model according to the first similarity matrix; Retraining the neural network model according to the error, and updating the second type center vector according to the error; In the retraining of the current batch, when the error is less than a preset threshold, the loop ends.
3. The method according to claim 1, characterized in that The step of retraining the trained neural network model according to the first similarity matrix includes: Determine a first preset number of third-category center vectors for each of the first sample feature vectors using a nearest neighbor algorithm according to the first similarity matrix; Determine a second preset number of fourth-category center vectors for each of the third-category center vectors using a nearest neighbor algorithm according to the addressing matrix; Calculate the similarity between any two of the first sample feature vectors and the second class center vector, and calculate the similarity between any two of the first sample feature vectors and the fourth class center vector, to obtain a second similarity matrix, wherein the second similarity matrix includes the similarities between all the first sample feature vectors and the second class center vector, and also includes the similarities between all the first sample feature vectors and the fourth class center vector; The trained neural network model is retrained according to the second similarity matrix.
4. The method according to claim 3, characterized in that The step of retraining the trained neural network model according to the first similarity matrix includes: Acquire a preset multiple of second samples belonging to the same class as each of the first samples from the input image, and extract a second sample feature vector of each of the second samples using the trained neural network model; Performing weighted summation on a preset multiple of the second sample feature vectors corresponding to each of the first samples, averaging the weighted summation results, and using the averaging result as the local sample center vector; Perform replacement operations, including: Using each of the local sample center vectors to replace the class center vector corresponding to the first sample corresponding to each of the local sample center vectors in the plurality of second class center vectors, and using each of the local sample center vectors to replace the class center vector corresponding to the first sample corresponding to each of the local sample center vectors in the plurality of fourth class center vectors; After performing the replacement operation, calculating the similarity between any two of the first sample feature vectors and the second type center vector, calculating the similarity between any two of the first sample feature vectors and the fourth type center vector, to obtain a third similarity matrix, wherein the third similarity matrix includes the similarities between all the first sample feature vectors and the second type center vector, and also includes the similarities between all the first sample feature vectors and the fourth type center vector; The retraining is performed on the trained neural network model according to the third similarity matrix.
5. The method according to claim 1, characterized in that The step of downsampling the plurality of first-category center vectors according to the addressing matrix and using a preset rule to obtain a plurality of second-category center vectors comprises: Calculate the similarity between any two of the first-category center vectors to obtain a fourth similarity matrix; According to the fourth similarity matrix, using a nearest neighbor algorithm, a third preset number of fifth-category center vectors are determined for each first-category center vector from all the first-category center vectors; According to a preset rule, marking each of the first-category central vectors and one of the fifth-category central vectors corresponding to each of the first-category central vectors as true; The class center vectors marked as true are placed into a preset queue to obtain a plurality of the second class center vectors.
6. The method according to claim 1, characterized in that The preset rules include: Taking the preset position as the traversal starting point, traverse the first type of center vectors according to the preset intervals.
7. A training device for a neural network model, characterized in that: include: A first training module is configured to train a neural network model using an input image; A first extraction module is configured to extract a first class center vector of each class in the input image using the trained neural network model; A first calculation module is configured to perform structured calculation on a plurality of the first-type center vectors to obtain addressing matrices corresponding to the plurality of the first-type center vectors; A downsampling module is configured to downsample the plurality of first-category center vectors according to the addressing matrix and a preset rule to obtain a plurality of second-category center vectors; A second extraction module is configured to obtain a batch sample number of first samples from the input image, and extract a first sample feature vector of each first sample using the trained neural network model; A second calculation module is configured to calculate the similarity between any two of the first sample feature vectors and the second class center vectors, and obtain a first similarity matrix corresponding to all the first sample feature vectors and the second class center vectors; A second training module is configured to retrain the trained neural network model according to the first similarity matrix; The first calculation module is also configured to calculate the similarity between any two of the first-category center vectors to obtain a fourth similarity matrix; based on the fourth similarity matrix, use a nearest neighbor algorithm to determine a third preset number of fifth-category center vectors for each of the first-category center vectors from all the first-category center vectors; mark each of the fifth-category center vectors to construct the addressing matrix corresponding to multiple first-category center vectors.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
