Neural network training method and device based on subspace learning and electronic equipment

Abstract

The application provides a neural network training method and device based on subspace learning and an electronic device, and relates to the technical field of computers.The method comprises the following steps: obtaining a first fusion image corresponding to a first sample image, wherein the first fusion image is obtained by fusing the first sample image and a first visual prompt output by a visual prompt module; inputting the first fusion image into an image processing module to obtain a first output result; converting a first loss into an adjustment vector by using subspace learning; adjusting a vector in the visual prompt module according to the adjustment vector; iteratively performing the above steps; and obtaining a neural network composed of the image processing module and the adjusted visual prompt module when a first end condition is reached.The training method provided by the application can accelerate the convergence speed of the neural network and shorten the training time of the neural network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and more particularly, to a neural network training method, device and electronic equipment based on subspace learning in the technical field of computers. BACKGROUND

[0002] At present, in the field of image processing, there are more and more neural networks based on visual prompts. Such neural networks include a visual prompt module and an image processing module. The visual prompt module is used to generate visual prompts based on images, and the image processing module can process images based on the visual prompts to obtain processing results.

[0003] Such neural networks generally use the BlackVIP (Black-Box Visual Prompting) method for training. In the training process, a pre-trained image processing module and a visual prompt module are first obtained, and then the visual prompt module is trained using sample images. Since the image processing module and the visual prompt module are involved in the training process, and only the visual prompt module is adjusted in the training process, the image processing module remains unchanged, so the convergence speed in the training process is slow, and a long time is required. SUMMARY

[0004] The present application provides a neural network training method, device and electronic equipment based on subspace learning, which can shorten the training time of neural networks in the field of image processing.

[0005] In a first aspect, a neural network training method based on subspace learning is provided. The neural network includes a pre-trained image processing module and a visual prompt module. The method includes:

[0006] A first sample image is obtained, and a first fusion image corresponding to the first sample image is obtained. The first fusion image is fused from the first sample image and a first visual prompt, and the first visual prompt is generated by the visual prompt module based on the first sample image.

[0007] The first fusion image is input into the image processing module to obtain a first output result output by the image processing module.

[0008] Subspace learning is used to convert a first loss into an adjustment vector. The first loss is obtained according to the deviation between the image annotation of the first sample image and the first output result.

[0009] The vectors in the visual prompt module are adjusted according to the adjustment vector.

[0010] iterating the above steps, when a first end condition is reached, a neural network composed of the image processing module and the adjusted visual prompt module is obtained.

[0011] In the embodiments of the present application, in the training process of the neural network, a first fusion image corresponding to a first sample image is obtained, the first fusion image is obtained by fusing the first sample image and a first visual prompt output by the visual prompt module, the first fusion image is input into the image processing module to obtain a first output result, the first loss is converted into an adjustment vector by using subspace learning, the vector in the visual prompt module is adjusted according to the adjustment vector, the above steps are iterated, when a first end condition is reached, a neural network composed of the image processing module and the adjusted visual prompt module is obtained. In the model training process, the high-dimensional loss can be mapped into a low-dimensional subspace to obtain an adjustment vector by using subspace learning, and the visual prompt module in the neural network model is adjusted according to the adjustment vector. Subspace learning can significantly reduce the amount of data in the training process, thereby accelerating the convergence speed of the neural network and shortening the training time of the neural network.

[0012] Optionally, the visual prompt module includes a decoder; and the converting the first loss into an adjustment vector by using subspace learning includes: converting the first loss into a first adjustment vector by using a first subspace learning; and the adjusting the control vector in the visual prompt module according to the adjustment vector includes: adjusting the control vector in the decoder according to the first adjustment vector.

[0013] Optionally, the visual prompt module includes a trigger vector for adjusting the input of the decoder.

[0014] The converting the first loss into an adjustment vector by using subspace learning includes: converting the first loss into a second adjustment vector by using a second subspace learning.

[0015] The adjusting the vector in the visual prompt module according to the adjustment vector includes: adjusting the trigger vector according to the second adjustment vector.

[0016] Optionally, the obtaining the neural network composed of the image processing module and the adjusted visual prompt module when the first end condition is reached includes:

[0017] When the first end condition is reached, the visual prompt module is optimized to obtain a neural network composed of the image processing module and the optimized visual prompt module.

[0018] Optionally, the optimizing the visual prompt module includes:

[0019] obtaining a second sample image, and obtaining a second fusion image corresponding to the second sample image; wherein the second fusion image is obtained by fusing the second sample image and a second visual prompt, and the second visual prompt is generated by the visual prompt module based on the second sample image;

[0020] inputting the second fusion image into the image processing module to obtain a second output result output by the image processing module;

[0021] converting a second loss into an optimization vector by using the subspace obtained through the subspace learning; wherein the second loss is obtained according to a deviation between an image label of the second sample image and the second output result;

[0022] optimizing a vector in the visual prompt module according to the optimization vector;

[0023] iterating the above process until a second end condition is reached, and obtaining a neural network composed of the image processing module and the optimized visual prompt module.

[0024] Optionally, the converting the second loss into the optimization vector by using the subspace obtained through the subspace learning comprises:

[0025] extending a dimension of the subspace obtained through the subspace learning to obtain a target subspace;

[0026] converting the second loss into the optimization vector according to the target subspace.

[0027] Optionally, the visual prompt module comprises a decoder.

[0028] The converting the second loss into the optimization vector by using the subspace obtained through the subspace learning comprises converting the second loss into a first optimization vector by using the subspace obtained through the subspace learning.

[0029] The optimizing the vector in the visual prompt module according to the optimization vector comprises optimizing a control vector in the decoder according to the first optimization vector.

[0030] Optionally, the visual prompt module comprises a trigger vector for adjusting an input of the decoder.

[0031] The converting the second loss into the optimization vector by using the subspace obtained through the subspace learning comprises converting the second loss into a second optimization vector by using the subspace obtained through the subspace learning.

[0032] The optimizing the vector in the visual prompt module according to the optimization vector includes optimizing the trigger vector according to the second optimization vector.

[0033] In a second aspect, a neural network training apparatus based on subspace learning is provided, the neural network comprising a pre-trained image processing module and a visual prompt module, and the apparatus comprising:

[0034] An acquisition module is configured to acquire a first sample image and acquire a first fusion image corresponding to the first sample image, wherein the first fusion image is fused by the first sample image and a first visual prompt, and the first visual prompt is generated by the visual prompt module based on the first sample image.

[0035] An input module is configured to input the first fusion image into the image processing module to obtain a first output result output by the image processing module.

[0036] A conversion module is configured to convert a first loss into an adjustment vector by using subspace learning, wherein the first loss is obtained according to a deviation between an image label of the first sample image and the first output result.

[0037] An adjustment module is configured to adjust a vector in the visual prompt module according to the adjustment vector.

[0038] An iteration module is configured to iterate the above steps, and when a first end condition is reached, obtain a neural network composed of the image processing module and the adjusted visual prompt module.

[0039] In a third aspect, an electronic device is provided, comprising a memory configured to store executable program code, and a processor configured to call and run the executable program code from the memory, so that the electronic device executes the method in the first aspect or any possible implementation manner of the first aspect.

[0040] In a fourth aspect, an executable program code product is provided, which comprises executable program code, when the executable program code is run on a computer, so that the computer executes the method in the first aspect or any possible implementation manner of the first aspect.

[0041] In a fifth aspect, a readable storage medium is provided, which stores executable program code, when the executable program code is run on a computer, so that the computer executes the method in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1is a flowchart of a neural network training method based on subspace learning provided by an embodiment of the present application.

[0043] Figure 2 is a principle diagram of neural network training based on subspace learning provided by an embodiment of the present application.

[0044] Figure 3 is a flowchart of another neural network training method based on subspace learning provided by an embodiment of the present application.

[0045] Figure 4 is a principle diagram of neural network training based on subspace learning provided by an embodiment of the present application.

[0046] Figure 5 is a structural diagram of a neural network training device based on subspace learning provided by an embodiment of the present application.

[0047] Figure 6 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the present application will be described in detail below with reference to the drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B: "and / or" in the text is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone, in addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0049] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features.

[0050] In a neural network for image processing, the visual cue-based neural network includes a visual cue module and an image processing module. In the image processing process, the visual cue module can generate a visual cue based on the image to be processed, and the visual cue and the image to be processed are superimposed to obtain a fusion image (also called superimposed image). The image processing module can process the fusion image to obtain a processing result corresponding to the image to be processed.

[0051] The image processing module can be a neural network for image processing, such as a convolutional neural network, a recurrent neural network, and a cyclic neural network. The image processing module can perform image processing such as image enhancement, image generation, image detection, and image segmentation based on the input image. The visual cue module can be, for example, a Transformer model, and other neural networks that can generate visual cues based on images.

[0052] The neural network based on the visual cue is generally trained using the BlackVIP method. In the training process, a pre-trained image processing module and a visual cue module are obtained. The pre-trained image processing module and the visual cue module refer to neural networks that have been trained on a large-scale dataset, learned rich feature representations, and have strong generalization ability. The neural networks can be used for transfer learning or fine-tuning on different tasks and datasets to adapt to new tasks.

[0053] In the training process, the sample image is first input into the visual cue module. The visual cue module generates a visual cue based on the sample image. The visual cue is superimposed on the sample image to obtain a fusion image. The fusion image is input into the image processing module to obtain the output result of the image processing module. Then, the loss is calculated according to the deviation between the image annotation of the sample image and the output result. The parameters of the visual cue module are adjusted according to the loss, and the parameters of the image processing module remain unchanged. The above process is iterated to complete the training of the neural network. Since the image processing module and the visual cue module are involved in the training process, and only the visual cue module is adjusted in the training process, the image processing module remains unchanged, the convergence speed in the training process is slow, and a long time is required.

[0054] In addition, the pre-trained image processing module is generally installed in a server. In the training process, the application programming interface (API) of the image processing module needs to be frequently called to input the fusion image into the image processing module and obtain the output result of the image processing module, so that more network resources are consumed and the cost is high.

[0055] To solve the above technical problems, an embodiment of the present application provides a neural network training method based on subspace learning. The purpose of subspace learning is to learn a mapping matrix between adjustable parameters and low-dimensional representation parameters to reduce resource consumption.

[0056] Referring to Figure 1 , Figure 1This is a flowchart illustrating a neural network training method based on subspace learning provided in an embodiment of this application. The execution entity of this method can be a computer, such as... Figure 1 As shown, the method may include the following steps:

[0057] Step 101: Obtain the first sample image and the first fused image corresponding to the first sample image.

[0058] Step 102: Input the first fused image into the image processing module to obtain the first output result from the image processing module.

[0059] The first fused image is obtained by fusing the first sample image and the first visual cue, and the first visual cue is generated by the visual cue module based on the first sample image.

[0060] In this embodiment, a neural network can first be constructed and then trained. For example, a pre-trained visual cue module and image processing module can be obtained. These pre-trained modules have been trained on a large-scale dataset, learning rich feature representations and possessing strong generalization capabilities. Then, the visual cue module and image processing module are installed on a computer. Alternatively, only the pre-trained visual cue module can be obtained, installed on the computer, and then the computer and server can be connected. The neural network is obtained by combining the visual cue module installed on the computer and the pre-trained image processing module installed on the server.

[0061] See Figure 2 , Figure 2 This is a schematic diagram illustrating the principle of neural network training based on subspace learning, provided in an embodiment of this application. For example... Figure 2 As shown, the neural network includes a visual cue module 21 and an image processing module 22. The visual cue module 21, for example, is a Transformer model, including an encoder 211 and a decoder 212. During the operation of the neural network, image 31 is input to the visual cue module 21 (i.e., encoder 211). Encoder 211 encodes image 31 to obtain an encoded vector f. The encoded vector is concatenated with a trigger vector P and then input to decoder 212. Decoder 212 decodes the vector formed by concatenating the trigger vector and the encoded vector to generate visual cue 32. Visual cue 32 is superimposed on image 31 to obtain a fused image. The fused image is input to image processing module 22 to obtain the output result. For example, when image processing module is used for face recognition, the output result of image processing module 22 is the face recognition result; when image processing module is used for image generation, the output result of image processing module 22 is the newly generated image.

[0062] In the training process, first, a first training sample set can be obtained, the first training sample set including a plurality of sample images and image labels of each sample image. For example, when the neural network is used for image classification, the first training sample set includes a plurality of sample images of each category and a category label of each sample image, and the category label is the image label.

[0063] After starting the training, each sample image (i.e., a first sample image) in the first training sample set can be obtained in turn, the first sample image (e.g., image 31) is input into the visual prompt module 21 to obtain a first visual prompt (e.g., visual prompt 32) output by the visual prompt module 21, then the first visual prompt is superimposed on the first sample image to obtain a fusion image (i.e., a first fusion image), and the first fusion image is input into the image processing module 22 to obtain an output result (i.e., a first output result) output by the image processing module 22.

[0064] In step 103, the first loss is converted into an adjustment vector by using subspace learning.

[0065] The first loss is obtained according to the deviation between the image label of the first sample image and the first output result.

[0066] In this embodiment, after obtaining the first output result, the first loss can be calculated according to the deviation between the first output result and the image label of the first sample image, and then the first loss is converted into an adjustment vector by using subspace learning. As shown in Figure 2 L is a loss calculated based on the deviation between the output result of the image processing module and the image label of the sample image, and L represents the first loss in the adjustment process. For example, when the neural network is used for image classification, after obtaining the first output result (which is a category label), the deviation between the first output result and the category label of the first sample image can be determined first, then the deviation is input into a loss function, the first loss corresponding to the deviation is determined through the loss function, and then the first loss is converted into an adjustment vector by using subspace learning.

[0067] Optionally, when the visual prompt module includes a decoder, the adjustment vector can include a first adjustment vector, and the first adjustment vector is used to adjust a control vector (the control vector is a low-dimensional representation vector) in the decoder. Correspondingly, step 103 can include: converting the first loss into a first adjustment vector by using first subspace learning.

[0068] As shown in Figure 2 After obtaining the first output result, the first loss L can be calculated according to the first output result and the image label of the first sample image, and the first loss L includes a vector . for the first subspace learning process The corresponding projection matrix is ​​used to calculate the vector. With projection matrix The product of these two vectors yields the first adjustment vector. × This allows the first loss (i.e., vector loss) to be learned through the first subspace. Convert to the first adjustment vector × .vector For high-dimensional vectors, through the projection matrix Vectors can be Project it into a low-dimensional subspace.

[0069] Here, i is a positive integer, representing the i-th first sample image input during training. During training, for each first sample image, a random projection matrix is ​​obtained. Through the projection matrix The corresponding vector Projecting this vector into a low-dimensional subspace yields the first adjustment vector. × For each first sample image, a one-dimensional vector can be learned. After i learning iterations, the learned i vectors can be concatenated to obtain a subspace. A subspace can also be called a subspace vector.

[0070] Optionally, during the process of learning the first adjustment vector using the first subspace, the orthogonality of two adjacent projection matrices can be controlled. For example, when i is greater than 1, the projection matrix corresponding to the i-th sample image can be controlled to be orthogonal. The projection matrix corresponding to the (i-1)th sample image Orthogonal. This helps ensure the diversity of visual cues output by the visual cue module during training, preventing the neural network from getting trapped in local optima.

[0071] Optionally, when the visual cue module includes a trigger vector for adjusting the input to the decoder, the adjustment vector includes a second adjustment vector, which is used to adjust the trigger vector in the visual cue module. Correspondingly, step 103 may include: using second subspace learning to convert the first loss into the second adjustment vector.

[0072] like Figure 2 As shown, after obtaining the first output result, the first loss L can be calculated based on the first output result and the image annotation of the first sample image. The first loss L includes vectors. . For the projection matrix in the second subspace learning process, calculate the vector. the product of the projection matrix , the second adjustment vector × may be obtained. In this way, the first loss (i.e., the vector ) can be converted into the second adjustment vector × by the second subspace learning. The vector is a high-dimensional vector, and the vector may be projected into a low-dimensional subspace by the projection matrix .

[0073] where i is an integer greater than 0, representing the i-th input sample image in the training process. In the training process, for each first sample image, a corresponding projection matrix may be obtained by the second subspace learning, and a corresponding vector may be projected into a low-dimensional subspace by the projection matrix , to obtain a second adjustment vector × . In the first subspace learning process, for each first sample image, a vector may be learned, and i vectors learned in i times of learning may be spliced to obtain a subspace .

[0074] Optionally, in the process of obtaining the second adjustment vector by the second subspace learning method, the orthogonality of two adjacent projection matrices can be controlled. For example, when i is greater than 1, the projection matrix corresponding to the i-th sample image can be controlled to be orthogonal to the projection matrix corresponding to the i-1-th sample image. In this way, it is helpful to ensure the diversity of the visual cues output by the visual cue module in the training process, and to avoid the neural network from falling into a local optimal solution.

[0075] It should be noted that the first subspace learning and the second subspace learning are two independent subspace learning processes, the first subspace learning corresponds to the projection matrix , and the second subspace learning corresponds to the projection matrix .

[0076] Optionally, when the adjustment vector includes the first adjustment vector and the second adjustment vector, the first loss including the vector and the vector may be calculated by the following formula (1) and formula (2).

[0077] (1);

[0078] (2);

[0079] Where A = { , }, Z={ , }, ={ , }, For loss function, Image annotation for the first sample image. This is the output of the image processing module. This represents the deviation between the output of the image processing module and the image annotation.

[0080] Step 104: Adjust the vectors in the visual cue module according to the adjustment vectors.

[0081] In this embodiment, after obtaining the adjustment vector corresponding to each sample image, the vector in the visual cue module can be adjusted once by adjusting the adjustment vector to obtain the adjusted visual cue module.

[0082] Optionally, when the adjustment vector includes a first adjustment vector, the control vector in the decoder can be adjusted based on the first adjustment vector. For example... Figure 2 As shown, This is the initial control vector in the decoder. The initial control vector can be obtained from decoder 212 before training begins. For each first sample image, a corresponding first adjustment vector is obtained. × Then, the first adjustment vector can be adjusted. × The vector is added to the initial control vector W0, and the resulting vector is used as the new control vector. The new control vector replaces the control vector in the decoder, thus completing one adjustment of the decoder 212.

[0083] Optionally, when the adjustment vector includes a second adjustment vector, the trigger vector can be adjusted based on the second adjustment vector. For example... Figure 2 As shown, This is the initial trigger vector in the visual cue module 21 (the trigger vector is a low-dimensional representation vector). The initial trigger vector can be obtained from the visual cue module 21 before training begins. For each first sample image, a corresponding second adjustment vector is obtained. × After that, you can Second adjustment vector × and the initial trigger vector The addition is performed, the vector obtained by the addition is taken as a new trigger vector, and the new trigger vector is used to replace the trigger vector in the visual prompt module 21, so as to complete one adjustment of the trigger vector.

[0084] Step 105, iterate the above steps, and when a first end condition is reached, obtain a neural network composed of the image processing module and the adjusted visual prompt module.

[0085] In this embodiment, for each sample image, steps 101 to 104 can be repeated until a preset first end condition is reached. Exemplarily, the first end condition is that the sample images in the first training sample set are used up. Then, during the training process, when the ith acquired first sample image is the last sample image in the first training sample set, the training can be ended, and at this time, a neural network composed of the image processing module and the adjusted visual prompt module can be obtained.

[0086] As shown in Figure 2 , for the last sample image in the first training sample set, after the adjustment of the visual prompt module 21 is completed, the neural network composed of the visual prompt module 21 and the image processing module 22 , can be put into use.

[0087] It should be noted that in the adjustment process, the parameters in the encoder remain unchanged, and only the trigger vector and the control vector in the decoder are adjusted. Alternatively, only the control vector in the decoder can be adjusted, and the trigger vector is not adjusted.

[0088] Referring to Figure 3 , Figure 3 is a flowchart of another neural network training method based on subspace learning provided by the embodiment of the present application. The execution subject of the method can be a computer, as shown in Figure 3 , the method can include the following steps:

[0089] Step 301, acquiring a first sample image and acquiring a first fusion image corresponding to the first sample image.

[0090] Step 302, inputting the first fusion image into the image processing module to obtain a first output result output by the image processing module.

[0091] Step 303, converting the first loss into an adjustment vector by using subspace learning.

[0092] Step 304, adjusting the vector in the visual prompt module according to the adjustment vector.

[0093] Wherein, the understanding of steps 301-304 can refer to Figure 1 , which will not be repeated here.

[0094] Step 305, iterate the above steps, and end the adjustment when the first end condition is reached.

[0095] Step 306, obtaining a second sample image and obtaining a second fusion image corresponding to the second sample image.

[0096] Step 307, inputting the second fusion image into the image processing module to obtain the second output result output by the image processing module.

[0097] Wherein, the second fusion image is obtained by fusing the second sample image and the second visual prompt, and the second visual prompt is generated by the visual prompt module based on the second sample image.

[0098] In this embodiment, when the first end condition is reached, a second training sample set can be obtained, and the The visual prompt module is optimized. Similarly, in the optimization process, each second sample image can be obtained from the second training sample image in turn, the second sample image is input into the visual prompt module, the visual prompt module outputs the second visual prompt based on the second sample image, the second visual prompt is superimposed on the second sample image to obtain a fusion image, the fusion image is input into the image processing module to obtain the second output result output by the image processing module. The process of obtaining the second output result can refer to obtaining the first output result, which will not be repeated here.

[0099] Step 308, converting the second loss into an optimization vector through the subspace obtained by subspace learning.

[0100] Step 309, optimizing the vector in the visual prompt module according to the optimization vector.

[0101] Wherein, the second loss is obtained according to the deviation between the image annotation of the second sample image and the second output result.

[0102] In this embodiment, after obtaining the second output result, the second loss can be calculated according to the deviation between the second output result and the image annotation of the second sample image, then the second loss is converted into an optimization vector through the subspace obtained by subspace learning, and the vector in the visual prompt module is optimized according to the optimization vector.

[0103] Optionally, the second loss is converted into an optimization vector using the subspace learned through the subspace, including: expanding the dimension of the subspace learned through the subspace to obtain a target subspace; and converting the second loss into an optimization vector based on the target subspace. The expanded dimension can supplement information that may be lost during the visual cue generation process, thereby addressing the information deficiency problem that may arise from low-dimensional projection.

[0104] Optionally, when the visual cue module includes a decoder, the optimization vector includes a first optimization vector, which can optimize the control vector in the decoder. Step 308 includes: converting the second loss into the first optimization vector through the subspace learned from the subspace. Step 309 includes: optimizing the control vector in the decoder based on the first optimization vector.

[0105] In one implementation, the subspace can be expanded to obtain the target subspace, the second loss can be converted into a first optimization vector based on the target subspace, and the control vector of the decoder can be optimized based on the first optimization vector.

[0106] See Figure 4 , Figure 4 This is a schematic diagram illustrating the principle of neural network training based on subspace learning, provided in an embodiment of this application. For example... Figure 4 As shown, after obtaining the second output result, the second loss L can be calculated based on the second output result and the image annotations of the second sample image. The second loss L includes vectors. .

[0107] During the expansion of the subspace, a random orthogonal matrix can be introduced. orthogonal matrix with subspace Orthogonal, orthogonal matrix Bi subspace Calculate orthogonal matrices with multiple dimensions d. with subspace The product of these terms yields the expanded target subspace. Then, the target subspace can be calculated. with vector The product of these two vectors yields the first optimized vector. × This allows access through the target subspace. The second loss ( Convert to the first optimization vector × Then, the first optimization vector can be optimized. × Initial control vector of decoder 212 The new control vector is obtained by adding the two vectors together. The new control vector is then used to replace the original control vector in the decoder 212, thereby optimizing the decoder 212.

[0108] In another implementation, the subspace can be computed directly. with vector The product of these two vectors yields the first optimized vector. × This allows us to access subspaces. The second loss ( Convert to the first optimization vector × Then, the first optimization vector can be optimized. × Initial control vector of decoder 212 The new control vector is obtained by adding the two vectors together. The new control vector is then used to replace the original control vector in the decoder 212, thereby optimizing the decoder 212.

[0109] Optionally, when the visual cue module includes a trigger vector for adjusting the input of the decoder, step 308 includes: converting the second loss into a second optimized vector through the subspace learned from the subspace; step 309 includes: optimizing the trigger vector according to the second optimized vector.

[0110] like Figure 4 As shown, after obtaining the second output result, the second loss L can be calculated based on the second output result and the image annotations of the second sample image. The second loss L includes vectors. During the expansion of the subspace, a random orthogonal matrix can be introduced. orthogonal matrix with subspace Orthogonal, orthogonal matrix Bi subspace Calculate orthogonal matrices with multiple dimensions d. with subspace The product of these terms yields the expanded target subspace. Then, the target subspace can be calculated. with vector The product of these two vectors yields the second optimized vector. × This allows access through the target subspace. The second loss ( Transform into the second optimization vector × Then, the second optimization vector can be optimized. × Initial control vector of decoder 212 The addition of the second optimization vector and the initial control vector of the decoder 212 results in a new trigger vector, which replaces the original trigger vector, and thus the trigger vector is optimized once.

[0111] In another embodiment, the subspace can be directly calculated to obtain a first optimization vector . In this way, the second loss can be converted into a second optimization vector . After that, the second optimization vector can be added to the initial control vector of the decoder 212 to obtain a new trigger vector, which replaces the original trigger vector, and thus the trigger vector is optimized once.

[0112] In the optimization process, a zero-order optimization strategy can be used to minimize the loss function by estimating the gradient. For example, simultaneous perturbation stochastic approximation can be used to estimate the gradient. Specifically, the gradient can be estimated by formula (3) as follows.

[0113] (3)

[0114] wherein is the objective function value calculated on the training sample set. represents the low-dimensional latent parameter. is the perturbation amplitude, which is used for random perturbation around the current parameter value. is a random direction sampled from a standard normal distribution.

[0115] Step 310, iterate the above process, and when the second end condition is reached, obtain the neural network composed of the image processing module and the optimized visual prompt module.

[0116] In this embodiment, after the first end condition is reached, steps 306-309 can be iteratively executed until a preset second end condition is reached. Illustratively, the second end condition can be that the number of iterations reaches a preset number. The preset number is 1, at which time after obtaining a second sample image and completing the optimization of the visual prompt module once, the optimization can be ended.

[0117] At this time, the sample images in the verification sample set can also be used to verify the neural network composed of the visual prompt module 21 and the image processing module 22, and when the verification is passed, the neural network composed of the visual prompt module 21 and the image processing module 22 can be put into use.​​​

[0118] In the embodiment of the present application, in the training process of the neural network, a first fusion image corresponding to a first sample image is obtained, the first fusion image is obtained by fusing the first sample image and a first visual prompt output by the visual prompt module, the first fusion image is input into the image processing module to obtain a first output result, the first loss is converted into an adjustment vector by using subspace learning, the vector in the visual prompt module is adjusted according to the adjustment vector, the above steps are iterated, and when a first end condition is reached, a neural network composed of the image processing module and the adjusted visual prompt module is obtained. In the model training process, the high-dimensional loss can be mapped into a low-dimensional subspace to obtain an adjustment vector by using subspace learning, and the visual prompt module in the neural network model is adjusted according to the adjustment vector. Subspace learning can significantly reduce the amount of data in the training process, thereby accelerating the convergence speed of the neural network and shortening the training time of the neural network.

[0119] Referring to Table 1, the parameters shown in row A of Table 1 are related parameters when the neural network is trained by using the BlackVIP method, and the parameters shown in row B of Table 1 are related parameters when the neural network is trained by using the training method of the neural network provided in the present application. As can be known from Table 1, compared with the BlackVIP method, the Params parameter (0.15k) corresponding to the training method provided in the present embodiment is smaller than the Params parameter (9.92k) corresponding to the BlackVIP method, and therefore the training method provided in the present application can significantly improve the convergence speed in the training process and shorten the training time compared with the BlackVIP method.

[0120]

[0121] Table 1

[0122] Referring to Table 2, the parameters shown in row A of Table 2 are related parameters when the neural network is trained by using the BlackVIP method, and the parameters shown in row B of Table 1 are related parameters when the neural network is trained by using the training method of the neural network provided in the present application. As can be known from Table 1, compared with the BlackVIP method, the training method provided in the present embodiment corresponds to each parameter smaller than the corresponding parameter of the BlackVIP method, and therefore the training method provided in the present application can significantly improve the convergence speed in the training process and shorten the training time compared with the BlackVIP method.

[0123] Table 2

[0124] Figure 5 is a structural schematic diagram of a neural network training device based on subspace learning provided in the embodiment of the present application. As Figure 5As shown, the apparatus 500 can include:

[0125] The acquisition module 501 is configured to acquire a first sample image and acquire a first fusion image corresponding to the first sample image, wherein the first fusion image is obtained by fusing the first sample image and a first visual prompt, and the first visual prompt is generated by the visual prompt module based on the first sample image.

[0126] The input module 502 is configured to input the first fusion image into the image processing module to obtain a first output result output by the image processing module.

[0127] The conversion module 503 is configured to convert a first loss into an adjustment vector by using subspace learning, wherein the first loss is obtained according to a deviation between an image label of the first sample image and the first output result.

[0128] The adjustment module 504 is configured to adjust a vector in the visual prompt module according to the adjustment vector.

[0129] The iteration module 505 is configured to iterate the above steps, and obtain a neural network composed of the image processing module and the adjusted visual prompt module when a first end condition is reached.

[0130] Optionally, the visual prompt module includes a decoder, the conversion module 503 is specifically configured to convert the first loss into a first adjustment vector by using first subspace learning, and the adjustment module 504 is specifically configured to adjust a control vector in the decoder according to the first adjustment vector.

[0131] Optionally, the visual prompt module includes a trigger vector for adjusting an input of the decoder, the conversion module 503 is specifically configured to convert the first loss into a second adjustment vector by using second subspace learning, and the adjustment module 504 is specifically configured to adjust the trigger vector according to the second adjustment vector.

[0132] Optionally, the iteration module 505 is specifically configured to optimize the visual prompt module when the first end condition is reached, and obtain a neural network composed of the image processing module and the optimized visual prompt module.

[0133] Optionally, the iteration module 505 is specifically configured to acquire a second sample image and acquire a second fusion image corresponding to the second sample image, wherein the second fusion image is obtained by fusing the second sample image and a second visual prompt, and the second visual prompt is generated by the visual prompt module based on the second sample image.

[0134] The second fused image is input into the image processing module to obtain the second output result output by the image processing module; the second loss is converted into an optimization vector through the subspace learned by the subspace; wherein, the second loss is obtained based on the deviation between the image annotation of the second sample image and the second output result; the vector in the visual cue module is optimized according to the optimization vector; the above steps are iterated, and when the second termination condition is reached, a neural network composed of the image processing module and the optimized visual cue module is obtained.

[0135] Optionally, the iterative module 505 is specifically used to expand the dimension of the subspace learned through the subspace to obtain a target subspace; and to convert the second loss into the optimization vector according to the target subspace.

[0136] Optionally, the visual cue module includes a decoder; the iterative module 505 is specifically used to convert the second loss into a first optimization vector through the subspace learned from the subspace; and to optimize the control vector in the decoder according to the first optimization vector.

[0137] Optionally, the visual cue module includes a trigger vector for adjusting the input of the decoder; the iterative module 505 is specifically used to convert the second loss into a second optimization vector through the subspace learned from the subspace; and to optimize the trigger vector according to the second optimization vector.

[0138] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 6 As shown, the electronic device 600 includes hardware components such as a central processing unit 601, a memory 602, an image processor 603, a communication bus 604, a user interface 605, and a network interface 606. The memory 602 stores executable program code, and the central processing unit 601 is used to call and execute this executable program code to perform the subspace-based neural network training method described in the example above. The image processor 603 is used to display the GUI, and the communication bus 604 is used to establish communication connections between the electronic device and other devices.

[0139] This embodiment can divide the device into functional modules based on the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0140] It should be noted that all related content of each step involved in the above method embodiments can be referred to the function description of the corresponding function module, and will not be repeated here.

[0141] It should be understood that the device provided by the embodiment is used to execute the above-mentioned neural network training method, and therefore can achieve the same effect as the above-mentioned implementation method.

[0142] In the case of using integrated units, the device can include a processing module and a storage module. When the device is applied to an electronic device, the processing module can be used to control and manage the actions of the electronic device. The storage module can be used to support the electronic device to execute mutual program codes and the like.

[0143] The processing module can be a processor or a controller, which can realize or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure of the present application. The processor can also be a combination of computing functions, such as including one or more microprocessor combinations, combinations of digital signal processing (digital signal processing, DSP) and microprocessors, etc. The storage module can be a memory.

[0144] In addition, the device provided by the embodiment of the present application can be a chip, a component or a module, the chip can include a connected processor and a memory; wherein the memory is used to store instructions, when the processor calls and executes the instructions, the chip can execute the neural network training method provided by the above-mentioned embodiment.

[0145] The embodiment also provides a readable storage medium, the readable storage medium stores executable program codes, when the executable program codes run on the computer, the computer executes the above-mentioned related method steps to realize the neural network training method provided by the above-mentioned embodiment.

[0146] The embodiment also provides an executable program code product, when the executable program code product runs on the computer, the computer executes the above-mentioned related steps to realize the neural network training method provided by the above-mentioned embodiment.

[0147] The device, readable storage medium, executable program code product or chip provided by the embodiment are used to execute the corresponding method provided above, and therefore the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0148] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0149] In the embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules or units is only a logical function division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed ones can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0150] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A neural network training method based on subspace learning, characterized in that, The neural network comprises a pre-trained image processing module and a visual cue module, the visual cue module comprises a decoder, and the method comprises: obtaining a first sample image and a first fusion image corresponding to the first sample image; wherein the first fusion image is fused by the first sample image and a first visual cue, and the first visual cue is generated by the visual cue module based on the first sample image; inputting the first fusion image into the image processing module to obtain a first output result output by the image processing module; The first loss is converted into a first adjustment vector by using first subspace learning, wherein the first loss is obtained according to deviation between image annotation of the first sample image and the first output result, and the first loss includes a vector ; adjusting a control vector in the decoder according to the first adjustment vector, the control vector being a low-dimensional representation vector; iterating the above steps, and when a first end condition is reached, optimizing the visual cue module to obtain a neural network composed of the image processing module and the optimized visual cue module; The first loss is converted into a first adjustment vector by using first subspace learning, including: obtaining a random projection matrix for each first sample image; projecting the vector into a low-dimensional subspace by using the projection matrix to obtain the first adjustment vector; learning a dimension vector for each first sample image; and splicing i learned vectors to obtain a subspace; i represents the i-th first sample image. The first loss is converted into a first adjustment vector by using first subspace learning, including: obtaining a random projection matrix for each first sample image; projecting the vector into a low-dimensional subspace by using the projection matrix to obtain the first adjustment vector; learning a dimension vector for each first sample image; and splicing i learned vectors to obtain a subspace; i represents the i-th first sample image. wherein the optimization of the visual cue module to obtain the neural network composed of the image processing module and the optimized visual cue module comprises: obtaining a second sample image and a second fusion image corresponding to the second sample image; wherein the second fusion image is fused by the second sample image and a second visual cue, and the second visual cue is generated by the visual cue module based on the second sample image; inputting the second fusion image into the image processing module to obtain a second output result output by the image processing module; introducing a random orthogonal matrix, the orthogonal matrix being orthogonal to the subspace, and the orthogonal matrix being more than d dimensions than the subspace; calculating the product of the orthogonal matrix and the subspace to obtain a target subspace; calculating the product of the target subspace and the vector included in the second loss to obtain a first optimization vector; wherein the second loss is obtained according to the deviation between the image annotation of the second sample image and the second output result; optimizing the control vector according to the first optimization vector; iterating the above steps, and when a second end condition is reached, obtaining a neural network composed of the image processing module and the optimized visual cue module.

2. The method of claim 1, wherein, The visual cue module comprises a trigger vector for adjusting the input of the decoder, and the method further comprises: using a second subspace learning to convert the first loss into a second adjustment vector; adjusting the trigger vector according to the second adjustment vector.

3. The method of claim 1, wherein, The visual cue module comprises a trigger vector for adjusting the input of the decoder, and the method further comprises: converting a second loss into a second optimization vector through the target subspace; optimizing the trigger vector according to the second optimization vector.

4. A neural network training apparatus based on subspace learning, characterized by, The neural network comprises a pre-trained image processing module and a visual cue module, the visual cue module comprises a decoder, and the device comprises: The acquisition module is configured to acquire a first sample image and acquire a first fusion image corresponding to the first sample image, wherein the first fusion image is obtained by fusing the first sample image and a first visual prompt, and the first visual prompt is generated by the visual prompt module based on the first sample image. The input module is configured to input the first fusion image into the image processing module to obtain a first output result output by the image processing module. The conversion module is configured to convert a first loss into a first adjustment vector by using first subspace learning, wherein the first loss is obtained according to a deviation between an image label of the first sample image and the first output result, and the first loss includes a vector ; The adjustment module is configured to adjust a control vector in the decoder according to the first adjustment vector, and the control vector is a low-dimensional representation vector. The iteration module is configured to iterate the above steps, and when a first end condition is reached, optimize the visual prompt module to obtain a neural network composed of the image processing module and the optimized visual prompt module. The conversion module is specifically configured to obtain a random projection matrix for each first sample image; project the vector into a low-dimensional subspace through the projection matrix to obtain the first adjustment vector; learn an i-dimensional vector for each first sample image, and splice the learned i vectors to obtain a subspace; i represents the i-th first sample image. The conversion module is specifically configured to obtain a random projection matrix for each first sample image; project the vector into a low-dimensional subspace through the projection matrix to obtain the first adjustment vector; learn an i-dimensional vector for each first sample image, and splice the learned i vectors to obtain a subspace; i represents the i-th first sample image. The iteration module is configured to acquire a second sample image and acquire a second fusion image corresponding to the second sample image, wherein the second fusion image is obtained by fusing the second sample image and a second visual prompt, and the second visual prompt is generated by the visual prompt module based on the second sample image; input the second fusion image into the image processing module to obtain a second output result output by the image processing module; introduce a random orthogonal matrix, wherein the orthogonal matrix is orthogonal to the subspace, and the orthogonal matrix has d more dimensions than the subspace; calculate the product of the orthogonal matrix and the subspace to obtain a target subspace; calculate the product of the target subspace and a vector included in a second loss to obtain a first optimization vector, wherein the second loss is obtained according to the deviation between the image annotation of the second sample image and the second output result; optimize the control vector according to the first optimization vector; iterate the above steps, and when a second end condition is reached, obtain a neural network composed of the image processing module and the optimized visual prompt module.

5. An electronic device, comprising: The electronic device comprises: a memory for storing executable program code; a processor for calling and running the executable program code from the memory, so that the electronic device executes the method of any one of claims 1 to 3.

Images