Neural network training method and device based on subspace learning and electronic equipment

Abstract

The invention provides a neural network training method and device based on subspace learning and electronic equipment, and relates to the technical field of computers, and the method comprises the steps: obtaining a first fusion image corresponding to a first sample image, and enabling the first fusion image to be obtained through the fusion of the first sample image and a first visual prompt output by a visual prompt module, and inputting the first fused image into an image processing module to obtain a first output result, converting the first loss into an adjustment vector by adopting subspace learning, adjusting a vector in a visual prompt module according to the adjustment vector, iterating the steps, and when a first end condition is met, outputting the first output result. And a neural network formed by combining the image processing module and the adjusted visual prompt module is obtained. According to the training method provided by the invention, the convergence speed of the neural network can be accelerated, and the training time of the neural network is shortened.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a neural network training method, device and electronic device based on subspace learning in the field of computer technology. Background Art

[0002] At present, in the field of image processing, there are more and more neural networks based on visual prompts. This type of neural network includes a visual prompt module and an image processing module. The visual prompt module is used to generate visual prompts based on images, and the image processing module can process images based on visual prompts to obtain processing results.

[0003] This type of neural network is generally trained using the BlackVIP (Black-Box Visual Prompting) method. During the training process, the pre-trained image processing module and visual prompting module are first obtained, and then the visual prompting module is trained using sample images. Since the training process involves the image processing module and the visual prompting module, and only the visual prompting module is adjusted during the training process, while the image processing module remains unchanged, the convergence speed during the training process is slow and takes a long time. Summary of the invention

[0004] The present application provides a neural network training method, device and electronic device based on subspace learning, which can shorten the training time of neural networks in the field of image processing.

[0005] In a first aspect, a neural network training method based on subspace learning is provided, wherein the neural network includes a pre-trained image processing module and a visual prompt module, and the method includes:

[0006] Acquire a first sample image, and acquire a first fused image corresponding to the first sample image; wherein the first fused image is obtained by fusing the first sample image and a first visual cue, and the first visual cue is generated by the visual cue module based on the first sample image;

[0007] Inputting the first fused image into the image processing module to obtain a first output result output by the image processing module;

[0008] Using subspace learning, converting a first loss into an adjustment vector; wherein the first loss is obtained according to a deviation between an image annotation of the first sample image and the first output result;

[0009] Adjusting the vector in the visual cue module according to the adjustment vector;

[0010] Iterate the above steps, and when the first end condition is reached, obtain a neural network composed of the image processing module and the adjusted visual prompt module.

[0011] In an embodiment of the present application, during the training process of the neural network, a first fused image corresponding to the first sample image is obtained, the first fused image is obtained by fusing the first sample image and the first visual cue output by the visual cue module, the first fused image is input into the image processing module to obtain a first output result, the first loss is converted into an adjustment vector using subspace learning, the vector in the visual cue module is adjusted according to the adjustment vector, the above steps are iterated, and when the first end condition is reached, a neural network composed of the image processing module and the adjusted visual cue module is obtained. During the model training process, the high-dimensional loss can be mapped to a low-dimensional subspace through subspace learning to obtain an adjustment vector, and the visual cue module in the neural network model can be adjusted according to the adjustment vector. Subspace learning can significantly reduce the amount of data in the training process, thereby accelerating the convergence speed of the neural network and shortening the training time of the neural network.

[0012] Optionally, the visual cue module includes a decoder; the adopting subspace learning to convert the first loss into an adjustment vector includes: adopting first subspace learning to convert the first loss into a first adjustment vector; the adjusting the control vector in the visual cue module according to the adjustment vector includes: adjusting the control vector in the decoder according to the first adjustment vector.

[0013] Optionally, the visual cue module includes a trigger vector for adjusting the input of the decoder;

[0014] The adopting subspace learning to convert the first loss into an adjustment vector includes: adopting second subspace learning to convert the first loss into a second adjustment vector;

[0015] The adjusting the vector in the visual prompt module according to the adjustment vector includes: adjusting the trigger vector according to the second adjustment vector.

[0016] Optionally, when the first end condition is reached, a neural network composed of the image processing module and the adjusted visual prompt module is obtained, comprising:

[0017] When the first end condition is reached, the visual prompt module is optimized to obtain a neural network composed of the image processing module and the optimized visual prompt module.

[0018] Optionally, the optimizing the visual prompt module includes:

[0019] Acquire a second sample image, and acquire a second fused image corresponding to the second sample image; wherein the second fused image is obtained by fusing the second sample image and a second visual cue, and the second visual cue is generated by the visual cue module based on the second sample image;

[0020] Inputting the second fused image into the image processing module to obtain a second output result output by the image processing module;

[0021] The second loss is converted into an optimization vector through the subspace obtained by the subspace learning; wherein the second loss is obtained according to the deviation between the image annotation of the second sample image and the second output result;

[0022] Optimizing the vector in the visual cue module according to the optimization vector;

[0023] The above process is iterated, and when the second end condition is reached, a neural network composed of the image processing module and the optimized visual prompt module is obtained.

[0024] Optionally, the subspace obtained by the subspace learning, converting the second loss into an optimization vector, comprises:

[0025] Expanding the dimension of the subspace obtained through the subspace learning to obtain a target subspace;

[0026] The second loss is converted into the optimization vector according to the target subspace.

[0027] Optionally, the visual cue module includes a decoder;

[0028] The subspace obtained through the subspace learning, converting the second loss into an optimization vector, comprises: converting the second loss into a first optimization vector through the subspace obtained through the subspace learning;

[0029] Optimizing the vector in the visual prompt module according to the optimization vector includes: optimizing the control vector in the decoder according to the first optimization vector.

[0030] Optionally, the visual cue module includes a trigger vector for adjusting the input of the decoder;

[0031] The subspace obtained through the subspace learning, converting the second loss into an optimization vector, comprises: converting the second loss into a second optimization vector through the subspace obtained through the subspace learning;

[0032] The optimizing the vector in the visual prompt module according to the optimization vector includes: optimizing the trigger vector according to the second optimization vector.

[0033] In a second aspect, a neural network training device based on subspace learning is provided, wherein the neural network includes a pre-trained image processing module and a visual prompt module, and the device includes:

[0034] an acquisition module, configured to acquire a first sample image and a first fused image corresponding to the first sample image; wherein the first fused image is obtained by fusing the first sample image and a first visual cue, and the first visual cue is generated by the visual cue module based on the first sample image;

[0035] An input module, used for inputting the first fused image into the image processing module to obtain a first output result output by the image processing module;

[0036] A conversion module, configured to convert a first loss into an adjustment vector by using subspace learning; wherein the first loss is obtained according to a deviation between an image annotation of the first sample image and the first output result;

[0037] An adjustment module, used for adjusting the vector in the visual prompt module according to the adjustment vector;

[0038] An iterative module is used to iterate the above steps and obtain a neural network composed of the image processing module and the adjusted visual prompt module when the first end condition is reached.

[0039] In a third aspect, an electronic device is provided, comprising a memory for storing executable program code; and a processor for calling and running the executable program code from the memory, so that the electronic device executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0040] In a fourth aspect, an executable program code product is provided, which includes: an executable program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.

[0041] In a fifth aspect, a readable storage medium is provided, which stores an executable program code. When the executable program code runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1It is a flowchart of a neural network training method based on subspace learning provided in an embodiment of the present application;

[0043] Figure 2 It is a schematic diagram of the principle of neural network training based on subspace learning provided in an embodiment of the present application;

[0044] Figure 3 It is a flowchart of another neural network training method based on subspace learning provided in an embodiment of the present application;

[0045] Figure 4 It is a schematic diagram of the principle of neural network training based on subspace learning provided in an embodiment of the present application;

[0046] Figure 5 It is a structural schematic diagram of a neural network training device based on subspace learning provided in an embodiment of the present application;

[0047] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0049] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0050] In a neural network for image processing, a neural network based on visual cues includes a visual cue module and an image processing module. During the image processing process, the visual cue module can generate a visual cue based on the image to be processed, and the visual cue is superimposed on the image to be processed to obtain a fused image (also called a superimposed image), and the image processing module can process the fused image to obtain a processing result corresponding to the image to be processed.

[0051] The image processing module may be a neural network for image processing such as a convolutional neural network, a recursive neural network, or a cyclic neural network, and the image processing module may perform image processing such as image enhancement, image generation, image detection, and image segmentation based on the input image. The visual cue module may be a Transformer model, or other neural networks that can generate visual cues based on images.

[0052] Neural networks based on visual cues are generally trained using the BlackVIP method. During the training process, the pre-trained image processing module and visual cue module are first obtained. The pre-trained image processing module and visual cue module refer to neural networks that have been trained on large-scale data sets, have learned rich feature representations, and have strong generalization capabilities. Transfer learning (Transfer Learning) or fine-tuning (Fine-Tuning) can be performed on different tasks and data sets to adapt to new tasks.

[0053] During the training process, the sample image is first input into the visual cue module. The visual cue module generates visual cues based on the sample image, and the visual cues are superimposed on the sample image to obtain a fused image. The fused image is input into the image processing module to obtain the output result of the image processing module. Then, the loss is calculated based on the deviation between the image annotation of the sample image and the output result, and the parameters of the visual cue module are adjusted based on the loss. The parameters of the image processing module remain unchanged, and the above process is iterated to complete the training of the neural network. Since the image processing module and the visual cue module are involved in the training process, and only the visual cue module is adjusted during the training process, and the image processing module remains unchanged, the convergence speed during the training process is slow and takes a long time.

[0054] Moreover, the pre-trained image processing module is generally installed in the server. During the training process, it is necessary to frequently call the application programming interface (API) of the image processing module to input the fused image to the image processing module and obtain the output results of the image processing module. Therefore, it consumes more network resources and has a high cost.

[0055] In order to solve the above technical problems, the embodiment of the present application provides a neural network training method based on subspace learning, wherein the purpose of subspace learning is to learn the mapping matrix between adjustable parameters and their low-dimensional representation parameters to reduce resource consumption.

[0056] See also Figure 1 , Figure 1is a flowchart of a neural network training method based on subspace learning provided in an embodiment of the present application. The execution subject of the method may be a computer, such as Figure 1 As shown, the method may include the following steps:

[0057] Step 101: Acquire a first sample image, and acquire a first fused image corresponding to the first sample image.

[0058] Step 102: Input the first fused image into an image processing module to obtain a first output result output by the image processing module.

[0059] The first fused image is obtained by fusing the first sample image and the first visual cue, and the first visual cue is generated by the visual cue module based on the first sample image.

[0060] In this embodiment, a neural network can be constructed first, and then the neural network can be trained. For example, a pre-trained visual cue module and an image processing module can be obtained first. The pre-trained image processing module and the visual cue module have been trained on a large-scale data set, and have learned a rich feature representation and have a strong generalization ability. Then, the visual cue module and the image processing module are installed in the computer. Alternatively, only the pre-trained visual cue module can be obtained, the visual cue module is installed in the computer, and then the computer and the server are connected, and the neural network is obtained by combining the visual cue module installed in the computer and the pre-trained image processing module installed in the server.

[0061] See also Figure 2 , Figure 2 Schematic diagram of the principle of neural network training based on subspace learning provided by the embodiment of the present application. Figure 2 As shown, the neural network includes a visual cue module 21 and an image processing module 22. The visual cue module 21 is, for example, a Transformer model, including an encoder 211 and a decoder 212. During the operation of the neural network, the image 31 is input into the visual cue module 21 (i.e., the encoder 211). The encoder 211 encodes the image 31 to obtain a coding vector f, which is spliced ​​with the trigger vector P (trigger vector) and then input into the decoder 212. The decoder 212 decodes the vector spliced ​​with the trigger vector and the coding vector to generate a visual cue 32. The visual cue 32 is superimposed on the image 31 to obtain a fused image, and the fused image is input into the image processing module 22 to obtain an output result. For example, when the image processing module is used for face recognition, the output result of the image processing module 22 is the face recognition result; when the image processing module is used for image generation, the output result of the image processing module 22 is a newly generated image.

[0062] During the training process, a first training sample set may be obtained, which includes a plurality of sample images and an image annotation of each sample image. For example, when the neural network is used for image classification, the first training sample set includes a plurality of sample images of various categories and a category label of each sample image, and the category label is the image annotation.

[0063] After starting training, each sample image (i.e., the first sample image) can be obtained in turn from the first training sample set, and the first sample image (e.g., image 31) can be input into the visual cue module 21 to obtain the first visual cue (e.g., visual cue 32) output by the visual cue module 21, and then the first visual cue can be superimposed on the first sample image to obtain a fused image (i.e., the first fused image), and the first fused image can be input into the image processing module 22 to obtain the output result (i.e., the first output result) output by the image processing module 22.

[0064] Step 103: Use subspace learning to convert the first loss into an adjustment vector.

[0065] The first loss is obtained according to the deviation between the image annotation of the first sample image and the first output result.

[0066] In this embodiment, after obtaining the first output result, the loss can be calculated based on the deviation between the first output result and the image annotation of the first sample image to obtain the first loss, and then the subspace learning method is used to convert the first loss into an adjustment vector. Figure 2 As shown, L is the loss calculated based on the deviation between the output result of the image processing module and the image annotation of the sample image, and L represents the first loss in the adjustment process. For example, when the neural network is used for image classification, after obtaining the first output result (which is a category label), the deviation between the first output result and the category label of the first sample image can be determined first, and then the deviation is input into the loss function, and the first loss corresponding to the deviation is determined by the loss function, and then the first loss is converted into an adjustment vector by subspace learning.

[0067] Optionally, when the visual cue module includes a decoder, the adjustment vector may include a first adjustment vector, and the first adjustment vector is used to adjust a control vector in the decoder (the control vector is a low-dimensional representation vector). Correspondingly, step 103 may include: using a first subspace learning to convert the first loss into a first adjustment vector.

[0068] like Figure 2 As shown, after obtaining the first output result, the first loss L can be calculated based on the first output result and the image annotation of the first sample image. The first loss L includes the vector Z W . A wi The first subspace learning process and the vector ZW The corresponding projection matrix calculates the vector Z W With the projection matrix A wi The product of can get the first adjustment vector A wi ×Z W , so that the first loss (i.e., vector Z) can be learned through the first subspace W ) is converted into the first adjustment vector A wi ×Z W . Vector Z W For a high-dimensional vector, through the projection matrix A wi The vector Z can be W Projected into a lower dimensional subspace.

[0069] Where i is an integer greater than 0, representing the i-th first sample image input during the training process. During the training process, for each first sample image, a random projection matrix A is obtained wi , through the projection matrix A wi The corresponding vector Z W Projected into a low-dimensional subspace, the first adjustment vector A is obtained wi ×Z W For each first sample image, a vector W of one dimension can be learned i , after i times of learning, the i vectors learned can be concatenated to obtain the subspace A subspace can also be called a subspace vector.

[0070] Optionally, in the process of acquiring the first adjustment vector by learning the first subspace, two adjacent projection matrices may be controlled to be orthogonal. For example, when i is greater than 1, the projection matrix A corresponding to the i-th sample image may be controlled to be wi The projection matrix A corresponding to the i-1th sample image w(i-1) This helps to ensure the diversity of visual cues output by the visual cue module during training and prevent the neural network from falling into a local optimal solution.

[0071] Optionally, when the visual cue module includes a trigger vector for adjusting the input of the decoder, the adjustment vector includes a second adjustment vector, and the second adjustment vector is used to adjust the trigger vector in the visual cue module. Correspondingly, step 103 may include: using the second subspace learning to convert the first loss into the second adjustment vector.

[0072] like Figure 2 As shown, after obtaining the first output result, the first loss L can be calculated based on the first output result and the image annotation of the first sample image. The first loss L includes the vector Z p . A piFor the projection matrix in the second subspace learning process, calculate the vector Z p With the projection matrix A pi The product of can get the second adjustment vector A pi ×Z p . In this way, the first loss (i.e., vector Z) can be learned through the second subspace p ) is converted into the second adjustment vector A pi ×Z p . Vector Z p For a high-dimensional vector, through the projection matrix A pi The vector Z can be pi Projected into a lower dimensional subspace.

[0073] Wherein, i is an integer greater than 0, representing the i-th sample image input during the training process. During the training process, for each first sample image, the second subspace learning can be used to obtain a corresponding projection matrix A pi , through the projection matrix A pi The corresponding vector Z p Projected into a low-dimensional subspace, the second adjustment vector A is obtained pi ×Z p In the first subspace learning process, for each first sample image, a vector P of one dimension can be learned i , after i times of learning, the i vectors learned can be concatenated to obtain the subspace

[0074] Optionally, in the process of obtaining the second adjustment vector using the second subspace learning method, two adjacent projection matrices may be controlled to be orthogonal. For example, when i is greater than 1, the projection matrix A corresponding to the i-th sample image may be controlled to be pi The projection matrix A corresponding to the i-1th sample image p(i-1) This helps to ensure the diversity of visual cues output by the visual cue module during training and prevent the neural network from falling into a local optimal solution.

[0075] It should be noted that the first subspace learning and the second subspace learning are two independent subspace learning processes. The first subspace learning corresponds to the projection matrix A wi , the second subspace learns the corresponding projection matrix A pi .

[0076] Optionally, when the adjustment vector includes a first adjustment vector and a second adjustment vector, the vector Z can be calculated by using the following formulas (1) and (2): W and vector Z p The first loss.

[0077] I = A × Z + a0 (1);

[0078]

[0079] Among them, A={A pi , A wi}, Z={Z p , Z W}, a0={P0,W0}, is the loss function, is the image annotation of the first sample image, y is the output result of the image processing module, P θ is the deviation between the output of the image processing module and the image annotation.

[0080] Step 104: Adjust the vector in the visual prompt module according to the adjustment vector.

[0081] In this embodiment, after the adjustment vector corresponding to each sample image is obtained, the vector in the visual prompt module may be adjusted once by the adjustment vector to obtain an adjusted visual prompt module.

[0082] Optionally, when the adjustment vector includes a first adjustment vector, the control vector in the decoder may be adjusted according to the first adjustment vector. Figure 2 As shown, W0 is the initial control vector in the decoder. Before starting training, the initial control vector W0 can be obtained from the decoder 212. wi ×Z W Afterwards, the first adjustment vector A can be wi ×Z W The control vector W0 is added to the initial control vector W0, and the vector obtained by the addition is used as a new control vector. The new control vector is used to replace the control vector in the decoder to complete an adjustment of the decoder 212.

[0083] Optionally, when the adjustment vector includes a second adjustment vector, the trigger vector may be adjusted according to the second adjustment vector. Figure 2 As shown, P0 is the initial trigger vector in the visual prompt module 21 (the trigger vector is a low-dimensional representation vector). Before starting training, the initial trigger vector P0 can be obtained from the visual prompt module 21. pi ×Z p Afterwards, the second adjustment vector A can be pi ×Z pThe initial trigger vector P0 is added, and the vector obtained by the addition is used as a new trigger vector. The new trigger vector is used to replace the trigger vector in the visual prompt module 21 to complete an adjustment of the trigger vector.

[0084] Step 105, iterate the above steps, and when the first end condition is reached, obtain a neural network composed of the image processing module and the adjusted visual prompt module.

[0085] In this embodiment, for each sample image, steps 101 to 104 may be repeated until a preset first end condition is reached. Exemplarily, the first end condition is that the sample images in the first training sample set are used up. In the training process, when the i-th first sample image obtained is the last sample image in the first training sample set, the training may be terminated, and at this time, a neural network composed of the image processing module and the adjusted visual cue module may be obtained.

[0086] like Figure 2 As shown, for the last sample image in the first training sample set, after completing the adjustment of the visual cue module 21, the neural network composed of the visual cue module 21 and the image processing module 22 is verified. After the verification is passed, the neural network composed of the visual cue module 21 and the image processing module 22 can be put into use.

[0087] The first end condition may also be other end conditions, which is not limited in this embodiment. It should be noted that during the adjustment process, the parameters in the encoder remain unchanged, and only the trigger vector and the control vector in the decoder are adjusted. Alternatively, only the control vector in the decoder may be adjusted without adjusting the trigger vector.

[0088] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of another neural network training method based on subspace learning provided in an embodiment of the present application. The execution subject of the method may be a computer, such as Figure 3 As shown, the method may include the following steps:

[0089] Step 301: Acquire a first sample image, and acquire a first fused image corresponding to the first sample image.

[0090] Step 302: Input the first fused image into an image processing module to obtain a first output result output by the image processing module.

[0091] Step 303: Use subspace learning to convert the first loss into an adjustment vector.

[0092] Step 304: Adjust the vector in the visual prompt module according to the adjustment vector.

[0093] For understanding of steps 301 to 304, please refer to Figure 1 , this embodiment will not be described in detail here.

[0094] Step 305: Iterate the above steps and end the adjustment when the first end condition is reached.

[0095] Step 306: Acquire a second sample image, and acquire a second fused image corresponding to the second sample image.

[0096] Step 307: input the second fused image into the image processing module to obtain a second output result output by the image processing module.

[0097] The second fused image is obtained by fusing the second sample image and the second visual cue, and the second visual cue is generated by the visual cue module based on the second sample image.

[0098] In this embodiment, when the first end condition is reached, a second training sample set can be obtained, and the visual prompt module can be optimized through the second sample image in the training sample set. Similarly, during the optimization process, each second sample image can be obtained from the second training sample image in turn, and the second sample image can be input into the visual prompt module. The visual prompt module outputs a second visual prompt based on the second sample image. The second visual prompt is superimposed on the second sample image to obtain a fused image. The fused image is input into the image processing module to obtain a second output result output by the image processing module. The process of obtaining the second output result can refer to obtaining the first output result, and this embodiment will not be repeated here.

[0099] Step 308: Convert the second loss into an optimization vector through the subspace obtained by subspace learning.

[0100] Step 309: Optimize the vector in the visual prompt module according to the optimized vector.

[0101] The second loss is obtained according to the deviation between the image annotation of the second sample image and the second output result.

[0102] In this embodiment, after obtaining the second output result, the second loss can be obtained by calculating the loss based on the deviation between the second output result and the image annotation of the second sample image, and then the second loss is converted into an optimization vector through the subspace obtained by subspace learning, and the vector in the visual cue module is optimized according to the optimization vector.

[0103] Optionally, the subspace obtained by subspace learning converts the second loss into an optimization vector, including: expanding the dimension of the subspace obtained by subspace learning to obtain a target subspace; and converting the second loss into an optimization vector according to the target subspace. The expanded dimension can supplement the information that may be lost in the process of generating visual cues, thereby solving the problem of insufficient information that may be caused by low-dimensional projection.

[0104] Optionally, when the visual cue module includes a decoder, the optimization vector includes a first optimization vector, and the first optimization vector can optimize the control vector in the decoder. Step 308 includes: converting the second loss into the first optimization vector through the subspace obtained by subspace learning. Step 309 includes: optimizing the control vector in the decoder according to the first optimization vector.

[0105] In one implementation, the subspace may be expanded first to obtain a target subspace, the second loss may be converted into a first optimization vector according to the target subspace, and the control vector of the decoder may be optimized according to the first optimization vector.

[0106] See also Figure 4 , Figure 4 Schematic diagram of the principle of neural network training based on subspace learning provided by the embodiment of the present application. Figure 4 As shown, after obtaining the second output result, the second loss L can be calculated based on the second output result and the image annotation of the second sample image. The second loss L includes the vector Z W .

[0107] In the process of expanding the subspace, a random orthogonal matrix can be introduced Orthogonal Matrix With subspace Orthogonal, orthogonal matrix Subspace Multiple d dimensions, calculate the orthogonal matrix With subspace The product of can get the expanded target subspace Afterwards, the target subspace can be calculated With vector Z W The product of , we get the first optimization vector In this way, we can pass through the target subspace The second loss (Z W ) is converted to the first optimization vector Afterwards, the first optimization vector A new control vector is obtained by adding it to the initial control vector W0 of the decoder 212 , and the new control vector is used to replace the original control vector in the decoder 212 , thereby achieving an optimization of the encoder 212 .

[0108] In another embodiment, the subspace can be directly calculated With vector Z W The product of gets the first optimization vector This can be done through the subspace The second loss (Z W ) is converted to the first optimization vector Afterwards, the first optimization vector A new control vector is obtained by adding it to the initial control vector W0 of the decoder 212 , and the new control vector is used to replace the original control vector in the decoder 212 , thereby achieving an optimization of the encoder 212 .

[0109] Optionally, when the visual cue module includes a trigger vector for adjusting the input of the decoder, step 308 includes: converting the second loss into a second optimization vector through the subspace obtained by subspace learning; step 309 includes: optimizing the trigger vector according to the second optimization vector.

[0110] like Figure 4 As shown, after obtaining the second output result, the second loss L can be calculated based on the second output result and the image annotation of the second sample image. The second loss L includes the vector Z p In the process of expanding the subspace, a random orthogonal matrix can be introduced Orthogonal Matrix With subspace Orthogonal, orthogonal matrix Subspace Multiple d dimensions, calculate the orthogonal matrix With subspace The product of can get the expanded target subspace Afterwards, the target subspace can be calculated With vector Z p The product of can get the second optimization vector In this way, we can pass through the target subspace The second loss (Z p ) is converted into the second optimization vector Afterwards, the second optimization vector A new trigger vector is obtained by adding the initial control vector P0 of the decoder 212, and the new trigger vector is used to replace the original trigger vector, thereby achieving an optimization of the trigger vector.

[0111] In another embodiment, the subspace can be directly calculated With vector Z p The product of , we get the first optimization vector This can be done through the subspace The second loss (Z p ) is converted into the second optimization vector Afterwards, the second optimization vector A new trigger vector is obtained by adding the initial control vector P0 of the decoder 212, and the new trigger vector is used to replace the original trigger vector, thereby achieving an optimization of the trigger vector.

[0112] In the optimization process, a zero-order optimization strategy can be used to minimize the loss function by estimating the gradient. For example, the gradient can be estimated using simultaneous perturbation stochastic approximation. Specifically, the gradient can be estimated using the following formula (3).

[0113]

[0114] in, is the objective function value calculated on the training sample set. p ,z W} represents the low-dimensional latent parameter. μ is the perturbation amplitude, which is used to perform random perturbations around the current parameter value. j is a random direction sampled from a standard normal distribution.

[0115] Step 310, iterate the above process, and when the second end condition is reached, obtain a neural network composed of the image processing module and the optimized visual prompt module.

[0116] In this embodiment, after the first end condition is reached, steps 306 to 309 may be iteratively executed until a preset second end condition is reached. Exemplarily, the second end condition may be that the number of iterations reaches a preset number of times. The preset number of times is 1, and at this time, after a second sample image is obtained and the optimization of the visual prompt module is completed once, the optimization may be terminated.

[0117] At this time, the neural network composed of the visual prompt module 21 and the image processing module 22 can also be verified using sample images in the verification sample set. After the verification is passed, the neural network composed of the visual prompt module 21 and the image processing module 22 can be put into use.

[0118] In an embodiment of the present application, during the training process of the neural network, a first fused image corresponding to the first sample image is obtained, the first fused image is obtained by fusing the first sample image and the first visual cue output by the visual cue module, the first fused image is input into the image processing module to obtain a first output result, the first loss is converted into an adjustment vector using subspace learning, the vector in the visual cue module is adjusted according to the adjustment vector, the above steps are iterated, and when the first end condition is reached, a neural network composed of the image processing module and the adjusted visual cue module is obtained. During the model training process, the high-dimensional loss can be mapped to a low-dimensional subspace through subspace learning to obtain an adjustment vector, and the visual cue module in the neural network model can be adjusted according to the adjustment vector. Subspace learning can significantly reduce the amount of data in the training process, thereby accelerating the convergence speed of the neural network and shortening the training time of the neural network.

[0119] Referring to Table 1, the parameters shown in row A of Table 1 are the relevant parameters when the neural network is trained using the BlackVIP method, and the parameters shown in row B of Table 1 are the relevant parameters when the neural network is trained using the neural network training method provided by the present application. As can be seen from Table 1, compared with the BlackVIP method, the Params parameter (0.15k) corresponding to the training method provided by this implementation is smaller than the Params parameter (9.92k) corresponding to the BlackVIP method. Therefore, the training method provided by the present application can significantly improve the convergence speed during the training process and shorten the training time compared with the BlackVIP method.

[0120]

[0121] Table 1

[0122] See Table 2, the parameters shown in row A of Table 2 are the relevant parameters when the neural network is trained using the BlackVIP method, and the parameters shown in row B of Table 1 are the relevant parameters when the neural network is trained using the neural network training method provided by the present application. It can be seen from Table 1 that, compared with the BlackVIP method, the parameters corresponding to the training method provided by this embodiment are all smaller than the parameters corresponding to the BlackVIP method. Therefore, the training method provided by the present application can significantly improve the convergence speed during the training process and shorten the training time compared with the BlackVIP method.

[0123]

[0124] Table 2

[0125] Figure 5 Schematic diagram of a neural network training device based on subspace learning provided in an embodiment of the present application. Figure 5 As shown, the apparatus 500 may include:

[0126] An acquisition module 501 is used to acquire a first sample image and acquire a first fused image corresponding to the first sample image; wherein the first fused image is obtained by fusing the first sample image and a first visual cue, and the first visual cue is generated by the visual cue module based on the first sample image;

[0127] An input module 502, configured to input the first fused image into the image processing module to obtain a first output result output by the image processing module;

[0128] A conversion module 503, configured to convert a first loss into an adjustment vector by using subspace learning; wherein the first loss is obtained according to a deviation between an image annotation of the first sample image and the first output result;

[0129] An adjustment module 504, configured to adjust the vector in the visual prompt module according to the adjustment vector;

[0130] The iteration module 505 is used to iterate the above steps and obtain a neural network composed of the image processing module and the adjusted visual prompt module when the first end condition is reached.

[0131] Optionally, the visual cue module includes a decoder; the conversion module 503 is specifically used to adopt a first subspace learning to convert the first loss into a first adjustment vector; the adjustment module 504 is specifically used to adjust the control vector in the decoder according to the first adjustment vector.

[0132] Optionally, the visual cue module includes a trigger vector for adjusting the input of the decoder; the conversion module 503 is specifically used to adopt a second subspace learning to convert the first loss into a second adjustment vector; the adjustment module 504 is specifically used to adjust the trigger vector according to the second adjustment vector.

[0133] Optionally, the iteration module 505 is specifically used to optimize the visual prompt module when the first end condition is reached, so as to obtain a neural network composed of the image processing module and the optimized visual prompt module.

[0134] Optionally, the iteration module 505 is specifically used to obtain a second sample image, and obtain a second fused image corresponding to the second sample image; wherein the second fused image is obtained by fusing the second sample image and a second visual cue, and the second visual cue is generated by the visual cue module based on the second sample image;

[0135] The second fused image is input into the image processing module to obtain a second output result output by the image processing module; the second loss is converted into an optimization vector through the subspace obtained by the subspace learning; wherein the second loss is obtained according to the deviation between the image annotation of the second sample image and the second output result; the vector in the visual cue module is optimized according to the optimization vector; the above process is iterated, and when the second end condition is reached, a neural network composed of the image processing module and the optimized visual cue module is obtained.

[0136] Optionally, the iteration module 505 is specifically used to expand the dimension of the subspace obtained through the subspace learning to obtain a target subspace; and according to the target subspace, convert the second loss into the optimization vector.

[0137] Optionally, the visual cue module includes a decoder; the iteration module 505 is specifically used to convert the second loss into a first optimization vector through the subspace obtained by the subspace learning; and optimize the control vector in the decoder according to the first optimization vector.

[0138] Optionally, the visual cue module includes a trigger vector for adjusting the input of the decoder; the iteration module 505 is specifically used to convert the second loss into a second optimization vector through the subspace obtained by the subspace learning; and optimize the trigger vector according to the second optimization vector.

[0139] Figure 6 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 600 includes: a central processing unit 601, a memory 602, an image processor 603, a communication bus 604, a user interface 605, a network interface 606 and other hardware components. The memory 602 stores an executable program code, and the central processing unit 601 is used to call and execute the executable program code to perform the neural network training method based on subspace learning in the above example. The image processor 603 is used to display a GUI, and the communication bus 604 is used to establish a communication connection between the electronic device and other devices.

[0140] In this embodiment, the functional modules of the device can be divided according to the above method example. For example, each functional module can be corresponded, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0141] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0142] It should be understood that the device provided in this embodiment is used to execute the above-mentioned neural network training method, and thus can achieve the same effect as the above-mentioned implementation method.

[0143] In the case of an integrated unit, the device may include a processing module and a storage module. When the device is applied to an electronic device, the processing module may be used to control and manage the actions of the electronic device. The storage module may be used to support the electronic device to execute mutual program codes, etc.

[0144] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits disclosed in the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0145] In addition, the device provided in the embodiments of the present application may specifically be a chip, a component or a module, and the chip may include a connected processor and a memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a neural network training method provided in the above embodiments.

[0146] This embodiment also provides a readable storage medium, which stores executable program code. When the executable program code runs on a computer, the computer executes the above-mentioned related method steps to implement a neural network training method provided by the above embodiment.

[0147] This embodiment also provides an executable program code product. When the executable program code product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement a neural network training method provided by the above embodiment.

[0148] Among them, the device, readable storage medium, executable program code product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0149] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0150] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0151] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A neural network training method based on subspace learning, characterized in that: The neural network includes a pre-trained image processing module and a visual cue module, and the method includes: Acquire a first sample image, and acquire a first fused image corresponding to the first sample image; wherein the first fused image is obtained by fusing the first sample image and a first visual cue, and the first visual cue is generated by the visual cue module based on the first sample image; Inputting the first fused image into the image processing module to obtain a first output result output by the image processing module; Using subspace learning, converting a first loss into an adjustment vector; wherein the first loss is obtained according to a deviation between an image annotation of the first sample image and the first output result; Adjusting the vector in the visual cue module according to the adjustment vector; Iterate the above steps, and when the first end condition is reached, obtain a neural network composed of the image processing module and the adjusted visual prompt module.

2. The method according to claim 1, characterized in that The visual cue module includes a decoder; The adopting subspace learning to convert the first loss into an adjustment vector includes: adopting first subspace learning to convert the first loss into a first adjustment vector; The adjusting the control vector in the visual prompt module according to the adjustment vector includes: adjusting the control vector in the decoder according to the first adjustment vector.

3. The method according to claim 2, characterized in that The visual cue module includes a trigger vector for adjusting an input to the decoder; The adopting subspace learning to convert the first loss into an adjustment vector includes: adopting second subspace learning to convert the first loss into a second adjustment vector; The adjusting the vector in the visual prompt module according to the adjustment vector includes: adjusting the trigger vector according to the second adjustment vector.

4. The method according to claim 2, characterized in that When the first end condition is reached, a neural network composed of the image processing module and the adjusted visual prompt module is obtained, comprising: When the first end condition is reached, the visual prompt module is optimized to obtain a neural network composed of the image processing module and the optimized visual prompt module.

5. The method according to claim 3, characterized in that The optimizing the visual prompt module comprises: Acquire a second sample image, and acquire a second fused image corresponding to the second sample image; wherein the second fused image is obtained by fusing the second sample image and a second visual cue, and the second visual cue is generated by the visual cue module based on the second sample image; Inputting the second fused image into the image processing module to obtain a second output result output by the image processing module; The second loss is converted into an optimization vector through the subspace obtained by the subspace learning; wherein the second loss is obtained according to the deviation between the image annotation of the second sample image and the second output result; Optimizing the vector in the visual cue module according to the optimization vector; The above process is iterated, and when the second end condition is reached, a neural network composed of the image processing module and the optimized visual prompt module is obtained.

6. The method according to claim 5, characterized in that The subspace obtained by the subspace learning converts the second loss into an optimization vector, including: Expanding the dimension of the subspace obtained through the subspace learning to obtain a target subspace; The second loss is converted into the optimization vector according to the target subspace.

7. The method according to claim 5, characterized in that The visual cue module includes a decoder; The subspace obtained through the subspace learning, converting the second loss into an optimization vector, comprises: converting the second loss into a first optimization vector through the subspace obtained through the subspace learning; Optimizing the vector in the visual prompt module according to the optimization vector includes: optimizing the control vector in the decoder according to the first optimization vector.

8. The method according to claim 5, characterized in that The visual cue module includes a trigger vector for adjusting an input to the decoder; The subspace obtained through the subspace learning, converting the second loss into an optimization vector, comprises: converting the second loss into a second optimization vector through the subspace obtained through the subspace learning; The optimizing the vector in the visual prompt module according to the optimization vector includes: optimizing the trigger vector according to the second optimization vector.

9. A neural network training device based on subspace learning, characterized in that: The neural network includes a pre-trained image processing module and a visual prompt module, and the device includes: an acquisition module, configured to acquire a first sample image and a first fused image corresponding to the first sample image; wherein the first fused image is obtained by fusing the first sample image and a first visual cue, and the first visual cue is generated by the visual cue module based on the first sample image; An input module, used for inputting the first fused image into the image processing module to obtain a first output result output by the image processing module; A conversion module, configured to convert a first loss into an adjustment vector by using subspace learning; wherein the first loss is obtained according to a deviation between an image annotation of the first sample image and the first output result; An adjustment module, used for adjusting the vector in the visual prompt module according to the adjustment vector; An iterative module is used to iterate the above steps and obtain a neural network composed of the image processing module and the adjusted visual prompt module when the first end condition is reached.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the method according to any one of claims 1 to 8.

Images