Image recognition method, model training method and device

Abstract

The present application relates to an image recognition method, a model training method, and an apparatus. The method comprises: obtaining an image to be recognized and a recognition task corresponding to the image to be recognized; determining a target convolution bypass and a target additional network unit corresponding to the recognition task, wherein the target convolution bypass and the target additional network unit are generated after training based on an image sample set corresponding to the recognition task; loading the target convolution bypass and the target additional network unit into a pre-trained model to generate an image recognition model corresponding to the recognition task; inputting the image to be recognized into the image recognition model, and obtaining a recognition result output by the image recognition model. In this way, for different recognition tasks, it is only necessary to load different convolution bypass and additional network units into the pre-trained model to perform image recognition corresponding to the recognition task. Compared with storing all model parameters corresponding to the recognition task, storage overhead is effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image recognition method, a model training method and a device. Background Art

[0002] Currently, large-scale pre-trained models are widely used in the field of computer vision. For downstream tasks such as image classification, object detection, and semantic segmentation, the model parameters of the pre-trained model can be fine-tuned using the labeled data corresponding to each downstream task. This allows the pre-trained model to better perform image recognition based on different downstream tasks after fine-tuning.

[0003] In related technologies, when fine-tuning the parameters of a pre-trained model, it is usually necessary to adjust all model parameters in the pre-trained model. Accordingly, for different downstream tasks, different fine-tuned model parameters need to be stored.

[0004] However, as the size of the pre-trained model continues to increase, the model parameters of the pre-trained model are also increasing, which leads to excessive storage overhead of the model parameters of the fine-tuned pre-trained model. Summary of the Invention

[0005] Based on this, it is necessary to provide an image recognition method, a model training method and a device that can effectively reduce the storage overhead of the model parameters of the pre-trained model in order to address the above technical problems.

[0006] In a first aspect, the present application provides an image recognition method. The method comprises:

[0007] Obtaining an image to be identified and a recognition task corresponding to the image to be identified;

[0008] Determining a target convolution bypass and a target additional network unit corresponding to the recognition task, wherein the target convolution bypass and the target additional network unit are generated after training based on an image sample set corresponding to the recognition task;

[0009] Loading the target convolution bypass and the target additional network unit into a pre-trained model to generate an image recognition model corresponding to the recognition task;

[0010] The image to be recognized is input into the image recognition model, and a recognition result output by the image recognition model is obtained.

[0011] In one embodiment, the pre-trained model includes at least one self-attention unit and at least one feed-forward neural network unit;

[0012] The loading of the target convolution bypass and the target additional unit into the pre-trained model comprises:

[0013] Adding a target convolution bypass to each self-attention unit and each feedforward neural network unit in the pre-trained model;

[0014] Add the target additional network unit at the target position in the pre-trained model.

[0015] In one embodiment, the input of the target convolution bypass is the input of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass;

[0016] The output of the target convolution bypass is used to add to the output of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass.

[0017] In one embodiment, the target convolution bypass includes a first convolution subunit and a second convolution subunit, and the convolution kernel of the first convolution subunit is smaller than the convolution kernel of the second convolution subunit.

[0018] In one embodiment, if the input of the pre-trained model is a serialized input, the target convolution bypass further includes a reconstruction subunit;

[0019] The reconstruction subunit is arranged before the second convolution subunit;

[0020] The reconstruction subunit is used to reconstruct the two-dimensional spatial structure of the graphic token in the input data to generate a reconstructed picture token, and input the reconstructed picture token and the identification token in the input data into the second convolution subunit.

[0021] In one embodiment, before determining the target convolution bypass and target additional network unit corresponding to the recognition task, the method further includes:

[0022] Obtaining an image sample set corresponding to the recognition task;

[0023] According to the identification task, the original additional network unit is confirmed;

[0024] Adding the original convolution bypass and the original additional network unit to the pre-trained model;

[0025] Under the condition that the model parameters of the pre-trained model are locked, the original convolution bypass and the original additional network unit in the pre-trained model are trained using the image sample set to obtain the trained target convolution bypass and target additional network unit;

[0026] According to the recognition task, the target convolution bypass and the target additional network unit are saved.

[0027] In a second aspect, the present application provides a model training method. The method comprises:

[0028] Obtain the image sample set corresponding to the recognition task;

[0029] According to the identification task, the original additional network unit is confirmed;

[0030] Adding the original convolution bypass and the original additional network unit to the pre-trained model;

[0031] Under the condition that the model parameters of the pre-trained model are locked, the original convolution bypass and the original additional network unit in the pre-trained model are trained using the image sample set to obtain a trained target convolution bypass and a target additional network unit, wherein the target convolution bypass and the target additional network unit are used to generate an image recognition model corresponding to the recognition task;

[0032] The target convolution bypass and the target additional network unit are saved.

[0033] In a third aspect, the present application provides an image recognition device. The method includes:

[0034] A first acquisition module is configured to acquire an image to be recognized and a recognition task corresponding to the image to be recognized; determine a target convolution bypass and a target additional network unit corresponding to the recognition task, wherein the target convolution bypass and the target additional network unit are generated after training based on an image sample set corresponding to the recognition task;

[0035] A loading module, configured to load the target convolution bypass and the target additional network unit into a pre-trained model to generate an image recognition model corresponding to the recognition task;

[0036] The recognition module is used to input the image to be recognized into the image recognition model and obtain the recognition result output by the image recognition model.

[0037] In one embodiment, the pre-trained model includes at least one self-attention unit and at least one feed-forward neural network unit;

[0038] The loading module is specifically used to add a target convolution bypass to each self-attention unit and each feedforward neural network unit in the pre-trained model; and add the target additional network unit at the target position in the pre-trained model.

[0039] In one embodiment, the input of the target convolution bypass is the input of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass;

[0040] The output of the target convolution bypass is used to add to the output of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass.

[0041] In one embodiment, the target convolution bypass includes a first convolution subunit and a second convolution subunit, and the convolution kernel of the first convolution subunit is smaller than the convolution kernel of the second convolution subunit.

[0042] In one embodiment, if the input of the pre-trained model is a serialized input, the target convolution bypass further includes a reconstruction subunit;

[0043] The reconstruction subunit is arranged before the second convolution subunit;

[0044] The reconstruction subunit is used to reconstruct the two-dimensional spatial structure of the graphic token in the input data to generate a reconstructed picture token, and input the reconstructed picture token and the identification token in the input data into the second convolution subunit.

[0045] In one embodiment, the image recognition device further includes:

[0046] A training module is used to obtain an image sample set corresponding to the recognition task; determine the original additional network unit according to the recognition task; add the original convolution bypass and the original additional network unit to the pre-trained model; use the image sample set to train the original convolution bypass and the original additional network unit in the pre-trained model while locking the model parameters of the pre-trained model to obtain the trained target convolution bypass and target additional network unit; and save the target convolution bypass and the target additional network unit according to the recognition task.

[0047] In a fourth aspect, the present application provides a model training device. The method includes:

[0048] The second acquisition module is used to obtain an image sample set corresponding to a recognition task; and according to the recognition task, determine the original additional network unit;

[0049] An adding module, configured to add the original convolution bypass and the original additional network unit to a pre-trained model;

[0050] A training module is configured to train the original convolution bypass and the original additional network unit in the pre-trained model using the image sample set while locking the model parameters of the pre-trained model, to obtain a trained target convolution bypass and a target additional network unit, wherein the target convolution bypass and the target additional network unit are used to generate an image recognition model corresponding to the recognition task;

[0051] A storage module is used to store the target convolution bypass and the target additional network unit.

[0052] In a fifth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the image recognition method described in the first aspect or the model training method described in the second aspect when executing the computer program.

[0053] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image recognition method described in the first aspect or the model training method described in the second aspect.

[0054] In a seventh aspect, the present application further provides a computer program product, comprising a computer program that, when executed by a processor, implements the image recognition method described in the first aspect or the model training method described in the second aspect.

[0055] The above-mentioned image recognition method, model training method, and device first obtain an image to be recognized and a recognition task corresponding to the image to be recognized. Secondly, the target convolution bypass and target additional network unit corresponding to the recognition task are determined. The target convolution bypass and target additional network unit are generated after training based on the image sample set corresponding to the recognition task. Thirdly, the target convolution bypass and target additional network unit are loaded into a pre-trained model to generate an image recognition model corresponding to the recognition task. Finally, the image to be recognized is input into the image recognition model, and the recognition result output by the image recognition model is obtained. In this way, for different recognition tasks, it is only necessary to load different convolution bypasses and additional network units into the pre-trained model to perform image recognition corresponding to the recognition task. Compared with storing all model parameters of the pre-trained model corresponding to the recognition task, storage overhead is effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 An application environment diagram of an image recognition method provided in an embodiment of the present application;

[0057] Figure 2 A flowchart of an image recognition method provided in an embodiment of the present application;

[0058] Figure 3 A schematic diagram of a convolution bypass provided in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of loading a convolution bypass into a pre-trained model provided in an embodiment of the present application;

[0060] Figure 5A schematic diagram of a model training process provided in an embodiment of the present application;

[0061] Figure 6 A flowchart of another image recognition method provided in an embodiment of the present application;

[0062] Figure 7 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0063] Figure 8 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0064] Figure 9 This is a diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0066] The image recognition method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. When an image recognition task is required, the terminal device 101 can send an image recognition request to the server. After receiving the image recognition request, the server 102 can extract the image to be recognized and the recognition task, and determine the corresponding convolution bypass and additional network unit according to the recognition task. Subsequently, the server 102 can load the convolution bypass and additional network unit into the pre-trained model to generate an image recognition model corresponding to the recognition task. Finally, the server 102 inputs the image to be recognized into the image recognition model, obtains the recognition result output by the image recognition model, and feeds back the image recognition result to the terminal device 101.

[0067] Terminals 101 and 103 may be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 102 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0068] In one embodiment, Figure 2 As shown, an image recognition method is provided, in which the image recognition method is applied to Figure 1 The following steps are used as an example to illustrate the server in the example:

[0069] S201: Obtain an image to be recognized and a recognition task corresponding to the image to be recognized.

[0070] In this application, when an image recognition task is required, the server can obtain the image to be recognized and the recognition task corresponding to the image to be recognized.

[0071] It should be understood that the embodiments of the present application do not limit how to obtain the image to be recognized and the recognition task corresponding to the image to be recognized. In some embodiments, when a user needs to perform image recognition, they can send an image recognition request to the server via a terminal device. The image recognition request may include the image to be recognized and the recognition task corresponding to the image to be recognized. The server can obtain the image to be recognized and the recognition task corresponding to the image to be recognized indicated by the user from the image recognition request to perform subsequent image recognition.

[0072] In other embodiments, the server may receive images to be recognized that are captured in real time and store them in a database. A user may send an image recognition instruction to the server via a terminal device, instructing the server to perform image recognition on images to be recognized within a certain time period and / or in certain scenarios. The image recognition instruction includes the recognition task specified by the user. The server then retrieves the images to be recognized from the database and obtains the recognition task from the image recognition instruction to perform subsequent image recognition.

[0073] The above-mentioned recognition tasks can be determined according to the user's image recognition requirements, and the present embodiment does not limit this. Exemplarily, the above-mentioned recognition tasks may include image classification tasks, object detection tasks, semantic segmentation tasks, etc.

[0074] S202. Determine a target convolution bypass and a target additional network unit corresponding to the recognition task, where the target convolution bypass and the target additional network unit are generated after training based on the image sample set corresponding to the recognition task.

[0075] In this step, after the server obtains the image to be recognized and the recognition task corresponding to the image to be recognized, it can determine the target convolution bypass and target additional network unit corresponding to the recognition task.

[0076] The target additional network unit may be a trained additional network unit corresponding to the recognition task. The embodiment of the present application does not limit the type of the target additional network unit. For example, if the recognition task is an image classification task, the target additional network unit may include a classification head attached to a pre-trained model; if the recognition task is an object detection task, the target additional network unit may include a detection head attached to a pre-trained model; if the recognition task is a semantic segmentation task, the target additional network unit may include a segmentation decoder attached to a pre-trained model.

[0077] The target convolution bypass may be a trained convolution bypass corresponding to the recognition task. In some embodiments, the target convolution bypass includes a first convolution subunit and a second convolution subunit, and the convolution kernel of the first convolution subunit is smaller than the convolution kernel of the second convolution subunit.

[0078] For example, the convolution kernel of the first convolution subunit may be 1*1, and the convolution kernel of the second convolution unit may be 3*3. For example, the convolution kernel of the first convolution unit may be 1*1, and the convolution kernel of the second convolution unit may be 2*2.

[0079] It should be understood that an activation function unit, such as a Gaussian Error Linear Unit (GELU) activation function unit, may be further provided between the first subconvolution subunit and the second convolution subunit.

[0080] It should be understood that the first convolution subunit and the second convolution subunit in the target convolution bypass can be one or more, respectively, and the embodiments of the present application do not limit the number of first convolution subunits and second convolution subunits. For example, the target convolution bypass includes two first convolution subunits and one second convolution subunit.

[0081] For example, Figure 3 The figure shows a schematic diagram of a convolution bypass provided in an embodiment of the present application. The convolution bypass includes two 1*1 convolution subunits and one 3*3 convolution subunit. The 3*3 convolution subunit is arranged between the two 1*1 convolution subunits, and a GELU activation function subunit is also arranged between different convolution subunits.

[0082] In other embodiments, if the input of the pre-trained model to be added corresponding to the target convolution bypass is a serialized input, the target convolution bypass may further include a reconstruction subunit. The reconstruction subunit may be arranged before the second convolution subunit, and the reconstruction subunit is used to reconstruct the two-dimensional spatial structure of the graphic token in the input data to generate a reconstructed picture token, and input the reconstructed picture token and the identification token in the input data into the second convolution subunit. The pre-trained model whose input is a serialized input may include a vision transformer model.

[0083] It should be noted that in the embodiments of the present application, different recognition tasks correspond to different convolutional bypasses and additional network units. Based on the image sample sets corresponding to different recognition tasks, the original convolutional bypasses and original additional network units can be trained to produce different fine-tuned convolutional bypasses and additional network units to adapt to different recognition tasks. Therefore, before performing image recognition, the server needs to determine the target convolutional bypass and target additional network unit corresponding to the recognition task.

[0084] S203. Load the target convolution bypass and target additional network units into the pre-trained model to generate an image recognition model corresponding to the recognition task.

[0085] In this step, after the server determines the target convolution bypass and target additional network unit corresponding to the recognition task, the target convolution bypass and target additional network unit can be loaded into the pre-trained model to generate an image recognition model corresponding to the recognition task.

[0086] The pre-trained model may be a model that first performs preliminary training using sample data and then adjusts parameters based on the preliminary training according to the lower-level recognition task. The preliminary training may use images with or without labeled information. The embodiments of the present application do not limit the type of pre-trained model. Exemplarily, the pre-trained model may include a Region-based Convolutional Neural Network (Mask R-CNN) model, a MobileNet model, a Vision Transformer model, and the like.

[0087] In some embodiments, the above-mentioned pre-training model may include at least one self-attention unit and at least one feedforward neural network unit, and the self-attention unit and the feedforward neural network unit constitute a multi-layer network structure of the pre-training model.

[0088] It should be understood that the number of self-attention units and feedforward neural network units in the above pre-training model can be determined according to actual conditions, and can be one or more, and the number of self-attention units and feedforward neural network units can be equal or unequal.

[0089] For example, when the pre-trained model is a Vision Transformer model, it may include 12 multi-head self-attention modules and 12 feedforward neural network modules. Furthermore, the pre-trained model may include one embedding unit and 13 normalization units. If the recognition task is a classification task, one classification head may also be added to the pre-trained model.

[0090] It should be understood that the embodiments of the present application do not limit how to load the target convolution bypass and the target additional network unit into the pre-trained model. In some embodiments, the server can add a target convolution bypass for each self-attention unit and each feedforward neural network unit in the pre-trained model, and add the target additional network unit at the target position in the pre-trained model.

[0091] For example, Figure 4As shown, the input of the target convolution bypass is the input of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass. The output of the target convolution bypass is used to add to the output of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass.

[0092] It should be understood that the above target position can be determined based on the target additional network unit. For example, if the recognition task is an image classification task, the target additional network unit is a classification head, and accordingly, the classification head can be added to the top of the pre-trained model.

[0093] S204: Input the image to be recognized into the image recognition model, and obtain the recognition result output by the image recognition model.

[0094] In this step, after the image recognition model corresponding to the recognition task is generated, the server can input the image to be recognized into the image recognition model and obtain the recognition result output by the image recognition model.

[0095] The recognition result can be determined based on the recognition task. For example, if the recognition task is an image classification task, the recognition result can be the classification result of the image to be recognized. If the recognition task is an object detection task, the recognition result can be the detection result of the image to be recognized.

[0096] The image recognition method provided in the embodiment of the present application first obtains an image to be recognized and a recognition task corresponding to the image to be recognized. Secondly, the target convolution bypass and target additional network unit corresponding to the recognition task are determined. The target convolution bypass and target additional network unit are generated after training based on the image sample set corresponding to the recognition task. Thirdly, the target convolution bypass and target additional network unit are loaded into a pre-trained model to generate an image recognition model corresponding to the recognition task. Finally, the image to be recognized is input into the image recognition model, and the recognition result output by the image recognition model is obtained. In this way, for different recognition tasks, it is only necessary to load different convolution bypass and additional network units into the pre-trained model to perform image recognition corresponding to the recognition task. Compared with storing all model parameters corresponding to the recognition task, the storage overhead is effectively reduced.

[0097] The following describes the training of convolution bypass and additional network units. Figure 5 A flow chart of a model training process provided in an embodiment of the present application is as follows: Figure 5 As shown, the model training method includes:

[0098] S501: Obtain an image sample set corresponding to a recognition task.

[0099] It should be understood that in this application, the image sample set may include images with labeled information. For example, if the recognition task is an image classification task, the image sample set may include multiple sample images X, each of which is labeled with its corresponding category label Y.

[0100] In some embodiments, different recognition tasks correspond to different image sample sets. The different image sample sets may include the same sample images carrying different annotation information, or different sample images carrying different annotation information.

[0101] For example, the image sample set corresponding to the image recognition task and the image sample set corresponding to the target detection task may contain the same sample images, but the same sample images may have different annotation information. The sample images in the image sample set corresponding to the image recognition task are annotated with image classification information, while the sample images in the image sample set corresponding to the target detection task are annotated with target detection information. For example, the sample images in the image sample set corresponding to the image recognition task and the sample images in the image sample set corresponding to the target detection task may also be different.

[0102] S502: According to the identification task, determine the original additional network unit.

[0103] The above-mentioned original additional network unit may be an additional network unit corresponding to the recognition task, and the original additional network unit has not been trained with the image sample set corresponding to the recognition task.

[0104] Exemplarily, if the recognition task is an image classification task, the above-mentioned original additional network unit may include a classification head attached to the pre-trained model; if the recognition task is an object detection task, the above-mentioned original additional network unit may include a detection head attached to the pre-trained model; if the recognition task is a semantic segmentation task, the above-mentioned original additional network unit may include a segmentation decoder attached to the pre-trained model.

[0105] S503. Add the original convolution bypass and the original additional network unit to the pre-trained model.

[0106] In some embodiments, the pre-trained model includes at least one self-attention unit and at least one feed-forward neural network unit. The server can add a primitive convolution bypass to each self-attention unit and each feed-forward neural network unit in the pre-trained model, and add the primitive additional network unit at the target position in the pre-trained model.

[0107] The input of the target convolution bypass is the input of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass. The output of the target convolution bypass is used to add to the output of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass.

[0108] For example, if the pre-trained model is a Vision Transformer model, each layer of the Vision Transformer model can be composed of a multi-head self-attention module, a feedforward neural network module, and two normalization layers. Accordingly, the input and output of each layer of the Vision Transformer model can be shown as in Formula (1) and Formula (2):

[0109] x1=x0+MHSA(LN(x0)) (1)

[0110] x2=x1+FFN(LN(x1)) (2)

[0111] Among them, x0 is the input data, x2 is the output data, x1 is the intermediate data, MHSA is the multi-head self-attention mechanism, FFN is the feedforward neural network, and LN is the normalization process.

[0112] After adding a convolution bypass to each multi-head self-attention module and each feedforward neural network module, the input and output of each layer of the Vision Transformer model can be shown as shown in Equation (3) and Equation (4):

[0113] x1=x0+MHSA(LN(x0)) +Convpass(LN(x0)) (1)

[0114] x2=x1+FFN(LN(x1))+Convpass(LN(x1)) (2)

[0115] Among them, x0 is the input data, x2 is the output data, x1 is the intermediate data, MHSA is the multi-head self-attention mechanism, FFN is the feedforward neural network, LN is the normalization process, and Convpass is the convolution bypass.

[0116] S504: Under the model parameters of the pre-trained model, the original convolution bypass and the original additional network unit in the pre-trained model are trained using the image sample set to obtain the trained target convolution bypass and target additional network unit.

[0117] It should be understood that the embodiments of the present application do not limit how to train the original convolution bypass and the original additional network unit.

[0118] In some embodiments, the server trains the convolution bypass and target additional network units using gradient descent by calculating the cross entropy classification loss on the image sample set while locking the model parameters of the pre-trained model.

[0119] S505. Save the target convolution bypass and target additional network unit according to the recognition task.

[0120] In some embodiments, the target convolution bypass and the target additional network unit can be classified and saved according to the corresponding recognition tasks.

[0121] In an embodiment of the present application, since only the target convolution bypass and target additional network unit corresponding to each recognition task are saved, compared with saving all the adjustment parameters of the pre-trained model, the storage overhead can be greatly reduced when using the pre-trained model.

[0122] In one embodiment, Figure 6 As shown, another image recognition method is provided, comprising the following steps:

[0123] S601: Obtain an image sample set corresponding to a recognition task.

[0124] S602: According to the identification task, determine the original additional network unit.

[0125] S603: Add the original convolution bypass and the original additional network unit to the pre-trained model.

[0126] S604: Under the model parameters of the pre-trained model, the original convolution bypass and the original additional network unit in the pre-trained model are trained using the image sample set to obtain the trained target convolution bypass and target additional network unit.

[0127] S605: Save the target convolution bypass and target additional network unit.

[0128] S606: Obtain the image to be recognized and the recognition task corresponding to the image to be recognized.

[0129] S607. Determine a target convolution bypass and a target additional network unit corresponding to the recognition task, where the target convolution bypass and the target additional network unit are generated after training based on the image sample set corresponding to the recognition task.

[0130] S608. Load the target convolution bypass and target additional network units into the pre-trained model to generate an image recognition model corresponding to the recognition task.

[0131] S609: Input the image to be recognized into the image recognition model, and obtain the recognition result output by the image recognition model.

[0132] The image recognition method provided in the embodiment of the present application first obtains an image to be recognized and a recognition task corresponding to the image to be recognized. Secondly, the target convolution bypass and target additional network unit corresponding to the recognition task are determined. The target convolution bypass and target additional network unit are generated after training based on the image sample set corresponding to the recognition task. Thirdly, the target convolution bypass and target additional network unit are loaded into a pre-trained model to generate an image recognition model corresponding to the recognition task. Finally, the image to be recognized is input into the image recognition model, and the recognition result output by the image recognition model is obtained. In this way, for different recognition tasks, it is only necessary to load different convolution bypass and additional network units into the pre-trained model to perform image recognition corresponding to the recognition task. Compared with storing all model parameters corresponding to the recognition task, the storage overhead is effectively reduced.

[0133] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0134] Based on the same inventive concept, embodiments of the present application also provide an image recognition device for implementing the aforementioned image recognition method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following image recognition device embodiments can be found in the above-described limitations on the image recognition method and will not be further elaborated here.

[0135] In one embodiment, Figure 7 As shown, an image recognition device is provided. The image recognition device 700 includes: a first acquisition module 701, a loading module 702, a recognition module 703 and a training module 704.

[0136] The first acquisition module 701 is used to obtain an image to be recognized and a recognition task corresponding to the image to be recognized; determine a target convolution bypass and a target additional network unit corresponding to the recognition task, where the target convolution bypass and the target additional network unit are generated after training based on an image sample set corresponding to the recognition task;

[0137] A loading module 702 is used to load the target convolution bypass and target additional network unit into the pre-trained model to generate an image recognition model corresponding to the recognition task;

[0138] The recognition module 703 is used to input the image to be recognized into the image recognition model and obtain the recognition result output by the image recognition model.

[0139] In one embodiment, the pre-trained model includes at least one self-attention unit and at least one feed-forward neural network unit.

[0140] The loading module 702 is specifically used to add a target convolution bypass to each self-attention unit and each feedforward neural network unit in the pre-trained model; and add a target additional network unit at the target position in the pre-trained model.

[0141] In one embodiment, the input of the target convolution bypass is the input of the self-attention unit or the feedforward neural network unit corresponding to the target convolution bypass.

[0142] The output of the target convolution bypass is used to add to the output of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass.

[0143] In one embodiment, the target convolution bypass includes a first convolution subunit and a second convolution subunit, and the convolution kernel of the first convolution subunit is smaller than the convolution kernel of the second convolution subunit.

[0144] In one embodiment, if the input of the pre-trained model is a serialized input, the target convolution bypass further includes a reconstruction subunit;

[0145] The reconstruction subunit is arranged before the second convolution subunit;

[0146] The reconstruction subunit is used to reconstruct the two-dimensional spatial structure of the graphic token in the input data to generate a reconstructed picture token, and input the reconstructed picture token and the recognition token in the input data into the second convolution subunit.

[0147] In one embodiment, the image recognition device 700 further includes:

[0148] The training module 704 is used to obtain an image sample set corresponding to the recognition task; determine the original additional network unit according to the recognition task; add the original convolution bypass and the original additional network unit to the pre-trained model; use the image sample set to train the original convolution bypass and the original additional network unit in the pre-trained model under the model parameters of the pre-trained model to obtain the trained target convolution bypass and target additional network unit; save the target convolution bypass and target additional network unit according to the recognition task.

[0149] Each module in the above-mentioned image recognition device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0150] Based on the same inventive concept, the present application also provides a model training device for implementing the aforementioned model training method. The solution to the problem provided by the device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following model training device embodiments can be found in the above-mentioned limitations on the model training method, and will not be repeated here.

[0151] In one embodiment, Figure 8 As shown, a model training device is provided, which includes: a second acquisition module 801, an adding module 802, a training module 803 and a storage module 804.

[0152] The second acquisition module 801 is used to obtain an image sample set corresponding to a recognition task; and determine the original additional network unit according to the recognition task;

[0153] An adding module 802 for adding the original convolution bypass and the original additional network unit to the pre-trained model;

[0154] A training module 803 is configured to train the original convolution bypass and the original additional network unit in the pre-trained model using the image sample set while locking the model parameters of the pre-trained model, thereby obtaining the trained target convolution bypass and target additional network unit, which are used to generate an image recognition model corresponding to the recognition task;

[0155] The storage module 804 is used to store the target convolution bypass and the target additional network unit.

[0156] Each module in the above-mentioned model training device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0157] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image recognition method or a model training method is implemented.

[0158] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0159] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0160] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0161] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0162] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image recognition method, characterized in that: The method comprises: Obtaining an image to be identified and a recognition task corresponding to the image to be identified; Determining a target convolution bypass and a target additional network unit corresponding to the recognition task, wherein the target convolution bypass and the target additional network unit are generated after training based on an image sample set corresponding to the recognition task; Loading the target convolution bypass and the target additional network unit into a pre-trained model to generate an image recognition model corresponding to the recognition task, wherein the pre-trained model includes at least one self-attention unit and at least one feedforward neural network unit; Inputting the image to be recognized into the image recognition model, and obtaining a recognition result output by the image recognition model; The step of loading the target convolution bypass and the target additional unit into the pre-trained model comprises: A target convolution bypass is added to each self-attention unit and each feedforward neural network unit in the pre-trained model; the input of the target convolution bypass is the input of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass; the output of the target convolution bypass is used to be added to the output of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass; Adding the target additional network unit at a target position in the pre-trained model; Among them, the input of each layer of the pre-training model is x1=x0+MHSA(LN(x0))+Convpass(LN(x0)); the output of each layer of the pre-training model is x2=x1+FFN(LN(x1))+Convpass(LN(x1)); x0 is input data, x2 is output data, x1 is intermediate data, MHSA is multi-head self-attention mechanism, FFN is feedforward neural network, LN is normalization processing, and Convpass is convolution bypass.

2. The method according to claim 1, characterized in that The target convolution bypass includes a first convolution subunit and a second convolution subunit, and the convolution kernel of the first convolution subunit is smaller than the convolution kernel of the second convolution subunit.

3. The method according to claim 2, characterized in that If the input of the pre-trained model is a serialized input, the target convolution bypass further includes a reconstruction subunit; The reconstruction subunit is arranged before the second convolution subunit; The reconstruction subunit is used to reconstruct the two-dimensional spatial structure of the graphic token in the input data to generate a reconstructed picture token, and input the reconstructed picture token and the identification token in the input data into the second convolution subunit.

4. The method according to any one of claims 1 to 3, characterized in that Before determining the target convolution bypass and target additional network unit corresponding to the recognition task, the method further includes: Obtaining an image sample set corresponding to the recognition task; According to the identification task, the original additional network unit is confirmed; Adding the original convolution bypass and the original additional network unit to the pre-trained model; Under the condition that the model parameters of the pre-trained model are locked, the original convolution bypass and the original additional network unit in the pre-trained model are trained using the image sample set to obtain the trained target convolution bypass and target additional network unit; According to the recognition task, the target convolution bypass and the target additional network unit are saved.

5. A model training method, characterized in that: The method comprises: Obtain the image sample set corresponding to the recognition task; According to the identification task, the original additional network unit is confirmed; Adding the original convolution bypass and the original additional network unit to a pre-trained model, wherein the pre-trained model includes at least one self-attention unit and at least one feed-forward neural network unit; Under the condition that the model parameters of the pre-trained model are locked, the original convolution bypass and the original additional network unit in the pre-trained model are trained using the image sample set to obtain a trained target convolution bypass and a target additional network unit, wherein the target convolution bypass and the target additional network unit are used to generate an image recognition model corresponding to the recognition task; Saving the target convolution bypass and the target additional network unit; A target convolution bypass is added to each self-attention unit and each feedforward neural network unit in the pre-trained model; the input of the target convolution bypass is the input of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass; the output of the target convolution bypass is used to be added to the output of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass; Adding the target additional network unit at a target position in the pre-trained model; Among them, the input of each layer of the pre-training model is x1=x0+MHSA(LN(x0))+Convpass(LN(x0)); the output of each layer of the pre-training model is x2=x1+FFN(LN(x1))+Convpass(LN(x1)); x0 is input data, x2 is output data, x1 is intermediate data, MHSA is multi-head self-attention mechanism, FFN is feedforward neural network, LN is normalization processing, and Convpass is convolution bypass.

6. An image recognition device, characterized in that: The device comprises: A first acquisition module is configured to acquire an image to be recognized and a recognition task corresponding to the image to be recognized; determine a target convolution bypass and a target additional network unit corresponding to the recognition task, wherein the target convolution bypass and the target additional network unit are generated after training based on an image sample set corresponding to the recognition task; A loading module, configured to load the target convolution bypass and the target additional network unit into a pre-trained model to generate an image recognition model corresponding to the recognition task, wherein the pre-trained model includes at least one self-attention unit and at least one feedforward neural network unit; A recognition module, configured to input the image to be recognized into the image recognition model and obtain a recognition result output by the image recognition model; The loading module is further configured to add a target convolution bypass to each self-attention unit and each feedforward neural network unit in the pre-trained model; the input of the target convolution bypass is the input of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass; the output of the target convolution bypass is configured to be added to the output of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass; and the target additional network unit is added to the target position in the pre-trained model; Among them, the input of each layer of the pre-training model is x1=x0+MHSA(LN(x0))+Convpass(LN(x0)); the output of each layer of the pre-training model is x2=x1+FFN(LN(x1))+Convpass(LN(x1)); x0 is input data, x2 is output data, x1 is intermediate data, MHSA is multi-head self-attention mechanism, FFN is feedforward neural network, LN is normalization processing, and Convpass is convolution bypass.

7. A model training device, characterized in that: The device comprises: The second acquisition module is used to obtain an image sample set corresponding to a recognition task; and according to the recognition task, determine the original additional network unit; An adding module, configured to add the original convolution bypass and the original additional network unit to a pre-trained model, wherein the pre-trained model includes at least one self-attention unit and at least one feedforward neural network unit; A training module is configured to train the original convolution bypass and the original additional network unit in the pre-trained model using the image sample set while locking the model parameters of the pre-trained model, to obtain a trained target convolution bypass and a target additional network unit, wherein the target convolution bypass and the target additional network unit are used to generate an image recognition model corresponding to the recognition task; A storage module, used to store the target convolution bypass and the target additional network unit; add a target convolution bypass to each self-attention unit and each feedforward neural network unit in the pre-trained model; the input of the target convolution bypass is the input of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass; the output of the target convolution bypass is used to add to the output of the self-attention unit or feedforward neural network unit corresponding to the target convolution bypass; add the target additional network unit to the target position in the pre-trained model; Among them, the input of each layer of the pre-training model is x1=x0+MHSA(LN(x0))+Convpass(LN(x0)); the output of each layer of the pre-training model is x2=x1+FFN(LN(x1))+Convpass(LN(x1)); x0 is input data, x2 is output data, x1 is intermediate data, MHSA is multi-head self-attention mechanism, FFN is feedforward neural network, LN is normalization processing, and Convpass is convolution bypass.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images