Image recognition method, device and electronic device

Abstract

The present invention discloses an image recognition method, device and electronic device, which belong to the field of computer vision. The image recognition method comprises: obtaining an image to be recognized and recognition instruction information; inputting the recognition instruction information into an embedding module to obtain a feature vector output by the embedding module; inputting the feature vector and the image to be recognized into a target model to obtain a target recognition result output by the target model. The image recognition method of the present invention can improve the recognition accuracy and recall rate of the target model for a specific recognition object while retaining the generalization ability of the target model, thereby realizing the practical application of the subsequent open set target detection model.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an image recognition method, device and electronic equipment. Background Art

[0002] Object detection is an important task in the field of computer vision. Open-set object detection and closed-set object detection are two concepts in the field of object detection. Among them, the closed-set object detection model is trained based on a specific category or a specific task. All object categories encountered during training and testing are known, and it has high accuracy and recall. The open-set object detection model is trained based on a large-scale data set and has good generalization, that is, it can detect any category, but the accuracy and recall on a specific data set (specific category or specific task) are lower than the closed-set object detection model. In related technologies, it is usually adopted to expand a specific data set and train an open-set object detection model to improve the accuracy and recall of the open-set object detection model on a specific data set. However, expanding a specific data set will cause the parameters of the open-set object detection model to change, thereby causing the open-set object detection model to lose the general knowledge (generalization) learned in the pre-training stage, thereby limiting the subsequent implementation of the open-set object detection model. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes an image recognition method, device and electronic device, which can improve the recognition accuracy and recall rate of the target model for a specific recognition object while retaining the generalization ability of the target model, thereby realizing the implementation of the subsequent open set target detection model.

[0004] In a first aspect, the present invention provides an image recognition method, the method comprising:

[0005] Obtaining the image to be recognized and recognition instruction information;

[0006] Inputting the recognition instruction information into an embedding module to obtain a feature vector output by the embedding module;

[0007] The feature vector and the image to be recognized are input into the target model to obtain the target recognition result output by the target model; wherein,

[0008] The embedding module is obtained by training based on the first sample data, and the target model is obtained by training based on the second sample data.

[0009] According to the image recognition method of the present invention, by obtaining an image to be recognized and recognition instruction information, inputting the recognition instruction information into an embedding module, obtaining a feature vector output by the embedding module, and then inputting the feature vector and the image to be recognized into a target model, obtaining a target recognition result output by the target model, the input features of a specific recognition object input to the target model can be adjusted based on the embedding module obtained by training a specific data set without changing the target model parameters, so as to identify an accurate target recognition result using the original target model parameters, thereby improving the recognition accuracy and recall rate of the target model for specific recognition objects on the basis of retaining the generalization ability of the target model, thereby realizing the practical application of subsequent open-set target detection models.

[0010] According to an embodiment of the present invention, before acquiring the image to be recognized and the recognition instruction information, the method further includes:

[0011] Acquire the first sample data, where the first sample data includes a first sample label;

[0012] Inputting the first sample label into the embedding module to obtain a feature vector output by the embedding module;

[0013] Under the condition that the model parameters of the target model are controlled to remain unchanged, the feature vector is input into the target model to obtain a first prediction recognition result output by the target model;

[0014] The embedding module is trained based on the first predicted recognition result and the first sample label.

[0015] According to an embodiment of the present invention, the step of training the embedding module based on the first predicted recognition result and the first sample label includes:

[0016] When the difference between the first predicted recognition result and the first sample label is greater than a target threshold, optimizing the embedding module and obtaining a feature vector output by the optimized embedding module;

[0017] Under the condition that the model parameters of the target model are controlled to remain unchanged, the feature vector output by the optimized embedding module is input into the target model to obtain a new first prediction recognition result output by the target model;

[0018] When the difference between the first predicted recognition result and the first sample label is not greater than a target threshold, the optimization of the embedding module is stopped.

[0019] According to an embodiment of the present invention, before acquiring the image to be recognized and the recognition instruction information, the method further includes:

[0020] Acquire the second sample data, where the second sample data includes a second sample image and a second sample label corresponding to the second sample image;

[0021] Based on the second sample image and the second sample label, the model parameters of the target model are trained.

[0022] According to an embodiment of the present invention, inputting the recognition instruction information into an embedding module to obtain a feature vector output by the embedding module includes:

[0023] The recognition instruction information is input into the embedding module, and the feature vector output by the embedding module after processing the recognition instruction information based on the Embedding technology is obtained.

[0024] According to an embodiment of the present invention, after acquiring the image to be recognized and the recognition instruction information, the method further includes:

[0025] When the category of the recognition instruction information is different from the category of the first sample data, the recognition instruction information and the image to be recognized are input into the target model to obtain the target recognition result output by the target model.

[0026] In a second aspect, the present invention provides an image recognition device, the device comprising:

[0027] A first processing module, used for acquiring an image to be recognized and recognition instruction information;

[0028] A second processing module, used for inputting the recognition instruction information into an embedding module, and obtaining a feature vector output by the embedding module;

[0029] The third processing module is used to input the feature vector and the image to be recognized into the target model to obtain the target recognition result output by the target model; wherein,

[0030] The embedding module is obtained by training based on the first sample data, and the target model is obtained by training based on the second sample data.

[0031] According to the image recognition device of the present invention, by acquiring an image to be recognized and recognition instruction information, inputting the recognition instruction information into an embedding module, acquiring a feature vector output by the embedding module, and then inputting the feature vector and the image to be recognized into a target model, and acquiring a target recognition result output by the target model, the input features of a specific recognition object input to the target model can be adjusted based on the embedding module obtained by training with a specific data set without changing the target model parameters, so as to identify accurate target recognition results using the original target model parameters, thereby improving the recognition accuracy and recall rate of the target model for specific recognition objects on the basis of retaining the generalization ability of the target model, thereby realizing the practical application of subsequent open-set target detection models.

[0032] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image recognition method as described in the first aspect above when executing the computer program.

[0033] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image recognition method as described in the first aspect above.

[0034] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the image recognition method as described in the first aspect above.

[0035] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0036] By obtaining an image to be identified and identification instruction information, inputting the identification instruction information into an embedding module, obtaining a feature vector output by the embedding module, and then inputting the feature vector and the image to be identified into a target model, and obtaining a target recognition result output by the target model, the input features of a specific recognition object input into the target model can be adjusted based on the embedding module trained by a specific data set without changing the target model parameters, so as to identify accurate target recognition results using the original target model parameters, thereby improving the recognition accuracy and recall rate of the target model for specific recognition objects while retaining the generalization ability of the target model, thereby realizing the practical application of subsequent open-set target detection models.

[0037] Furthermore, by training the embedding module based on the first prediction recognition result and the first sample label while controlling the model parameters of the target model to remain unchanged, it is possible to maintain the generalization ability of the target model while achieving the training of the embedding module that enables the target model to predict a specific data set with higher accuracy and recall.

[0038] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0040] Figure 1 This is one of the flowcharts of the image recognition method provided by the embodiment of the present invention;

[0041] Figure 2 This is the second flow chart of the image recognition method provided by the embodiment of the present invention;

[0042] Figure 3 This is a third flow chart of the image recognition method provided by an embodiment of the present invention;

[0043] Figure 4 is a schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention;

[0044] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0046] The terms "first", "second", etc. in the specification and claims of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0047] The image recognition method, image recognition device, electronic device and readable storage medium provided by the embodiments of the present invention are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0048] The image recognition method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0049] The terminal includes but is not limited to portable communication devices such as mobile phones or tablet computers. It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer.

[0050] The image recognition method provided in the embodiment of the present invention may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the image recognition method. The electronic devices mentioned in the embodiment of the present invention include but are not limited to mobile phones, tablet computers, computers, cameras and wearable devices. The image recognition method provided in the embodiment of the present invention is described below using the electronic device as an example of the execution subject.

[0051] like Figure 1 As shown, the image recognition method includes: step 110, step 120 and step 130.

[0052] Step 110: Obtain the image to be recognized and recognition instruction information;

[0053] In this step, the image to be identified may be an image containing any features.

[0054] For example, an image of a kitten sitting on a chair, an image of three lions crouching on a rock outcropping, or a class photo.

[0055] The recognition instruction information is text information customized by the user based on the image to be recognized, and the text information can be in the form of words or phrases.

[0056] For example, if the image to be identified is an image of a kitten sitting on a chair, the identification instruction information may be “detect the cat in the image”, “the chair in the image”, “find the cat’s tail” or “cat”, etc.

[0057] It should be noted that the category of the recognition instruction information is the same as the category of the text information of the specific data set involved in the training embedding module in the following embodiments.

[0058] Among them, the same category can be understood as that the recognition instruction information and the text information of the specific data set involved in the training embedding module exist in the same object; wherein, the object is the recognition object of the target model, and the object involved in the text information of the specific data set is the specific recognition object of the target model.

[0059] For example, the text information of a specific data set includes "cat on the chair", "person on the far left" and "mouse on the floor", among which "cat", "person", "mouse", "chair" or "floor" are specific recognition objects of the target model. Then, when the recognition instruction information includes specific recognition objects such as "cat", "person", "chair", "floor" or "mouse", the category of the recognition instruction information is considered to be the same as the category of the text information of the specific data set involved in the training embedding module.

[0060] It can be understood that the image to be identified is an image having features of a specific identification object, and the identification instruction information is text information related to the specific identification object.

[0061] Step 120: input the recognition instruction information into the embedding module to obtain the feature vector output by the embedding module;

[0062] like Figure 2 As shown, in this step, the embedding module (trainable nn.Embedding) is a module used to fine-tune the target model based on a specific dataset.

[0063] The specific data set is a text information data set related to one or more specific recognition objects.

[0064] The target model is an open set target detection model, for example, the Grounding DINO model.

[0065] Fine-tuning the target model based on a specific dataset means training the specific dataset based on the embedding module while keeping the target model parameters unchanged, so as to enhance the prediction accuracy and recall rate of the target model for specific recognition objects.

[0066] For example, when the original target model cannot recognize cats or has a poor recognition effect on cats, a specific data set related to cats can be trained based on the embedding module to enable the target model to recognize cats or improve the accuracy and recall of cat recognition results.

[0067] It is understandable that the embedding module can achieve supplementary training of the target model on a specific data set while keeping the model parameters of the target model unchanged, so that the target model can recognize a specific object or improve the prediction accuracy and recall rate of the specific object.

[0068] The specific implementation method of training a specific data set based on the embedding module will be described in the following embodiments, and the present invention will not be described in detail here.

[0069] The feature vector is a feature vector used to represent digital features of identification instruction information.

[0070] It can be understood that the embedding module can be used to extract features from the recognition instruction information.

[0071] In some embodiments, inputting the recognition instruction information into the embedding module and obtaining the feature vector output by the embedding module may include:

[0072] The recognition instruction information is input into the embedding module, and the feature vector output by the embedding module based on the Embedding (embedded model) technology is obtained.

[0073] In this embodiment, the Embedding technology is a technology that maps discrete high-dimensional data (such as recognition instruction information) to a low-dimensional continuous vector space (feature vector). During the mapping process, a vector (feature vector) composed of real numbers is generated for the computer system to capture the potential relationship and structure of the original data.

[0074] It can be understood that, in the actual execution process, the recognition instruction information can be mapped into a feature vector in the embedding module based on the Embedding technology.

[0075] According to the image recognition method provided by the embodiment of the present invention, by processing recognition instruction information based on Embedding technology to obtain feature vectors, high-dimensional discretized text information can be mapped into low-dimensional feature vectors, thereby realizing the quantification of the potential relationships and structural features of the text information, thereby facilitating the computer system to capture and compare the features of the text information.

[0076] Step 130: input the feature vector and the image to be recognized into the target model to obtain the target recognition result output by the target model; wherein the embedding module is obtained by training based on the first sample data, and the target model is obtained by training based on the second sample data.

[0077] In this embodiment, the target recognition result is a recognition result image of a specific recognition object by a target model.

[0078] In the actual execution process, the target recognition result is a recognition result image in which the image to be recognized is used as the background image and a specific recognition object is framed by an anchor frame in the image to be recognized.

[0079] For example, there are three kittens side by side in the image to be identified, and the identification instruction information is "detect the leftmost cat". Then the target recognition result can be a recognition result image of the leftmost cat in the image framed by the anchor frame, with the image to be identified as the background image.

[0080] The first sample data and the second sample data are different data samples.

[0081] In the actual execution process, the embedding module is obtained after the target model is trained based on the second sample data, the target model parameters are frozen (that is, the target model parameters are kept unchanged), and then the target model is trained based on the first sample data. The details will be explained in the following embodiments, and the present invention will not be elaborated here.

[0082] During the research process, the inventors found that in the related technology, the specific data set is usually expanded and the open set target detection model is trained to improve the accuracy and recall rate of the open set target detection model on the specific data set. However, expanding the specific data set will cause the parameters of the open set target detection model to change, thereby causing the open set target detection model to lose the general knowledge (generalization) learned in the pre-training stage, thereby limiting the subsequent practical application of the open set target detection model.

[0083] In the present invention, by obtaining an image to be identified and recognition instruction information of the same type as a specific data set, inputting the recognition instruction information into an embedding module, and obtaining a feature vector output by the embedding module, feature extraction of a specific recognition object in the recognition instruction information can be achieved; the feature vector and the image to be identified are then input into a target model to obtain a target recognition result output by the target model. Without changing the target model parameters, the input features of the specific recognition object input into the target model can be adjusted based on the embedding module trained by the specific data set, so as to identify accurate target recognition results using the original target model parameters, thereby improving the recognition accuracy and recall rate of the target model for the specific recognition object while retaining the generalization ability of the target model, thereby realizing the practical application of the subsequent open set target detection model.

[0084] According to the image recognition method provided by the embodiment of the present invention, by obtaining an image to be recognized and recognition instruction information, inputting the recognition instruction information into an embedding module, obtaining a feature vector output by the embedding module, and then inputting the feature vector and the image to be recognized into a target model, and obtaining a target recognition result output by the target model, the input features of a specific recognition object input to the target model can be adjusted based on the embedding module obtained by training with a specific data set without changing the target model parameters, so as to identify an accurate target recognition result using the original target model parameters, thereby improving the recognition accuracy and recall rate of the target model for a specific recognition object on the basis of retaining the generalization ability of the target model, thereby realizing the practical application of the subsequent open set target detection model.

[0085] The training process of the embedding module is described below.

[0086] In some embodiments, before acquiring the image to be recognized and the recognition instruction information, the method may further include:

[0087] Acquire first sample data, where the first sample data includes a first sample label;

[0088] Inputting the first sample label into the embedding module, and obtaining a feature vector output by the embedding module;

[0089] Under the condition that the model parameters of the target model are controlled to remain unchanged, the feature vector is input into the target model to obtain a first prediction recognition result output by the target model;

[0090] Based on the first predicted recognition result and the first sample label, an embedding module is trained.

[0091] In this embodiment, the first sample data is training sample data of the embedding module, that is, a specific data set.

[0092] The first sample data includes a first sample label.

[0093] The first sample label is a plurality of text information customized by the user based on the specific recognition object. The plurality of text information may be in the form of words or phrases, wherein the specific recognition object may be any object.

[0094] For example, the first sample label may include "there is a computer on the table", "a car parked on the side of the road", or "lion".

[0095] The first sample label is in the form of embedding (word embedding, mapping words into numerical vectors).

[0096] In some embodiments, the first sample label may be a pre-randomly generated embedding.

[0097] In other embodiments, the first sample label may also be an embedding obtained after feature extraction by a text feature extraction module (text backbone) of the target model.

[0098] In the actual implementation process, the first sample label can be understood as the supplementary text information training sample data of the target model.

[0099] The first predicted recognition result is a text information predicted recognition result obtained based on the first sample label prediction.

[0100] During the actual execution process, the target model can be trained in advance, and after the trained target model is obtained, the embedding module can be embedded into the target model; during the training process, the target model can receive the feature vector sent by the embedding module after processing the features of a specific recognition object, and predict the first predicted recognition result while controlling the model parameters of the target model to remain unchanged, thereby realizing the training of the embedding module based on the first predicted recognition result and the first sample label predicted by the target model.

[0101] It is understandable that, while keeping the model parameters of the target model unchanged, the target model can be supplemented with training for a specific data set based on the embedding module, thereby enabling the target model to recognize new specific recognition objects or improve the recognition accuracy and recall rate of the target model for specific recognition objects without affecting the original recognition ability of the target model.

[0102] According to the image recognition method provided by an embodiment of the present invention, by training the embedding module based on the first predicted recognition result and the first sample label while controlling the model parameters of the target model to remain unchanged, it is possible to maintain the generalization ability of the target model while achieving the training of the embedding module that enables the target model to predict a specific data set with higher accuracy and recall rate.

[0103] In some embodiments, training the embedding module based on the first predicted recognition result and the first sample label may include:

[0104] When the difference between the first predicted recognition result and the first sample label is greater than a target threshold, optimizing the embedding module and obtaining a feature vector output by the optimized embedding module;

[0105] Under the condition that the model parameters of the target model remain unchanged, the feature vector output by the optimized embedding module is input into the target model to obtain a new first prediction recognition result output by the target model;

[0106] When the difference between the first predicted recognition result and the first sample label is not greater than the target threshold, the optimization of the embedding module is stopped.

[0107] In this embodiment, the difference degree is the degree of difference between the first predicted recognition result and the first sample label.

[0108] The target threshold is a threshold used to measure the difference between the first predicted recognition result and the first sample label.

[0109] Target thresholds can be based on user-defined settings.

[0110] During the actual execution process, when the difference is greater than the target threshold, it is considered that the difference between the first predicted recognition result and the first sample label is large, and the embedding module training is not completed; when the difference is not greater than the target threshold, it is considered that the difference between the first predicted recognition result and the first sample label is small, and the embedding module training is completed.

[0111] The optimization embedding module can adjust the training parameters of the optimization embedding module.

[0112] For example, adjusting parameters such as the learning rate of the embedding module.

[0113] In the actual execution process, during the training process of the embedding module, the difference between the first predicted recognition result and the first sample label can be continuously judged, wherein, when the difference between the first predicted recognition result and the first sample label is greater than the target threshold, the learning rate and other parameters of the embedding module are adjusted to optimize the embedding module, thereby optimizing the feature vector output by the embedding module to optimize the first predicted recognition result output by the target model, so as to reduce the difference between the first predicted recognition result and the first sample label, and this cycle is repeated until the difference between the first predicted recognition result and the first sample label is not greater than the target threshold, then the optimization of the embedding module is stopped and the training of the embedding module is completed.

[0114] In some embodiments, the difference between the first predicted recognition result and the first sample label can be calculated based on the loss function.

[0115] In this embodiment, the loss function may be a function for calculating the gap between the predicted value and the true value, that is, a function for calculating the gap between the first predicted recognition result and the first sample label.

[0116] It can be understood that the difference between the first predicted recognition result and the first sample label can be reflected based on the loss function.

[0117] In the actual execution process, the difference between the first predicted recognition result and the first sample label is not greater than the target threshold, which can be manifested as the convergence of the loss function. That is, when the loss function converges, it is considered that the embedding module training is completed.

[0118] According to the image recognition method provided by the embodiment of the present invention, by judging the difference between the first predicted recognition result and the first sample label, whether the embedding module has been trained can be judged, so that the training process and training results can be made transparent, which is convenient for controlling and adjusting the training process; and based on continuously optimizing the embedding module, the first predicted recognition result can be made closer to the first sample label, that is, the predicted value can be made closer to the true value, thereby realizing the optimized training of the embedding module.

[0119] In some embodiments, before acquiring the image to be recognized and the recognition instruction information, the method may further include:

[0120] Acquire second sample data, where the second sample data includes a second sample image and a second sample label corresponding to the second sample image;

[0121] Based on the second sample image and the second sample label, the model parameters of the target model are trained.

[0122] In this embodiment, the second sample data is training sample data of the target model.

[0123] The second sample data may include a second sample image and a second sample label corresponding to the second sample image.

[0124] The second sample label is a plurality of text information customized by the user based on the second sample image, and the plurality of text information may be in the form of words or phrases.

[0125] For example, when the second sample image is an image of three lions squatting side by side in the grass, the second sample label may include "lion", "lion's ear", "lion's tail", "the leftmost lion" or "grass", etc.

[0126] During the actual execution process, the second sample image may be an image containing multiple object features, and the second sample label may be text information corresponding to the second sample image. It can be understood that based on the second sample image and the second sample label, the model parameters of the target model are trained to obtain a target model with good generalization ability (capable of identifying multiple objects).

[0127] It should be noted that the first sample label may be completely different from the object included in the text information of the second sample label, so that the first sample label can be used as a supplementary training sample for a new specific recognition object of the target model, and the target model can predict the new specific recognition object based on the embedding module; or it may be the same as part of the specific recognition objects included in the text information of the second sample label, so that the first sample label can be used as a supplementary training sample for the part of the specific recognition objects of the target model, and the prediction accuracy and recall rate of the target model for the part of the specific recognition objects can be improved based on the embedding module.

[0128] According to the image recognition method provided by an embodiment of the present invention, by training the model parameters of the target model based on the second sample image and the second sample label, a target model with good generalization ability can be obtained, thereby facilitating the subsequent recognition of non-specific identification objects based on the target model.

[0129] In some embodiments, after obtaining the image to be recognized and the recognition instruction information, the method may further include:

[0130] When the category of the recognition instruction information is different from the category of the first sample data, the recognition instruction information and the image to be recognized are input into the target model to obtain the target recognition result output by the target model.

[0131] In this embodiment, the category of the recognition instruction information is different from the category of the first sample data. It can be understood that the recognition instruction information and the text information included in the first sample data do not have the same object.

[0132] It can be understood that when the recognition instruction information and the text information included in the first sample data do not have the same object, the recognition instruction information and the image to be recognized can be input into the target model, and based on the generalization ability of the target model, the recognition of the image to be recognized based on the recognition instruction information can be achieved.

[0133] For example, Figure 2 and Figure 3 As shown, when the recognition instruction information includes objects such as "dog", "car" or "flower", and the text information included in the first sample data includes "cat", "person" or "mouse", the recognition instruction information and the image to be recognized can be input into the target model, and based on the generalization ability of the target model, the recognition of objects such as "dog", "car" or "flower" in the image to be recognized can be achieved.

[0134] According to the image recognition method provided by an embodiment of the present invention, by inputting recognition instruction information and an image to be recognized into a target model when the category of the recognition instruction information is different from the category of the first sample data, and obtaining a target recognition result output by the target model, recognition of non-specific recognition objects can be achieved based on the target model that retains generalization capabilities.

[0135] The image recognition method provided in the embodiment of the present invention may be executed by an image recognition device. In the embodiment of the present invention, an image recognition device executing the image recognition method is taken as an example to illustrate the image recognition device provided in the embodiment of the present invention.

[0136] The embodiment of the present invention also provides an image recognition device.

[0137] like Figure 4 As shown, the image recognition device includes: a first processing module 410 , a second processing module 420 and a third processing module 430 .

[0138] The first processing module 410 is used to obtain the image to be recognized and the recognition instruction information;

[0139] The second processing module 420 is used to input the recognition instruction information into the embedding module and obtain the feature vector output by the embedding module;

[0140] The third processing module 430 is used to input the feature vector and the image to be recognized into the target model to obtain the target recognition result output by the target model; wherein,

[0141] The embedding module is obtained by training based on the first sample data, and the target model is obtained by training based on the second sample data.

[0142] According to the image recognition device provided by the embodiment of the present invention, by obtaining an image to be recognized and recognition instruction information, inputting the recognition instruction information into an embedding module, obtaining a feature vector output by the embedding module, and then inputting the feature vector and the image to be recognized into a target model, and obtaining a target recognition result output by the target model, the input features of a specific recognition object input to the target model can be adjusted based on the embedding module trained by a specific data set without changing the target model parameters, so as to identify an accurate target recognition result using the original target model parameters, thereby improving the recognition accuracy and recall rate of the target model for specific recognition objects while retaining the generalization ability of the target model, thereby realizing the practical application of subsequent open-set target detection models.

[0143] In some embodiments, the second processing module 420 may also be used to:

[0144] The recognition instruction information is input into the embedding module, and the feature vector output by the embedding module based on the Embedding technology is obtained.

[0145] In some embodiments, the apparatus may further include a fourth processing module, configured to:

[0146] Acquire first sample data, where the first sample data includes a first sample label;

[0147] Inputting the first sample label into the embedding module, and obtaining a feature vector output by the embedding module;

[0148] Under the condition that the model parameters of the target model are controlled to remain unchanged, the feature vector is input into the target model to obtain a first prediction recognition result output by the target model;

[0149] Based on the first predicted recognition result and the first sample label, an embedding module is trained.

[0150] In some embodiments, the fourth processing module may also be used to:

[0151] When the difference between the first predicted recognition result and the first sample label is greater than a target threshold, optimizing the embedding module and obtaining a feature vector output by the optimized embedding module;

[0152] Under the condition that the model parameters of the target model remain unchanged, the feature vector output by the optimized embedding module is input into the target model to obtain a new first prediction recognition result output by the target model;

[0153] When the difference between the first predicted recognition result and the first sample label is not greater than the target threshold, the optimization of the embedding module is stopped.

[0154] In some embodiments, the apparatus may further include a fifth processing module, configured to:

[0155] Acquire second sample data, where the second sample data includes a second sample image and a second sample label corresponding to the second sample image;

[0156] Based on the second sample image and the second sample label, the model parameters of the target model are trained.

[0157] In some embodiments, the apparatus may further include a sixth processing module, configured to:

[0158] When the category of the recognition instruction information is different from the category of the first sample data, the recognition instruction information and the image to be recognized are input into the target model to obtain the target recognition result output by the target model.

[0159] The image recognition device in the embodiment of the present invention may be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than a terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It may also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present invention.

[0160] The image recognition device in the embodiment of the present invention may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present invention.

[0161] The image recognition device provided by the embodiment of the present invention can realize Figures 1 to 3 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0162] like Figure 5As shown, an embodiment of the present invention further provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, each process of the above-mentioned image recognition method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0163] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0164] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned image recognition method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0165] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0166] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the above-mentioned image recognition method when executed by a processor.

[0167] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0168] An embodiment of the present invention further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image recognition method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0169] It should be understood that the chip mentioned in the embodiment of the present invention may also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0170] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0172] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0173] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0174] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. An image recognition method, characterized in that: include: Obtaining the image to be recognized and recognition instruction information; Inputting the recognition instruction information into an embedding module to obtain a feature vector output by the embedding module; The feature vector and the image to be recognized are input into the target model to obtain the target recognition result output by the target model; wherein, The embedding module is obtained by training based on the first sample data, and the target model is obtained by training based on the second sample data; Before acquiring the image to be recognized and the recognition instruction information, the method further includes: Acquire the first sample data, where the first sample data includes a first sample label; Inputting the first sample label into the embedding module to obtain a feature vector output by the embedding module; Under the condition that the model parameters of the target model are controlled to remain unchanged, the feature vector is input into the target model to obtain a first prediction recognition result output by the target model; The embedding module is trained based on the first predicted recognition result and the first sample label.

2. The image recognition method according to claim 1, characterized in that: The step of training the embedding module based on the first predicted recognition result and the first sample label includes: When the difference between the first predicted recognition result and the first sample label is greater than a target threshold, optimizing the embedding module and obtaining a feature vector output by the optimized embedding module; Under the condition that the model parameters of the target model are controlled to remain unchanged, the feature vector output by the optimized embedding module is input into the target model to obtain a new first prediction recognition result output by the target model; When the difference between the first predicted recognition result and the first sample label is not greater than a target threshold, the optimization of the embedding module is stopped.

3. The image recognition method according to claim 1 or 2, characterized in that: Before acquiring the image to be recognized and the recognition instruction information, the method further includes: Acquire the second sample data, where the second sample data includes a second sample image and a second sample label corresponding to the second sample image; Based on the second sample image and the second sample label, the model parameters of the target model are trained.

4. The image recognition method according to claim 1 or 2, characterized in that: The step of inputting the recognition instruction information into an embedding module and obtaining a feature vector output by the embedding module includes: The recognition instruction information is input into the embedding module, and the feature vector output by the embedding module after processing the recognition instruction information based on the Embedding technology is obtained.

5. The image recognition method according to claim 1 or 2, characterized in that: After obtaining the image to be recognized and the recognition instruction information, the method further includes: When the category of the recognition instruction information is different from the category of the first sample data, the recognition instruction information and the image to be recognized are input into the target model to obtain the target recognition result output by the target model.

6. An image recognition device, characterized in that: include: A first processing module, used for acquiring an image to be recognized and recognition instruction information; A second processing module, used for inputting the recognition instruction information into an embedding module, and obtaining a feature vector output by the embedding module; The third processing module is used to input the feature vector and the image to be recognized into the target model to obtain the target recognition result output by the target model; wherein, The embedding module is obtained by training based on the first sample data, and the target model is obtained by training based on the second sample data; A fourth processing module, configured to obtain the first sample data before obtaining the image to be recognized and the recognition instruction information, wherein the first sample data includes a first sample label; Inputting the first sample label into the embedding module to obtain a feature vector output by the embedding module; Under the condition that the model parameters of the target model are controlled to remain unchanged, the feature vector is input into the target model to obtain a first prediction recognition result output by the target model; The embedding module is trained based on the first predicted recognition result and the first sample label.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image recognition method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image recognition method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image recognition method according to any one of claims 1 to 5 is implemented.

Images