A training method for an image recognition model and an image recognition method

Abstract

This application discloses a training method for an image recognition model and an image recognition method. The training method for the image recognition model includes: acquiring image data and auxiliary information for the image data; generating visual prototype data based on the image data and the auxiliary information; acquiring training samples and specific visual prototype data corresponding to the training samples; training an original image recognition model using the training samples; determining training loss data generated during training based on the specific visual prototype data; and training until the training loss data meets preset conditions to obtain a trained image recognition model. The image recognition method includes: acquiring a target image to be recognized; recognizing the target image based on the trained image recognition model to obtain a recognition result. Using this method, the interference of noisy images in the training samples on the training of the image recognition model is suppressed, improving the learning efficiency and accuracy of the image recognition model.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method, apparatus, and device for training an image recognition model. This application also relates to an image recognition method, apparatus, and device. This application further relates to an image recognition system. Background Technology

[0002] Image recognition currently has wide applications, and neural networks are a commonly used image recognition model. Generally, image data is used as training samples to train the original neural network, resulting in a trained neural network. An image containing the content to be recognized is then used as input to the trained neural network to identify relevant information about the content. The quality and quantity of training samples are crucial for model training. In practice, text tags can be used to search for image data using search engines, which can meet the requirement of a sufficient number of training samples. However, the searched image data often contains a lot of noisy images, which can interfere with model training. Current technology involves manually labeling the searched image data to reduce noise in the training samples. However, manual labeling is inefficient and costly.

[0003] Therefore, how to suppress the interference of noisy images in the training samples on the training of image recognition models is a problem that needs to be solved. Summary of the Invention

[0004] The image recognition model training method and image recognition method provided in this application embodiment solve the interference of noisy images in the training samples on the training of the image recognition model, and improve the learning efficiency and accuracy of the image recognition model.

[0005] This application provides an image recognition method, including:

[0006] A trained image recognition model is obtained. The trained image recognition model is trained by using training samples to train the original image recognition model. During the training process, training loss data generated during training is determined based on specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets preset conditions. The specific visual prototype data is visual prototype data generated based on image data and auxiliary information of the image data.

[0007] Acquire the target image to be identified;

[0008] Based on the trained image recognition model, the target image is recognized to obtain the recognition result.

[0009] Optionally, it also includes: generating the visual prototype data based on the matching degree between the image data and the auxiliary information of the image data.

[0010] Optionally, generating the visual prototype data based on the matching degree between the image data and the auxiliary information of the image data includes:

[0011] Based on the auxiliary information of the image data, a specific type of text classification relationship is generated;

[0012] Generate the specific type of visual classification relationship based on the image data;

[0013] The matching degree between the text classification relationship and the visual classification relationship is determined by a graph matching algorithm, and visual prototype data corresponding to the specific type is generated.

[0014] Optionally, generating a specific type of text classification relationship based on the auxiliary information of the image data includes:

[0015] Extract the specific type of text description information from the auxiliary information;

[0016] Extract text vectors from the text description information;

[0017] The specific type of text classification relationship is generated based on the similarity between the text vectors.

[0018] Optionally, generating the specific type of visual classification relationship based on the image data includes:

[0019] The image data is classified using a neural network classification model to obtain the classification result;

[0020] Image data whose confidence scores for a specific type in the classification results meet a preset confidence threshold are used as image data to generate the visual prototype data corresponding to that specific type.

[0021] Based on the image data that participated in generating the visual prototype data corresponding to the specific type, a visual classification relationship corresponding to the specific type is generated.

[0022] Optionally, the step of determining the matching degree between the visual classification relationship and the text classification relationship using a graph matching algorithm, and generating visual prototype data corresponding to the specific type, includes:

[0023] The similarity between the visual classification relationship and the text classification relationship of the specific type is determined by the relative entropy matching algorithm, and is used as the matching degree.

[0024] The weights of the image features are determined based on the matching degree, and the visual prototype data is generated based on the weights and the image features; wherein, the image features are the features of the image data that participate in generating the visual prototype data.

[0025] Optionally, determining the training loss data generated during training based on the specific visual prototype data includes:

[0026] Determine the similarity between the training samples and the specific visual prototype data;

[0027] The similarity is used as the noise weight of the training sample, and the noise weight is used with the cross-entropy loss function to form a weighted cross-entropy loss function;

[0028] The training loss data generated in each training session is calculated based on the weighted cross-entropy loss function.

[0029] Optionally, determining the similarity between the training sample and the specific visual prototype data includes: calculating the distance value between the training sample and the specific visual prototype data, and using the distance value as the similarity.

[0030] This application also provides a method for training an image recognition model, including:

[0031] Acquire image data and auxiliary information of the image data, and generate visual prototype data based on the image data and the auxiliary information;

[0032] Acquire training samples and the specific visual prototype data corresponding to the training samples;

[0033] The original image recognition model is trained using the training samples. During the training process, the training loss data generated during training is determined based on the specific visual prototype data, and the training continues until the training loss data meets the preset conditions, thus obtaining the trained image recognition model.

[0034] Optionally, generating visual prototype data based on the image data and the auxiliary information includes: generating the visual prototype data based on the matching degree between the image data and the auxiliary information of the image data.

[0035] Optionally, generating the visual prototype data based on the matching degree between the image data and the auxiliary information of the image data includes:

[0036] Based on the auxiliary information of the image data, a specific type of text classification relationship is generated;

[0037] Generate the specific type of visual classification relationship based on the image data;

[0038] The matching degree between the text classification relationship and the visual classification relationship is determined by a graph matching algorithm, and visual prototype data corresponding to the specific type is generated.

[0039] Optionally, generating a specific type of text classification relationship based on the auxiliary information of the image data includes:

[0040] Extract the specific type of text description information from the auxiliary information;

[0041] Extract text vectors from the text description information;

[0042] The specific type of text classification relationship is generated based on the similarity between the text vectors.

[0043] Optionally, generating the specific type of visual classification relationship based on the image data includes:

[0044] The image data is classified using a neural network classification model to obtain the classification result;

[0045] Image data whose confidence scores for a specific type in the classification results meet a preset confidence threshold are used as image data to generate the visual prototype data corresponding to that specific type.

[0046] Based on the image data that participated in generating the visual prototype data corresponding to the specific type, a visual classification relationship corresponding to the specific type is generated.

[0047] Optionally, the step of determining the matching degree between the visual classification relationship and the text classification relationship using a graph matching algorithm, and generating visual prototype data corresponding to the specific type, includes:

[0048] The similarity between the visual classification relationship and the text classification relationship of the specific type is determined by the relative entropy matching algorithm, and is used as the matching degree.

[0049] The weights of the image features are determined based on the matching degree, and the visual prototype data is generated based on the weights and the image features; wherein, the image features are the features of the image data that participate in generating the visual prototype data.

[0050] Optionally, determining the training loss data generated during training based on the specific visual prototype data includes:

[0051] Determine the similarity between the training samples and the specific visual prototype data;

[0052] The similarity is used as the noise weight of the training sample, and the noise weight is used with the cross-entropy loss function to form a weighted cross-entropy loss function;

[0053] The training loss data generated in each training session is calculated based on the weighted cross-entropy loss function.

[0054] Optionally, determining the similarity between the training sample and the specific visual prototype data includes: calculating the distance value between the training sample and the specific visual prototype data, and using the distance value as the similarity.

[0055] Optionally, the auxiliary information includes at least one of the following: hierarchical semantic information corresponding to the image, search ranking information of the image, search source information of the image, description information of the image, tag information of the image, and title information of the image.

[0056] Optionally, generating visual prototype data based on the image data and the auxiliary information includes:

[0057] The image data and / or the auxiliary information are categorized to obtain: image types; and the image data and auxiliary information included in each image type;

[0058] Based on the image data and auxiliary information included in each image type, generate visual prototype data corresponding to each image type;

[0059] The step of obtaining the specific visual prototype data corresponding to the training samples includes:

[0060] Determine the image type matched by the training sample, and use the visual prototype data corresponding to the image type as the specific visual prototype data corresponding to the training sample.

[0061] This application also provides an image recognition system, including: a client and a server;

[0062] The client is configured to acquire a target image to be identified, send the target image to the server, receive and display the recognition result information for the target image provided by the server.

[0063] The server is configured to receive the target image to be recognized sent by the client; acquire a trained image recognition model, wherein the trained image recognition model is trained using training samples to train the original image recognition model, and during the training process, training loss data generated during training is determined based on specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets preset conditions; wherein the specific visual prototype data is visual prototype data generated based on image data and auxiliary information of the image data; and recognize the target image according to the trained image recognition model, obtain the recognition result, and send it to the client.

[0064] Optionally, the server determines the image recognition service level to be provided to the client based on the user level of the logged-in user accessing the server, and recognizes the target image based on the image recognition service level and the trained image recognition model.

[0065] This application also provides an image recognition device, including:

[0066] The model acquisition unit is used to acquire the trained image recognition model. The trained image recognition model is obtained by training the original image recognition model using training samples. During the training process, the training loss data generated during training is determined based on the specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets the preset conditions. The specific visual prototype data is visual prototype data generated based on image data and auxiliary information of the image data.

[0067] The input unit is used to acquire the target image to be identified;

[0068] The recognition unit is used to recognize the target image based on the trained image recognition model and obtain the recognition result.

[0069] This application embodiment also provides a training device for an image recognition model, comprising:

[0070] A visual prototype generation unit is used to acquire image data and auxiliary information of the image data, and generate visual prototype data based on the image data and the auxiliary information.

[0071] The training data acquisition unit is used to acquire training samples and specific visual prototype data corresponding to the training samples.

[0072] The training unit is used to train the original image recognition model using the training samples. During the training process, it determines the training loss data generated during training based on the specific visual prototype data, and trains until the training loss data meets the preset conditions to obtain the trained image recognition model.

[0073] This application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program, which, when run by the processor, executes the method provided in this application.

[0074] This application also provides a storage device storing a computer program, which, when run by a processor, executes the method provided in this application.

[0075] This application also provides an image recognition method, including:

[0076] An image recognition model is obtained by training an original image recognition model using training samples. During the training process, training loss data generated during training is determined based on specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets preset conditions. The specific visual prototype data is visual prototype data generated based on image data and auxiliary information of the image data.

[0077] Acquire the target image to be identified;

[0078] The target image is identified according to the image recognition model to obtain the recognition result.

[0079] Compared with the prior art, this application has the following advantages:

[0080] This application provides an image recognition method, apparatus, and device that recognizes a target image based on a trained image recognition model to obtain a recognition result. The trained image recognition model is trained using training samples. During training, training loss data is determined based on specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets preset conditions. The specific visual prototype data is generated based on image data and auxiliary information of the image data. Generating visual prototype data based on auxiliary information of the image data and using it to calculate the training loss data for each training iteration allows for noise filtering of the image data. This eliminates the need for pre-annotation of training samples, suppresses interference from noisy images in the training samples, and improves model learning efficiency and recognition accuracy.

[0081] This application provides a training method, apparatus, and device for an image recognition model. It generates visual prototype data based on image data and auxiliary information from the image data. During the training of the original image recognition model using training samples, training loss data generated during training is determined based on the specific visual prototype data, and training continues until the training loss data meets preset conditions, resulting in a trained image recognition model. Generating visual prototype data based on auxiliary information from the image data and using it to calculate the training loss data for each training iteration allows for noise filtering of the image data. This suppresses the interference of noisy images in the training samples on training, improving model learning efficiency and model recognition accuracy.

[0082] This application provides an image recognition system, comprising: a client and a server; the client sends a target image to be recognized to the server; receives and displays recognition result information for the target image provided by the server; the server receives the target image to be recognized; recognizes the target image according to a trained image recognition model, obtains the recognition result, and sends it to the client; the trained image recognition model trains an original image recognition model using training samples, and during the training process, determines training loss data based on specific visual prototype data corresponding to the training samples, and trains until the training loss data meets preset conditions; wherein, the specific visual prototype data is visual prototype data generated based on image data and auxiliary information of the image data. By recognizing and / or searching images containing content to be recognized, relevant information about the content to be recognized in the image is obtained, providing users with a convenient way to obtain information. Attached Figure Description

[0083] Figure 1 This is a flowchart of the image recognition processing for an application scenario provided in the embodiments of this application;

[0084] Figure 2 This is a schematic diagram of a system for training an image recognition model provided in an embodiment of this application;

[0085] Figure 3 This is a flowchart of an image recognition method provided in the first embodiment of this application;

[0086] Figure 4 This is a flowchart illustrating a training method for an image recognition model provided in the second embodiment of this application.

[0087] Figure 5 This is a schematic diagram of the image auxiliary information included in the method provided in the embodiments of this application;

[0088] Figure 6 This is a flowchart illustrating the training process of the image recognition model included in the embodiments of this application;

[0089] Figure 7 This is a schematic diagram of an image recognition system provided in the third embodiment of this application;

[0090] Figure 8 This is a schematic diagram of an image recognition device provided in the fourth embodiment of this application;

[0091] Figure 9 This is a schematic diagram of a training device for an image recognition model provided in the fifth embodiment of this application;

[0092] Figure 10 This is a schematic diagram of the electronic device provided in this application. Detailed Implementation

[0093] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0094] This application provides an image recognition method, apparatus, electronic device, and storage device. It also provides a method, apparatus, electronic device, and storage device for training an image recognition model. Furthermore, it provides an image recognition system. These will be described in detail in the following embodiments.

[0095] To facilitate understanding, the application scenarios of the method provided in this application embodiment are first given. The training method of the image recognition model uses images with side information as training samples to train a trained image recognition model. The use of side information reduces the interference of noisy images in the training samples on the model training. Using the image recognition method, an image containing the content to be recognized is used as input to the trained image recognition model, accurately identifying the content to be recognized in the image. The image recognition model is a neural network model, preferably a Convolutional Neural Network (CNN). In practical applications, it can be used, but is not limited to, scenarios such as object recognition from photos and image search. In this embodiment, object recognition from photos refers to capturing or scanning an image containing an unknown object, identifying relevant information about the unknown object using image recognition technology, and outputting it. Image search refers to capturing or scanning a target object of interest using a camera, identifying keywords corresponding to the target object in the image or video using image recognition technology, and searching for and outputting other relevant information about the target object based on the keywords. For example, by using a smartphone's image search function client to take a picture of clothes of interest with the camera, websites where the clothes can be purchased can be found. Please refer to... Figure 1The figure illustrates an image recognition processing flow for an application scenario, including: S101, the client captures or scans an image containing the object to be recognized. The client can be an application software (APP) installed on the terminal or a mini-program on the terminal. For example, a browser installed on a smartphone that supports image search is an example of a client. The user triggers the corresponding page element of the browser interface's photo capture or scan function to capture or scan the object to be recognized, obtaining an image of the object to be recognized. S102, the client sends the image containing the object to be recognized to the server for recognition and / or search. S103, the server obtains the trained image recognition model. S104, the server uses the image as input to the image recognition model and outputs recognition result information. S105, the client obtains the recognition result information and displays it. For example, in the example in the figure, the recognition and search results are related to the Australian Silky Terrier. S106, the client displays the recognition result information.

[0096] Please refer to this again. Figure 2 The figure shows a system for training the image recognition model, comprising: an image capture module 201 for capturing image data with auxiliary information, including noisy images. In practical applications, image data with auxiliary information can be captured from a search engine based on text tags. A visual prototype generation module 202 for generating visual prototype data from the captured image data and the auxiliary information contained therein. A training module 203 for selecting training samples from the image data, obtaining visual prototype data according to the type of the training samples, constructing noise weights using the similarity between the visual prototype data and the training samples, constructing a weighted cross-entropy loss function using the noise weights and the cross-entropy loss function, and training the image recognition model until the training loss data of the weighted cross-entropy loss function meets preset conditions, thus obtaining the trained image recognition model.

[0097] Example 1 is combined with the following Figure 3 The image recognition method provided in the first embodiment of this application will be described. Figure 3 The image recognition method shown includes steps S301 to S303.

[0098] Step S301: Obtain the trained image recognition model. The trained image recognition model is obtained by training the original image recognition model using training samples. During the training process, the training loss data generated during training is determined based on the specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets the preset conditions. The specific visual prototype data is visual prototype data generated based on the image data and the auxiliary information of the image data.

[0099] In this embodiment, the image recognition model is a neural network model, preferably a convolutional neural network. Training samples for training the image recognition model can be obtained by crawling image data with auxiliary information from a search engine using text tags. The crawled image data, without manual annotation, generally contains a large amount of noisy images. Noisy images refer to images whose actual content does not match the keywords entered during the semantic query, but whose tags match the image's tags, thus resulting in the search results. (See reference...) Figure 2 Image capture results captured by the image capture module 101: Based on "phalarope" (a type of bird), among the multiple images retrieved, some are correctly labeled images as shown in "Correct Label", while others are incorrectly labeled noisy images as shown in "Noise Label".

[0100] The auxiliary information refers to information beyond the image data used to describe the image, which can be textual information or other prior knowledge. It includes at least one of the following: hierarchical semantic information (WordNet) corresponding to the image, the image's search ranking information (rank), the image's search source information (source), the image's description information (description), the image's tags information (tags), and the image's title information (title). This auxiliary information can be used to suppress the interference of noisy images on the image recognition model.

[0101] In this embodiment, the original image recognition model is trained using captured image data with auxiliary information. Specifically, this includes: generating visual prototype data based on the matching degree between the image data and the auxiliary information of the image data; determining training loss data based on the specific visual prototype data corresponding to the training samples during the training process; training the original image recognition model until the training loss data meets preset conditions, thus obtaining a trained image recognition model. The process from the original image recognition model to the trained image recognition model can be divided into two stages: the generation stage of visual prototype data and the model training stage.

[0102] In this embodiment, during the visual prototype data generation stage, a Visual based Class Realization Graph and a Textual based Class Realization Graph are constructed based on the image data and auxiliary information of the images, respectively. Then, a Visual Prototype for each type is constructed based on the Graph Matching algorithm. Specifically, the following processes are included: generating a specific type of text classification relationship based on the auxiliary information of the image data; generating the specific type of visual classification relationship based on the image data; determining the matching degree between the text classification relationship and the visual classification relationship using the graph matching algorithm, and generating the visual prototype data corresponding to the specific type.

[0103] The step of generating a specific type of text classification relationship based on the auxiliary information of the image data includes: extracting text description information of the specific type from the auxiliary information; extracting text vectors from the text description information; and generating the specific type of text classification relationship based on the similarity relationship between the text vectors. For example, text descriptions of each type can be extracted from the hierarchical semantic information (WordNet) corresponding to the image data, text vectors can be extracted using BERT (Bidirectional Encoder Representation from Transformers), and text classification relationships can be constructed based on the similarity relationship between the text vectors.

[0104] The step of generating the specific type of visual classification relationship based on the image data includes: classifying the image data using a neural network classification model to obtain classification results; determining image data in the classification results whose confidence scores for a specific type meet a preset confidence threshold, and using these as image data to participate in generating the visual prototype data corresponding to the specific type; and generating the visual classification relationship corresponding to the specific type based on the image data used to generate the visual prototype data corresponding to the specific type. For example, a neural network classification model can be trained using image data, and the K images with the highest probability scores from each category can be extracted to construct the visual classification relationship.

[0105] The step of determining the matching degree between the visual classification relationship and the text classification relationship using a graph matching algorithm to generate visual prototype data corresponding to the specific type includes: determining the similarity between the visual classification relationship and the text classification relationship of the specific type using a relative entropy matching algorithm, as the matching degree; determining the weights of image features based on the matching degree; and generating the visual prototype data based on the weights and the image features; wherein the image features are the features of the image data participating in the generation of the visual prototype data. The relative entropy matching algorithm is also known as the Kullback-Leibler divergence matching algorithm. Graph matching algorithms determine the matching degree between image data and auxiliary information of image data. A higher matching degree indicates more reliable image data, and the greater the weight of the image features of that image data when constructing visual prototype data, that is, the higher the contribution of the image features to the visual prototype data. Therefore, using visual prototype data to construct a loss function for training can suppress interference from noisy images with low reliability.

[0106] In this embodiment, during the model training phase, the image recognition model is trained based on Weighted Cross Entropy (WCE). The visual prototype data is used to construct the weights of the WCE, thereby reducing the interference of noisy images on model training. WCE introduces weights into the cross-entropy loss; specifically, each image data used as a training sample is multiplied by a weight, and then the training loss is calculated. The specific steps for determining the training loss data generated in each training session include: determining the similarity between the training sample and the specific visual prototype data; using the similarity as the noise weight of the training sample; using the noise weight and the cross-entropy loss function to construct the WCE weighted cross-entropy loss function; and calculating the training loss data generated in each training session based on the WCE weighted cross-entropy loss function. Specifically, determining the similarity between the training sample and the specific visual prototype data includes: calculating the distance value between the training sample and the specific visual prototype data, and using the distance value as the similarity.

[0107] Step S302: Obtain the target image to be identified.

[0108] In this embodiment, the target image can be captured or scanned by a shooting device, or selected and acquired from a device storing target images. The shooting device can be a camera; or various devices with cameras, such as cameras, mobile phones, tablets, and visual robots. It can capture or scan objects that need to be identified or searched, such as unknown plants, animals, people, goods, cars, etc. For example, it can capture the target image of a trademark or a product of unknown purpose. In practice, the target image can be acquired by driving the camera through a client installed on the shooting device, such as an application software (APP) or a mini-program on a terminal. For example, a browser with image search functionality installed on a smartphone is an example of a client. The user triggers the corresponding photo or scan function page element on the browser interface to capture or scan the object to be identified, thus obtaining the target image.

[0109] Step S303: Based on the trained image recognition model, the target image is recognized to obtain the recognition result.

[0110] This embodiment also includes: outputting the recognition result information and displaying it to the user. Please refer to [further details]. Figure 1 The captured image is of a "dog", and the recognition result is "Australian Silky Terrier". It can also output descriptive information of "Australian Silky Terrier" and display it to the user on the client.

[0111] It should be noted that the methods provided in this embodiment and other embodiments of this application are not limited to image recognition and image search scenarios, but are also applicable to other image recognition scenarios. Unless otherwise specified, the features given in this embodiment and other embodiments of this application can be combined with each other, and steps S301 and S302 or similar terms do not limit the order in which the steps must be performed.

[0112] The method provided in the first embodiment has now been described. This method identifies a target image using a trained image recognition model to obtain a recognition result. The trained image recognition model is trained using training samples. During training, training loss data is determined based on specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets preset conditions. The specific visual prototype data is generated based on image data and auxiliary information from the image data. Generating visual prototype data based on auxiliary information from the image data and using it to calculate the training loss data for each training iteration allows for noise filtering of the image data. This eliminates the need for pre-annotation of training samples, suppresses interference from noisy images in the training samples, and thus improves model learning efficiency and recognition accuracy.

[0113] Example 2: Based on the above examples, the second example of this application provides a training method for an image recognition model, which is described below in conjunction with... Figures 4 to 6 The method is described below. Figure 4 The training method for the image recognition model shown includes steps S401 to S403.

[0114] Step S401: Obtain image data and auxiliary information of the image data, and generate visual prototype data based on the image data and the auxiliary information.

[0115] In this embodiment, the image recognition model is a neural network model, preferably a convolutional neural network. The training samples for training the image recognition model can be obtained by using text tags to crawl image data with auxiliary information from a search engine. The crawled image data, without manual annotation, generally contains a large amount of noisy images. Noisy images refer to images whose actual content does not match the keywords entered during the semantic query, but whose tags match the image's labels, thus resulting in a search result.

[0116] The auxiliary information (Side Information) refers to information describing the image beyond the image data itself, which can be textual information or other prior knowledge. It includes at least one of the following: hierarchical semantic information (WordNet) corresponding to the image, image search ranking information (rank), image search source information (source), image description information (description), image tag information (tags), and image title information (title). This auxiliary information can be used to suppress the interference of noisy images on the image recognition model. The WordNet is hierarchical semantic information, for example, animal -> mammal -> feline -> tiger -> Siberian tiger. Please refer to [reference needed]. Figure 5 The figure provides an example of auxiliary information. The auxiliary information corresponding to the image in the figure includes:

[0117] Source, meaning the source of the image search is an online search engine;

[0118] Search Rank: 1, meaning the image is ranked 1st in the search results;

[0119] Description: Four Australian Silky Terrier pups brought to the studio today;

[0120] Tags: Australian Silky Terrier Puppies Olympus E500;

[0121] Title: Australian Terrier Pups

[0122] The hierarchical semantic information (WordNet) of this image is entity -> physical entity -> terrie -> Australian Silky Terrier.

[0123] In this embodiment, the original image recognition model is trained using captured image data with auxiliary information. Specifically, this includes: generating visual prototype data based on the matching degree between the image data and the auxiliary information of the image data; determining training loss data based on the specific visual prototype data corresponding to the training samples during the training process; training the original image recognition model until the training loss data meets preset conditions, thus obtaining a trained image recognition model. The process from the original image recognition model to the trained image recognition model can be divided into two stages: the generation stage of visual prototype data and the model training stage.

[0124] In this embodiment, during the visual prototype data generation stage, the visual prototype data is generated based on the matching degree between the image data and the auxiliary information of the image data. Specifically, this includes the following processes:

[0125] Based on the auxiliary information of the image data, a specific type of text classification relationship is generated;

[0126] Generate the specific type of visual classification relationship based on the image data;

[0127] The matching degree between the text classification relationship and the visual classification relationship is determined by a graph matching algorithm, and visual prototype data corresponding to the specific type is generated.

[0128] It should be noted that the order in which specific types of text classification relationships are generated and specific types of visual classification relationships are generated is not restricted.

[0129] The step of generating a specific type of text classification relationship based on the auxiliary information of the image data includes: extracting text description information of the specific type from the auxiliary information; extracting text vectors from the text description information; and generating the specific type of text classification relationship based on the similarity relationship between the text vectors. For example, text descriptions of each type can be extracted from the hierarchical semantic information (WordNet) corresponding to the image data, text vectors can be extracted using BERT (Bidirectional Encoder Representation from Transformers), and text classification relationships can be constructed based on the similarity relationship between the text vectors.

[0130] The step of generating the specific type of visual classification relationship based on the image data includes: classifying the image data using a neural network classification model to obtain classification results; determining image data in the classification results whose confidence scores for a specific type meet a preset confidence threshold, and using these as image data to participate in generating the visual prototype data corresponding to the specific type; and generating the visual classification relationship corresponding to the specific type based on the image data used to generate the visual prototype data corresponding to the specific type. For example, a neural network classification model can be trained using image data, and the K images with the highest probability scores from each category can be extracted to construct the visual classification relationship.

[0131] The step of determining the matching degree between the visual classification relationship and the text classification relationship using a graph matching algorithm to generate visual prototype data corresponding to the specific type includes: determining the similarity between the visual classification relationship and the text classification relationship of the specific type using a relative entropy matching algorithm, as the matching degree; determining the weights of image features based on the matching degree; and generating the visual prototype data based on the weights and the image features; wherein the image features are the features of the image data participating in the generation of the visual prototype data. The relative entropy matching algorithm is also known as the Kullback-Leibler divergence matching algorithm. Graph matching algorithms determine the matching degree between image data and auxiliary information of image data. A higher matching degree indicates more reliable image data, and the greater the weight of the image features of that image data when constructing visual prototype data, that is, the higher the contribution of the image features to the visual prototype data. Therefore, using visual prototype data to construct a loss function for training can suppress interference from noisy images with low reliability.

[0132] Please refer to Figure 6The diagram includes a visual prototype generation phase 601 and a training phase 602. The visual prototype generation phase includes the following steps: inputting specific type image data 602-1 into a visual encoder for processing; this specific type of image data consists of multiple images retrieved from a search engine using specific text tags; the information processed by the visual encoder is then processed by an inter-class similarity (ICS) intermediate layer to generate a visual classification relationship 602-2 (i.e., establishing a visual-based class relation graph); inputting auxiliary information 602-3 corresponding to the specific type of image data into a textual encoder for processing, such as the descriptive information shown in the diagram: a short-haired tailless cat originating from the Isle of Man; the information processed by the textual encoder is then processed by an inter-class similarity (ICS) intermediate layer to generate a textual classification relationship 602-4 (i.e., establishing a textual-based class relation graph). The visual classification relationship and the text classification relationship are processed by a graph matching algorithm to generate the visual prototype data 602-5 (i.e., Category Visual Prototype) of the specific type. The visual prototype data is a matrix obtained by weighting the image features of each image data in the specific type. The weight of the image feature of each image data is determined by the matching degree between the visual classification relationship and the text classification relationship of the image data.

[0133] In this embodiment, visual prototype data is generated through the following process: classifying the image data and / or the auxiliary information to obtain: image types; and image data and auxiliary information included in each image type; generating visual prototype data corresponding to each image type based on the image data and auxiliary information included in each image type; further, obtaining specific visual prototype data corresponding to the training sample includes: determining the image type matched by the training sample, and using the visual prototype data corresponding to the image type as the specific visual prototype data corresponding to the training sample.

[0134] Step S402: Obtain training samples and specific visual prototype data corresponding to the training samples.

[0135] In this embodiment, during the model training phase, training samples and corresponding specific visual prototype data are acquired, and the original image recognition model is trained. Specifically, the image type matching the image data in the training samples is determined, and the visual prototype data corresponding to the image type is used as the specific visual prototype data corresponding to the training samples.

[0136] Step S403: Use the training samples to train the original image recognition model. During the training process, determine the training loss data generated during training based on the specific visual prototype data, and train until the training loss data meets the preset conditions to obtain the trained image recognition model.

[0137] In this embodiment, an image recognition model is trained based on Weighted Cross Entropy (WCE). The visual prototype data is used to construct the weights of the WCE, thereby reducing the interference of noisy images on model training. WCE introduces weights into the cross-entropy loss; specifically, each image data used as a training sample is multiplied by a weight, and then the training loss is calculated. The step of determining the training loss data generated during training based on the specific visual prototype data includes:

[0138] Determine the similarity between the training samples and the specific visual prototype data;

[0139] The similarity is used as the noise weight of the training sample, and the noise weight is used with the cross-entropy loss function to form a weighted cross-entropy loss function;

[0140] The training loss data generated in each training session is calculated based on the weighted cross-entropy loss function.

[0141] Specifically, determining the similarity between the training sample and the specific visual prototype data includes: calculating the distance value between the training sample and the specific visual prototype data, and using the distance value as the similarity.

[0142] Please refer to this again. Figure 6 The training phase 602 process in the diagram includes: inputting image data used as training samples into a visual encoder for processing; processing the encoded information by a query embedding layer; and inputting the information processed by the query embedding layer into a fully connected layer (FC layer), using the output information of the FC layer to construct a cross-entropy loss function H(y = c|x). The information processed by the query embedding layer is also used to calculate the similarity between the image features of the training samples and the visual prototype data, specifically using L2 distance to represent the similarity.

[0143] d = ||g(.) - v k (.)‖2);

[0144] Among them, v k (.) represents the vector constituting the visual prototype data; g(.) represents the features of the training sample images. The L2 distance is used to construct the noise weight: The weighted cross-entropy loss function is then obtained from the noise weights and the cross-entropy loss function: Loss = w * H(c|x).

[0145] The method provided in this embodiment has been described above. This method generates visual prototype data based on image data and auxiliary information of the image data. During the training of the original image recognition model using training samples, training loss data generated during training is determined based on the specific visual prototype data, and training continues until the training loss data meets preset conditions, resulting in a trained image recognition model. Generating visual prototype data based on auxiliary information of the image data and using it to calculate the training loss data for each training iteration allows for noise filtering of the image data. This suppresses the interference of noisy images in the training samples on training, improving model learning efficiency and model recognition accuracy.

[0146] Based on the above embodiments, the third embodiment of this application provides an image recognition system, which is described below in conjunction with... Figure 7 The system will be described. Figure 7 The image recognition system shown includes: a client 701 and a server 702;

[0147] The client is configured to acquire a target image to be identified, send the target image to the server, receive and display the recognition result information for the target image provided by the server.

[0148] The server is configured to receive the target image to be recognized sent by the client; acquire a trained image recognition model, wherein the trained image recognition model is trained using training samples to train the original image recognition model, and during the training process, training loss data generated during training is determined based on specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets preset conditions; wherein the specific visual prototype data is visual prototype data generated based on image data and auxiliary information of the image data; and recognize the target image according to the trained image recognition model, obtain the recognition result, and send it to the client.

[0149] In this embodiment, the server provides image recognition services. The client can access the image recognition services provided by the server. The client triggers a capture or scan of the object to be recognized to obtain the target image, or triggers the selection of the target image from a device storing the target image. The object to be recognized can be an animal, plant, commodity, car, etc., or it can be a trademark of a commodity, a commodity of unknown purpose, etc. The client sends the obtained target image to the server, and obtains the recognition result information of the target image through the image recognition service provided by the server based on the image recognition model. In practical applications, the client can be a device with camera and / or scanning functions, or a device storing image information, such as a mobile phone, camera, camcorder, computer, laptop, tablet, PAD, portable terminal, VR device, all-in-one device, fax machine, smart home device, etc. The server can be a device providing image recognition services in the cloud, or it can be a non-cloud device. The server can be a server independently deploying the image recognition service, or it can be a server with a distributed deployment of the image recognition service. In a distributed deployment, at least two recognition servers and a recognition result decision server are required to provide image recognition using the image recognition model. The two recognition servers send their respective recognition results to the recognition result decision server, which determines the final recognition result and provides the recognition result information to the client; or, the recognition result decision server further searches for relevant information based on the recognition result and provides the recognition result and related information to the client.

[0150] In this embodiment, the server determines the image recognition service level to be provided to the client based on the user level of the logged-in user accessing the server, and recognizes the target image based on the image recognition service level and the trained image recognition model. In one embodiment, the recognition accuracy is determined based on the user level, and the target image is recognized according to the trained image recognition model based on the recognition accuracy. In another embodiment, the types of objects that can be recognized by the user level are determined, and the target image is recognized using the trained image recognition model based on the object types. For example, if the logged-in user's user level is a regular user and the image recognition service for plant species is enabled, then only the plant species contained in the target image are recognized. As another example, if the logged-in user's user level is an advanced user and the image recognition service for plant and animal species is enabled, then the plant species and / or animal species contained in the target image are recognized. In one embodiment, the recognition priority corresponding to the user level is determined, and the target image is recognized using the trained image recognition model based on the recognition priority. Therefore, image recognition services can be provided to users with higher recognition priority during peak recognition periods.

[0151] In this embodiment, the interaction between the client and the server includes: the client sending an image containing the object to be identified to the server for identification and / or searching; the server obtaining the trained image recognition model; the server using the image as input to the image recognition model and outputting recognition result information; and the client obtaining and displaying the recognition result information.

[0152] This concludes the description of the system provided in this embodiment. The system obtains relevant information about the content to be identified in an image by recognizing and / or searching for such content, providing users with a convenient way to obtain information.

[0153] Embodiment 4 corresponds to Embodiment 1. This fourth embodiment of the application provides an image recognition device, which is described below in conjunction with... Figure 8 The device will be described. Figure 8 The image recognition device shown includes:

[0154] The model acquisition unit 801 is used to acquire a trained image recognition model. The trained image recognition model is obtained by training the original image recognition model using training samples. During the training process, the training loss data generated during training is determined based on the specific visual prototype data corresponding to the training samples, and the training continues until the training loss data meets the preset conditions. The specific visual prototype data is visual prototype data generated based on image data and auxiliary information of the image data.

[0155] Input unit 802 is used to acquire the target image to be identified;

[0156] The recognition unit 803 is used to recognize the target image according to the trained image recognition model and obtain the recognition result.

[0157] Optionally, the device further includes a training unit, which is used to generate the visual prototype data based on the matching degree between the image data and the auxiliary information of the image data.

[0158] Optionally, the training unit is specifically used to: generate a specific type of text classification relationship based on the auxiliary information of the image data; generate the specific type of visual classification relationship based on the image data; determine the matching degree between the text classification relationship and the visual classification relationship through a graph matching algorithm, and generate visual prototype data corresponding to the specific type.

[0159] Optionally, the training unit is specifically used to: extract the specific type of text description information from the auxiliary information; extract text vectors from the text description information; and generate the specific type of text classification relationship based on the similarity relationship between the text vectors.

[0160] Optionally, the training unit is specifically used to: classify the image data using a neural network classification model to obtain classification results; determine image data whose confidence scores for a specific type in the classification results meet a preset confidence threshold, and use them as image data to participate in generating visual prototype data corresponding to the specific type; and generate a visual classification relationship corresponding to the specific type based on the image data participating in generating the visual prototype data corresponding to the specific type.

[0161] Optionally, the training unit is specifically used to: determine the similarity between the visual classification relationship and the text classification relationship of the specific type through a relative entropy matching algorithm, as the matching degree; determine the weight of the image features according to the matching degree; and generate the visual prototype data according to the weight and the image features; wherein, the image features are the features of the image data that participate in generating the visual prototype data.

[0162] Optionally, the training unit is specifically used to: determine the similarity between the training sample and the specific visual prototype data; use the similarity as the noise weight of the training sample, and use the noise weight and the cross-entropy loss function to form a weighted cross-entropy loss function; and calculate the training loss data generated for each training session based on the weighted cross-entropy loss function.

[0163] Optionally, the training unit is specifically used to: calculate the distance value between the training sample and the specific visual prototype data, and use the distance value as the similarity.

[0164] In actual deployment, the aforementioned units can be deployed on the same entity or on different entities. For example, when deployed separately, the input unit can be deployed on the client side, and the model acquisition unit and recognition unit can be deployed on the server side. The client sends the obtained target image to the server side, and obtains the recognition result information of the target image through the image recognition service provided by the server side based on the image recognition model. The client can be a device with camera and / or scanning functions, or a device storing image information, such as a mobile phone, camera, camcorder, computer, laptop, tablet, PAD, portable terminal, VR device, all-in-one device, fax machine, smart home device, etc. The server side can be a device providing image recognition services in the cloud or a non-cloud device. The server side can be a server that independently deploys the image recognition service or a server that distributes the image recognition service. In a distributed deployment, at least two recognition servers and a recognition result decision server are required to provide image recognition using the image recognition model. The two recognition servers send their respective recognition results to the recognition result decision server, which determines the final recognition result and provides the recognition result information to the client; or, the recognition result decision server further searches for relevant information based on the recognition result and provides the recognition result and related information to the client.

[0165] Example 5 corresponds to Example 2. This fifth embodiment of the application provides a training device for an image recognition model, which is described below in conjunction with... Figure 9 The device will be described. Figure 9 The training apparatus for the image recognition model shown includes:

[0166] Visual prototype generation unit 901 is used to acquire image data and auxiliary information of the image data, and generate visual prototype data based on the image data and the auxiliary information.

[0167] The training data acquisition unit 902 is used to acquire training samples and specific visual prototype data corresponding to the training samples;

[0168] The training unit 903 is used to train the original image recognition model using the training samples. During the training process, it determines the training loss data generated during training based on the specific visual prototype data, and trains until the training loss data meets the preset conditions to obtain the trained image recognition model.

[0169] Optionally, the visual prototype generation unit 901 is specifically used to generate the visual prototype data based on the matching degree between the image data and the auxiliary information of the image data.

[0170] Optionally, the visual prototype generation unit 901 is specifically used to: generate a specific type of text classification relationship based on the auxiliary information of the image data; generate the specific type of visual classification relationship based on the image data; determine the matching degree between the text classification relationship and the visual classification relationship through a graph matching algorithm, and generate visual prototype data corresponding to the specific type.

[0171] Optionally, the visual prototype generation unit 901 is specifically used to: extract the specific type of text description information from the auxiliary information; extract text vectors from the text description information; and generate the specific type of text classification relationship based on the similarity relationship between the text vectors.

[0172] Optionally, the visual prototype generation unit 901 is specifically used to: classify the image data using a neural network classification model to obtain a classification result; determine image data whose confidence scores for a specific type in the classification result meet a preset confidence threshold, and use them as image data to participate in generating visual prototype data corresponding to the specific type; and generate a visual classification relationship corresponding to the specific type based on the image data participating in generating the visual prototype data corresponding to the specific type.

[0173] Optionally, the visual prototype generation unit 901 is specifically used to: determine the similarity between the visual classification relationship and the text classification relationship of the specific type through a relative entropy matching algorithm, as the matching degree; determine the weight of the image features according to the matching degree; and generate the visual prototype data according to the weight and the image features; wherein, the image features are the features of the image data that participate in generating the visual prototype data.

[0174] Optionally, the training unit 903 is specifically used to: determine the similarity between the training sample and the specific visual prototype data; use the similarity as the noise weight of the training sample, and use the noise weight and the cross-entropy loss function to form a weighted cross-entropy loss function; and calculate the training loss data generated for each training session based on the weighted cross-entropy loss function.

[0175] Optionally, the training unit 903 is specifically used to: calculate the distance value between the training sample and the specific visual prototype data, and use the distance value as the similarity.

[0176] Optionally, the auxiliary information includes at least one of the following: hierarchical semantic information corresponding to the image, search ranking information of the image, search source information of the image, description information of the image, tag information of the image, and title information of the image.

[0177] Optionally, the visual prototype generation unit 901 is specifically used to: classify the image data and / or the auxiliary information to obtain: image types; and image data and auxiliary information included in each image type; and generate visual prototype data corresponding to each image type based on the image data and auxiliary information included in each image type; the training data acquisition unit 902 is specifically used to: determine the image type matched by the training sample, and use the visual prototype data corresponding to the image type as the specific visual prototype data corresponding to the training sample.

[0178] Based on the above embodiments, the sixth embodiment of this application provides an electronic device. Figure 10 This is a schematic diagram of the electronic device, which includes: a memory 1001 and a processor 1002; the memory is used to store a computer program, which, after being run by the processor, executes the method provided in the embodiments of this application.

[0179] Based on the above embodiments, the seventh embodiment of this application provides a storage device. For relevant parts, please refer to the corresponding descriptions in the above embodiments. The schematic diagram of the storage device is similar. Figure 10 The storage device stores a computer program, which, when executed by a processor, performs the method described in the embodiments of this application.

[0180] Based on the above embodiments, the eighth embodiment of this application provides an image recognition method, including:

[0181] An image recognition model is obtained by training an original image recognition model using training samples. During the training process, training loss data generated during training is determined based on specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets preset conditions. The specific visual prototype data is visual prototype data generated based on image data and auxiliary information of the image data.

[0182] Acquire the target image to be identified;

[0183] The target image is identified according to the image recognition model to obtain the recognition result.

[0184] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0185] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0186] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0187] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0188] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. An image recognition method, characterized in that, include: A trained image recognition model is obtained. The trained image recognition model is trained using training samples. During the training process, training loss data is determined based on specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets preset conditions. The specific visual prototype data is generated based on image data and auxiliary information of the image data. The visual prototype data is used to filter noise from the image data. The auxiliary information includes hierarchical semantic information. The visual prototype data is generated based on the matching degree between a specific type of text classification relationship and a visual classification relationship generated based on the image data and the hierarchical semantic information. Acquire the target image to be identified; Based on the trained image recognition model, the target image is recognized to obtain the recognition result.

2. The method according to claim 1, characterized in that, Also includes: The visual prototype data is generated based on the matching degree between the image data and the auxiliary information of the image data.

3. The method according to claim 2, characterized in that, The generation of the visual prototype data based on the matching degree between the image data and the auxiliary information of the image data includes: Based on the auxiliary information of the image data, a specific type of text classification relationship is generated; Generate the specific type of visual classification relationship based on the image data; The matching degree between the text classification relationship and the visual classification relationship is determined by a graph matching algorithm, and visual prototype data corresponding to the specific type is generated.

4. The method according to claim 3, characterized in that, The step of generating a specific type of text classification relationship based on the auxiliary information of the image data includes: Extract the specific type of text description information from the auxiliary information; Extract text vectors from the text description information; The specific type of text classification relationship is generated based on the similarity between the text vectors.

5. The method according to claim 4, characterized in that, The step of generating the specific type of visual classification relationship based on the image data includes: The image data is classified using a neural network classification model to obtain the classification result; Image data whose confidence scores for a specific type in the classification results meet a preset confidence threshold are used as image data to generate the visual prototype data corresponding to that specific type. Based on the image data that participated in generating the visual prototype data corresponding to the specific type, a visual classification relationship corresponding to the specific type is generated.

6. The method according to claim 5, characterized in that, The step of determining the matching degree between the visual classification relationship and the text classification relationship using a graph matching algorithm, and generating visual prototype data corresponding to the specific type, includes: The similarity between the visual classification relationship and the text classification relationship of the specific type is determined by the relative entropy matching algorithm, and is used as the matching degree. The weights of the image features are determined based on the matching degree, and the visual prototype data is generated based on the weights and the image features; wherein, the image features are the features of the image data that participate in generating the visual prototype data.

7. The method according to claim 1, characterized in that, The step of determining the training loss data generated during training based on the specific visual prototype data includes: Determine the similarity between the training samples and the specific visual prototype data; The similarity is used as the noise weight of the training sample, and the noise weight is used with the cross-entropy loss function to form a weighted cross-entropy loss function; The training loss data generated in each training session is calculated based on the weighted cross-entropy loss function.

8. The method according to claim 7, characterized in that, Determining the similarity between the training samples and the specific visual prototype data includes: Calculate the distance value between the training sample and the specific visual prototype data, and use the distance value as the similarity.

9. A method for training an image recognition model, characterized in that, include: Acquire image data and auxiliary information of the image data, generate visual prototype data based on the image data and the auxiliary information, the visual prototype data is used to filter noise from the image data, the auxiliary information includes hierarchical semantic information, and the visual prototype data is generated based on the matching degree between a specific type of text classification relationship and a visual classification relationship generated based on the image data and the hierarchical semantic information. Acquire training samples and the specific visual prototype data corresponding to the training samples; The original image recognition model is trained using the training samples. During the training process, the training loss data generated during training is determined based on the specific visual prototype data, and the training continues until the training loss data meets the preset conditions, thus obtaining the trained image recognition model.

10. The method according to claim 9, characterized in that, The step of generating visual prototype data based on the image data and the auxiliary information includes: The visual prototype data is generated based on the matching degree between the image data and the auxiliary information of the image data.

11. The method according to claim 10, characterized in that, The generation of the visual prototype data based on the matching degree between the image data and the auxiliary information of the image data includes: Based on the auxiliary information of the image data, a specific type of text classification relationship is generated; Generate the specific type of visual classification relationship based on the image data; The matching degree between the text classification relationship and the visual classification relationship is determined by a graph matching algorithm, and visual prototype data corresponding to the specific type is generated.

12. The method according to claim 11, characterized in that, The step of generating a specific type of text classification relationship based on the auxiliary information of the image data includes: Extract the specific type of text description information from the auxiliary information; Extract text vectors from the text description information; The specific type of text classification relationship is generated based on the similarity between the text vectors.

13. The method according to claim 12, characterized in that, The step of generating the specific type of visual classification relationship based on the image data includes: The image data is classified using a neural network classification model to obtain the classification result; Image data whose confidence scores for a specific type in the classification results meet a preset confidence threshold are used as image data to generate the visual prototype data corresponding to that specific type. Based on the image data that participated in generating the visual prototype data corresponding to the specific type, a visual classification relationship corresponding to the specific type is generated.

14. The method according to claim 13, characterized in that, The step of determining the matching degree between the visual classification relationship and the text classification relationship using a graph matching algorithm, and generating visual prototype data corresponding to the specific type, includes: The similarity between the visual classification relationship and the text classification relationship of the specific type is determined by the relative entropy matching algorithm, and is used as the matching degree. The weights of the image features are determined based on the matching degree, and the visual prototype data is generated based on the weights and the image features; wherein, the image features are the features of the image data that participate in generating the visual prototype data.

15. The method according to claim 9, characterized in that, The step of determining the training loss data generated during training based on the specific visual prototype data includes: Determine the similarity between the training samples and the specific visual prototype data; The similarity is used as the noise weight of the training sample, and the noise weight is used with the cross-entropy loss function to form a weighted cross-entropy loss function; The training loss data generated in each training session is calculated based on the weighted cross-entropy loss function.

16. The method according to claim 15, characterized in that, Determining the similarity between the training samples and the specific visual prototype data includes: Calculate the distance value between the training sample and the specific visual prototype data, and use the distance value as the similarity.

17. The method according to claim 9, characterized in that, The auxiliary information includes at least one of the following: hierarchical semantic information corresponding to the image, search ranking information of the image, search source information of the image, description information of the image, tag information of the image, and title information of the image.

18. The method according to claim 9, characterized in that, The step of generating visual prototype data based on the image data and the auxiliary information includes: The image data and / or the auxiliary information are categorized to obtain: image types; and the image data and auxiliary information included in each image type; Based on the image data and auxiliary information included in each image type, generate visual prototype data corresponding to each image type; The step of obtaining the specific visual prototype data corresponding to the training samples includes: Determine the image type matched by the training sample, and use the visual prototype data corresponding to the image type as the specific visual prototype data corresponding to the training sample.

19. An image recognition system, characterized in that, include: Client and server; The client is configured to acquire a target image to be identified, send the target image to the server, receive and display the recognition result information for the target image provided by the server. The server is used to receive the target image to be identified sent by the client; acquire the trained image recognition model, which is trained by using training samples to train the original image recognition model. During the training process, the training loss data generated during training is determined based on the specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets preset conditions. The specific visual prototype data is generated based on image data and auxiliary information of the image data. The visual prototype data is used to filter noise from the image data. The auxiliary information includes hierarchical semantic information. The visual prototype data is generated based on the matching degree between a specific type of text classification relationship and a visual classification relationship generated by the image data and the hierarchical semantic information. The target image is identified using the trained image recognition model, and the recognition result is sent to the client.

20. The system according to claim 19, characterized in that, The server determines the image recognition service level to be provided to the client based on the user level of the logged-in user accessing the server, and recognizes the target image based on the image recognition service level and the trained image recognition model.

21. An image recognition device, characterized in that, include: The model acquisition unit is used to acquire the trained image recognition model. The trained image recognition model is obtained by training the original image recognition model using training samples. During the training process, the training loss data generated during training is determined based on the specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets the preset conditions. The specific visual prototype data is visual prototype data generated based on image data and auxiliary information of the image data. An input unit is used to acquire a target image to be identified. The visual prototype data is used to filter noise from the image data. The auxiliary information includes hierarchical semantic information. The visual prototype data is generated based on the image data and the hierarchical semantic information, and is generated by the matching degree between a specific type of text classification relationship and a visual classification relationship. The recognition unit is used to recognize the target image based on the trained image recognition model and obtain the recognition result.

22. A training device for an image recognition model, characterized in that, include: A visual prototype generation unit is used to acquire image data and auxiliary information of the image data, generate visual prototype data based on the image data and the auxiliary information, and use the visual prototype data to filter noise from the image data. The auxiliary information includes hierarchical semantic information. The visual prototype data is generated based on the matching degree between a specific type of text classification relationship and a visual classification relationship generated based on the image data and the hierarchical semantic information. The training data acquisition unit is used to acquire training samples and specific visual prototype data corresponding to the training samples. The training unit is used to train the original image recognition model using the training samples. During the training process, it determines the training loss data generated during training based on the specific visual prototype data, and trains until the training loss data meets the preset conditions to obtain the trained image recognition model.

23. An electronic device, characterized in that, include: Memory, and processor; the memory is used to store a computer program, the computer program being processed by the processor. After running, the method described in any one of claims 1-18 shall be executed.

24. A storage device, characterized in that, The device contains a computer program, which, when executed by a processor, performs the method described in any one of claims 1-18.

25. An image recognition method, characterized in that, include: An image recognition model is obtained by training an original image recognition model using training samples. During the training process, training loss data is determined based on specific visual prototype data corresponding to the training samples, and training continues until the training loss data meets preset conditions. The specific visual prototype data is generated based on image data and auxiliary information of the image data. The visual prototype data is used to filter noise from the image data. The auxiliary information includes hierarchical semantic information. The visual prototype data is generated based on the matching degree between a specific type of text classification relationship and a visual classification relationship, based on the image data and the hierarchical semantic information. Acquire the target image to be identified; The target image is identified according to the image recognition model to obtain the recognition result.

Images