Image recognition method and device, storage medium and electronic equipment

Abstract

The invention relates to an image recognition method and device, a storage medium and electronic equipment. The method comprises the steps that after a to-be-recognized target image is obtained, the target image and a label set are input into an image recognition model, the image recognition model is a model trained by using a sample image, a text description of the sample image and a label description of a label of the sample image, the label set comprises a plurality of labels, and the label set comprises a plurality of labels; the labels of the sample images belong to a plurality of labels; obtaining a target label of a target image output by the image recognition model, wherein the target label belongs to a plurality of labels; and determining the target label as an entity contained in the target image. According to the invention, the technical problem that the model in the prior art is poor in recognition accuracy of an air mirror image is solved.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and in particular to an image recognition method, device, storage medium and electronic device. Background Art

[0002] In the existing technology, in the empty shot algorithm recognition business, there are many empty shot pictures with romantic colors, such as beautiful scenes after snow and magnificent palaces. How to distinguish these romantic pictures from ordinary snowy days and ordinary palaces so that the model can understand the meaning of romance is a difficult problem that needs to be solved urgently, because what we really need to classify and reuse are these empty shots with romantic elements.

[0003] Traditional algorithms do not have the ability to recognize such "atmosphere-type" empty scenes. How to make the model understand the meaning of romance is the key to identifying high-quality romantic empty scenes. Therefore, it is necessary to propose a model that can recognize empty scenes containing romantic elements. However, the existing technology cannot accurately distinguish between empty scenes and ordinary scenes, and the recognition accuracy is poor. Summary of the invention

[0004] The present application provides an image recognition method, device, storage medium and electronic device to solve the technical problem of poor recognition accuracy of the model in the prior art for the empty mirror image.

[0005] In a first aspect, the present application provides an image recognition method, comprising: after acquiring a target image to be identified, inputting the target image and a label set into an image recognition model, wherein the image recognition model is a model trained using a sample image, a text description of the sample image, and a label description of the label of the sample image, the label set includes multiple labels, and the label of the sample image belongs to the multiple labels; acquiring a target label of the target image output by the image recognition model, and the target label belongs to the multiple labels; and determining the target label as an entity contained in the target image.

[0006] In a second aspect, the present application provides an image recognition device, comprising: an input module, for inputting the target image and a label set into an image recognition model after acquiring a target image to be identified, wherein the image recognition model is a model trained using a sample image, a text description of the sample image, and a label description of the label of the sample image, the label set includes multiple labels, and the label of the sample image belongs to the multiple labels; an acquisition module, for acquiring a target label of the target image output by the image recognition model, and the target label belongs to the multiple labels; a determination module, for determining the target label as an entity contained in the target image.

[0007] As an optional example, the above-mentioned device also includes: a training module, which is used to obtain the above-mentioned sample image, the text description of the above-mentioned sample image and the label description of the above-mentioned sample image before obtaining the above-mentioned target image; extract the image features of the above-mentioned sample image, the text features of the above-mentioned text description and the label features of the above-mentioned label description; fuse the above-mentioned image features with the above-mentioned text features to obtain a first fused feature; fuse the above-mentioned image features with the above-mentioned label features to obtain a second fused feature; and use the above-mentioned first fused feature and the above-mentioned second fused feature to train the above-mentioned image recognition model.

[0008] As an optional example, the above-mentioned training module includes: a first training unit, used to randomly block some features of the above-mentioned image features to obtain the blocked image features; randomly block some features of the above-mentioned text features to obtain the blocked text features; and fuse the blocked image features with the blocked text features to obtain the above-mentioned first fused features.

[0009] As an optional example, the above-mentioned training module includes: a second training unit, used to randomly block some features of the above-mentioned image features to obtain the blocked image features; randomly block some features of the above-mentioned label features to obtain the blocked label features; and fuse the blocked image features with the blocked label features to obtain the above-mentioned second fused features.

[0010] As an optional example, the training module includes: a calculation unit for calculating a first loss of the first fused feature and the label of the text description; calculating a second loss of the second fused feature and the label of the label description; performing a weighted sum of the first loss and the second loss to obtain a target loss; and adjusting the image recognition model according to the target loss.

[0011] As an optional example, the above-mentioned device also includes: an identification module, which is used to, after acquiring the target image to be identified, input the above-mentioned target image and the label set into the image recognition model, extract the image features of the above-mentioned target image and the label features of each label in the above-mentioned label set; fuse the above-mentioned image features with the label features of each label to obtain the target fusion features, wherein each label corresponds to one of the above-mentioned target fusion features; identify each of the above-mentioned target fusion features through the above-mentioned image recognition model to obtain the above-mentioned target label of the above-mentioned target image.

[0012] As an optional example, the determination module includes: a classification unit, configured to classify the target image according to the number and types of entities contained in the target image.

[0013] In a third aspect, the present application provides an electronic device comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the memory stores a computer program, and the processor is configured to implement any one of the above-mentioned image recognition methods when executing the computer program.

[0014] In a fourth aspect, the present application further provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute any of the above-mentioned image recognition methods of the present application.

[0015] The above-mentioned technical scheme provided by the embodiment of the present application has the following advantages over the prior art: the scheme provided by the embodiment of the present application, after acquiring the target image to be identified, inputs the above-mentioned target image and the label set into the image recognition model, wherein the above-mentioned image recognition model is a model trained using a sample image, a text description of the above-mentioned sample image, and a label description of the label of the above-mentioned sample image, the above-mentioned label set includes multiple labels, and the label of the above-mentioned sample image belongs to the above-mentioned multiple labels; the target label of the above-mentioned target image output by the above-mentioned image recognition model is obtained, and the above-mentioned target label belongs to the above-mentioned multiple labels; the above-mentioned target label is determined as the entity contained in the above-mentioned target image, so that when training the image recognition model, the model can be trained by sample images, text descriptions of sample images and label descriptions of sample images, thereby improving the model's perception ability of the text descriptions and label descriptions of sample images, and the trained image recognition model has higher recognition accuracy for empty mirror images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0019] Figure 1 A flowchart of an image recognition method provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of another image recognition method provided in an embodiment of the present application;

[0021] Figure 3 A flowchart of another image recognition method provided in an embodiment of the present application;

[0022] Figure 4 A flowchart of another image recognition method provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0026] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0027] In order to solve the technical problem in the prior art that the existing model has poor recognition accuracy for empty mirror images, the present application provides an image recognition method, which can improve the recognition accuracy of the image recognition model for empty mirror images.

[0028] Figure 1 A flowchart of an image recognition method provided in an embodiment of the present application. Figure 1 As shown, the above-mentioned image recognition method includes:

[0029] S102, after acquiring a target image to be recognized, inputting the target image and the label set into an image recognition model, wherein the image recognition model is a model trained using a sample image, a text description of the sample image, and a label description of a label of the sample image, the label set includes a plurality of labels, and the label of the sample image belongs to the plurality of labels;

[0030] S104, obtaining a target label of the target image output by the image recognition model, where the target label belongs to the multiple labels;

[0031] S106: Determine the target tag as an entity included in the target image.

[0032] The empty shot mentioned in this embodiment is a shot that does not contain any characters, and is mainly used to show natural scenes or object details without involving characters in the plot. Empty shots can be divided into two types: scenery and objects. Scenery empty shots usually use long-range or panoramic shots to introduce the environment in which the story takes place; object empty shots use close-ups and other small shots to describe the details of objects. This shooting technique is often used at the beginning of a film to explain the general environment in which the story takes place, to create an atmosphere, and to set the tone of the film. In addition, empty shots can also be used in the middle of a film as a transition when the topic changes, or to express the emotions of characters from the side. It should be noted that characters can also be included in the empty shot picture, but the characters are not the main components in the empty shot picture, but the content that serves as a side foil.

[0033] The above-mentioned image recognition method of this embodiment can be applied in the process of identifying entities contained in an image, or in the process of classifying a video, or in the process of identifying whether it is an empty space image.

[0034] The above-mentioned target image is the image to be identified. The image to be identified can be a batch uploaded image or a video frame captured from a video. If it is a batch uploaded image, each image of the batch uploaded image is identified. If it is a video frame in a video, a video frame can be captured from the video for identification. The entity in the image is identified by identifying the image, or it is identified whether the image is an empty space image, and the video is classified by identifying the video frame.

[0035] The above-mentioned image recognition model is a model used to recognize the target image. The image recognition model is pre-trained with sample images, and the recognition accuracy is higher than a certain value. The image recognition model is used to recognize the type of the target image and output the target label of the target image.

[0036] When training an image recognition model, it is necessary to input not only sample images, but also text descriptions and label descriptions of the sample images. The text description of the sample image is the text describing the content contained in the sample image, and the label description is the explanation of the entity included in the sample image. For example, the entity is a tree, a house, a well, a snowflake, etc. The label description is used to describe the content of the entity, such as a house: a building with walls, a roof, doors, windows, for people to live in or for other purposes. The label description is used to represent the content of the label.

[0037] The trained image recognition model can input a target image, recognize the target image, and output the target labels contained in the target image, identifying each label as an entity contained in the target image.

[0038] For example, the above-mentioned image recognition method is applied in the process of classifying videos. After obtaining the video to be classified, this embodiment can extract the key frames of the video. Extracting the key frames of the video can reduce the number of images to be recognized. Then, each key frame is input into the image recognition model, and the image recognition model outputs the target label of the key frame. The label identifies the entity contained in the key frame, thereby determining which entities are contained in the key frame. The key frames are classified by the entities, so that the classification result of the entire video is determined according to the classification result of each key frame.

[0039] The solution provided by the embodiment of the present application, after acquiring the target image to be identified, inputs the target image and the label set into the image recognition model, wherein the image recognition model is a model trained using a sample image, a text description of the sample image, and a label description of the label of the sample image, the label set includes multiple labels, and the label of the sample image belongs to the multiple labels; the target label of the target image output by the image recognition model is obtained, and the target label belongs to the multiple labels; the target label is determined as an entity contained in the target image, so that when training the image recognition model, the model can be trained through sample images, text descriptions of sample images, and label descriptions of sample images, thereby improving the model's perception of the text descriptions and label descriptions of sample images, and the trained image recognition model has higher recognition accuracy for empty mirror images.

[0040] As an alternative example, Figure 2 As shown, before acquiring the target image, the method further includes:

[0041] S202, obtaining the sample image, a text description of the sample image, and a label description of the sample image;

[0042] S204, extracting image features of the sample image, text features of the text description, and label features of the label description;

[0043] S206, fusing the image feature with the text feature to obtain a first fused feature;

[0044] S208, fusing the image feature with the label feature to obtain a second fused feature;

[0045] S210: Use the first fusion feature and the second fusion feature to train the image recognition model.

[0046] In this embodiment, before acquiring the target image, the image recognition model needs to be trained. The image recognition model can be trained using a batch of sample images, in which each sample image has a text description for describing the image of the sample image, and also includes a label description for describing the meaning of the label of the sample image.

[0047] When training the image recognition model, the image recognition model can obtain the image features of the sample image, the text features of the text description, and the label features of the label description. After obtaining the three features, the image features can be fused with the above text features to obtain a first fused feature, and the image features can be fused with the above label features to obtain a second fused feature, and the first fused feature and the second fused feature can be used to train the image recognition model.

[0048] In one example, the sample images can be divided into different batches. For the sample images of the same batch, the first fusion feature and the second fusion feature of each sample image are obtained, and the image recognition model is trained. After the image recognition model is trained with the sample images of the current batch, the image recognition model can be further trained using the sample images of the next batch.

[0049] As an optional example, the above-mentioned image features are fused with the above-mentioned text features to obtain the first fused features, including: randomly masking some features of the above-mentioned image features to obtain masked image features; randomly masking some features of the above-mentioned text features to obtain masked text features; and fusing the masked image features with the masked text features to obtain the above-mentioned first fused features.

[0050] In this embodiment, when some features in the image features are blocked, the main purpose is to focus the image recognition model on important features through the attention mechanism. By blocking some features in the image features and text features, and then fusing them to obtain the first fused features, and training the image recognition model, the image recognition model can learn the impact of different areas of blocked image features and text features on the recognition results, so that which features are more important can be identified, and the weight of these features will be higher, while for less important features, the weight will be lower.

[0051] When masking the image features and text features as described above, some of the features may be randomly masked. As another example, for image features, it may be determined whether to mask or the range of feature masking based on the pixel values ​​of the pixels in the image. The features corresponding to the pixels of a specific color in the pixel values ​​may be used as the features to be masked, thereby obtaining the masked image features. For text features, it may be determined whether to mask based on the part of speech, such as nouns, verbs, etc. By determining whether to mask based on the part of speech, the features corresponding to some meaningless or unimportant words may be masked.

[0052] As an optional example, the above-mentioned image features are fused with the above-mentioned label features to obtain the second fused features, including: randomly occluding some features in the above-mentioned image features to obtain the occluded image features; randomly occluding some features in the above-mentioned label features to obtain the occluded label features; and fusing the occluded image features with the occluded label features to obtain the above-mentioned second fused features.

[0053] In this embodiment, when some features in the image features are blocked, the main purpose is to focus the image recognition model on important features through the attention mechanism. By blocking some features in the image features and label features, and then fusing them to obtain the second fused features, and training the image recognition model, the image recognition model can learn the impact of different areas of blocked image features and label features on the recognition results, so that which features are more important can be identified, and the weight of these features will be higher, while for less important features, the weight will be lower.

[0054] When masking the image features and label features as described above, some of the features may be randomly masked. As another example, for image features, it may be determined whether to mask or the range of feature masking based on the pixel values ​​of the pixels in the image. The features corresponding to the pixels of a specific color in the pixel values ​​may be used as the features to be masked, thereby obtaining the masked image features. For label features, it may be determined whether to mask based on the part of speech, such as nouns, verbs, etc. By determining whether to mask based on the part of speech, the features corresponding to some meaningless or unimportant words may be masked.

[0055] As an optional example, the above-mentioned use of the first fusion feature and the second fusion feature to train the above-mentioned image recognition model includes: calculating the first loss of the above-mentioned first fusion feature and the label tag of the above-mentioned text description; calculating the second loss of the above-mentioned second fusion feature and the label tag of the above-mentioned label description; taking a weighted sum of the above-mentioned first loss and the above-mentioned second loss to obtain a target loss; and adjusting the above-mentioned image recognition model according to the above-mentioned target loss.

[0056] In this embodiment, for the first fusion feature and the second fusion feature, the first loss of the first fusion feature and the label of the text description can be calculated, and the second loss of the second fusion feature and the label of the label description can be calculated. The first loss and the second loss are weighted and summed to obtain the target loss. The image recognition model is adjusted according to the size of the target loss and the loss threshold. If the target loss is less than the loss threshold, the model parameters do not need to be adjusted. If the target loss is greater than or equal to the loss threshold, the model parameters can be adjusted.

[0057] As an optional example, after acquiring the target image to be identified, after inputting the target image and the label set into the image recognition model, the method further includes: extracting image features of the target image and label features of each label in the label set; fusing the image features with the label features of each label to obtain target fusion features, wherein each label corresponds to one target fusion feature; and identifying each of the target fusion features through the image recognition model to obtain the target label of the target image.

[0058] In this embodiment, after the image recognition model is trained, it can be used to recognize the target image. In this embodiment, the target image and the label set can be input into the image recognition model, and the image recognition model matches the corresponding label for the target image from the label set. The label set here does not include the label description of the label. That is to say, when training the image recognition model, the sample image, the text description of the sample image, and the label description of the label of the sample image are used, and when using the image recognition model, the target image and the label set are input, the label set includes the label but not the label description of the label, and the target recognition model matches the target label from the label set.

[0059] As an optional example, determining the target label as the entity included in the target image includes: classifying the target image according to the number and types of entities included in the target image.

[0060] For example, in this embodiment, after matching the label of the target image, the entity identified by the label is the entity of the target image. According to the identified entity, the target image can be classified. For example, the identified entities may include animals, plants, rain and snow, buildings, etc. The target image is classified by the type and number of entities included. In the classified image set, some images in the image set include animals, and some images in the image set include plants, thereby dividing the images.

[0061] In this embodiment, when extracting features, a pre-trained convolutional neural network (such as ResNet, VGG, EfficientNet, MobileNet) can be used to extract image features, input the image for feature extraction, and add an attention mechanism based on the extracted features to preliminarily achieve the purpose of enhancing the feature representation, and the feature dimension of the extracted features remains unchanged at 1024; a pre-trained language model (such as BERT, GPT, RoBERTa) is used to extract text features, such as using ResNet50, or RoBERTa, and an attention mechanism is added based on the extracted features to preliminarily achieve the purpose of enhancing the features, and the feature dimension remains unchanged at 768 (both text description and label description can be extracted based on RoBERTa); then the image features, text features, and label features are aligned through the attention mechanism, and the target output is generated. The aligned features here have two parts, namely, "image feature-text description feature" alignment and "image feature-label description feature" alignment. For example, the image is a little girl making a snowman, and the corresponding text description is "a girl making a snowman in the snow", and the corresponding label description is based on Based on the description of the label generated by LLM (snowman: a human-shaped sculpture made of snow, usually made by children and families in winter outdoor activities; girl: usually refers to female children aged from childhood to early adolescence); by training the model through feature alignment, the target output of "image feature-text description feature" can be obtained, for example: from multiple outputs of "girl making a snowman, boy riding a bicycle, cat sleeping on the sofa, dog playing with a ball, etc.", "girl making a snowman" is hit; similarly, the target output of "image feature-label description feature" is obtained, that is, in "snowman: a human-shaped sculpture made of snow, usually made by children and families in winter outdoor activities; girl: usually refers to female children aged from childhood to early adolescence); Human-shaped sculptures made of snow, usually made by children and families in outdoor activities in winter", "Girl: usually refers to female children aged from childhood to early adolescence", "Cat: commonly known as domestic cat, is a domesticated mammal belonging to the cat family, with soft hair and sharp claws" and other label descriptions, and the descriptions of "snowman" and "girl" are hit; in this way, the training of image-text description-label alignment is realized, so that the model can recognize the corresponding text description and label description from the image, thereby realizing the recognition of romantic empty shot materials and improving the recognition ability of the model.

[0062] During inference, the input is images and labels. For example, the image content is a girl making a snowman, and the labels are "snowman", "girl", "boy", "building", "tree", "bicycle", "snow scene", "lantern", "rainstorm", etc. By fusing and classifying the image features and label features, the softmax classification layer can finally hit the labels of "girl", "snowman", and "snow scene". These labels with higher scores can be used to accurately represent the content of the current picture (generally sorted by score, taking the top 3 or top 5 as the final output).

[0063] Figure 3 Schematic diagram of model training in this embodiment.

[0064] The first step is to input the sample image and use the most basic network ResNet50 to extract features to obtain the corresponding image features with a feature dimension of 1024.

[0065] The second step is to input the text description. The text description is the description content generated based on manual labeling of all the images in the training set. Each image has its corresponding image description (text description). RoBERTa is used for feature extraction to obtain 768-dimensional features.

[0066] In the third step, we input label descriptions and extract named entities, that is, the content in the picture, for each picture in the training set. We manually annotate and obtain a large number of labels, such as heavy rain, heavy snow, lanterns, crowds, flowers, starry sky, courtyard, etc. Then, based on these labels, we use LLM (large-scale language model) to generate descriptions of these labels. We use the same model RoBERTa as in step 2 for feature extraction (sharing parameters with step 2), and also obtain 768-dimensional features.

[0067] In the fourth step, the features extracted in the first three steps are enhanced through the attention mechanism. The feature dimension remains unchanged after feature enhancement, which only makes the model pay more attention to important feature information and improves the feature representation ability.

[0068] In the fifth step, the features extracted in step 1 and step 2 are enhanced and then fused (concat together). The feature dimension is 1792. After passing through the Alignment Decoder, that is, the fused attention mechanism layer, it can be considered that the feature dimension of this layer is consistent with that of the fused feature. The feature information is the attention map. The specific value of the map is the attention parameter, which can be adjusted through feedback propagation. The feature dimension after the alignment decoder is still 1792. The attention mechanism serves the purpose of fusion and feature enhancement. Finally, the final output is obtained through the FC layer and the softmax layer, and the loss1 is calculated with the label.

[0069] In the sixth step, the features extracted in steps 1 and 3 are also fused, passed through the AlignmentDecoder alignment decoder, and then through FC and softmax to obtain the final output, and the loss2 is calculated with the label. Note that steps 5 and 6 share the same alignment decoder and shared parameters.

[0070] The seventh step is to calculate the final loss by weighting the loss, and train the model through feedback propagation. The weighted value is the empirical value loss = a*loss1+(1-a)*loss2.

[0071] Figure 4 Schematic diagram of the target image recognized by the model in this embodiment.

[0072] Input the target image and label library (the label library here is different from the label description used in the training stage. In order to integrate LLM knowledge into the text description and enhance the model's understanding of romantic elements in the training stage, the label description is based on the label library and generated by LLM, while the label in the label library is used here. For example: there is "cat" in the label library. This cat is input using the label text used in the stage, but the description of the label "cat" generated by LLM is input in the training stage: "Cat, commonly known as domestic cat, is a domesticated mammal belonging to the cat family with soft hair and sharp claws"), and the corresponding features are obtained through image feature extraction and text feature extraction. Then the extracted image features and label features are subjected to the attention mechanism to achieve feature enhancement. Then the image features and label features are merged, and the fused attention mechanism is performed through the alignment decoder to obtain the final features (1792 dimensions). Finally, the fused features are fused through FC and softmax to obtain the final output, and the top3 or top5 of the classification is taken as the final output to obtain the final target label.

[0073] Figure 5 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of the present application. Figure 5 As shown, the above-mentioned image recognition device includes:

[0074] An input module 502 is used to input the target image to be recognized and the label set into the image recognition model after acquiring the target image to be recognized, wherein the image recognition model is a model trained using a sample image, a text description of the sample image, and a label description of the label of the sample image, the label set includes multiple labels, and the label of the sample image belongs to the multiple labels;

[0075] An acquisition module 504 is used to acquire a target label of the target image output by the image recognition model, where the target label belongs to the multiple labels;

[0076] The determination module 506 is used to determine the target tag as an entity included in the target image.

[0077] The empty shot mentioned in this embodiment is a shot that does not contain any characters, and is mainly used to show natural scenes or object details without involving characters in the plot. Empty shots can be divided into two types: scenery and objects. Scenery empty shots usually use long-range or panoramic shots to introduce the environment in which the story takes place; object empty shots use close-ups and other small shots to describe the details of objects. This shooting technique is often used at the beginning of a film to explain the general environment in which the story takes place, to create an atmosphere, and to set the tone of the film. In addition, empty shots can also be used in the middle of a film as a transition when the topic changes, or to express the emotions of characters from the side. It should be noted that characters can also be included in the empty shot picture, but the characters are not the main components in the empty shot picture, but the content that serves as a side foil.

[0078] The above-mentioned image recognition method of this embodiment can be applied in the process of identifying entities contained in an image, or in the process of classifying a video, or in the process of identifying whether it is an empty space image.

[0079] The above-mentioned target image is the image to be identified. The image to be identified can be a batch uploaded image or a video frame captured from a video. If it is a batch uploaded image, each image of the batch uploaded image is identified. If it is a video frame in a video, a video frame can be captured from the video for identification. The entity in the image is identified by identifying the image, or it is identified whether the image is an empty space image, and the video is classified by identifying the video frame.

[0080] The above-mentioned image recognition model is a model used to recognize the target image. The image recognition model is pre-trained with sample images, and the recognition accuracy is higher than a certain value. The image recognition model is used to recognize the type of the target image and output the target label of the target image.

[0081] When training an image recognition model, it is necessary to input not only sample images, but also text descriptions and label descriptions of the sample images. The text description of the sample image is the text describing the content contained in the sample image, and the label description is the explanation of the entity included in the sample image. For example, the entity is a tree, a house, a well, a snowflake, etc. The label description is used to describe the content of the entity, such as a house: a building with walls, a roof, doors, windows, for people to live in or for other purposes. The label description is used to represent the content of the label.

[0082] The trained image recognition model can input a target image, recognize the target image, and output the target labels contained in the target image, identifying each label as an entity contained in the target image.

[0083] For example, the above-mentioned image recognition method is applied in the process of classifying videos. After obtaining the video to be classified, this embodiment can extract the key frames of the video. Extracting the key frames of the video can reduce the number of images to be recognized. Then, each key frame is input into the image recognition model, and the image recognition model outputs the target label of the key frame. The label identifies the entity contained in the key frame, thereby determining which entities are contained in the key frame. The key frames are classified by the entities, so that the classification result of the entire video is determined according to the classification result of each key frame.

[0084] The solution provided by the embodiment of the present application, after acquiring the target image to be identified, inputs the target image and the label set into the image recognition model, wherein the image recognition model is a model trained using a sample image, a text description of the sample image, and a label description of the label of the sample image, the label set includes multiple labels, and the label of the sample image belongs to the multiple labels; the target label of the target image output by the image recognition model is obtained, and the target label belongs to the multiple labels; the target label is determined as an entity contained in the target image, so that when training the image recognition model, the model can be trained through sample images, text descriptions of sample images, and label descriptions of sample images, thereby improving the model's perception of the text descriptions and label descriptions of sample images, and the trained image recognition model has higher recognition accuracy for empty mirror images.

[0085] As an optional example, the above-mentioned device also includes: a training module, which is used to obtain the above-mentioned sample image, the text description of the above-mentioned sample image and the label description of the above-mentioned sample image before obtaining the above-mentioned target image; extract the image features of the above-mentioned sample image, the text features of the above-mentioned text description and the label features of the above-mentioned label description; fuse the above-mentioned image features with the above-mentioned text features to obtain a first fused feature; fuse the above-mentioned image features with the above-mentioned label features to obtain a second fused feature; and use the above-mentioned first fused feature and the above-mentioned second fused feature to train the above-mentioned image recognition model.

[0086] In this embodiment, before acquiring the target image, the image recognition model needs to be trained. The image recognition model can be trained using a batch of sample images, in which each sample image has a text description for describing the image of the sample image, and also includes a label description for describing the meaning of the label of the sample image.

[0087] When training the image recognition model, the image recognition model can obtain the image features of the sample image, the text features of the text description, and the label features of the label description. After obtaining the three features, the image features can be fused with the above text features to obtain a first fused feature, and the image features can be fused with the above label features to obtain a second fused feature, and the first fused feature and the second fused feature can be used to train the image recognition model.

[0088] In one example, the sample images can be divided into different batches. For the sample images of the same batch, the first fusion feature and the second fusion feature of each sample image are obtained, and the image recognition model is trained. After the image recognition model is trained with the sample images of the current batch, the image recognition model can be further trained using the sample images of the next batch.

[0089] As an optional example, the above-mentioned training module includes: a first training unit, used to randomly block some features of the above-mentioned image features to obtain the blocked image features; randomly block some features of the above-mentioned text features to obtain the blocked text features; and fuse the blocked image features with the blocked text features to obtain the above-mentioned first fused features.

[0090] In this embodiment, when some features in the image features are blocked, the main purpose is to focus the image recognition model on important features through the attention mechanism. By blocking some features in the image features and text features, and then fusing them to obtain the first fused features, and training the image recognition model, the image recognition model can learn the impact of different areas of blocked image features and text features on the recognition results, so that which features are more important can be identified, and the weight of these features will be higher, while for less important features, the weight will be lower.

[0091] When masking the image features and text features as described above, some of the features may be randomly masked. As another example, for image features, it may be determined whether to mask or the range of feature masking based on the pixel values ​​of the pixels in the image. The features corresponding to the pixels of a specific color in the pixel values ​​may be used as the features to be masked, thereby obtaining the masked image features. For text features, it may be determined whether to mask based on the part of speech, such as nouns, verbs, etc. By determining whether to mask based on the part of speech, the features corresponding to some meaningless or unimportant words may be masked.

[0092] As an optional example, the above-mentioned training module includes: a second training unit, used to randomly block some features of the above-mentioned image features to obtain the blocked image features; randomly block some features of the above-mentioned label features to obtain the blocked label features; and fuse the blocked image features with the blocked label features to obtain the above-mentioned second fused features.

[0093] In this embodiment, when some features in the image features are blocked, the main purpose is to focus the image recognition model on important features through the attention mechanism. By blocking some features in the image features and label features, and then fusing them to obtain the second fused features, and training the image recognition model, the image recognition model can learn the impact of different areas of blocked image features and label features on the recognition results, so that which features are more important can be identified, and the weight of these features will be higher, while for less important features, the weight will be lower.

[0094] When masking the image features and label features as described above, some of the features may be randomly masked. As another example, for image features, it may be determined whether to mask or the range of feature masking based on the pixel values ​​of the pixels in the image. The features corresponding to the pixels of a specific color in the pixel values ​​may be used as the features to be masked, thereby obtaining the masked image features. For label features, it may be determined whether to mask based on the part of speech, such as nouns, verbs, etc. By determining whether to mask based on the part of speech, the features corresponding to some meaningless or unimportant words may be masked.

[0095] As an optional example, the training module includes: a calculation unit for calculating a first loss of the first fused feature and the label of the text description; calculating a second loss of the second fused feature and the label of the label description; performing a weighted sum of the first loss and the second loss to obtain a target loss; and adjusting the image recognition model according to the target loss.

[0096] In this embodiment, for the first fusion feature and the second fusion feature, the first loss of the first fusion feature and the label of the text description can be calculated, and the second loss of the second fusion feature and the label of the label description can be calculated. The first loss and the second loss are weighted and summed to obtain the target loss. The image recognition model is adjusted according to the size of the target loss and the loss threshold. If the target loss is less than the loss threshold, the model parameters do not need to be adjusted. If the target loss is greater than or equal to the loss threshold, the model parameters can be adjusted.

[0097] As an optional example, the above-mentioned device also includes: an identification module, which is used to, after acquiring the target image to be identified, input the above-mentioned target image and the label set into the image recognition model, extract the image features of the above-mentioned target image and the label features of each label in the above-mentioned label set; fuse the above-mentioned image features with the label features of each label to obtain the target fusion features, wherein each label corresponds to one of the above-mentioned target fusion features; identify each of the above-mentioned target fusion features through the above-mentioned image recognition model to obtain the above-mentioned target label of the above-mentioned target image.

[0098] In this embodiment, after the image recognition model is trained, it can be used to recognize the target image. In this embodiment, the target image and the label set can be input into the image recognition model, and the image recognition model matches the corresponding label for the target image from the label set. The label set here does not include the label description of the label. That is to say, when training the image recognition model, the sample image, the text description of the sample image, and the label description of the label of the sample image are used, and when using the image recognition model, the target image and the label set are input, the label set includes the label but not the label description of the label, and the target recognition model matches the target label from the label set.

[0099] As an optional example, the determination module includes: a classification unit, configured to classify the target image according to the number and types of entities contained in the target image.

[0100] For example, in this embodiment, after matching the label of the target image, the entity identified by the label is the entity of the target image. According to the identified entity, the target image can be classified. For example, the identified entities may include animals, plants, rain and snow, buildings, etc. The target image is classified by the type and number of entities included. In the classified image set, some images in the image set include animals, and some images in the image set include plants, thereby dividing the images.

[0101] For other examples of this embodiment, please refer to the above examples and will not be described in detail here.

[0102] like Figure 6 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0103] Memory 113, used for storing computer programs;

[0104] In one embodiment of the present application, the processor 111 is used to implement the image recognition method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113.

[0105] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, an image recognition method provided by any of the aforementioned method embodiments is implemented.

[0106] The device embodiments described above are merely illustrative, wherein the units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the above methods of each embodiment or some parts of the embodiments.

[0108] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "above" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0109] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An image recognition method, characterized in that: include: After acquiring a target image to be recognized, inputting the target image and the label set into an image recognition model, wherein the image recognition model is a model trained using a sample image, a text description of the sample image, and a label description of a label of the sample image, wherein the label set includes a plurality of labels, and the label of the sample image belongs to the plurality of labels; Obtaining a target label of the target image output by the image recognition model, where the target label belongs to the multiple labels; The target tag is determined as an entity contained in the target image.

2. The method according to claim 1, characterized in that Before acquiring the target image, the method further includes: Acquire the sample image, a text description of the sample image, and a label description of the sample image; Extracting image features of the sample image, text features of the text description, and label features of the label description; Fusing the image feature with the text feature to obtain a first fused feature; Fusing the image feature with the label feature to obtain a second fused feature; The image recognition model is trained using the first fusion feature and the second fusion feature.

3. The method according to claim 2, characterized in that The fusing the image feature with the text feature to obtain a first fused feature comprises: Randomly masking some of the image features to obtain masked image features; Randomly masking some of the text features to obtain masked text features; The occluded image feature and the occluded text feature are fused to obtain the first fused feature.

4. The method according to claim 2, characterized in that: The fusing the image feature with the label feature to obtain a second fused feature comprises: Randomly masking some of the image features to obtain masked image features; Randomly masking some of the label features to obtain masked label features; The occluded image feature and the occluded label feature are fused to obtain the second fused feature.

5. The method according to claim 2, characterized in that: The using the first fusion feature and the second fusion feature to train the image recognition model comprises: Calculate the first loss of the first fusion feature and the label of the text description; Calculate the second loss of the second fused feature and the label described by the label; Performing a weighted summation on the first loss and the second loss to obtain a target loss; The image recognition model is adjusted according to the target loss.

6. The method according to claim 1, characterized in that After acquiring the target image to be recognized, the target image and the label set are input into the image recognition model, and the method further includes: Extracting image features of the target image and label features of each label in the label set; Fusion of the image features with the label features of each label to obtain a target fusion feature, wherein each label corresponds to one target fusion feature; Each of the target fusion features is identified through the image recognition model to obtain the target label of the target image.

7. The method according to claim 1, characterized in that The step of determining the target tag as an entity contained in the target image comprises: The target image is classified according to the quantity and type of entities contained in the target image.

8. An image recognition device, characterized in that: include: An input module, configured to input the target image and the label set into an image recognition model after acquiring the target image to be recognized, wherein the image recognition model is a model trained using a sample image, a text description of the sample image, and a label description of the label of the sample image, the label set includes a plurality of labels, and the label of the sample image belongs to the plurality of labels; An acquisition module, used for acquiring a target label of the target image output by the image recognition model, wherein the target label belongs to the multiple labels; A determination module is used to determine the target tag as an entity contained in the target image.

9. An electronic device, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein a computer program is stored in the memory, and when the processor executes the computer program, the image recognition method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the image recognition method described in any one of claims 1 to 7 of the present application.

Images