Recognition model generation method, recognition method, and related device

Abstract

Embodiments of the present disclosure relate to the technical field of image processing, and more particularly, to a recognition model generation method, a recognition method, and related equipment. The recognition model generation method comprises: decoupling a main task into multiple sub-tasks according to multiple target objects included in a preset scene, the main task being configured to recognize the preset scene, and each sub-task being configured to recognize the target objects; constructing a recognition model, the recognition model comprising at least one main task model configured to execute the main task, and multiple sub-task models configured to execute the sub-tasks; and training the constructed recognition model using multiple image samples, the image samples being labeled with first label information corresponding to the main task and second label information corresponding to each sub-task. The technical solution of the present disclosure can effectively improve the recognition accuracy of abstract scenes.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of image processing technology, and more specifically, the embodiments of this disclosure relate to a recognition model generation method, a recognition method, and related equipment. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure set forth in the claims, and the description herein is not acknowledged as prior art simply because it is included in this section.

[0003] In scene recognition of images and videos, scenes are abstract concepts rather than identifiable entities. Therefore, current methods for scene recognition in images and videos present challenges. Summary of the Invention

[0004] However, the following approaches are commonly used to achieve scene recognition in images and videos: Approach 1: Using a single-task recognition algorithm to complete scene recognition; Approach 2: Detecting the region to be recognized in the image or video, and then using a single-task recognition algorithm within the detected region to complete scene recognition. However, both Approach 1 and Approach 2 essentially use a single-task recognition algorithm to achieve scene recognition. Since a scene is an abstract concept, the accuracy of recognizing abstract concepts using a single-task recognition algorithm is usually low.

[0005] Therefore, there is a great need for an improved method for generating recognition models, a recognition method, and related equipment to improve the accuracy of scene recognition.

[0006] In this context, embodiments of the present invention aim to provide a method for generating a recognition model, a recognition method, and related equipment.

[0007] According to one aspect of this disclosure, a method for generating a recognition model is provided, wherein the generated recognition model is used to identify the content information represented by a corresponding image to determine its matching with a preset scene, including:

[0008] Based on the multiple target objects included in the preset scenario, the main task is decoupled into multiple sub-tasks. The main task is configured to identify the preset scenario, and each sub-task is configured to identify the target object.

[0009] Construct an identification model, which includes at least one main task model for performing the main task and multiple subtask models for performing the subtasks;

[0010] The constructed recognition model is trained using multiple image samples, each image sample being labeled with a first label corresponding to the main task and a second label corresponding to each sub-task; wherein,

[0011] The first label information is used to distinguish the first positive example image and the first negative example image corresponding to the main task among the multiple image samples.

[0012] In one exemplary embodiment of this disclosure, decoupling the main task into multiple sub-tasks according to the multiple target objects included in the preset scenario includes:

[0013] The objects common to each of the first positive example images are identified as the multiple target objects, so that the main task is decoupled into multiple sub-tasks based on the multiple target objects.

[0014] In one exemplary embodiment of this disclosure, determining the objects common to each of the first positive example images as the plurality of target objects, and decoupling the main task into a plurality of sub-tasks based on the plurality of target objects, further includes:

[0015] Based on the correlation relationships among the multiple target objects, at least one group of related objects and / or at least one independent object among the target objects are determined;

[0016] The main task is decoupled into multiple subtasks based on the at least one related object group and / or at least one independent object; wherein, a corresponding related object group corresponds to a subtask, and a corresponding independent object corresponds to a subtask.

[0017] In one exemplary embodiment of this disclosure, decoupling the main task into multiple sub-tasks according to the multiple target objects included in the preset scenario includes:

[0018] Determine multiple target objects included in the preset scene based on at least one of the following:

[0019] The definition of the preset scene, the frequency of occurrence of the corresponding object in the image sample, and the semantic features corresponding to the preset scene.

[0020] In one exemplary embodiment of this disclosure, in the constructed recognition model, the feature data output by each of the sub-task models is input to the main task model after being fused by the fusion layer.

[0021] In one exemplary embodiment of this disclosure, the fusion operation includes:

[0022] Add the feature data output by each of the sub-task models together.

[0023] In one exemplary embodiment of this disclosure, the fusion operation includes:

[0024] The feature data output by each of the sub-task models are concatenated.

[0025] In one exemplary embodiment of this disclosure, the fusion operation includes:

[0026] The output features of the sub-task models corresponding to at least one group of related objects are fused to obtain intermediate features;

[0027] The output features of the subtask model corresponding to at least one independent object are fused with the intermediate features.

[0028] In one exemplary embodiment of this disclosure, in the constructed recognition model, each of the sub-task models shares an input processing layer, the output data of which is input to each of the sub-task models. The input processing layer includes at least one convolutional layer and / or at least one pooling layer.

[0029] The input processing layer has at least one pooling layer used to perform downsampling on the image data input to the input processing layer.

[0030] At least one convolutional layer of the input processing layer is used to extract a shared feature map from the image data input to the input processing layer. The shared feature map represents the low-level image information contained in the image sample. The shared feature map is used to input into each of the sub-task models to identify the target object.

[0031] In one exemplary embodiment of this disclosure, each of the subtask models includes at least one convolutional layer and / or at least one pooling layer.

[0032] The subtask model has at least one pooling layer used to perform downsampling on the image data input to the subtask model.

[0033] At least one convolutional layer of the subtask model is used to extract feature data from the image data input to the subtask model. The extracted feature data is the feature data of the target object to be identified by the subtask model.

[0034] In one exemplary embodiment of this disclosure, the main task model includes a feature processing layer and an output processing layer, which are sequentially connected. The feature processing layer includes at least one convolutional layer and / or at least one pooling layer. The output processing layer is used to calculate the recognition probability of the preset scene based on the feature data output by the feature processing layer. The recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene.

[0035] Wherein, at least one pooling layer of the feature processing layer is used to perform downsampling operation on the feature data input to the main task model;

[0036] At least one convolutional layer in the feature processing layer is used to extract feature data from the feature data input to the main task model. The extracted feature data is the feature data of the preset scene that the main task model is to identify.

[0037] In one exemplary embodiment of this disclosure, training the constructed recognition model using multiple image samples includes:

[0038] The multiple image samples are input into the constructed recognition model;

[0039] Based on the feature data output by each subtask model and the second label information of the image sample, calculate the subtask loss function for each subtask model;

[0040] The main task loss function of the main task model is calculated based on the output data of the main task model and the first label information of the image sample.

[0041] The loss function of the recognition model is calculated based on the sub-task loss function and the main task loss function;

[0042] The constructed recognition model is trained based on the loss function of the recognition model.

[0043] In one exemplary embodiment of this disclosure, the loss function of the recognition model is the sum of the weighted sum of the sub-task loss functions and the main task loss function, and the step of training the constructed recognition model according to the loss function of the recognition model includes:

[0044] Based on the loss function of the recognition model, gradient backpropagation is used to train the constructed recognition model.

[0045] Specifically, during each gradient backpropagation, the weights of the loss functions of each subtask are updated according to the gradient loss of the recognition model, and the weights of the loss function of the main task are updated according to the gradient loss of the recognition model.

[0046] In one exemplary embodiment of this disclosure, the constructed recognition model is trained using a gradient normalization algorithm based on the loss function of the recognition model.

[0047] In one exemplary embodiment of this disclosure, calculating the subtask loss function for each subtask model based on the feature data output by each subtask model includes:

[0048] Based on the feature data output by the sub-task model, the recognition probability of the target object of the sub-task model is calculated. The recognition probability of the target object is the probability that the image data input to the sub-task model contains the target object.

[0049] Based on the recognition probability of the target object in the sub-task model, determine the recognition result of the target object in the sub-task model;

[0050] The subtask loss function of the subtask model is calculated based on the recognition result of the target object of the subtask model and the second label information of the image sample.

[0051] In one exemplary embodiment of this disclosure, calculating the main task loss function of the main task model based on the output data of the main task model includes:

[0052] Based on the recognition probability of the preset scene output by the main task model, the recognition result of the preset scene of the main task model is determined, and the recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene;

[0053] The main task loss function of the main task model is calculated based on the recognition results of the preset scene of the main task model and the first label information of the image sample.

[0054] In one exemplary embodiment of this disclosure, before training the constructed recognition model using multiple image samples, the method further includes:

[0055] The image samples are scaled to the same standard size.

[0056] According to one aspect of this disclosure, an identification method is provided, comprising:

[0057] The image to be recognized is input into the recognition model, which is generated using the recognition model generation method described above.

[0058] The recognition result of the image to be recognized is obtained based on the output of the recognition model.

[0059] In one exemplary embodiment of this disclosure, obtaining the recognition result of the image to be recognized based on the output of the recognition model includes:

[0060] Based on the output of the recognition model, the probability that the image to be recognized matches the preset scene is determined, or whether the image to be recognized matches the preset scene is determined.

[0061] In one exemplary embodiment of this disclosure, obtaining the recognition result of the image to be recognized based on the output of the recognition model includes:

[0062] Based on the output of the recognition model, the preset scene that the image to be recognized conforms to is identified.

[0063] In one exemplary embodiment of this disclosure, during training, the main task model and the sub-task model further include their respective loss function calculation layers, and during recognition, the loss function calculation layers are not operational.

[0064] According to one aspect of this disclosure, a recognition model generation apparatus is provided, wherein the generated recognition model is used to recognize the content information represented by a corresponding image to determine its matching with a preset scene, including:

[0065] A decoupling module is used to decouple the main task into multiple sub-tasks according to the multiple target objects included in the preset scenario. The main task is configured to identify the preset scenario, and each sub-task is configured to identify the target object.

[0066] A building module is used to build a recognition model, the recognition model including at least one main task model for performing the main task, and multiple subtask models for performing the subtasks;

[0067] The training module is used to train the constructed recognition model using multiple image samples, wherein the image samples are labeled with first label information corresponding to the main task and second label information corresponding to each sub-task; wherein,

[0068] The first label information is used to distinguish the first positive example image and the first negative example image corresponding to the main task among the multiple image samples.

[0069] In one exemplary embodiment of this disclosure, the decoupling module includes:

[0070] The target object determination module is used to determine the objects common to each of the first positive example images as the multiple target objects, so as to decouple the main task into multiple sub-tasks based on the multiple target objects.

[0071] In one exemplary embodiment of this disclosure, the target object determination module is used to determine at least one related object group and / or at least one independent object among the target objects based on the correlation relationship between the plurality of target objects; and to decouple the main task into a plurality of subtasks based on the at least one related object group and / or at least one independent object; wherein, a corresponding related object group corresponds to a subtask, and a corresponding independent object corresponds to a subtask.

[0072] In one exemplary embodiment of this disclosure, the decoupling module is used to determine a plurality of target objects included in a preset scene based on at least one of the following: the definition of the preset scene, the frequency of occurrence of the corresponding objects in the image samples, and the semantic features corresponding to the preset scene.

[0073] In one exemplary embodiment of this disclosure, in the constructed recognition model, the feature data output by each of the sub-task models is input to the main task model after being fused by the fusion layer.

[0074] In one exemplary embodiment of this disclosure, the fusion operation includes: adding the feature data output by each of the subtask models.

[0075] In one exemplary embodiment of this disclosure, the fusion operation includes: concatenating the feature data output by each of the subtask models.

[0076] In one exemplary embodiment of this disclosure, the fusion operation includes:

[0077] The output features of the sub-task models corresponding to at least one group of related objects are fused to obtain intermediate features;

[0078] The output features of the subtask model corresponding to at least one independent object are fused with the intermediate features.

[0079] In one exemplary embodiment of this disclosure, in the constructed recognition model, each of the sub-task models shares an input processing layer, the output data of which is input to each of the sub-task models. The input processing layer includes at least one convolutional layer and / or at least one pooling layer.

[0080] The input processing layer has at least one pooling layer used to perform downsampling on the image data input to the input processing layer.

[0081] At least one convolutional layer of the input processing layer is used to extract a shared feature map from the image data input to the input processing layer. The shared feature map represents the low-level image information contained in the image sample. The shared feature map is used to input into each of the sub-task models to identify the target object.

[0082] In one exemplary embodiment of this disclosure, each of the subtask models includes at least one convolutional layer and / or at least one pooling layer.

[0083] The subtask model has at least one pooling layer used to perform downsampling on the image data input to the subtask model.

[0084] At least one convolutional layer of the subtask model is used to extract feature data from the image data input to the subtask model. The extracted feature data is the feature data of the target object to be identified by the subtask model.

[0085] In one exemplary embodiment of this disclosure, the main task model includes a feature processing layer and an output processing layer, which are sequentially connected. The feature processing layer includes at least one convolutional layer and / or at least one pooling layer. The output processing layer is used to calculate the recognition probability of the preset scene based on the feature data output by the feature processing layer. The recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene.

[0086] Wherein, at least one pooling layer of the feature processing layer is used to perform downsampling operation on the feature data input to the main task model;

[0087] At least one convolutional layer in the feature processing layer is used to extract feature data from the feature data input to the main task model. The extracted feature data is the feature data of the preset scene that the main task model is to identify.

[0088] In one exemplary embodiment of this disclosure, the training module includes:

[0089] The input module is used to input the multiple image samples into the constructed recognition model;

[0090] The subtask loss function calculation module is used to calculate the subtask loss function of each subtask model based on the feature data output by each subtask model and the second label information of the image sample.

[0091] The main task loss function calculation module is used to calculate the main task loss function of the main task model based on the output data of the main task model and the first label information of the image sample.

[0092] The model loss function calculation module is used to calculate the loss function of the recognition model based on the sub-task loss function and the main task loss function;

[0093] The loss training module is used to train the constructed recognition model according to the loss function of the recognition model.

[0094] In one exemplary embodiment of this disclosure, the loss function of the recognition model is the sum of the weighted sum of the sub-task loss functions and the main task loss function, and the loss training module is used for:

[0095] Based on the loss function of the recognition model, gradient backpropagation is used to train the constructed recognition model.

[0096] Specifically, during each gradient backpropagation, the weights of the loss functions of each subtask are updated according to the gradient loss of the recognition model, and the weights of the loss function of the main task are updated according to the gradient loss of the recognition model.

[0097] In one exemplary embodiment of this disclosure, the constructed recognition model is trained using a gradient normalization algorithm based on the loss function of the recognition model.

[0098] In one exemplary embodiment of this disclosure, the subtask loss function calculation module is used for:

[0099] Based on the feature data output by the sub-task model, the recognition probability of the target object of the sub-task model is calculated. The recognition probability of the target object is the probability that the image data input to the sub-task model contains the target object.

[0100] Based on the recognition probability of the target object in the sub-task model, determine the recognition result of the target object in the sub-task model;

[0101] The subtask loss function of the subtask model is calculated based on the recognition result of the target object of the subtask model and the second label information of the image sample.

[0102] In one exemplary embodiment of this disclosure, the main task loss function calculation module is used for:

[0103] Based on the recognition probability of the preset scene output by the main task model, the recognition result of the preset scene of the main task model is determined, and the recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene;

[0104] The main task loss function of the main task model is calculated based on the recognition results of the preset scene of the main task model and the first label information of the image sample.

[0105] In one exemplary embodiment of this disclosure, it further includes:

[0106] The scaling module is used to scale the image sample to the same standard size.

[0107] According to one aspect of this disclosure, an identification device is provided, comprising:

[0108] The model input module is used to input the image to be recognized into the recognition model, which is generated using the recognition model generation method described above.

[0109] The recognition module is used to obtain the recognition result of the image to be recognized based on the output of the recognition model.

[0110] In one exemplary embodiment of this disclosure, the recognition module is used to identify the probability that the image to be recognized conforms to the preset scene or to identify whether the image to be recognized conforms to the preset scene based on the output of the recognition model.

[0111] In one exemplary embodiment of this disclosure, the recognition module is used to identify a preset scene that the image to be recognized conforms to based on the output of the recognition model.

[0112] In one exemplary embodiment of this disclosure, during training, the main task model and the sub-task model further include their respective loss function calculation layers, and during recognition, the loss function calculation layers are not operational.

[0113] According to one aspect of this disclosure, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements: a recognition model generation method and / or a recognition method.

[0114] According to one aspect of this disclosure, an electronic device is provided, comprising:

[0115] Processor; and

[0116] Memory for storing the executable instructions of the processor;

[0117] The processor is configured to execute the recognition model generation method and / or recognition method described above by executing the executable instructions.

[0118] According to the recognition model generation method and recognition method of this disclosure, a preset scene is decoupled into multiple identifiable target objects. This allows the main task to be configured to recognize the preset scene, and the sub-tasks to be configured to recognize the target objects. A recognition model is constructed, including at least one main task model for performing the main task and multiple sub-task models for performing the sub-tasks. This model is then trained, thereby assisting the main task in recognition through multiple sub-tasks. Therefore, on the one hand, while ensuring the accuracy of abstract scene recognition, the requirements for the number of image samples and label information are relatively low; on the other hand, it can improve the training and recognition efficiency of the recognition model while reducing its complexity. Attached Figure Description

[0119] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0120] Figure 1 A flowchart illustrating an identification model generation method according to an embodiment of the present disclosure is shown schematically;

[0121] Figure 2A schematic diagram of a training-time recognition model according to one embodiment of the present disclosure is shown.

[0122] Figure 3 A schematic diagram of a training-time recognition model according to another embodiment of the present disclosure is shown;

[0123] Figure 4 A schematic diagram of a training-time recognition model according to yet another embodiment of the present disclosure is shown;

[0124] Figure 5 A flowchart illustrating an identification method according to an embodiment of the present disclosure is shown schematically;

[0125] Figure 6 schematically shown Figure 4 The diagram shown illustrates the recognition model during recognition.

[0126] Figure 7 A block diagram of an identification model generation apparatus according to an embodiment of the present disclosure is shown schematically;

[0127] Figure 8 A block diagram of an identification device according to an embodiment of the present disclosure is shown schematically;

[0128] Figure 9 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown; and

[0129] Figure 10 A block diagram of an electronic device according to a disclosed embodiment is shown schematically.

[0130] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0131] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0132] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0133] According to embodiments of this disclosure, a method for generating a recognition model, a recognition method, and related equipment are provided.

[0134] In this document, any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0135] The principles and spirit of this disclosure are explained in detail below with reference to several representative embodiments. Invention Overview

[0137] The inventors have discovered that scene recognition in images and videos presents challenges because scenes are abstract concepts, not tangible, recognizable entities.

[0138] Currently, the following approaches are generally used to achieve scene recognition in images and videos: Approach 1: Using a single-task recognition algorithm to complete scene recognition; Approach 2: Detecting the region to be recognized in the image or video and then using a single-task recognition algorithm within the detected region to complete scene recognition. However, both Approach 1 and Approach 2 essentially use a single-task recognition algorithm for scene recognition. Since a scene is an abstract concept, improving recognition accuracy requires iterative optimization using a large amount of labeled data. On the one hand, a large amount of labeled data requires significant human resources. On the other hand, iterative optimization using a large amount of data is not only time-consuming but also computationally expensive. Furthermore, for scenes with a small number of images to be recognized, the recognition accuracy of Approach 1 and Approach 2 cannot be improved, making them unsuitable for scene recognition. Additionally, during image annotation, training samples often contain noisy data due to annotation quality issues. When there is a lot of noisy data and a small number of image samples, it becomes difficult for a single-task recognition algorithm to distinguish between relevant and irrelevant features within the model, further affecting the accuracy of recognition. On the other hand, since the scene is an abstract concept, when a single task recognition algorithm makes a misjudgment, it is difficult to directly locate the cause of the misjudgment in order to make targeted optimizations.

[0139] In view of the above, the basic idea of ​​this invention is as follows: by decoupling a preset scene into multiple identifiable target objects, the main task is configured to identify the preset scene, and the sub-tasks are configured to identify the target objects. A recognition model is constructed, including at least one main task model for executing the main task and multiple sub-task models for executing the sub-tasks, and trained. This allows the recognition of the main task to be assisted by multiple sub-tasks. Since the target objects are identifiable entities, the training of each sub-task model does not require a large number of image samples or second label information to achieve high recognition accuracy. The main task model is assisted by sub-task models that can provide target object information, requiring no large number of image samples or first label information, and without increasing model complexity or the number of iterations, to achieve high recognition accuracy for abstract scenes. Therefore, the recognition model generation method and recognition method of this disclosure, while ensuring the recognition accuracy of abstract scenes, have low requirements for the number of image samples and label information; they can improve the training and recognition efficiency of the recognition model while reducing its complexity.

[0140] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below.

[0141] Exemplary methods

[0142] The following is combined Figure 1 This document describes a recognition model generation method according to exemplary embodiments of the present disclosure. The generated recognition model is used to identify the content information represented by a corresponding image to determine its matching with a preset scene. (Reference) Figure 1 As shown, the recognition model generation method may include the following steps:

[0143] S110. According to the multiple target objects included in the preset scenario, the main task is decoupled into multiple sub-tasks, the main task is configured to identify the preset scenario, and each sub-task is configured to identify the target object.

[0144] S120. Construct an identification model, the identification model including at least one main task model for performing the main task, and multiple sub-task models for performing the sub-tasks.

[0145] S130. The constructed recognition model is trained using multiple image samples, wherein the image samples are labeled with first label information corresponding to the main task and second label information corresponding to each sub-task.

[0146] The first label information is used to distinguish the first positive example image and the first negative example image corresponding to the main task among the multiple image samples.

[0147] In the recognition model generation method of this disclosure, a preset scene is decoupled into multiple identifiable target objects. This allows the main task to be configured to recognize the preset scene, and the sub-tasks to be configured to recognize the target objects. A recognition model is constructed, comprising at least one main task model for performing the main task and multiple sub-task models for performing the sub-tasks. This model is then trained, thereby assisting the main task in recognition through multiple sub-tasks. Therefore, on the one hand, while ensuring the accuracy of abstract scene recognition, the requirements for the number of image samples and label information are relatively low; on the other hand, the complexity of the recognition model can be reduced while improving the training and recognition efficiency of the recognition model.

[0148] According to exemplary embodiments of this disclosure, the preset scenario can be an abstract scenario. An abstract scenario is a scenario that cannot be directly defined by an identifiable object. For example, a preset scenario can be a live streaming scenario, which cannot be identified by semantics, name, or a single identifiable object. Similarly, a preset scenario can be a live-streaming sales scenario, which also cannot be identified by its semantics, name, or a single identifiable object.

[0149] According to exemplary embodiments of this disclosure, the object / target object can be an identifiable object. The object / target object may include entity elements, entity elements with defined attributes, abstract elements with defined attributes, etc. For example, the object / target object may include food (entity element), a red apple (entity element with defined attributes), a red flag (entity element with defined attributes), multiple red circles (abstract elements with defined attributes), etc., but this disclosure is not intended to limit it.

[0150] According to an exemplary embodiment of this disclosure, the first label information can annotate the scene of each image sample. Thus, when the first label information matches a preset scene, the image sample is a first positive example image; when the first label information does not match the preset scene, the image sample is a first negative example image. The first label information can also indicate whether each image sample belongs to a preset scene. Thus, when the first label information indicates that an image sample belongs to a preset scene, the image sample is a first positive example image; when the first label information indicates that an image sample does not belong to a preset scene, the image sample is a first negative example image.

[0151] According to an exemplary embodiment of this disclosure, the second label information can annotate the objects contained in each image sample. Thus, when the second label information matches the target object to be identified by the subtask, the image sample is a second positive example image for that subtask; when the second label information does not match the target object to be identified by the subtask, the image sample is a second negative example image for that subtask. The second label information can also indicate whether each image sample contains a target object. Thus, when the second label information indicates that an image sample contains the target object to be identified by the subtask, the image sample is a second positive example image for that subtask; when the second label information indicates that an image sample does not contain the target object to be identified by the subtask, the image sample is a second negative example image for that subtask.

[0152] According to an exemplary embodiment of this disclosure, step S110 may include identifying objects common to each of the first positive example images as the plurality of target objects, so as to decouple the main task into a plurality of sub-tasks based on the plurality of target objects.

[0153] Since the preset scene is an abstract scene, as stated above in this disclosure, it is difficult to directly define the abstract scene in a tangible and recognizable way, making it difficult to conceive of any relationship between the abstract scene and the elements in the image to be identified. Therefore, current abstract scene identification uses a single-task classification model to learn the correlation between the entire image features and the abstract scene. Furthermore, since the classification model is an image processing model such as a convolutional neural network, those skilled in the art cannot know from this model which image features / image elements can contribute to the identification of the abstract scene. Under this premise, the exemplary embodiment of this disclosure decouples the main task into multiple sub-tasks by identifying the common objects in each of the first positive example images as the multiple target objects, thereby interpreting and defining the abstract scene through entity objects. This allows the correlation / causal relationship between the objects in the image and the abstract scene to be obtained in advance before the classification model of the abstract scene is identified, thus assisting the single-task classification model by obtaining the correlation / causal relationship between the objects in the image and the abstract scene in advance. Further, the above steps represent and select objects in the image that have a correlation / causal relationship with the abstract scene by using the common objects in each of the first positive example images.

[0154] Therefore, step S110 can be used to identify the common objects in each of the first positive example images as the multiple target objects, thereby decoupling the main task. It can be understood that when an object exists in each of the first positive example images, the image of the preset scene will generally also include that object. Therefore, using these objects as target objects to assist in the recognition of the preset scene can effectively improve the recognition accuracy of the preset scene.

[0155] Specifically, in the exemplary embodiments of this disclosure, objects common to each of the first positive example images can be directly identified as the multiple target objects, and each subtask can correspond to one target object. For example, in a live streaming scenario, when the common objects in each of the first positive example images are multiple people and multiple different backgrounds, the multiple people and multiple backgrounds can be regarded as two target objects, thereby setting two subtasks to identify people and backgrounds respectively.

[0156] In some other exemplary embodiments of this disclosure, determining the objects common to each of the first positive example images as the plurality of target objects, and decoupling the main task into a plurality of sub-tasks based on the plurality of target objects, may further include: determining at least one related object group and / or at least one independent object among the target objects based on the correlation relationship between the plurality of target objects; decoupling the main task into a plurality of sub-tasks based on the at least one related object group and / or at least one independent object; wherein, a corresponding related object group corresponds to a sub-task, and a corresponding independent object corresponds to a sub-task.

[0157] Specifically, objects in a related object group are correlated with each other. Independent objects are uncorrelated with objects in other related object groups or with other independent objects. In an exemplary embodiment of this disclosure, the correlation between objects can be determined based on the probability of the objects appearing in an image sample. For example, when the probability of two objects appearing in the same image sample is equal to the product of the probability of one object appearing in the image sample and the probability of the other object appearing in the image sample, it indicates that the two objects are uncorrelated.

[0158] In some exemplary embodiments of this disclosure, an object (to represent the relevant object group) can be selected from each relevant object group as the target object to be identified in a subtask, and each independent object can be used as the target object to be identified in other subtasks. For example, among target objects A, B, and C, target objects A and B are related, target objects C and A are not related, and target objects B are not related. Thus, target objects A and B form a relevant object group, and target object C forms an independent object. Target object C can be used as the target object to be identified in one subtask, and a target object can be selected from the relevant object group (target objects A and B) as the target object to be identified in another subtask. In this exemplary embodiment, on the one hand, compared to generating the same number of subtasks for each target object, the number of subtasks can be reduced by using the form of relevant object groups, thereby reducing the size and complexity of the identification model; on the other hand, target objects that are easier to identify, have lower complexity in the corresponding subtask model, and have higher identification accuracy in the corresponding subtask model can be selected from the relevant object group, thereby further optimizing the identification model. This simplifies the recognition model while improving its recognition efficiency.

[0159] According to an exemplary embodiment of this disclosure, decoupling the main task into multiple sub-tasks based on the multiple target objects included in the preset scenario may include: determining the multiple target objects included in the preset scenario based on at least one of the following: the definition of the preset scenario, the frequency of occurrence of the corresponding objects in the image samples, and the semantic features corresponding to the preset scenario. In an exemplary embodiment of determining the multiple target objects included in the preset scenario according to the definition of the preset scenario, taking "live connection" as an example, the direct definition of "live connection" is "multiple people from different places appearing in the same frame". According to this definition, there needs to be "people". In order to distinguish multiple people in the same place in a normal live broadcast, people from different places in a live connection will generally have "different backgrounds". At the same time, there needs to be "people" to perform "live connection". Therefore, according to the definition of the scenario, the target objects in the "live connection" scenario can be determined to be "multiple people" and "multiple backgrounds". In an exemplary embodiment of determining multiple target objects included in a preset scene based on the frequency of occurrence of corresponding objects in image samples, taking "live-streaming sales" as an example, in image samples with the scene of "live-streaming sales," the frequencies of "large quantity of goods" and "price figures" are the highest. Therefore, the target objects in the "live-streaming sales" scene can be determined to be "large quantity of goods" and "price figures." Thus, in this exemplary embodiment of determining multiple target objects included in a preset scene based on the frequency of occurrence of corresponding objects in image samples, all objects in each image sample can be obtained, and the frequency ranking of each object can be obtained by statistically analyzing the frequency of occurrence of each object. The top N objects in the frequency ranking can be selected as target objects, where N can be an integer greater than 0, and the value of N can be set as needed. Further, in this exemplary embodiment of determining multiple target objects included in a preset scene based on the frequency of occurrence of corresponding objects in image samples, all objects in each image sample can be obtained, and the objects with a frequency greater than M% of the highest frequency of occurrence can be selected as target objects, where M can be a constant greater than 50 and less than or equal to 100, and the value of M can be set as needed. In an exemplary embodiment of determining multiple target objects included in a preset scenario based on the semantic features corresponding to the preset scenario, taking "live streaming" as an example, the direct definition of "live streaming" is "multiple people from different places streaming together." According to this definition, the noun included is "people," the verb is "live streaming," and the adjective is "different places." To distinguish it from normal live streaming involving multiple people, live streaming typically involves "different backgrounds." Therefore, based on the semantic features of this scenario, the target objects in the "live streaming" scenario can be determined to be "multiple people" and "multiple backgrounds."Therefore, in the exemplary embodiment of determining multiple target objects contained in a preset scene based on the semantic features of the preset scene, it is necessary to obtain the entity content such as the subject and object contained in the definition of the preset scene as the target objects. When the subject / object is missing, it can be supplemented by supplementation. Simultaneously, in order to distinguish it from other images of the same type, subjects and / or objects containing adjectives can also be used as target objects. In these exemplary embodiments, the content information contained in the preset scene can be enriched, thereby determining the target objects based on this content information to obtain target objects more suitable for preset scene recognition.

[0160] According to an exemplary embodiment of this disclosure, in the constructed recognition model, the feature data output by each sub-task model is fused by a fusion layer before being input to the main task model. The position of the fusion layer can be combined with... Figures 2 to 4 Examples of the multiple recognition models 210, 310, and 410 shown are for reference. The fusion layer is used to integrate the feature data output by each of the sub-task models, making it easier for the main task model to read, thereby obtaining the information recognized by each sub-task model. This information can assist the main task model in performing its recognition task and improve the recognition accuracy of the main task model.

[0161] In some exemplary embodiments of this disclosure, the fusion operation may include adding the feature data output by each of the sub-task models. In other exemplary embodiments of this disclosure, the fusion operation may include concatenating the feature data output by each of the sub-task models. Different concatenation methods may be selected according to different application scenarios, and this disclosure is not intended to limit the scope of the fusion operation.

[0162] In some exemplary embodiments of this disclosure, multiple target objects can be selected from a group of related objects, so that one group of related objects can correspond to multiple sub-tasks. In this exemplary embodiment, the fusion operation may include fusing the output features of the sub-task models corresponding to at least one group of related objects to obtain intermediate features; and fusing the output features of the sub-task models corresponding to at least one independent object with the intermediate features. This ensures that related feature data are also interconnected in the fusion operation, so that the relevance of the target objects can be considered during the training of the recognition model to improve the recognition accuracy.

[0163] According to some exemplary embodiments of this disclosure, in the constructed recognition model, each of the sub-task models can share an input processing layer. This can be combined with... Figure 2 and Figure 4The embodiments of recognition models 210 and 410 shown are for reference. The output data of the input processing layer is input to each of the sub-task models. The input processing layer may include at least one convolutional layer and / or at least one pooling layer, wherein the at least one pooling layer of the input processing layer is used to perform a downsampling operation on the image data input to the input processing layer; the at least one convolutional layer of the input processing layer is used to extract a shared feature map from the image data input to the input processing layer. The shared feature map represents the low-level image information contained in the image sample, and the shared feature map is used to input to each of the sub-task models to recognize the target object. The types of layers, the number of layers, and the computational parameters of the layers (such as the size of the convolutional kernel) included in the input processing layer are not limited thereto. By sharing the input processing layer, memory usage can be reduced, the results of multiple sub-task models can be obtained in a single forward computation, improving inference speed, and information sharing and supplementation between different tasks can be achieved, improving the performance of the recognition model. According to some other exemplary embodiments of this disclosure, the sub-task models in the constructed recognition model may also not share the input processing layer. In other words, image samples can be directly input into each sub-task model without going through an input processing layer, and can be combined with... Figure 3 The identification model 310 shown is for reference.

[0164] According to an exemplary embodiment of this disclosure, each sub-task model includes at least one convolutional layer and / or at least one pooling layer, wherein the at least one pooling layer of the sub-task model is used to perform a downsampling operation on the image data input to the sub-task model; the at least one convolutional layer of the sub-task model is used to extract feature data from the image data input to the sub-task model, and the extracted feature data is the feature data of the target object to be identified by the sub-task model. The structure of the layers of the sub-task model can be combined with... Figure 4 The identification model 410 shown is for reference. Depending on the target object to be identified and the composition of the data, the type of layers, the number of layers, and the computational parameters of the layers (such as the size of the convolution kernel) contained in each subtask model can be adjusted, but this disclosure is not intended to limit it.

[0165] According to an exemplary embodiment of this disclosure, the main task model may include a feature processing layer and an output processing layer. It can be combined with... Figure 4The recognition model 410 shown is used for reference. The feature processing layer and the output processing layer are connected sequentially. The feature processing layer includes at least one convolutional layer and / or at least one pooling layer. The output processing layer is used to calculate the recognition probability of the preset scene based on the feature data output by the feature processing layer. The recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene. The at least one pooling layer of the feature processing layer is used to perform downsampling on the feature data input to the main task model. The at least one convolutional layer of the feature processing layer is used to extract feature data from the feature data input to the main task model. The extracted feature data is the feature data of the preset scene to be recognized by the main task model. The output processing layer can be a fully connected layer, which acts as a classifier in the model, thereby calculating the recognition probability of the preset scene based on the feature data output by the feature processing layer. Depending on the preset scene to be recognized and the feature data of the acquired target object, the type of layers, the number of layers, and the calculation parameters of the layers (such as the size of the convolutional kernel) included in the feature processing layer of the main task model can be adjusted, but this disclosure is not intended to limit this.

[0166] According to an exemplary embodiment of this disclosure, step S130, which uses multiple image samples to train the constructed recognition model, may include: inputting the multiple image samples into the constructed recognition model; calculating a subtask loss function for each subtask model based on the feature data output by each subtask model and the second label information of the image samples; calculating a main task loss function for the main task model based on the output data of the main task model and the first label information of the image samples; calculating a loss function for the recognition model based on the subtask loss function and the main task loss function; and training the constructed recognition model based on the loss function of the recognition model. Specifically, the subtask loss function, the main task loss function, and the loss function of the recognition model can serve as supervision and guidance for the training of the recognition model, so as to improve the recognition accuracy of the recognition model.

[0167] According to an exemplary embodiment of this disclosure, the loss function of the recognition model can be the sum of the weighted sum of the sub-task loss functions and the main task loss function, which can be expressed as the following formula:

[0168] Loss = w a loss taskA +w b loss taskB +…w n loss taskN +loss Maintask ,

[0169] in, Loss is the loss function of the recognition model. taskA Let loss be the loss function of subtask model A. taskB Let loss be the loss function of subtask model B. taskN Let loss be the loss function of sub-task model N. Maintask is the loss function of the main task model, and w is the weight of each sub-task model.

[0170] Therefore, training the constructed recognition model based on the loss function of the recognition model can include: training the constructed recognition model using gradient backpropagation based on the loss function of the recognition model, wherein during each gradient backpropagation, the weights of the loss functions of each sub-task are updated according to the gradient loss of the recognition model, and the weights of the main task loss function are updated according to the gradient loss of the recognition model. In this exemplary embodiment, the weights of each sub-task can be dynamically adjusted according to the gradient, thereby balancing different sub-tasks and controlling the update speed of different sub-tasks.

[0171] According to an exemplary embodiment of this disclosure, the constructed recognition model is trained using a gradient normalization algorithm based on the loss function of the recognition model. Using a gradient normalization algorithm to train the constructed recognition model avoids the problem of different magnitudes of loss gradients for different tasks, which could cause some sub-tasks to dominate gradient backpropagation, leading to the model overlearning those sub-tasks and neglecting other sub-tasks, thus affecting the recognition progress of the recognition model. Furthermore, it also avoids the problem of inconsistent convergence speeds for different sub-tasks, which could affect the model training process.

[0172] According to an exemplary embodiment of this disclosure, calculating the subtask loss function of each subtask model based on the feature data output by each subtask model may include: calculating the recognition probability of the target object of the subtask model based on the feature data output by the subtask model, wherein the recognition probability of the target object is the probability that the image data input to the subtask model contains the target object; determining the recognition result of the target object of the subtask model based on the recognition probability of the target object of the subtask model; and calculating the subtask loss function of the subtask model based on the recognition result of the target object of the subtask model and the second label information of the image sample. Combined with... Figure 4The recognition model 410 shown is used for reference. The feature data output by each sub-task model is not only input to the main task model, but also to its respective fully connected layer. The fully connected layer acts as a classifier in the model, thereby calculating the recognition probability of the target object based on the feature data output by the sub-task model. Therefore, the recognition result of the target object of the sub-task model can be determined based on the recognition probability of the target object. The recognition result can be, for example, that the image sample includes the target object to be recognized by the sub-task model; or that the image sample does not include the target object to be recognized by the sub-task model. Simultaneously, the second label information can be used to distinguish the second positive example image and the second negative example image corresponding to each sub-task among the multiple image samples. When the recognition result is that the image sample includes the target object to be recognized by the sub-task model, and the input image sample is the second positive example image, it indicates that the sub-task model's recognition of the current image sample is accurate; when the recognition result is that the image sample does not include the target object to be recognized by the sub-task model, and the input image sample is the second negative example image, it indicates that the sub-task model's recognition of the current image sample is accurate. Therefore, the sub-task loss function of the sub-task model can be calculated based on whether the sub-task model accurately identifies each image sample.

[0173] According to an exemplary embodiment of this disclosure, calculating the main task loss function of the main task model based on the output data of the main task model may include: determining the recognition result of the preset scene of the main task model based on the recognition probability of the preset scene output by the main task model, wherein the recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene; and calculating the main task loss function of the main task model based on the recognition result of the preset scene of the main task model and the first label information of the image sample. The calculation method of the main task loss function of the main task model is similar to that of the sub-task model.

[0174] According to an exemplary embodiment of this disclosure, before training the constructed recognition model using multiple image samples, step S130 may further include scaling the image samples to the same standard size. This allows for image sample preprocessing, thereby ensuring a balance between model training speed and model recognition accuracy. The specific standard scale can be determined based on actual task requirements. In some exemplary embodiments of this disclosure, image sample preprocessing may further include normalizing the image sample data to ensure consistency with the data distribution during model training.

[0175] Figures 2 to 4This disclosure merely illustrates several recognition models provided herein and is not intended to limit the scope of the disclosure. Other implementations of convolutional neural network models, multi-expert models, decision tree models, etc., as sub-task models and / or main task models are all within the scope of protection of this disclosure.

[0176] The recognition model generation method provided in this disclosure is particularly suitable for scenarios with a small number of image samples and high annotation costs. For example, it can be applied to the recognition of prohibited images. The recognition of prohibited images utilizes computing devices to process, analyze, and understand images to select and identify images containing uncivilized behavior and other spam information from a vast amount of internet images. The recognition of prohibited images can purify the internet and ensure the safety of the information received by the public while enjoying the convenience of the internet. Therefore, higher requirements are placed on the accuracy of prohibited content recognition. In application scenarios with a small number of image samples, such as the recognition of prohibited images, the recognition model generation method provided in this disclosure decouples the main task from the content dimension. Multiple decoupled sub-tasks assist and enhance the learning of the main task, and the sub-tasks and the main task are jointly trained and integrated into a single recognition model. The training of the recognition model is completed directly from the input to the output, fully utilizing the auxiliary enhancement effect of the decoupled sub-tasks on the main task. This effectively avoids the learning difficulties, misjudgments, and poor interpretability problems caused by directly using a single-task classification model.

[0177] The recognition model generation method disclosed herein can obtain feature data from multiple independent but related subtasks even when there are few image samples, thereby assisting the main task in learning and improving the accuracy of the model, especially the recognition accuracy of abstract scenes (scenes with high-level semantic features).

[0178] The recognition model generation method disclosed herein is based on content decoupling and multi-task assistance for model training. Compared with recognition schemes without content decoupling, it can also improve the recognition accuracy of the recognition model.

[0179] The following is combined Figure 5 This describes an identification method according to exemplary embodiments of the present disclosure.

[0180] refer to Figure 5 As shown, the identification method may include the following steps:

[0181] S510. Input the image to be recognized into the recognition model, which is generated using the recognition model generation method described above.

[0182] S520. Based on the output of the recognition model, obtain the recognition result of the image to be recognized.

[0183] According to an exemplary embodiment of this disclosure, step S520 may include: identifying the probability that the image to be identified conforms to the preset scene or identifying whether the image to be identified conforms to the preset scene based on the output of the recognition model.

[0184] According to an exemplary embodiment of this disclosure, the recognition model can output the probability that the image to be recognized belongs to a preset scene. When the probability value output by the recognition model is greater than a set probability threshold, the image to be recognized can be considered to conform to the preset scene; when the probability value output by the recognition model is not greater than the set probability threshold, the image to be recognized can be considered not to conform to the preset scene.

[0185] According to an exemplary embodiment of this disclosure, the recognition model can directly output whether the image to be recognized conforms to the preset scene. In this exemplary embodiment, the conversion between the probability that the image to be recognized conforms to the preset scene and the recognition result of whether the image to be recognized conforms to the preset scene can be internalized and trained in the recognition model, so that the recognition model can directly output whether the image to be recognized conforms to the preset scene.

[0186] According to an exemplary embodiment of this disclosure, step S520 may include: identifying a preset scene that the image to be identified conforms to based on the output of the recognition model. In an exemplary embodiment, the recognition model may directly output the scene that the image to be identified conforms to, thereby determining whether the image to be identified conforms to the preset scene based on the matching of the scene output by the recognition model with the preset scene.

[0187] According to an exemplary embodiment of this disclosure, during training, the main task model and the sub-task model further include their respective loss function calculation layers; during recognition, these loss function calculation layers are not operational. This exemplary embodiment can be referred to... Figure 4 and Figure 6 The recognition model shown Figure 6 schematically shown Figure 4 The diagram shows the recognition model during recognition. Figure 4 and Figure 6 The recognition model 410, during training, requires loss function calculation layers a through e to calculate sub-task loss functions based on fully connected layers a through c, and main task loss functions based on the output of the output processing layer. Finally, the loss function of the recognition model is calculated based on the sub-task and main task loss functions. The loss function can be used only during model training; therefore, the fully connected layers and loss function calculation layers do not operate during model recognition. Thus, the loss function is not used during training. Figure 6 These layers are shown in the recognition model 610. The above method can improve model recognition efficiency and reduce model computation.

[0188] According to an exemplary embodiment of this disclosure, after the recognition model has completed training, each of the loss function calculation layers can be deleted to reduce the amount of data in the recognition model and reduce storage requirements.

[0189] According to an exemplary embodiment of this disclosure, after the recognition model completes training, the recognition model with the loss function calculation layers removed can be saved locally to the recognition system for local access. When iterative optimization of the recognition model is required, the recognition model with loss function calculation layers in the cloud / server can be used for iterative training to update the parameters of the recognition model and improve the recognition accuracy.

[0190] In summary, the method provided in this disclosure decouples a preset scene into multiple identifiable target objects. This allows the main task to be configured to identify the preset scene, and the sub-tasks to be configured to identify the target objects. A recognition model is constructed, comprising at least one main task model for executing the main task and multiple sub-task models for executing the sub-tasks. This model is then trained, thereby assisting the main task in identification through multiple sub-tasks. Since the target objects are identifiable entities, the training of each sub-task model does not require a large number of image samples or second label information to achieve high recognition accuracy. The main task model is assisted by sub-task models that provide target object information, eliminating the need for a large number of image samples and first label information, and avoiding increased model complexity and training iterations, while still achieving high recognition accuracy for abstract scenes. Therefore, the recognition model generation method and recognition method of this disclosure, while ensuring the accuracy of abstract scene recognition, have low requirements for the number of image samples and label information; they can improve the training and recognition efficiency of the recognition model while reducing its complexity.

[0191] Exemplary device

[0192] After introducing the identification model generation method and identification method of exemplary embodiments of this disclosure, the following will refer to... Figure 7 The identification model generation apparatus of exemplary embodiments of the present disclosure will be described.

[0193] refer to Figure 7 As shown, the recognition model generation apparatus 700 of this exemplary embodiment may include: a decoupling module 710, a construction module 720, and a training module 730. The generated recognition model is used to identify the content information represented by the corresponding image to determine its matching with a preset scene, wherein...

[0194] The decoupling module 710 can be used to decouple the main task into multiple sub-tasks according to the multiple target objects included in the preset scenario. The main task is configured to identify the preset scenario, and each sub-task is configured to identify the target object.

[0195] The construction module 720 can be used to construct an identification model, which includes at least one main task model for performing the main task and multiple sub-task models for performing the sub-tasks.

[0196] The training module 730 can be used to train the constructed recognition model using multiple image samples, wherein the image samples are labeled with first label information corresponding to the main task and second label information corresponding to each sub-task.

[0197] The first label information is used to distinguish the first positive example image and the first negative example image corresponding to the main task among the multiple image samples.

[0198] According to an exemplary embodiment of this disclosure, the decoupling module 710 may include: a target object determination module, configured to determine the objects common to each of the first positive example images as the plurality of target objects, so as to decouple the main task into a plurality of sub-tasks based on the plurality of target objects.

[0199] According to an exemplary embodiment of this disclosure, the target object determination module can be used to determine at least one related object group and / or at least one independent object among the target objects based on the correlation relationship between the plurality of target objects; and to decouple the main task into a plurality of subtasks based on the at least one related object group and / or at least one independent object; wherein, a corresponding related object group corresponds to a subtask, and a corresponding independent object corresponds to a subtask.

[0200] According to an exemplary embodiment of this disclosure, the decoupling module 710 can be used to determine multiple target objects included in a preset scene based on at least one of the following: the definition of the preset scene, the frequency of occurrence of the corresponding objects in the image samples, and the semantic features corresponding to the preset scene.

[0201] According to an exemplary embodiment of this disclosure, in the constructed recognition model, the feature data output by each of the sub-task models can be input to the main task model after being fused by the fusion layer.

[0202] According to an exemplary embodiment of this disclosure, the fusion operation may include: adding the feature data output by each of the subtask models.

[0203] According to an exemplary embodiment of this disclosure, the fusion operation may include: concatenating the feature data output by each of the subtask models.

[0204] According to an exemplary embodiment of this disclosure, the fusion operation may include: fusing the output features of a subtask model corresponding to at least one group of related objects to obtain intermediate features; and fusing the output features of a subtask model corresponding to at least one independent object with the intermediate features.

[0205] According to an exemplary embodiment of this disclosure, in the constructed recognition model, each of the sub-task models can share an input processing layer. The output data of the input processing layer is input to each of the sub-task models. The input processing layer may include at least one convolutional layer and / or at least one pooling layer. The at least one pooling layer of the input processing layer can be used to perform downsampling on the image data input to the input processing layer. The at least one convolutional layer of the input processing layer can be used to extract a shared feature map from the image data input to the input processing layer. The shared feature map represents the underlying image information contained in the image sample. The shared feature map is input to each of the sub-task models to recognize the target object.

[0206] According to an exemplary embodiment of this disclosure, each subtask model may include at least one convolutional layer and / or at least one pooling layer, wherein the at least one pooling layer of the subtask model can be used to perform downsampling on the image data input to the subtask model; the at least one convolutional layer of the subtask model can be used to extract feature data from the image data input to the subtask model, wherein the extracted feature data is the feature data of the target object to be identified by the subtask model.

[0207] According to an exemplary embodiment of this disclosure, the main task model may include a feature processing layer and an output processing layer, which may be connected sequentially. The feature processing layer may include at least one convolutional layer and / or at least one pooling layer. The output processing layer may be used to calculate the recognition probability of the preset scene based on the feature data output by the feature processing layer. The recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene. The at least one pooling layer of the feature processing layer may be used to perform downsampling on the feature data input to the main task model. The at least one convolutional layer of the feature processing layer may be used to extract feature data from the feature data input to the main task model. The extracted feature data is the feature data of the preset scene to be recognized by the main task model.

[0208] According to an exemplary embodiment of this disclosure, the training module 730 may include: an input module for inputting the plurality of image samples into the constructed recognition model; a subtask loss function calculation module for calculating a subtask loss function for each subtask model based on the feature data output by each subtask model and the second label information of the image samples; a main task loss function calculation module for calculating a main task loss function for the main task model based on the output data of the main task model and the first label information of the image samples; a model loss function calculation module for calculating a loss function of the recognition model based on the subtask loss function and the main task loss function; and a loss training module for training the constructed recognition model based on the loss function of the recognition model.

[0209] According to an exemplary embodiment of this disclosure, the loss function of the recognition model can be the sum of the weighted sum of the sub-task loss functions and the main task loss function. The loss training module can be used to: train the constructed recognition model by gradient backpropagation according to the loss function of the recognition model, wherein, during each gradient backpropagation, the weights of each sub-task loss function are updated according to the gradient loss of the recognition model, and the weights of the main task loss function are updated according to the gradient loss of the recognition model.

[0210] According to an exemplary embodiment of this disclosure, the constructed recognition model can be trained using a gradient normalization algorithm based on the loss function of the recognition model.

[0211] According to an exemplary embodiment of this disclosure, the subtask loss function calculation module can be used to: calculate the recognition probability of the target object of the subtask model based on the feature data output by the subtask model, wherein the recognition probability of the target object is the probability that the image data input to the subtask model contains the target object; determine the recognition result of the target object of the subtask model based on the recognition probability of the target object of the subtask model; and calculate the subtask loss function of the subtask model based on the recognition result of the target object of the subtask model and the second label information of the image sample.

[0212] According to an exemplary embodiment of this disclosure, the main task loss function calculation module can be used to: determine the recognition result of the preset scene of the main task model based on the recognition probability of the preset scene output by the main task model, wherein the recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene; and calculate the main task loss function of the main task model based on the recognition result of the preset scene of the main task model and the first label information of the image sample.

[0213] According to an exemplary embodiment of this disclosure, the apparatus may further include: a scaling module for scaling the size of the image sample to the same standard size.

[0214] Next, refer to Figure 8 The identification device 800 of the exemplary embodiments of the present disclosure will be described.

[0215] refer to Figure 8 As shown, the identification device 800 of the exemplary embodiments of this disclosure may include: a model input module 810 and an identification module 820. Wherein,

[0216] The model input module 810 can be used to input the image to be recognized into the recognition model, which is generated using the recognition model generation method described above.

[0217] The recognition module 820 can be used to obtain the recognition result of the image to be recognized based on the output of the recognition model.

[0218] According to an exemplary embodiment of this disclosure, the recognition module 820 can be used to identify the probability that the image to be recognized conforms to the preset scene or to identify whether the image to be recognized conforms to the preset scene based on the output of the recognition model.

[0219] According to an exemplary embodiment of this disclosure, the recognition module 820 can be used to recognize a preset scene that the image to be recognized conforms to based on the output of the recognition model.

[0220] According to an exemplary embodiment of this disclosure, during training, the main task model and the sub-task model may further include their respective loss function calculation layers, and during recognition, each of the loss function calculation layers may not be operational.

[0221] Since the recognition model generation apparatus and the various functional modules of the recognition apparatus in this embodiment are the same as those in the above-described recognition model generation method and recognition method, they will not be described again here.

[0222] Exemplary storage media

[0223] Having introduced the identification model generation method and identification method, as well as the identification model generation apparatus and identification apparatus, according to exemplary embodiments of this disclosure, the following refers to... Figure 9 The storage medium of the exemplary embodiments of this disclosure will be described.

[0224] refer to Figure 9As shown, a program product 900 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a device such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0225] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0226] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0227] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0228] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0229] Exemplary electronic devices

[0230] Having described the storage medium of exemplary embodiments of this disclosure, the following references are made. Figure 10 An electronic device according to an exemplary embodiment of the present disclosure will be described.

[0231] Figure 10 The electronic device 1000 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0232] like Figure 10 As shown, the electronic device 1000 is manifested in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010), and a display unit 1040.

[0233] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1010 can perform, as follows: Figure 1 The steps are shown in the figure.

[0234] Storage unit 1020 may include volatile storage units, such as random access memory (RAM) 10201 and / or cache memory 10202, and may further include read-only memory (ROM) 10203.

[0235] Storage unit 820 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0236] Bus 1030 may include a data bus, an address bus, and a control bus.

[0237] Electronic device 1000 can also communicate with one or more external devices 1100 (e.g., keyboard, pointing device, Bluetooth device, etc.) via input / output (I / O) interface 1050. Electronic device 1000 also includes a display unit 1040 connected to input / output (I / O) interface 1050 for display purposes. Furthermore, electronic device 1000 can communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 1060. As shown, network adapter 1060 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0238] It should be noted that although the identification model generation apparatus and identification apparatus are described in detail above, such division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0239] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0240] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for generating a recognition model, characterized in that, The generated recognition model is used to identify the content information represented by the corresponding image to determine its matching with the preset scene, including: Based on the multiple target objects included in the preset scenario, the main task is decoupled into multiple sub-tasks. The main task is configured to identify the preset scenario, and each sub-task is configured to identify the target object. Construct an identification model, which includes at least one main task model for performing the main task and multiple subtask models for performing the subtasks; The constructed recognition model is trained using multiple image samples, each image sample being labeled with a first label corresponding to the main task and a second label corresponding to each sub-task; wherein, The first label information is used to distinguish the first positive example image and the first negative example image corresponding to the main task among the multiple image samples. The first positive example image is an image sample whose first label information is consistent with the preset scene, and the first negative example image is an image sample whose first label information is inconsistent with the preset scene. The step of decoupling the main task into multiple sub-tasks based on the multiple target objects included in the preset scenario includes: The common objects in each of the first positive example images are identified as the multiple target objects, so that the main task can be decoupled into multiple sub-tasks based on the multiple target objects; Determine multiple target objects included in the preset scene based on at least one of the following: The definition of the preset scene, the frequency of occurrence of the corresponding object in the image sample, and the semantic features corresponding to the preset scene are all defined. The frequency of occurrence of the corresponding object in the image sample is determined by the frequency ranking of all objects in the multiple image samples.

2. The recognition model generation method according to claim 1, characterized in that, Determining the common objects in each of the first positive example images as the multiple target objects, and decoupling the main task into multiple sub-tasks based on the multiple target objects, further includes: Based on the correlation relationships among the multiple target objects, at least one group of related objects and / or at least one independent object among the target objects are determined; The main task is decoupled into multiple subtasks based on at least one related object group and / or at least one independent object; In this context, a corresponding group of related objects corresponds to a subtask, and a corresponding independent object corresponds to a subtask.

3. The recognition model generation method according to claim 1, characterized in that, In the constructed recognition model, the feature data output by each sub-task model is input to the main task model after being fused by the fusion layer.

4. The recognition model generation method according to claim 3, characterized in that, The fusion operation includes: Add the feature data output by each of the sub-task models together.

5. The recognition model generation method according to claim 3, characterized in that, The fusion operation includes: The feature data output by each of the sub-task models are concatenated.

6. The recognition model generation method according to claim 3, characterized in that, The fusion operation includes: The output features of the sub-task models corresponding to at least one group of related objects are fused to obtain intermediate features; The output features of the subtask model corresponding to at least one independent object are fused with the intermediate features.

7. The recognition model generation method according to claim 1, characterized in that, In the constructed recognition model, each sub-task model shares an input processing layer. The output data of the input processing layer is input to each sub-task model. The input processing layer includes at least one convolutional layer and / or at least one pooling layer. The input processing layer has at least one pooling layer used to perform downsampling on the image data input to the input processing layer. At least one convolutional layer of the input processing layer is used to extract a shared feature map from the image data input to the input processing layer. The shared feature map represents the low-level image information contained in the image sample. The shared feature map is used to input into each of the sub-task models to identify the target object.

8. The recognition model generation method according to claim 1, characterized in that, Each of the subtask models includes at least one convolutional layer and / or at least one pooling layer. The subtask model has at least one pooling layer used to perform downsampling on the image data input to the subtask model. At least one convolutional layer of the subtask model is used to extract feature data from the image data input to the subtask model. The extracted feature data is the feature data of the target object to be identified by the subtask model.

9. The recognition model generation method according to claim 1, characterized in that, The main task model includes a feature processing layer and an output processing layer, which are connected sequentially. The feature processing layer includes at least one convolutional layer and / or at least one pooling layer. The output processing layer is used to calculate the recognition probability of the preset scene based on the feature data output by the feature processing layer. The recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene. Wherein, at least one pooling layer of the feature processing layer is used to perform downsampling operation on the feature data input to the main task model; At least one convolutional layer in the feature processing layer is used to extract feature data from the feature data input to the main task model. The extracted feature data is the feature data of the preset scene that the main task model is to identify.

10. The recognition model generation method according to claim 1, characterized in that, The step of training the constructed recognition model using multiple image samples includes: The multiple image samples are input into the constructed recognition model; Based on the feature data output by each subtask model and the second label information of the image sample, calculate the subtask loss function for each subtask model; The main task loss function of the main task model is calculated based on the output data of the main task model and the first label information of the image sample. The loss function of the recognition model is calculated based on the sub-task loss function and the main task loss function; The constructed recognition model is trained based on the loss function of the recognition model.

11. The recognition model generation method according to claim 10, characterized in that, The loss function of the recognition model is the sum of the weighted sum of the sub-task loss functions and the main task loss function. Training the constructed recognition model based on the loss function of the recognition model includes: Based on the loss function of the recognition model, gradient backpropagation is used to train the constructed recognition model. Specifically, during each gradient backpropagation, the weights of the loss functions of each subtask are updated according to the gradient loss of the recognition model, and the weights of the loss function of the main task are updated according to the gradient loss of the recognition model.

12. The recognition model generation method according to claim 10, characterized in that, The constructed recognition model is trained using a gradient normalization algorithm based on the loss function of the recognition model.

13. The recognition model generation method according to claim 10, characterized in that, The step of calculating the subtask loss function for each subtask model based on the feature data output by each subtask model includes: Based on the feature data output by the sub-task model, the recognition probability of the target object of the sub-task model is calculated. The recognition probability of the target object is the probability that the image data input to the sub-task model contains the target object. Based on the recognition probability of the target object in the sub-task model, determine the recognition result of the target object in the sub-task model; The subtask loss function of the subtask model is calculated based on the recognition result of the target object of the subtask model and the second label information of the image sample.

14. The recognition model generation method according to claim 10, characterized in that, The step of calculating the main task loss function of the main task model based on the output data of the main task model includes: Based on the recognition probability of the preset scene output by the main task model, the recognition result of the preset scene of the main task model is determined, and the recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene; The main task loss function of the main task model is calculated based on the recognition results of the preset scene of the main task model and the first label information of the image sample.

15. The recognition model generation method according to any one of claims 1 to 14, characterized in that, Before training the constructed recognition model using multiple image samples, the process also includes: The image samples are scaled to the same standard size.

16. A method for identification, characterized in that, include: The image to be recognized is input into the recognition model, which is generated using the recognition model generation method as described in any one of claims 1 to 15; The recognition result of the image to be recognized is obtained based on the output of the recognition model.

17. The identification method according to claim 16, characterized in that, The step of obtaining the recognition result of the image to be recognized based on the output of the recognition model includes: Based on the output of the recognition model, the probability that the image to be recognized matches the preset scene is determined, or whether the image to be recognized matches the preset scene is determined.

18. The identification method according to claim 16, characterized in that, Based on the output of the recognition model, the recognition result of the image to be recognized is obtained as follows: Based on the output of the recognition model, the preset scene that the image to be recognized conforms to is identified.

19. The identification method according to any one of claims 16 to 18, characterized in that, During training, the main task model and the sub-task model of the recognition model each include their own loss function calculation layers. During recognition, the loss function calculation layers do not operate.

20. A recognition model generation device, characterized in that, The generated recognition model is used to identify the content information represented by the corresponding image to determine its matching with the preset scene, including: A decoupling module is used to decouple the main task into multiple sub-tasks according to the multiple target objects included in the preset scenario. The main task is configured to identify the preset scenario, and each sub-task is configured to identify the target object. A building module is used to build a recognition model, the recognition model including at least one main task model for performing the main task, and multiple subtask models for performing the subtasks; The training module is used to train the constructed recognition model using multiple image samples, wherein the image samples are labeled with first label information corresponding to the main task and second label information corresponding to each sub-task; wherein, The first label information is used to distinguish the first positive example image and the first negative example image corresponding to the main task among the multiple image samples. The first positive example image is an image sample whose first label information is consistent with the preset scene, and the first negative example image is an image sample whose first label information is inconsistent with the preset scene. The decoupling module includes: a target object determination module, used to determine the objects common to each first positive example image as the multiple target objects, so as to decouple the main task into multiple sub-tasks based on the multiple target objects; The decoupling module is used to determine multiple target objects contained in a preset scene based on at least one of the following: the definition of the preset scene, the frequency of occurrence of the corresponding objects in the image samples, and the semantic features corresponding to the preset scene.

21. The recognition model generation apparatus according to claim 20, characterized in that, The target object determination module is used to determine at least one related object group and / or at least one independent object among the target objects based on the correlation relationship between the multiple target objects; and to decouple the main task into multiple subtasks based on the at least one related object group and / or at least one independent object; wherein, a corresponding related object group corresponds to a subtask, and a corresponding independent object corresponds to a subtask.

22. The recognition model generation apparatus according to claim 20, characterized in that, In the constructed recognition model, the feature data output by each sub-task model is input to the main task model after being fused by the fusion layer.

23. The recognition model generation apparatus according to claim 22, characterized in that, The fusion operation includes: Add the feature data output by each of the sub-task models together.

24. The recognition model generation apparatus according to claim 22, characterized in that, The fusion operation includes: The feature data output by each of the sub-task models are concatenated.

25. The recognition model generation apparatus according to claim 22, characterized in that, The fusion operation includes: The output features of the sub-task models corresponding to at least one group of related objects are fused to obtain intermediate features; The output features of the subtask model corresponding to at least one independent object are fused with the intermediate features.

26. The recognition model generation apparatus according to claim 20, characterized in that, In the constructed recognition model, each sub-task model shares an input processing layer. The output data of the input processing layer is input to each sub-task model. The input processing layer includes at least one convolutional layer and / or at least one pooling layer. The input processing layer has at least one pooling layer used to perform downsampling on the image data input to the input processing layer. At least one convolutional layer of the input processing layer is used to extract a shared feature map from the image data input to the input processing layer. The shared feature map represents the low-level image information contained in the image sample. The shared feature map is used to input into each of the sub-task models to identify the target object.

27. The recognition model generation apparatus according to claim 20, characterized in that, Each of the subtask models includes at least one convolutional layer and / or at least one pooling layer. The subtask model has at least one pooling layer used to perform downsampling on the image data input to the subtask model. At least one convolutional layer of the subtask model is used to extract feature data from the image data input to the subtask model. The extracted feature data is the feature data of the target object to be identified by the subtask model.

28. The recognition model generation apparatus according to claim 20, characterized in that, The main task model includes a feature processing layer and an output processing layer, which are connected sequentially. The feature processing layer includes at least one convolutional layer and / or at least one pooling layer. The output processing layer is used to calculate the recognition probability of the preset scene based on the feature data output by the feature processing layer. The recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene. Wherein, at least one pooling layer of the feature processing layer is used to perform downsampling operation on the feature data input to the main task model; At least one convolutional layer in the feature processing layer is used to extract feature data from the feature data input to the main task model. The extracted feature data is the feature data of the preset scene that the main task model is to identify.

29. The recognition model generation apparatus according to claim 20, characterized in that, The training module includes: The input module is used to input the multiple image samples into the constructed recognition model; The subtask loss function calculation module is used to calculate the subtask loss function of each subtask model based on the feature data output by each subtask model and the second label information of the image sample. The main task loss function calculation module is used to calculate the main task loss function of the main task model based on the output data of the main task model and the first label information of the image sample. The model loss function calculation module is used to calculate the loss function of the recognition model based on the sub-task loss function and the main task loss function; The loss training module is used to train the constructed recognition model according to the loss function of the recognition model.

30. The recognition model generation apparatus according to claim 29, characterized in that, The loss function of the recognition model is the sum of the weighted sum of the sub-task loss functions and the main task loss function. The loss training module is used for: Based on the loss function of the recognition model, gradient backpropagation is used to train the constructed recognition model. Specifically, during each gradient backpropagation, the weights of the loss functions of each subtask are updated according to the gradient loss of the recognition model, and the weights of the loss function of the main task are updated according to the gradient loss of the recognition model.

31. The recognition model generation apparatus according to claim 29, characterized in that, The constructed recognition model is trained using a gradient normalization algorithm based on the loss function of the recognition model.

32. The recognition model generation device according to claim 29, characterized in that, The subtask loss function calculation module is used for: Based on the feature data output by the sub-task model, the recognition probability of the target object of the sub-task model is calculated. The recognition probability of the target object is the probability that the image data input to the sub-task model contains the target object. Based on the recognition probability of the target object in the sub-task model, determine the recognition result of the target object in the sub-task model; The subtask loss function of the subtask model is calculated based on the recognition result of the target object of the subtask model and the second label information of the image sample.

33. The recognition model generation apparatus according to claim 29, characterized in that, The main task loss function calculation module is used for: Based on the recognition probability of the preset scene output by the main task model, the recognition result of the preset scene of the main task model is determined, and the recognition probability of the preset scene is the probability that the feature data input to the main task model is the preset scene; The main task loss function of the main task model is calculated based on the recognition results of the preset scene of the main task model and the first label information of the image sample.

34. The recognition model generation apparatus according to any one of claims 20 to 33, characterized in that, Also includes: The scaling module is used to scale the image sample to the same standard size.

35. An identification device, characterized in that, include: The model input module is used to input the image to be recognized into the recognition model, wherein the recognition model is generated using the recognition model generation method as described in any one of claims 1 to 15; The recognition module is used to obtain the recognition result of the image to be recognized based on the output of the recognition model.

36. The identification device according to claim 35, characterized in that, The recognition module is used to identify the probability that the image to be recognized matches the preset scene or to identify whether the image to be recognized matches the preset scene based on the output of the recognition model.

37. The identification device according to claim 35, characterized in that, The recognition module is used to identify the preset scene that the image to be recognized conforms to based on the output of the recognition model.

38. The identification device according to any one of claims 35 to 37, characterized in that, During training, the main task model and the sub-task model of the recognition model each include their own loss function calculation layers. During recognition, the loss function calculation layers do not operate.

39. A storage medium having a computer program stored thereon, characterized in that, The computer program is executed by the processor to achieve the following: The method for generating a recognition model according to any one of claims 1 to 15; and / or The identification method according to any one of claims 16 to 19.

40. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the executable instructions: The method for generating a recognition model according to any one of claims 1 to 15; and / or The identification method according to any one of claims 16 to 19.

Images