A model training, image recognition method, device, equipment and storage medium

Abstract

This specification discloses a model training, image recognition method, apparatus, device and storage medium for privacy protection. The deviation between the probability value of a sample image belonging to each parent category and the probability value of a sample image belonging to a subcategory under each category can be used as a constraint condition to ensure that the probability value of a sample image output by the image recognition model belonging to any parent category and the probability value of a sample image output by the image recognition model belonging to a subcategory under this parent category are consistent. Therefore, during the training of the image recognition model, the image recognition model can be trained using fine-grained sample images while using some coarse-grained sample images. Therefore, the training effect of the image recognition model can be improved based on a given limited number of fine-grained sample images.

Description

Technical Field

[0001] This specification relates to the field of image recognition technology, and in particular to a model training, image recognition method, device, equipment and storage medium. Background Art

[0002] With the development of machine learning technology, image recognition models have been widely used in various fields. In the process of training image recognition models, the commonly used training method is supervised training. In the supervised training process, it is necessary to construct labeled data that matches the recognition granularity of the image recognition model (for example, coarse-grained animal species recognition, such as cats, dogs, etc., and fine-grained animal breed recognition, such as Huskies, Golden Retrievers, etc.) for training the image recognition model.

[0003] Among them, it is more difficult to obtain fine-grained annotated data for training image recognition models. For example, fine-grained image annotation requires high requirements on the quality of the images and the background knowledge possessed by the annotators, which leads to a limited amount of fine-grained annotated data obtained for training image recognition models, and thus makes the training effect of the image recognition model poor. Summary of the Invention

[0004] This specification provides a model training, image recognition method, device, equipment and storage medium to partially solve the problems existing in the existing technology.

[0005] This manual adopts the following technical solutions:

[0006] This specification provides a model training method, which includes:

[0007] Get a sample image;

[0008] Inputting the sample image into an image recognition model to determine, by the image recognition model, a probability value of the sample image belonging to each preset category, wherein each category includes parent categories and subcategories divided according to different granularity levels;

[0009] For each parent category, determining a first loss corresponding to the parent category based on a deviation between a probability value that the sample image belongs to the parent category and a probability value that the sample image belongs to a subcategory under the parent category;

[0010] Determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as optimization objectives.

[0011] Optionally, determining the probability value of the sample image belonging to each preset category by the image recognition model specifically includes:

[0012] determining image features of the sample image by using the image recognition model;

[0013] For each category, determining a correlation feature between the image feature and an initial category feature of the category, and adjusting the initial category feature using the correlation feature to obtain an adjusted category feature corresponding to the initial category feature;

[0014] According to the adjusted category feature, a probability value of the sample image belonging to the category corresponding to the adjusted category feature is determined.

[0015] Optionally, the image recognition model includes: an image feature extraction network and a feature interaction network;

[0016] Determining the image features of the sample image by using the image recognition model specifically includes:

[0017] encoding the sample image through the image feature extraction network of the image recognition model to determine image features of the sample image;

[0018] For each category, determining the correlation feature between the image feature and the initial category feature of the category, and adjusting the initial category feature using the correlation feature to obtain an adjusted category feature corresponding to the initial category feature, specifically including:

[0019] Through the feature interaction network of the image recognition model, for each category, the association features between the image features and the initial category features of the category are determined, and the initial category features are adjusted through the association features to obtain adjusted category features corresponding to the initial category features.

[0020] Optionally, before training the image recognition model, the method further comprises: determining the label parent category corresponding to the sample image, and taking at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as optimization objectives.

[0021] For each parent category, determining a second loss corresponding to the parent category based on a deviation between a probability value of the sample image belonging to the parent category and a probability value of the sample image belonging to a subcategory under the parent category, and based on a deviation between a probability value of the sample image belonging to the parent category and a probability value of the sample image belonging to a subcategory under another parent category;

[0022] Determining the label parent category corresponding to the sample image, and training the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as optimization objectives, specifically comprising:

[0023] Determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category and the second loss corresponding to each parent category as optimization objectives.

[0024] Optionally, before training the image recognition model, the method further comprises: determining the label parent category corresponding to the sample image, and taking at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as optimization objectives.

[0025] For each granularity level, the category to which the sample image actually belongs in the granularity level is used as the target category;

[0026] determining a third loss according to a difference between a probability value that the sample image belongs to the target category and a probability value that the sample image belongs to other categories in the granularity level;

[0027] Determining the label parent category corresponding to the sample image, and training the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as optimization objectives, specifically comprising:

[0028] Determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss and each third loss corresponding to each parent category as optimization objectives.

[0029] This specification provides an image recognition method, including:

[0030] Obtain the image to be recognized;

[0031] Inputting the image to be identified into a pre-trained image recognition model to determine the probability value of the image to be identified belonging to each preset category through the image recognition model, wherein the image recognition model is trained using the above-mentioned model training method;

[0032] According to the probability value, a recognition result of the image to be recognized is determined, and a task is executed according to the recognition result.

[0033] This specification provides a model training device, including:

[0034] An acquisition module, used for acquiring a sample image;

[0035] a first determination module, configured to input the sample image into an image recognition model to determine, through the image recognition model, a probability value of the sample image belonging to each preset category, wherein each category includes a parent category and subcategories divided according to different granularity levels;

[0036] a second determination module, configured to determine, for each parent category, a first loss corresponding to the parent category based on a deviation between a probability value that the sample image belongs to the parent category and a probability value that the sample image belongs to a subcategory under the parent category;

[0037] A training module is used to determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as optimization objectives.

[0038] Optionally, the first determination module is specifically used to determine the image features of the sample image through the image recognition model; for each category, determine the association features between the image features and the initial category features of the category, and adjust the initial category features through the association features to obtain adjusted category features corresponding to the initial category features; based on the adjusted category features, determine the probability value that the sample image belongs to the category corresponding to the adjusted category features.

[0039] Optionally, the image recognition model includes: an image feature extraction network and a feature interaction network;

[0040] The first determination module is specifically used to encode the sample image through the image feature extraction network of the image recognition model to determine the image features of the sample image; determine the association features between the image features and the initial category features of the category for each category through the feature interaction network of the image recognition model, and adjust the initial category features through the association features to obtain adjusted category features corresponding to the initial category features.

[0041] Optionally, the second determination module is specifically used to, for each parent category, determine the second loss corresponding to the parent category based on the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under the parent category, and based on the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under other parent categories; determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category, and minimizing the first loss corresponding to each parent category and the second loss corresponding to each parent category as optimization goals.

[0042] Optionally, the second determination module is specifically used to, for each granularity level, take the category to which the sample image actually belongs in the granularity level as the target category; determine the third loss based on the difference between the probability value of the sample image belonging to the target category and the probability values ​​of the sample image belonging to other categories in the granularity level; determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss and each third loss corresponding to each parent category as optimization goals.

[0043] This specification provides a data query device, including:

[0044] An image acquisition module, used to acquire an image to be identified;

[0045] a recognition module, configured to input the image to be recognized into a pre-trained image recognition model, so as to determine, by means of the image recognition model, a probability value that the image to be recognized belongs to each preset category, wherein the image recognition model is trained using the above-mentioned model training method;

[0046] An execution module is used to determine a recognition result of the image to be recognized according to the probability value, and perform a task according to the recognition result.

[0047] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned model training and image recognition methods.

[0048] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned model training and image recognition methods are implemented.

[0049] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0050] In the model training method provided in this specification, a sample image is first obtained and the sample image is input into an image recognition model to determine the probability value of the sample image belonging to each preset category through the image recognition model, wherein each category includes parent categories and subcategories divided according to different granularity levels. For each parent category, the first loss corresponding to the parent category is determined based on the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under the parent category, and the label parent category corresponding to the sample image is determined. The image recognition model is trained with at least minimizing the deviation between the probability of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as optimization goals.

[0051] It can be seen from the above method that the deviation between the probability value of the sample image belonging to each parent category and the probability value of the sample image belonging to the subcategory under each category can be used as a constraint condition to ensure that the probability value of the sample image output by the image recognition model belongs to any parent category and the probability value of the sample image output by the image recognition model belongs to the subcategory under this parent category are consistent. In this way, in the process of training the image recognition model, the image recognition model can be trained using fine-grained sample images while using some coarse-grained sample images to train the image recognition model. Therefore, the training effect of the image recognition model can be improved based on a given limited number of fine-grained sample images. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification.

[0053] In the picture:

[0054] Figure 1 A flowchart of a model training method provided in this specification;

[0055] Figure 2 A schematic diagram of different categories at different granularity levels provided in this specification;

[0056] Figure 3 A schematic diagram of the image recognition model provided in this specification;

[0057] Figure 4 A flowchart of an image recognition method provided in this specification;

[0058] Figure 5 A schematic diagram of a model training device provided in this specification;

[0059] Figure 6 A schematic diagram of an image recognition device provided in this specification;

[0060] Figure 7 This manual provides a corresponding Figure 1 Schematic diagram of the electronic device. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0062] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0063] Figure 1 This is a flow chart of a model training method provided in this specification, which includes the following steps:

[0064] S100: Acquire a sample image.

[0065] In this specification, when the business platform needs to train the image recognition model, it can first obtain sample images for training the image recognition model, and then train the image recognition model with the obtained sample images.

[0066] The above sample images can be sample images of different categories at different granularity levels, such as Figure 2 shown.

[0067] Figure 2 A schematic diagram of the different categories in the different granularity levels provided in this specification.

[0068] Combine Figure 2 It can be seen that when identifying an image, it is necessary to determine the category to which the image belongs in order to obtain the recognition result. When determining the category to which the image belongs, there can be different granularity levels, such as Figure 2 The first granularity level: birds, ships, etc.; the second granularity level: albatross, eagles, merchant ships, aircraft carriers, etc.; the third granularity level: Laysan albatross, black-footed albatross, Gyrfalcon, falcon, Nimitz-class aircraft carrier, Enterprise-class aircraft carrier, ocean-going ship, coastal ship, etc.

[0069] The categories Albatross and Eagle at the second granularity level have an affiliation with the category Bird at the first granularity level. That is, Albatross and Eagle belong to Bird. In other words, the category Bird at the first granularity level is the parent category of the categories Albatross and Eagle at the second granularity level, and the categories Albatross and Eagle at the second granularity level are subcategories of the category Bird at the first granularity level. Similarly, the Laysan Albatross, Black-footed Albatross, and Albatross have an affiliation, and the Gerschinensis and Falcon have an affiliation with Eagle.

[0070] It should be noted that the sample images obtained by the above-mentioned business platform may include fine-grained sample images and some coarse-grained sample images, thereby improving the training effect of the image recognition model based on the given fine-grained sample images.

[0071] In this specification, the execution entity used to implement the model training method can refer to a designated device such as a server set up on the business platform, or it can refer to a terminal device such as a desktop computer, a laptop computer, etc. For the sake of convenience of description, the model training method provided in this specification is explained below using the server as the execution entity as an example.

[0072] S102: Inputting the sample image into an image recognition model to determine the probability value of the sample image belonging to each preset category through the image recognition model, wherein each category includes a parent category and a subcategory divided according to different granularity levels.

[0073] Furthermore, the server can input the acquired sample image into the image recognition model to determine the probability value of the sample image belonging to each preset category through the image recognition model. Here, each category includes parent categories and subcategories divided according to different granularity levels, as shown in the following example: Figure 3 shown.

[0074] Figure 3 This is a schematic diagram of the image recognition model provided in this manual.

[0075] Combine Figure 3 As can be seen, the image recognition model includes: an image feature extraction network and a feature interaction network. The server can encode the sample image through the image feature extraction network of the image recognition model to determine the image features of the sample image. The server can then use the feature interaction network of the image recognition model to determine, for each category, the association features between the image features and the initial category features of that category. The server can then adjust the initial category features using the association features to obtain adjusted category features corresponding to the initial category features. Based on the adjusted category features, the server can then determine the probability value of the sample image belonging to the category corresponding to the adjusted category features.

[0076] It should be noted that the above-mentioned correlation features are used to characterize the image regions related to the category contained in the sample image.

[0077] Furthermore, the feature interaction network of the image recognition model includes an encoding layer, a decoding layer, and a linear projection layer. The method by which the server obtains adjusted category features corresponding to initial category features through the feature interaction network of the image recognition model may be that the server may input image features of a sample image and initial category features of each category into the feature interaction network, thereby determining self-attention codes corresponding to the image features of the sample image through the encoding layer of the feature interaction network.

[0078] Specifically, the server can determine the attention weight of each pixel in the sample image based on the importance of each pixel in the sample image to the entire sample image, and then determine the self-attention encoding of the sample image based on the attention weight of each pixel contained in the sample image and the image features of the sample image.

[0079] In addition, the server can also determine the attention weight of each convolution kernel based on the importance of each convolution kernel used when encoding the sample image, and then determine the self-attention encoding of the sample image based on the attention weight of each convolution kernel and the image features of the sample image determined by each convolution kernel.

[0080] Furthermore, after the server determines the self-attention code corresponding to the image features of the sample image through the encoding layer of the feature interaction network, it can determine the association features between the image features of the sample image and the initial category features of each category based on the self-attention code corresponding to the determined image features of the sample image and the initial category features of each category through the decoding layer of the feature interaction network, and then adjust the initial category features of each category through the association features to obtain the adjusted category features corresponding to the initial category features of each category, wherein the adjusted category features here can be used as the initialization category features input together with the image features of the sample image when the image features of a new sample image are input into the feature interaction network of the image recognition model next time.

[0081] The method by which the server determines the correlation features between the image features and the initial category features of each category through the feature interaction network may be an interactive learning method based on the cross-attention mechanism.

[0082] Furthermore, after the server determines the adjusted category features corresponding to the image features of the sample image through the decoding layer of the feature interaction network, it can input the determined adjusted category features of each category into the linear projection layer of the feature interaction network to determine the probability value of the sample image belonging to each category through the linear projection layer.

[0083] From the above content, it can be seen that in the process of training the image recognition model through sample images, the category features of each category can be continuously adjusted based on the associated features related to the sample images and each category.

[0084] S104: For each parent category, determine a first loss corresponding to the parent category according to a deviation between a probability value of the sample image belonging to the parent category and a probability value of the sample image belonging to a subcategory under the parent category.

[0085] S106: Determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as optimization goals.

[0086] In this specification, the server can determine the first loss corresponding to each category based on the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under the parent category, wherein the smaller the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under the parent category, the smaller the determined first loss.

[0087] Furthermore, the server can determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as the optimization goal. The label parent category here can be the parent category to which the sample image actually belongs.

[0088] From the above content, it can be seen that the server can use the deviation between the probability value of the sample image belonging to each parent category and the probability value of the sample image belonging to the subcategory under each category as a constraint condition, so that the probability value of the sample image output by the image recognition model belonging to any parent category and the probability value of the sample image output by the image recognition model belonging to the subcategory under this parent category are consistent, and in the process of adjusting the category features of each category, each subcategory can learn part of the category features of the parent category of the subcategory, thereby increasing the robustness of the image recognition model.

[0089] In addition, the server can also determine the second loss corresponding to each parent category based on the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under the parent category, and based on the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under other parent categories.

[0090] Among them, the smaller the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under the parent category compared to the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under other parent categories, the smaller the second loss.

[0091] Then, the label parent category corresponding to the sample image can be determined, and the image recognition model can be trained with the optimization goal of at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category, and minimizing the first loss corresponding to each parent category and the second loss corresponding to each parent category.

[0092] Additionally, the server may determine, for each granularity level, a third loss based on the difference between the probability value of the sample image belonging to the target category and the probability values ​​of the sample image belonging to other categories at the granularity level. The greater the difference between the probability value of the sample image belonging to the target category and the probability values ​​of the sample image belonging to other categories at the granularity level, the smaller the third loss.

[0093] Then, the label parent category corresponding to the sample image can be determined, and the image recognition model can be trained with the optimization goal of at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss and each third loss corresponding to each parent category.

[0094] It should be noted that the server can train the image recognition model by at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the labeled parent category, the first loss mentioned above, and at least one of the second loss and the third loss. That is, the second loss and the third loss mentioned above can be used alone together with the first loss to train the image recognition model, or together with another loss, the first loss to train the image recognition model.

[0095] For example, the server can perform a weighted fusion of the first loss, second loss, and third loss to obtain the target loss of the image recognition model. The image recognition model can then be trained with the optimization goal of minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the target loss.

[0096] From the above content, it can be seen that the affiliation relationship between categories at different granularity levels can be used to train the image recognition model, thereby improving the training effect of the image recognition model under the condition of limited samples.

[0097] In addition, in actual application scenarios, the cost of collecting high-quality sample images is often high, which leads to high costs for training image recognition models. The server can use the above method to reduce the requirements for sample images for training image recognition models, thereby reducing the cost of training image recognition models.

[0098] In order to explain the above content in detail, this specification also provides a method for performing image recognition using an image recognition model trained by the above method, specifically as follows: Figure 4 shown.

[0099] Figure 4 The following is a flow chart of an image recognition method provided in this specification, which includes the following steps:

[0100] S400: Acquire an image to be recognized;

[0101] S402: Inputting the image to be identified into a pre-trained image recognition model to determine the probability value of the image to be identified belonging to each preset category through the image recognition model, wherein the image recognition model is trained using the above-mentioned model training method;

[0102] S404: Determine a recognition result of the image to be recognized according to the probability value, and perform a task according to the recognition result.

[0103] In this specification, when the server needs to perform image recognition, it can obtain the image to be recognized and input the image to be recognized into a pre-trained image recognition model to determine the probability value of the image to be recognized belonging to each preset category through the image recognition model, wherein the image recognition model is trained by the above-mentioned model training method, and then the recognition result of the image to be recognized can be determined based on the determined probability value of the image to be recognized belonging to each preset category, and the task can be executed based on the recognition result.

[0104] The aforementioned tasks can be, for example, generating corresponding voice data based on the recognition results of the image to be recognized, and playing the generated voice data through the device used by the user. Another example is selecting products that match the recognition results from various products and recommending them to the user.

[0105] It can be seen from the above content that the recognition result of the image to be recognized can be obtained through the trained image recognition model.

[0106] The above is a model training and image recognition method provided in one or more embodiments of this specification. Based on the same idea, this specification also provides corresponding model training and image recognition devices, such as Figure 5 、 6 shown.

[0107] Figure 5 A schematic diagram of a model training device provided in this specification, including:

[0108] An acquisition module 501 is used to acquire a sample image;

[0109] A first determination module 502 is configured to input the sample image into an image recognition model to determine, through the image recognition model, a probability value of the sample image belonging to each preset category, wherein each category includes a parent category and subcategories divided according to different granularity levels;

[0110] A second determination module 503 is configured to determine, for each parent category, a first loss corresponding to the parent category based on a deviation between a probability value of the sample image belonging to the parent category and a probability value of the sample image belonging to a subcategory under the parent category;

[0111] The training module 504 is used to determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as the optimization goal.

[0112] Optionally, the first determination module 502 is specifically used to determine the image features of the sample image through the image recognition model; for each category, determine the association features between the image features and the initial category features of the category, and adjust the initial category features through the association features to obtain adjusted category features corresponding to the initial category features; based on the adjusted category features, determine the probability value that the sample image belongs to the category corresponding to the adjusted category features.

[0113] Optionally, the image recognition model includes: an image feature extraction network and a feature interaction network;

[0114] The first determination module 502 is specifically used to encode the sample image through the image feature extraction network of the image recognition model to determine the image features of the sample image; determine the association features between the image features and the initial category features of the category for each category through the feature interaction network of the image recognition model, and adjust the initial category features through the association features to obtain adjusted category features corresponding to the initial category features.

[0115] Optionally, the second determination module 503 is specifically used to, for each parent category, determine the second loss corresponding to the parent category based on the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under the parent category, and based on the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under other parent categories; determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category, and minimizing the first loss corresponding to each parent category and the second loss corresponding to each parent category as optimization goals.

[0116] Optionally, the second determination module 503 is specifically used to, for each granularity level, take the category to which the sample image actually belongs in the granularity level as the target category; determine the third loss based on the difference between the probability value of the sample image belonging to the target category and the probability value of the sample image belonging to other categories in the granularity level; determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss and each third loss corresponding to each parent category as optimization goals.

[0117] Figure 6 A schematic diagram of a model training device provided in this specification, including:

[0118] Image acquisition module 601, used to acquire the image to be identified;

[0119] Recognition module 602, configured to input the image to be recognized into a pre-trained image recognition model to determine the probability value of the image to be recognized belonging to each preset category through the image recognition model, wherein the image recognition model is trained using the above-mentioned model training method;

[0120] The execution module 603 is configured to determine a recognition result of the image to be recognized according to the probability value, and perform a task according to the recognition result.

[0121] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 A model training and image recognition method is provided.

[0122] This manual also provides Figure 7 The one shown corresponds to Figure 1 Schematic diagram of the electronic equipment. Figure 4 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Model training and image recognition methods. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0123] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0124] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0125] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0126] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0127] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable application detection device to produce a machine, so that the instructions executed by the processor of the computer or other programmable application detection device generate instructions for implementing the process in the flowchart. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable application detection device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable application detection device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0131] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0132] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0133] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0134] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0135] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0137] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0138] The above are merely examples of the present invention and are not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A model training method, comprising: Get a sample image; Inputting the sample image into an image recognition model to determine, by the image recognition model, a probability value of the sample image belonging to each preset category, wherein each category includes parent categories and subcategories divided according to different granularity levels; For each parent category, determining a first loss corresponding to the parent category based on a deviation between a probability value that the sample image belongs to the parent category and a probability value that the sample image belongs to a subcategory under the parent category; For each parent category, determining a second loss corresponding to the parent category based on a deviation between a probability value of the sample image belonging to the parent category and a probability value of the sample image belonging to a subcategory under the parent category, and based on a deviation between a probability value of the sample image belonging to the parent category and a probability value of the sample image belonging to a subcategory under another parent category; Determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as optimization objectives; Among them, the image recognition model is trained with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the labeled parent category and minimizing the first loss corresponding to each parent category as the optimization goal, specifically including: training the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the labeled parent category and minimizing the first loss corresponding to each parent category and the second loss corresponding to each parent category as the optimization goal.

2. The method according to claim 1, wherein determining the probability value of the sample image belonging to each preset category by the image recognition model specifically comprises: determining image features of the sample image by using the image recognition model; For each category, determining a correlation feature between the image feature and an initial category feature of the category, and adjusting the initial category feature using the correlation feature to obtain an adjusted category feature corresponding to the initial category feature; According to the adjusted category feature, a probability value of the sample image belonging to the category corresponding to the adjusted category feature is determined.

3. The method according to claim 2, wherein the image recognition model comprises: Image feature extraction network, feature interaction network; Determining the image features of the sample image by using the image recognition model specifically includes: encoding the sample image through the image feature extraction network of the image recognition model to determine image features of the sample image; For each category, determining the correlation feature between the image feature and the initial category feature of the category, and adjusting the initial category feature using the correlation feature to obtain an adjusted category feature corresponding to the initial category feature, specifically including: Through the feature interaction network of the image recognition model, for each category, the association features between the image features and the initial category features of the category are determined, and the initial category features are adjusted through the association features to obtain adjusted category features corresponding to the initial category features.

4. The method of claim 1, further comprising determining the labeled parent category corresponding to the sample image, and optimizing at least the deviation between the probability value of the sample image belonging to each parent category and the labeled parent category and minimizing the first loss corresponding to each parent category, before training the image recognition model: For each granularity level, the category to which the sample image actually belongs in the granularity level is used as the target category; determining a third loss according to a difference between a probability value that the sample image belongs to the target category and a probability value that the sample image belongs to other categories in the granularity level; Determining the label parent category corresponding to the sample image, and training the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as optimization objectives, specifically comprising: Determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss and each third loss corresponding to each parent category as optimization objectives.

5. An image recognition method, comprising: Obtain the image to be recognized; Inputting the image to be identified into a pre-trained image recognition model to determine the probability value of the image to be identified belonging to each preset category through the image recognition model, wherein the image recognition model is trained by the method of any one of claims 1 to 4 above; According to the probability value, a recognition result of the image to be recognized is determined, and a task is executed according to the recognition result.

6. A model training device comprising: An acquisition module, used for acquiring a sample image; a first determination module, configured to input the sample image into an image recognition model to determine, through the image recognition model, a probability value of the sample image belonging to each preset category, wherein each category includes a parent category and subcategories divided according to different granularity levels; a second determination module, configured to determine, for each parent category, a first loss corresponding to the parent category based on a deviation between a probability value that the sample image belongs to the parent category and a probability value that the sample image belongs to a subcategory under the parent category; a training module, configured to determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss corresponding to each parent category as optimization objectives; The second determination module is specifically used to, for each parent category, determine the second loss corresponding to the parent category based on the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under the parent category, and based on the deviation between the probability value of the sample image belonging to the parent category and the probability value of the sample image belonging to the subcategory under other parent categories; determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category, and minimizing the first loss corresponding to each parent category and the second loss corresponding to each parent category as optimization goals.

7. In the device as described in claim 6, the first determination module is specifically used to determine the image features of the sample image through the image recognition model; for each category, determine the association features between the image features and the initial category features of the category, and adjust the initial category features through the association features to obtain adjusted category features corresponding to the initial category features; based on the adjusted category features, determine the probability value of the sample image belonging to the category corresponding to the adjusted category features.

8. The apparatus of claim 7, wherein the image recognition model comprises: Image feature extraction network, feature interaction network; The first determination module is specifically used to encode the sample image through the image feature extraction network of the image recognition model to determine the image features of the sample image; determine the association features between the image features and the initial category features of the category for each category through the feature interaction network of the image recognition model, and adjust the initial category features through the association features to obtain adjusted category features corresponding to the initial category features.

9. In the device as described in claim 6, the second determination module is specifically used to, for each granularity level, take the category to which the sample image actually belongs in the granularity level as the target category; determine the third loss based on the difference between the probability value of the sample image belonging to the target category and the probability value of the sample image belonging to other categories in the granularity level; determine the label parent category corresponding to the sample image, and train the image recognition model with at least minimizing the deviation between the probability value of the sample image belonging to each parent category and the label parent category and minimizing the first loss and each third loss corresponding to each parent category as optimization goals.

10. An image recognition device, comprising: An image acquisition module, used to acquire an image to be identified; a recognition module, configured to input the image to be recognized into a pre-trained image recognition model to determine, by means of the image recognition model, a probability value that the image to be recognized belongs to each preset category, wherein the image recognition model is trained using the method of any one of claims 1 to 4; An execution module is used to determine a recognition result of the image to be recognized according to the probability value, and perform a task according to the recognition result.

11. A computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 4 when executed by a processor.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 4 when executing the program.

Images