Image processing model training method, processing method, device, equipment and medium

Abstract

The present application provides an image processing model training method, processing method, device, equipment and medium, which relate to the field of artificial intelligence technology, and in particular to the field of machine learning technology, and are used to improve the accuracy of the output categories of the image processing model. When training the image processing model, the image processing model training method trains the feature extraction module in the image processing model to learn the features required for outputting the corresponding categories at at least three different granularity levels, so that the feature extraction module has the ability to learn the features required for the categories at at least three different granularity levels, so that when the image processing model subsequently outputs the category of the image at any granularity level, the output results are obtained based on the features required for the category output at at least three different granularity levels, and the output results are limited by the granularity hierarchy relationship between at least three different granularity levels, thereby improving the accuracy of the output results corresponding to the image processing model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, in particular to the field of machine learning technology, and provides an image processing model training method, processing method, device, equipment and medium. Background Art

[0002] Artificial intelligence relies on a large number of images to develop, and how to retrieve the required images from a large number of images is a technical problem that needs to be solved urgently. In order to improve the efficiency of image retrieval, a large number of images are usually clustered.

[0003] At present, the K-means clustering method is generally used to cluster images. The specific process involved is: select multiple cluster centers from each local feature in multiple images; cluster all images according to these multiple cluster centers; use the mean of the local features belonging to the same cluster as the new cluster center, until the Euclidean distance between the new cluster center and the previous cluster center is less than the specified value, obtain multiple cluster centers, cluster the image to the cluster center with the shortest Euclidean distance to its local features, so as to complete the clustering process of multiple images.

[0004] However, when clustering images using this clustering method, the clustering results are highly dependent on the determined cluster centers. Once the determined cluster centers are unreasonable, the clustering results determined based on the Euclidean distance between the local features of the image and the cluster centers will have low accuracy. Summary of the invention

[0005] Embodiments of the present application provide an image processing model training method, processing method, apparatus, device and medium for improving the accuracy of the category of an image output by an image processing model.

[0006] On the one hand, a method for training an image processing model is provided, comprising:

[0007] Based on the sample image set, the image processing model to be trained is iteratively trained until a first convergence condition is met, wherein in one iterative training process, the following operations are performed:

[0008] Inputting a first sample image in the sample image set into a feature extraction module in the image processing model to obtain a first sample feature corresponding to the first sample image;

[0009] Inputting the first sample feature into a category recognition module in the image processing model to obtain predicted categories corresponding to the first sample image at at least three clustering granularities, wherein the at least three clustering granularities include clustering granularities of at least three different granularity levels;

[0010] Based on the obtained predicted categories and the real categories corresponding to the first sample image at the at least three clustering granularities, a training loss value is determined, and the parameters of the feature extraction module and the category recognition module are adjusted based on the training loss value, wherein the training loss value is used to represent the error between the obtained predicted categories and the corresponding real categories.

[0011] On the one hand, an embodiment of the present application provides an image processing method, including:

[0012] Obtaining a trained image processing model, wherein the trained image processing model is obtained by any of the image processing model training methods discussed above, and the trained image processing model includes a feature extraction module and a category recognition module;

[0013] Inputting the image to be processed into the feature extraction module to obtain the target features corresponding to the image to be processed;

[0014] The target feature is input into the category recognition module to obtain the target category of the image to be processed, wherein the target category belongs to the predicted category corresponding to each of the image to be processed output by the category recognition module at at least three clustering granularities, and the at least three clustering granularities include clustering granularities of at least three different granularity levels.

[0015] On the one hand, an embodiment of the present application provides an image processing model training device, the device comprising:

[0016] An image acquisition module, used to acquire a sample image set;

[0017] The model training module is used to iteratively train the image processing model to be trained based on the sample image set until a first convergence condition is met, wherein in one iterative training process, the following operations are performed:

[0018] Inputting a first sample image in the sample image set into a feature extraction module in the image processing model to obtain a first sample feature corresponding to the first sample image;

[0019] Inputting the first sample feature into a category recognition module in the image processing model to obtain predicted categories corresponding to the first sample image at at least three clustering granularities, wherein the at least three clustering granularities include clustering granularities of at least three different granularity levels;

[0020] Based on the obtained predicted categories and the real categories corresponding to the first sample image at the at least three clustering granularities, a training loss value is determined, and the parameters of the feature extraction module and the category recognition module are adjusted based on the training loss value, wherein the training loss value is used to represent the error between the obtained predicted categories and the corresponding real categories.

[0021] In a possible embodiment, the category identification module includes a plurality of category identification branch modules; and the model training module is specifically used for:

[0022] The first sample features are respectively input into each category recognition branch module in the category recognition module to obtain the predicted categories output by each category recognition branch module, wherein each predicted category output by each category recognition branch module includes at least corresponding categories under at least three clustering granularities of different granularity levels.

[0023] In a possible embodiment, the multiple category identification branch modules include at least a first category identification branch module, a second category identification branch module and a third category identification branch module; the model training module is specifically used for:

[0024] Inputting the first sample features into the first category recognition branch module respectively to obtain the predicted category corresponding to the first sample image at the first clustering granularity; and

[0025] Inputting the first sample features into the second category recognition branch module respectively to obtain the predicted category corresponding to the first sample image at a second clustering granularity, wherein the granularity level of the second clustering granularity is smaller than the granularity level of the first clustering granularity; and

[0026] The first sample features are respectively input into the third category recognition branch module to obtain the predicted category corresponding to the first sample image at the third clustering granularity, wherein the granularity level of the third clustering granularity is smaller than the granularity level of the second clustering granularity.

[0027] In a possible embodiment, the model training module is specifically used for:

[0028] Before adjusting the parameters of the feature extraction module and the category recognition module based on the training loss value, the feature extraction module and the first category recognition branch module are iteratively trained based on the sample image set until the feature extraction module and the first category recognition branch module meet the second convergence condition, wherein, in one iterative training process, the following operations are performed:

[0029] Determine a loss value corresponding to the first clustering granularity based on a predicted category corresponding to a second sample image in the sample image set at the first clustering granularity and a true category corresponding to the second sample image at the first clustering granularity;

[0030] Based on the loss value corresponding to the first clustering granularity, the parameters of the feature extraction module and the first category identification branch module are adjusted.

[0031] In a possible embodiment, the feature extraction module includes a pre-trained first feature extraction submodule and a second feature extraction submodule; and the model training module is specifically used for:

[0032] Inputting the first sample image into the first feature extraction submodule to obtain underlying features of the first sample image, wherein the underlying features are used to describe features presented by the first sample image on a visual layer;

[0033] Inputting the bottom-level features into the second feature extraction submodule to obtain high-level abstract features of the first sample image, wherein the high-level abstract features are used to describe features presented by the first sample image at the conceptual level;

[0034] Based on the training loss value, adjust the parameters of the second feature extraction submodule.

[0035] In a possible embodiment, the model training module is specifically used for:

[0036] Before determining the training loss value based on the obtained predicted categories and the real categories corresponding to the first sample images at the at least three clustering granularities, the real categories corresponding to the first sample images at the at least three clustering granularities are obtained in the following manner:

[0037] Determine K sample features from each sample feature as K candidate cluster centers, wherein each sample feature is output by each sample image in the sample image set after being output by the feature extraction module, and K is a positive integer;

[0038] For the K candidate cluster centers, iterative updating is performed in the following manner until the distance between the same candidate cluster centers outputted from two iterative updates is not greater than a first preset value: clustering each of the sample features to a candidate cluster center that is closest to the corresponding sample feature among the K candidate cluster centers, and updating the K candidate cluster centers based on the mean values ​​of the sample features corresponding to the K candidate cluster centers;

[0039] The K candidate cluster centers outputted by the last iterative update are used as K target candidate cluster centers, and the first sample feature is clustered to the target cluster center that is closest to the first sample feature among the K target cluster centers to obtain the true category corresponding to the first sample feature.

[0040] The present application provides an image processing device, including:

[0041] A model acquisition module is used to obtain a trained image processing model;

[0042] A feature acquisition module, used for inputting the image to be processed into the feature extraction module in the trained image processing model to obtain the target features corresponding to the image to be processed;

[0043] A category acquisition module is used to input the target features into the category recognition module in the trained image processing model to obtain the target category of the image to be processed, wherein the target category belongs to the predicted categories corresponding to the image to be processed output by the category recognition module at at least three clustering granularities, and the at least three clustering granularities include clustering granularities of at least three different granularity levels.

[0044] In a possible embodiment, the device further includes an image clustering module, wherein the image clustering module is used to:

[0045] Based on the target category, determining a candidate data set to which the image to be processed belongs;

[0046] Determining the similarity between the local features of the image to be processed and the local features of the reference images in the candidate data set;

[0047] If the determined similarity is greater than or equal to a second preset value, the image to be processed is clustered into the candidate data set.

[0048] An embodiment of the present application provides a computer device, including:

[0049] at least one processor, and

[0050] a memory communicatively coupled to the at least one processor;

[0051] The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements any image processing model training method or image processing method discussed above by executing the instructions stored in the memory.

[0052] An embodiment of the present application provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes any image processing model training method or image processing method discussed above.

[0053] Since the embodiment of the present application adopts the above technical solution, it has at least the following technical effects:

[0054] In an embodiment of the present application, an image processing model is trained for the purpose of outputting categories at at least three different granularity levels, so that the feature extraction module in the image processing model can learn how to extract the features required for category output at at least three different granularity levels, which is equivalent to ensuring that the image processing model can learn the feature distribution of the image at different levels, such as the global feature distribution and local feature distribution of the image, by combining the classification tasks at the coarse, medium and fine granularity levels. In this way, when the image processing model subsequently outputs categories, it actually relies on the features required for category output at at least three different granularity levels for output, which is equivalent to integrating the coarse, medium and fine granularity features into the current category output task to achieve the correlation between the current category output and the feature distribution of the image at different levels, thereby improving the accuracy of the output of the image processing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 An example diagram of the categories of images at different clustering granularities provided in the embodiments of the present application;

[0056] Figure 2A Application scenarios of the image processing model training method provided in the embodiment of the present application Figure 1 ;

[0057] Figure 2B Figure 2 is an application scenario of the image processing model training method provided in an embodiment of the present application;

[0058] Figure 3 A flowchart of an image processing model training method provided in an embodiment of the present application;

[0059] Figure 4 An example diagram of a process for identifying predicted categories of a first sample image at three clustering granularities provided in an embodiment of the present application;

[0060] Figure 5 Schematic diagram of the structure of the image processing model provided in the embodiment of the present application Figure 1 ;

[0061] Figure 6 A second structural diagram of an image processing model provided in an embodiment of the present application;

[0062] Figure 7An example diagram of the process of determining cluster centers based on the K-means method provided in an embodiment of the present application;

[0063] Figure 8 A flowchart of an image processing method provided in an embodiment of the present application;

[0064] Fig. 9 An example diagram of a process for determining a target category of an image to be processed provided in an embodiment of the present application;

[0065] Fig.10 A schematic diagram of the structure of an image processing model training device provided in an embodiment of the present application;

[0066] Fig.11 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0067] Fig.12 A schematic diagram of a computer device provided in an embodiment of the present application Figure 1 ;

[0068] Fig.13 A second structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to better understand the technical solution provided by the embodiments of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0070] In order to help those skilled in the art better understand the technical solution of the present application, the terms involved in the present application are introduced below.

[0071] 1. Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0072] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0073] 2. Computer Vision Technology (CV): Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0074] 3. Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0075] 4. Image recognition: Identify images at different levels. For example, coarse clustering granularity classification can ignore specific instances of objects and only consider large categories of objects (such as people or animals) to identify and give the category to which the object belongs. A typical example is the recognition task in the large-scale general object recognition open source dataset imagenet, which identifies which of 1,000 categories an object belongs to.

[0076] 5. Imagenet: A large-scale open source dataset for general object recognition, including multiple images. The Imagenet pre-trained model refers to the parameters of the model obtained by training the model based on the images in Imagenet.

[0077] 6. Image multi-category recognition: Use computers to identify whether an image has the categories corresponding to each attribute. An image may have multiple attributes, and the multi-category recognition task can determine which attributes of the image correspond to which categories.

[0078] 7. Clustering granularity and granularity level: Clustering granularity refers to the degree of coarseness or fineness of image classification. Granularity level is used to characterize the coarseness or fineness corresponding to clustering granularity, which can be further understood as the category level corresponding to the classification task. The larger the granularity level corresponding to the clustering granularity, the coarser the degree of image classification is, and the smaller the granularity level corresponding to the clustering granularity, the finer the degree of image classification is. Different clustering granularities can correspond to one granularity level, and different clustering granularities can also correspond to different granularity levels. For example, animals can be divided into four categories: cats, dogs, wolves, and sheep. Animals can also be divided into three categories: bears, tigers, and lions. Tigers and cats actually belong to different clustering granularities in this division method, but belong to the same granularity level. The number of categories included in each clustering granularity can be one or more. Therefore, in the classification task, it can be determined which category an image can correspond to at each clustering granularity. The number of categories included in each clustering granularity can be set according to actual needs.

[0079] In order to facilitate the description of different clustering granularities, in the embodiment of the present application, different clustering granularities can be described as the first clustering granularity, the second clustering granularity, the third clustering granularity, ... the Nth clustering granularity, etc. The granularity level of the first clustering granularity in the embodiment of the present application is greater than the granularity level of the second clustering granularity, and the granularity level of the second clustering granularity is greater than the granularity level of the third clustering granularity. Correspondingly, the first clustering granularity can also be called a coarse clustering granularity, the second clustering granularity can also be called a medium clustering granularity, and the third clustering granularity can also be called a fine clustering granularity.

[0080] In order to explain the concept of clustering granularity more clearly, Figure 1 The following is an example of the categories of the images shown in the figure at different clustering granularities:

[0081] Figure 1 Image a in the figure belongs to animals in the two categories of "people" and "animals" at the first clustering granularity; image a belongs to "dog" in the three categories of "dog", "cat" and "wolf" at the second clustering granularity; image a belongs to "Shih Tzu" in the three categories of "golden retriever", "shepherd" and "Shih Tzu" at the third clustering granularity. Figure 1 In the figure, the dotted rectangular box shows the corresponding categories of image a at each clustering granularity.

[0082] 8. Predicted category and true category: A category can be understood as the classification to which an image belongs at the corresponding clustering granularity. The predicted category in this application refers to the category corresponding to the image output by the image processing model during the training of the image processing model, and the true category refers to the reference of the true category of the image during the training of the image processing model. It should be noted that the true category is used as the true category of the image when training the image processing model, and is not necessarily the correct classification result of the image.

[0083] 9. Sample image set: refers to a set of sample images used for training a model, and the sample image set includes one or more sample images. The first sample image and the second sample image in the embodiment of the present application are both sample images selected from the sample image set, and the first sample image and the second sample image may be the same or different. Correspondingly, the feature corresponding to the first sample image is called the first sample feature, and the feature corresponding to the second sample image is called the second sample feature.

[0084] 10. Bottom-level features and high-level abstract features: Bottom-level features refer to the features of an image at the visual level, which can be obtained by performing a first operation on the image, such as a filtering operation. Bottom-level features include color features, texture features, or shape features of the image. High-level abstract features refer to the features of an image at the conceptual level. High-level abstract features can express what is closest to human understanding and can be used to classify images. For example, describing an image with bottom-level features means: the image is blue, and describing an image with high-level abstract features means: the image is the sea. High-level abstract features can be obtained by performing a second operation on bottom-level features, such as one or more of one or more convolution operations and pooling operations.

[0085] 11. Global features and local features: Global features are used to describe the overall features of an image. Corresponding to global features, local features are used to describe the features of a local area of ​​an image. Specific examples of global features include the shape, structure or deep features of an image. Deep features can also be called complex features. Shape features are used to describe the overall shape of an image. Shape features can be, for example, the quantified results of image contour information. Structural features can be, for example, the quantified results of geometric structures in an image. For example, the structural features of an image can be extracted through a graph convolutional neural network. Deep features can be the combined results of multiple features of the image to be annotated, or can be obtained by further processing one or more features in the image to be annotated, such as convolution processing or other deep learning processing. Among them, in the embodiments of the present application, local features can also be called detail features.

[0086] 12. Training loss value: It is used to represent the error between each predicted category of the sample image obtained by the image processing model and the corresponding true category. The training loss value can be the sum of the errors between each predicted category and the true category in the sample image, or it can be the overall performance of the errors between the predicted category and the true category under multiple clustering categories of the sample image.

[0087] It should be noted that, in the embodiments of the present application, “multiple” refers to two or more, “at least one” refers to one or more, and “at least three” refers to three or more.

[0088] In the scenario of video retrieval or image retrieval, it involves retrieving the required data from a large amount of data, that is, it involves a large-scale image retrieval process. Large-scale image retrieval can use bucket retrieval. The main idea of ​​bucket retrieval is to divide a large number of images into multiple non-overlapping data sets, generally one data set corresponds to one bucket. When performing image retrieval, you can find images that match a specific image from the bucket corresponding to the specific image, thereby improving image retrieval efficiency.

[0089] Each data set involved in bucket retrieval is generated after clustering images. Usually, a bucket has a cluster center. For example, there are 1 million sample images, and all images correspond to 10,000 buckets, then the corresponding cluster centers are 10,000. Therefore, accurately bucketing sample images has a great impact on the final result of image retrieval. The best bucketing result is to divide images with similar features into the same bucket, so that the recall rate of each bucket is high, and the number of images included in each bucket is evenly distributed. The benefit of a balanced distribution of the number of images is that it can avoid too many images in some buckets, resulting in a long subsequent image retrieval. An extreme imbalanced state, for example: all images are distributed in the same bucket, will affect the efficiency of subsequent image retrieval. Therefore, how to accurately cluster each image and how to improve the balanced distribution of the number of images in each bucket are two important issues that need to be solved in the image clustering process.

[0090] Combined with the content discussed in the previous background technology, one solution is to cluster images using the K-means clustering method. However, the K-means clustering method generally uses randomly initialized cluster centers, which randomly generates initialized cluster centers, which can easily cause unreasonable distribution of cluster centers and great uncertainty. In addition, when initializing and determining cluster centers, the characteristics of the image are not considered, and thus it cannot be guaranteed that semantically similar images are clustered into one category, resulting in inaccurate clustering results. Moreover, since the clustering process only relies on multiple set cluster centers for clustering, it is also easy to cause uneven distribution of subsequent clustering results.

[0091] In view of this, an embodiment of the present application provides an image processing model training method. When training an image processing model, the image processing model training method trains a category recognition module in the image processing model to output the corresponding categories at the cluster granularity of at least three granularity levels. This requires the feature extraction module in the image processing model to learn the features required for the classification tasks at at least three granularity levels, thereby improving the learning ability of the feature extraction module for the features required for the classification tasks at each granularity level. When the image processing model is used to determine the category of an image, it can be ensured that the feature extraction module can improve the features required for the classification of the image at different granularity levels, thereby enabling the classification recognition module to combine the features of the image at each granularity level to output the classification of the image. Because the classification recognition module can combine the features required for the classification of the image at different granularity levels for classification, it is equivalent to the classification recognition module combining the features of the image in many aspects for classification, thereby ensuring the accuracy of image classification. Moreover, since the corresponding classification at each granularity level itself satisfies the distribution relationship between the upper and lower granularity levels, the image processing model is trained in combination with the corresponding classification tasks at at least three granularity levels, which constrains the categories at each cluster granularity output by the image processing model, and can further improve the accuracy of image classification.

[0092] Based on the above design ideas, the application scenarios of the image processing model training method of the embodiment of the present application are introduced below.

[0093] Please refer to Figure 2A , which is an application scenario of the image processing model training method provided in the embodiment of the present application Figure 1 The application scenario diagram includes a first server 210 , a database 220 , a terminal 230 , and an application 231 running in the terminal 230 .

[0094] The database 220 may be deployed in the first server 210, or may be deployed in a device relatively independent of the first server 210. The database 220 stores a large number of sample images, which may be obtained by the first server 210 from network resources and written into the database, or may be obtained by the first server 210 from the application 231 and written into the database. For example, the first server 210 is a background server corresponding to the application 231 running in the terminal 230, and the application 231 generates a large number of images during operation. The terminal 230 and the first server 210 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application. The first server 210 may obtain the corresponding images from the terminal 230 and write these images into the database 220. The application 231 may be, for example, a content service application or a game application, which is not limited in this application.

[0095] When the first server 210 is training the image processing model, the first server 210 may obtain sample images from the database 220. The first server 210 trains the image processing model based on these sample images, and the training process of the image processing model involved will be discussed below.

[0096] After the first server 210 obtains the trained image processing model, it can obtain the image to be processed from the terminal 230. The first server 210 can use the trained image processing model to classify the image to be processed to obtain the target category of the image to be processed, and cluster the image to be processed based on the target category of the image to be processed. The process of determining the target category of the image to be processed will be discussed below. After the image to be processed is clustered, it is convenient to perform image retrieval, etc.

[0097] Among them, the trained image processing model can be widely used in various specific application scenarios, for example, applied to video retrieval classification. For example, after obtaining the classification of the image to be processed in the video, the video can be stored in the corresponding bucket based on the category corresponding to the video, so as to facilitate the later retrieval of various types of videos. It can also be applied to virtual reality (VR), augmented reality (AR), self-driving, smart furniture, smart office, smart wear, smart transportation, smart city, drone, remote medical, robots and other application scenarios involving image processing, and this application does not limit this.

[0098] In another possible application scenario, please refer to Figure 2B , which is an application scenario diagram 2 of the image processing model training method provided in an embodiment of the present application, the application scenario diagram includes a first server 210, a database 220, a terminal 230, an application 231 running in the terminal 230, and a second server 240.

[0099] Figure 2B The contents of the first server 210 and the database 220 in the scene diagram shown can be referred to Figure 2A The contents of the introduction will not be repeated here. Figure 2A The difference is, Figure 2B The device for training the image processing model and the device for using the image processing model are not the same. Specifically, after the first server 210 obtains the trained image processing model, the trained image processing model can be sent to the second server 240. The second server 240 determines the category of the image to be processed based on the trained image processing model.

[0100] The terminal discussed above may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, gaming device, smart TV or smart wearable device, etc., and this application is not limited to this.

[0101] The first server 210 and the second server 240 discussed above can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0102] Based on the above Figure 2A or Figure 2B The application scenario shown below takes the first server executing the image processing model training method in the embodiment of the present application as an example to introduce the image processing model training method involved in the embodiment of the present application. Figure 3 , is a flowchart of the image processing model training method provided in an embodiment of the present application, and the flowchart specifically includes:

[0103] S31, based on the sample image set, iteratively train the image processing model to be trained until a first convergence condition is met, wherein in one iterative training process, the following operations are performed:

[0104] S311, inputting a first sample image in the sample image set into a feature extraction module in the image processing model to obtain a first sample feature corresponding to the first sample image;

[0105] S312, inputting the first sample feature into a category recognition module in the image processing model to obtain the predicted categories corresponding to the first sample image at at least three clustering granularities;

[0106] S313, determining a training loss value based on each of the obtained predicted categories and the real categories corresponding to the first sample image at at least three clustering granularities;

[0107] S314, adjusting parameters of the feature extraction module and the category recognition module based on the training loss value.

[0108] The first server can obtain a set of sample images from a database, and iteratively train the image processing model to be trained based on the set of sample images. The image processing model to be trained is a network model built based on a target image processing task, and the target image processing task refers to a specific application task corresponding to the image processing model. The image processing model to be trained has initial parameters, and the initial parameters include one or two of randomized parameters and parameters obtained based on pre-training. For example, some parameters in the image processing model are obtained randomly, and another part of the parameters in the image processing model is obtained through training. However, the initial parameters of the image processing model may not be effective in achieving a specific target image processing task, so it is necessary to train the image processing model to obtain a trained image processing model. The trained image processing model can better handle the target image processing task because it has learned a large number of sample images.

[0109] The image processing model in the embodiment of the present application includes a feature extraction module and a category recognition module. The image processing model is trained iteratively for multiple times, which can actually be understood as multiple iterative training of the feature extraction module and the category recognition module in the image processing model.

[0110] When the image processing model is trained multiple times, one or more first sample images in the sample image set can be used for training in each iterative training. The meaning of the first sample image can refer to the content discussed above and will not be repeated here. The following is an example of an iterative training process:

[0111] The first server inputs the first sample image into a feature extraction module to obtain a first sample feature of the first sample image; the first server inputs the first sample feature into a category recognition module, and the category recognition module outputs the predicted categories corresponding to the first sample image at at least three clustering granularities.

[0112] The feature extraction module is used to extract the features of the image. After the feature extraction module processes the first sample image, the first sample feature corresponding to the first sample image can be obtained. The first sample feature refers to the image feature of the first sample image. The first sample feature includes one or both of the underlying features and high-level abstract features of the first sample image. The meanings of the underlying features and high-level abstract features can be referred to the content discussed above and will not be repeated here. Which specific feature of the first sample image is extracted is related to the specific network structure of the feature extraction module.

[0113] The category recognition module outputs classification based on the features output by the feature extraction module, specifically based on the first sample features, and then outputs the predicted categories corresponding to the first sample image at at least three clustering granularities, wherein the meanings of the clustering granularity and the predicted category can be referred to the content discussed above and will not be repeated here.

[0114] The first server can determine the training loss value according to each predicted category obtained and the real category corresponding to each of the first sample images at at least three clustering granularities. The meaning of the real category can refer to the content discussed above and will not be repeated here. The training loss value is used to represent the difference between the predicted category and the real category at each clustering granularity. The first server adjusts the parameters of the image processing model according to the training loss value to complete an iterative training process.

[0115] By analogy, the first server can perform multiple iterations of training on the image processing model, and when the image processing model satisfies the first convergence condition, a trained image processing model is obtained. The first convergence condition is, for example, that the number of iterations of training the image processing model reaches a first preset number, or the training loss value of the image processing model reaches a first preset loss value, etc., which is not limited in this application.

[0116] For example, see Figure 4 , which is an example diagram of the process of identifying the predicted category of the first sample image at three clustering granularities, the first server Figure 4 The first sample image b is input into the feature extraction module to obtain the first sample feature 400. Based on the first sample feature 400, the category of the first sample image at the coarse clustering granularity is person, the category at the medium clustering granularity is face, and the category at the fine clustering granularity is melon-seed face.

[0117] exist Figure 3 In the illustrated embodiment, when training the image processing model, the image processing model can learn the categories under the clustering granularity of at least three different granularity levels, so that the same image processing model can recognize the categories under three or more granularity levels. Compared with the feature output category of the image processing model based on the category under a single clustering granularity, the feature extraction module in the image processing model in the embodiment of the present application learns the features required to identify the categories under the clustering granularity of various granularity levels, ensuring the accuracy of the categories output by the image processing model. Moreover, in the embodiment of the present application, the categories under multiple granularity levels are set. Since there is a certain constraint relationship between the different granularity levels, the classification of the image processing model output is indirectly constrained, which can further improve the accuracy of the category output by the image processing model. Moreover, since the number of categories under each clustering granularity can be flexibly set, and the category of the image under multiple clustering granularities can be determined, it is convenient to cluster the images flexibly in the future, and the distribution of the images can be made more balanced, which is convenient for the subsequent rapid retrieval of related images.

[0118] Before the first server executes S31, a sample image set may be obtained. The first server may obtain the sample image set from a database. The sample image set in the database may be obtained by the first server in the following manner:

[0119] Method 1: The first server can select images needed by itself from network resources as sample images.

[0120] The first server can filter sample images required by itself from network resources based on the screening rules. There can be multiple screening rules. For example, the screening rule can be to filter out images whose image quality meets the preset quality conditions. The image quality meets the preset quality conditions, such as image clarity and color saturation meet one or two of the corresponding thresholds. For example, the screening rule can be to filter out images whose correlation with the target image processing task meets the preset correlation threshold. For example, if the target image processing task is to filter short videos, the first server can filter out some video frames from the short videos as sample images.

[0121] Method 2: The first server may obtain sample images from other devices.

[0122] Specifically, the first server can obtain sample images from devices related to the target image processing task according to the target image processing task corresponding to the image processing model. For example, if the target image processing task corresponding to the image processing model is determined to be classifying game videos, the first server can obtain sample images from a background server device related to the game.

[0123] Method 3: The first server combines the above methods 1 and 2 to obtain a sample image.

[0124] Regardless of the method used by the first server to obtain the sample image, the first server may directly obtain the sample video. Therefore, in the embodiment of the present application, the first server can filter out the sample image from the sample video.

[0125] Specifically, the first server may randomly sample the sample video to obtain a sample image. The first server may also sample the sample video at a preset sampling interval to obtain multiple candidate images. The first server may directly use the multiple candidate images as sample images, and may obtain the sample image simply and quickly.

[0126] Furthermore, in order to improve the effectiveness of generating the sample image, the first server may filter out candidate images with higher similarity among the multiple candidate images to obtain the sample image.

[0127] Specifically, after the first server screens out multiple candidate images from the sample video, it can determine the similarity between every two candidate images, for example, respectively extract the image feature vectors of the two candidate images, and calculate the similarity between the two image feature vectors. Specifically, the cosine similarity or Euclidean distance between the two image feature vectors can be used to characterize the similarity between the two image feature vectors. After obtaining the similarity between every two candidate images, if there are any two candidate images whose similarity is greater than the preset similarity, one of the candidate images is eliminated. Elimination can be understood as deleting the candidate image and not using it as a sample image. By analogy, the remaining candidate images are used as sample images. Among them, the preset similarity is a preset similarity threshold, and the specific value can be set according to demand, for example, a value of 0.95. In an embodiment of the present application, candidate images with higher similarity can be screened out, and the overfitting of the image processing model caused by training the model with candidate images with higher similarity can be avoided.

[0128] After obtaining the sample image set, the first server can execute S31. In the embodiment of the present application, the image processing model needs to output the corresponding categories at least three different levels of clustering granularity at one time, which can be understood as the image processing model needs to complete multiple classification tasks at one time. Each classification task actually involves two parts, namely, feature extraction and category output. In the embodiment of the present application, in order to enable the output of the category output at each granularity level to share the features required for the category output at other granularity levels, the image processing model in the embodiment of the present application includes a feature extraction module and multiple category identification branch modules respectively connected to the feature extraction module, wherein each category identification branch module is used to output a category at a clustering granularity, and multiple category branch modules can output the corresponding categories at at least three levels of clustering granularity.

[0129] When training the image processing model, the first sample image can be input into the feature extraction module to obtain the first sample feature of the first sample image, and the first sample feature can be input into each category recognition branch module respectively to obtain the category output by each category recognition branch module.

[0130] In this way, each category recognition branch module relies on the features output by the feature extraction module to achieve the classification task, which is equivalent to multiple category recognition branch modules sharing the parameters of the feature extraction module. In other words, the feature extraction module has learned the parameters required for extracting the classification at each clustering granularity, so that each category recognition branch module can output the image category based on the features at multiple clustering granularities, thereby improving the recognition accuracy of the image category.

[0131] To more clearly illustrate the structure of the image processing model in the embodiment of the present application, please refer to Figure 5The structure of the image processing model shown Figure 1 The image processing model includes a feature extraction module, and a plurality of category recognition branch modules connected in parallel with the feature extraction module, wherein the plurality of category recognition branch modules include a first category recognition branch module, a second category recognition branch module, ... an Nth category recognition branch module, where N is a positive integer greater than or equal to 3. The first server may input the first sample feature into the first category recognition branch module, the second category recognition branch module, and the Nth category recognition branch module, respectively, so as to obtain category 1, category 2, ... category N output by the first category recognition branch module, the second category recognition branch module, and the Nth category recognition branch module.

[0132] In a possible embodiment, the more category identification branch modules there are, the more sample features required for classification tasks at clustering granularities the feature extraction module can integrate, but this will also increase the amount of training in the training process. Therefore, it is very important to reasonably set the number of category identification branch modules. Therefore, in an embodiment of the present application, the number of category identification branch modules is set to three, and the multiple category identification branch modules specifically include a first category identification branch module, a second category identification branch module, and a third category identification branch module. These three category identification branch modules are respectively used to output corresponding categories at three different levels of clustering granularity.

[0133] Specifically, the first category identification branch module can be used to output the predicted category corresponding to the first sample image at the first cluster granularity, the second category identification branch module is used to output the predicted category corresponding to the second sample image at the second cluster granularity, and the third category identification branch module is used to output the predicted category corresponding to the third sample image at the third cluster granularity. The granularity level of the first cluster granularity is greater than the granularity level of the second cluster granularity, and the granularity level of the second cluster granularity is greater than the granularity level of the third cluster granularity.

[0134] The following is an analysis of the role of adding multiple clustering granularities in category recognition in the embodiment of the present application:

[0135] 1. The purpose and effect of adding coarse-grained clustering: For the global data of the image, if target clustering learning is performed based on the global data, the clustering result still has a certain degree of randomness like K-means. Therefore, global coarse-grained clustering is added in the embodiment of the present application, which can make the image processing model perceive the distribution of the global features of the image first. For example, the animals and people in the three categories of coarse-grained clustering will be clustered into the corresponding coarse-grained categories respectively, so that the image processing model pre-learns the coarse-grained clustering data existing in the overall data, and can preliminarily divide objects with large differences into different coarse-grained categories. Since the parameters of the feature extraction module will be changed during coarse-grained learning, the overall distribution of the image learned by coarse-grained learning can be transmitted back to the feature extraction module, so that the feature extraction module in the image processing model has the ability to characterize the overall distribution of the data.

[0136] 2. The purpose and effect of adding fine-grained clustering: If fine-grained clustering is not added, direct target clustering is likely to result in large semantic dissimilarity in the target clusters. Fine-grained clustering can make similar images within the clusters closer to each other, so that the fine-grained clustering center can more specifically (split) represent things. The parameters are learned by the image processing model and fed back to the feature extraction module, so that the feature extraction module can adjust the target cluster center to the effect of semantic distinction under the action of fine-grained distribution, thereby improving the accuracy of the output category of the image processing model.

[0137] Furthermore, the granularity level of the first cluster granularity is adjacent to the granularity level of the second cluster granularity, and the granularity level of the second cluster granularity is adjacent to the granularity level of the third cluster granularity. Adjacent can be understood as the granularity levels of the two cluster granularities being adjacent in the upper and lower granularity level distribution relationship. Since the granularity levels are adjacent, it means that the span between the two granularity levels is smaller, which means that the correlation between the features required for the corresponding output of the categories under the two adjacent granularity levels is stronger. Therefore, in the embodiment of the present application, the three granularity levels are set to be adjacent to each other, which can help the feature extraction module learn features with stronger correlation, and further improve the accuracy of the categories output by the category recognition module.

[0138] If the category recognition module includes multiple category recognition branch modules, any two of which can actually be considered relatively independent. Then, when training these category recognition branch modules, these category recognition branch modules can be trained flexibly. The following is an example of how to train each category recognition branch module:

[0139] Method 1: Based on the training loss value, adjust the parameters of each category recognition branch module and feature extraction module until the image processing model meets the first convergence condition.

[0140] In an embodiment of the present application, when the image processing model outputs each predicted category of the first sample image, the training loss value can be determined based on the each predicted category of the first sample image and the true category corresponding to the first sample image, and the image processing model can be adjusted directly based on the training loss value until it converges.

[0141] The training method in the embodiment of the present application is simple and direct, and the output of the feature extraction module in the trained image processing model can meet the output requirements of each category recognition branch module, and can ensure the accuracy of subsequent output categories.

[0142] Method 2: Train the target category branch module in multiple category recognition branch modules, and then train each category recognition branch module and feature extraction module based on the training loss value.

[0143] The target category branch module can be any partial category identification branch module among multiple category identification branch modules, or it can be a category identification branch module among multiple category identification branch modules whose granularity level is greater than a preset granularity level. The preset granularity level can be determined based on the number of granularity levels that can be output by the multiple category identification branch modules.

[0144] If the multiple category recognition branch modules include a first category recognition branch module, a second category recognition branch module, and a third category recognition branch module, the first server may first train the first category recognition branch module until the first category recognition branch module meets the second convergence condition, and the second convergence condition is, for example, that the number of iterative trainings on the first category recognition branch module reaches a second preset number of times, or the loss value corresponding to the first clustering granularity is less than or equal to the second preset loss value. Based on the training loss value, the first server trains the trained first category recognition branch module, the second category recognition branch module, the third category recognition branch module, and the feature extraction module until the feature extraction module and the category recognition module meet the first convergence condition.

[0145] Specifically, when training the first category recognition branch module, an iterative training method may also be used. The following takes one iterative training as an example to illustrate the process of training the first category recognition branch module:

[0146] The first server determines a second sample image from the sample image set. The meaning of the second sample image can refer to the content discussed above and will not be repeated here. The first server can input the second sample image into the feature extraction module to obtain the second sample feature. The meaning of the second sample feature can refer to the first sample feature discussed above and will not be repeated here. The first server inputs the second sample feature into the first category identification branch module to obtain the predicted category corresponding to the second sample image. The first server determines the loss value corresponding to the first clustering granularity based on the predicted category of the second sample image and the true category of the second sample image, and adjusts the parameters of the feature extraction module and the first category identification branch module based on the loss value corresponding to the first clustering granularity to complete an iterative training process.

[0147] By analogy, the first server iteratively trains the first category identification branch module until the feature extraction module and the first category identification branch module meet the second convergence condition. The second convergence condition can refer to the content discussed above and will not be repeated here.

[0148] After obtaining the trained feature extraction module and the first category recognition branch module, the trained feature extraction module, the first category recognition branch module, and the category recognition branch modules among the multiple category recognition branch modules except the first category recognition branch module are trained based on the training loss value to obtain a trained image processing model.

[0149] As an embodiment, the first category identification branch module may be implemented by a fully connected layer. The specific structure of the first category identification branch module may refer to the following Table 1:

[0150] Table 1

[0151] Layer name Output size Layer Fc_cluster 1xN Full connection layer

[0152] Please refer to Table 1, the first category branch module includes a fully connected layer, and the fully connected layer can output a (1xN) vector, wherein each value in the vector represents the probability that the first sample image belongs to a certain category at the first granularity level.

[0153] As an embodiment, the second category identification branch module may be implemented by a fully connected layer. The specific structure of the second category identification branch module may refer to the following Table 2:

[0154] Table 2

[0155] Layer name Output size Layer Fc_cluster1 1x(N / 10) Full connection layer

[0156] Please refer to Table 2, the second category recognition branch module specifically includes a fully connected layer, which can output a vector of (1x(N / 10), where each value in the vector represents the probability that the first sample image belongs to a certain category at the second granularity level.

[0157] As an embodiment, the third category identification branch module may be implemented by a fully connected layer, and the specific structure of the second category identification branch module may refer to the following Table 3:

[0158] Table 3

[0159] Layer name Output size Layer Fc_cluster2 1x(N*10) Full connection layer

[0160] Please refer to Table 3, the second category recognition branch module specifically includes a fully connected layer, which can output a vector of (1x(N*10), where each value in the vector represents the probability that the first sample image belongs to a certain category at the third granularity level.

[0161] In an embodiment of the present application, a category recognition branch module with a larger granularity level can be trained. Since the feature extraction module can first learn the global classification characteristics in the classification task with a larger granularity level, the classification with a larger granularity level output by the image processing model is more reasonable, which is conducive to improving the accuracy of the subsequent training of the image processing model.

[0162] As an embodiment, sampling any training sequence involves calculating the training loss value. In the embodiment of the present application, the first server can perform weighted summation on the loss value at each clustering granularity to obtain the training loss value. The following is an example of calculating the loss value at a clustering granularity:

[0163] After the first server inputs the first sample feature into the category recognition branch module at the cluster granularity, the predicted category at the cluster granularity can be obtained. The first server obtains the loss value of the first sample image at the cluster granularity based on the corresponding true category at the cluster granularity and the predicted category at the cluster granularity. The loss value of the cluster granularity refers to the error between the true category and the predicted category at the cluster granularity during the classification process. When the image processing model outputs the predicted category of the first sample image, it can directly output the category to which the first sample image belongs at the cluster granularity, or output the predicted probability that the first sample image belongs to each category at the cluster granularity. When determining the loss value of the cluster granularity, the error between the predicted probability output by the image processing model and the true category can be used to characterize the loss value corresponding to the cluster granularity.

[0164] The loss value of the clustering granularity may be calculated, for example, using a cross entropy loss function, an L2 norm calculation formula, and the like. For example, the loss value at the clustering granularity is expressed as follows by referring to the cross entropy loss function shown in the following formula:

[0165]

[0166] Among them, C represents the number of categories included in the clustering granularity, i represents the i-th category under the clustering granularity, and p(x i ) represents the true probability of the first sample image corresponding to the i-th category at the clustering granularity. If the first sample image belongs to the i-th category, then p(x i ) takes the value of 1, otherwise, it takes the value of 0; q(x i ) represents the predicted probability that the first sample image output by the image processing model belongs to the i-th category at the clustering granularity, and H(p, q) represents the value of the calculated cross entropy loss function.

[0167] For example, the first clustering granularity includes people and animals, and the true category of the first sample image is a person at the first clustering granularity. Then, under the person category, the corresponding p(x i ) takes the value of 1, which corresponds to p(x i ) can be 0. For example, the corresponding output of the image processing model is (0.3, 0.7), where 0.3 represents the predicted probability that the first sample image belongs to an animal, and 0.7 represents the predicted probability that the first sample image belongs to a person.

[0168] By analogy, after the first server calculates the corresponding loss value at each clustering granularity, it performs weighted summation on each loss value to obtain the training loss value of the image processing model. When performing weighted summation on each loss value, the weighted weights of each loss value may be the same or different. For example, the weighted weights of each loss value may be set according to the importance of each granularity level. The higher the importance corresponding to the granularity level, the greater the weighted weight of the corresponding loss value at the granularity level. The importance of the granularity level may be determined according to the target image processing task. For example, if the target image processing task focuses on the category of the first granularity level output by the image processing model, then the weighted weight of the loss value corresponding to the first granularity level is greater.

[0169] The following continues to take the category recognition module including the first category recognition branch module, the second category recognition branch module and the third category recognition branch module as an example to illustrate the calculation of the training loss value. The first server calculates the loss value between the predicted category output by the first category recognition branch module and the actual category corresponding to the first category recognition branch module as L1, which can be understood as the loss value under the coarse clustering granularity classification. The first server calculates the loss value between the predicted category output by the second category recognition branch module and the actual category corresponding to the second category recognition branch module as L2, which can be understood as the loss value under the medium clustering granularity classification. The first server calculates the loss value between the predicted category output by the third category recognition branch module and the actual category corresponding to the third category recognition branch module as L3, which can be understood as the loss value under the fine clustering granularity classification. The first server can calculate "L1+L2+L3" to obtain the training loss value.

[0170] In the above embodiment, the training loss value is obtained by weighted summing the loss values ​​at each clustering granularity, which can ensure that each category recognition branch module in the trained image processing model can meet its own classification loss and improve the accuracy of the output category of each category recognition branch module. In addition, the weighted weights of the loss values ​​corresponding to each clustering granularity can be flexibly set to meet different training requirements.

[0171] As an embodiment, in addition to considering the loss values ​​at each clustering granularity, the training loss value may also add consistency loss, which is used to indicate that the distribution of the output results of each sample image after the image processing model can be consistent with the distribution of the output results of each sample image after image enhancement processing and then input into the image processing model. Image enhancement refers to performing one or more image processing on the image, such as Gaussian noise, salt and pepper noise, cropping, rotation, watermarking, hue conversion, etc. In the case of adding consistency loss, the calculation formula of the training loss value is as follows:

[0172] L=a1*L1+a2*L2+a3*L3+a4*Lcon (2)

[0173] Among them, L1, L2 and L3 can be calculated using the above formula (1), and the embodiment of the present application does not limit the specific calculation method of L1, L2 and L3.

[0174] Among them, a1, a2 and a3 are the weighted weights corresponding to the loss values ​​at three different clustering granularities, L1, L2 and L3 are the loss values ​​at three different clustering granularities, a4 is the weighted weight corresponding to the consistency loss, Lcon is the consistency loss, and L is the training loss value.

[0175] In an embodiment of the present application, when calculating the training loss value, not only the loss value at each clustering granularity is considered, but also the consistency loss is considered, so that the processing result of the image processing model for the sample image is consistent with the processing result of the enhanced sample image, so that the image processing model can still output stably after the image is enhanced, thereby increasing the stability of the image processing model.

[0176] The following example uses the above consistency loss as an example:

[0177] For a sample image A0, after image enhancement, multiple enhanced sample images are generated, for example, 5 enhanced samples A1~A5. After A0 and A1~A5 are input into the image processing model, 6 corresponding predicted classifications P0~P5 under a certain clustering granularity are obtained. The purpose of introducing consistency loss is to make the image processing model keep the distribution of predicted classifications P1~P5 for the enhanced sample image consistent with the predicted classification P0 corresponding to the original sample image, so as to ensure the robustness of the output results of the image processing model, that is, a sample image can be clustered into the same category as the original sample image even after image enhancement. A method for calculating consistency loss is as follows:

[0178]

[0179] Where Dkln represents the consistency loss corresponding to the nth enhanced sample image.

[0180]

[0181] Where P(X i ) is the sample image X i The output result at a certain clustering granularity after inputting the image processing model, Q(X i ) is the output result of the sample image after image enhancement and then input into the image processing model at the corresponding clustering granularity.

[0182] As an embodiment, the weighted weights of the loss values ​​at each clustering granularity may be the same. For example, referring to the above formula (2), the values ​​of a1, a2 and a3 may be the same. The value of a4 may be a fixed value, such as 1.

[0183] Furthermore, the value of a1 is adjusted according to the specific training situation of the image processing model. For example, the greater the difference between the sum of the loss values ​​at each clustering granularity in the current round of iterative training and the sum of the loss values ​​at each clustering granularity in the previous round of iterative training, the smaller the corresponding value of a1.

[0184] Specifically, in the two tasks of clustering and consistency learning, since the goal of consistency learning is to allow the clustering model to cluster to the prediction category corresponding to the sample image before processing under various image enhancement methods, consistency learning needs to run through all rounds of image processing model learning iterations. Since the clustering category will be updated after a certain number of iterations, when the clustering category is first updated, the clustering effect caused by the change in clustering category will deteriorate, which will cause the L1, L2, and L3 clustering losses adjusted by a1 to increase instantly. At this time, the value of a1 needs to be reduced so that the clustering loss is relatively stable after a1 weighting, thereby avoiding excessive fluctuations in consistency loss. Note Lc = L1 + L2 + L3. Therefore, the value of a1 can refer to the following formula:

[0185]

[0186] Among them, t can represent the current round of iterative training, and (t-1) represents the previous round of iterative training.

[0187] In the embodiment of the present application, the weighted weights corresponding to the loss values ​​at the clustering granularity are flexibly set, which can avoid the situation where the image processing model has a poor effect due to the weighted weights being set too large or too small.

[0188] In another possible embodiment, the training loss value can be calculated using a training loss function under multiple classifications. For example, the error between the predicted classification and the corresponding true classification at each clustering granularity is directly summed. This eliminates the need to separately calculate the loss value at each clustering granularity, which simplifies the calculation relatively.

[0189] After calculating the training loss value, the parameters of the image processing model can be adjusted based on the training loss value. For example, the first server can use the Adam algorithm to adjust the parameters of the image processing model. Specifically, the first server calculates the time step according to the training loss values ​​of the previous iteration and the current iteration, and obtains the parameters of the image processing model after the current iteration training based on the calculated time step and the parameters of the image processing model after the previous iteration training. The calculation formula involved is as follows:

[0190] v t =γv t-1 +ηΔθJ(θ) (6)

[0191] θ=θ-v t (7)

[0192] Among them, θ is the parameter of the image processing model, η is the learning rate, J(θ) is the training loss value, and γ is a constant term, also known as momentum. t-1 is the variance of the gradient change of the training loss value corresponding to the previous iteration training, v tis the variance of the gradient change of the training loss value corresponding to this iterative training.

[0193] In a possible embodiment, the process of training the image processing model also involves how to determine the true category of each first sample image at each clustering granularity. The following takes determining the true category of the first sample image at a clustering granularity as an example to introduce the process of calculating the true category of the first sample image at the clustering granularity:

[0194] Calculation method 1:

[0195] Based on the labeling operation, the true category of the first sample image at the clustering granularity is obtained.

[0196] The user can directly perform a labeling operation on the first sample image on the first server. The labeling operation may be, for example, the user inputting category information of the first sample image at the clustering granularity on the first sample image, or, for example, the user selecting the category label of the first sample image from multiple existing category labels. The first server responds to the labeling operation to obtain the true category of the first sample image at the clustering granularity.

[0197] The first server may also obtain the labeled first sample image from the terminal, which is equivalent to obtaining the real category of the first sample image. The terminal may obtain the labeled first sample image based on the labeling operation performed by the user.

[0198] In an embodiment of the present application, the sample image can be directly labeled to obtain the true category of the sample image. In this way, the true category with high accuracy can be obtained, so that the image processing model can be subsequently supervised trained based on the labeled sample image, thereby improving the accuracy of the output category of the image processing model.

[0199] Calculation method 2:

[0200] Based on a preset clustering method, a true category of the first sample image is determined.

[0201] For example, the first server may determine the true category of the first sample image according to the K-means clustering method. The following is an example of a process of determining the true category according to the K-means clustering method.

[0202] S1.1, from each sample feature, determine K sample features as K candidate cluster centers.

[0203] The first server can obtain sample features of each sample image in the sample image set through the feature extraction module, and these sample features include the first sample feature and the second sample feature discussed above. The first server can determine K sample features from these sample features, where K is a positive integer, and the value of K is the same as the number of categories set at the clustering granularity. For example, if the number of categories set at the clustering granularity is 10, then the value of K is 10.

[0204] The first server can randomly select K sample features from these sample features as K candidate cluster centers, and can relatively easily determine multiple cluster centers. The first server can also map these sample features to coordinate points in a preset coordinate system, thereby dividing the coordinate point distribution area into multiple units in turn, and sequentially taking the coordinate points in the preset number of units as K candidate cluster centers, so that relatively dispersed K candidate cluster centers can be obtained, which is conducive to the subsequent determination of more accurate K candidate cluster centers.

[0205] S1.2, for the K candidate cluster centers, iterative update is performed in the following manner until the distance between the same candidate cluster centers output by two iterative updates is no greater than a first preset value: each sample feature is clustered to the candidate cluster center among the K candidate cluster centers that is closest to the corresponding sample feature, and the K candidate cluster centers are updated based on the mean of the sample features corresponding to the K candidate cluster centers.

[0206] After determining the K candidate cluster centers, the first server may iteratively update the K candidate cluster centers until the distance between the same candidate cluster centers output from two iterative updates is less than or equal to a first preset value. The following is an explanation of an iterative update process:

[0207] S1.2.1, cluster each sample feature based on K candidate cluster centers.

[0208] The first server may determine the distance between each sample feature in each sample feature and each candidate cluster center in the K candidate cluster centers, where the distance is, for example, Euclidean distance or cosine distance, etc. For each sample feature, K distances may be obtained correspondingly, and the sample feature is classified as the candidate cluster center corresponding to the minimum distance among the K distances, which is equivalent to preliminarily determining the candidate cluster center corresponding to the sample feature.

[0209] S1.2.2, update the candidate cluster center based on the mean of the sample features belonging to the same candidate cluster center.

[0210] After clustering each sample feature, the first server can obtain sample features belonging to each candidate cluster center in the K candidate cluster centers, which is equivalent to the first server obtaining each sample feature belonging to each candidate cluster center. The first server determines the mean between the sample features under each cluster, where the mean here is, for example, the average value between the sample features under the cluster, and the first server updates the mean under the cluster to the candidate cluster center under the cluster.

[0211] By analogy, the first server may iteratively update each of the K candidate cluster centers by using steps S1.2.1 to S1.2.2.

[0212] After obtaining a cluster center output by this iterative update, the first server can determine whether the distance between the cluster center output by this iterative update and the corresponding cluster center output by the previous iterative update is less than or equal to a first preset value. The distance is calculated by, for example, Euclidean distance or cosine distance. If the distance between the cluster center output by this iterative update and the corresponding cluster center output by the previous iterative update is less than or equal to the first preset value, it is equivalent to determining that the distance between the same candidate cluster center output by two iterative updates is not greater than the first preset value. The so-called "the same candidate cluster center output by two iterative updates" refers to the candidate cluster center targeted by the two iterative updates, indicating that the error between the cluster center output by this iterative update and the cluster center output by the previous iterative update is small, and the cluster center output by this iterative update is determined as the target cluster center. Among them, the corresponding cluster center of the last iterative update output refers to the cluster center of the iterative update output before this iterative update. For example, if the cluster center of the current iterative update output is the cluster center after the Sth update, then the corresponding cluster center of the last iterative update output is the cluster center after the (S-1)th update, and S is a positive integer greater than or equal to 2.

[0213] If the distance between the cluster center output by this iteration update and the corresponding cluster center output by the last iteration update is greater than the first preset value, it means that the error between the cluster center output by this iteration update and the cluster center output by the last iteration update is large, so the cluster center output by this iteration update can be updated according to the mean of the sample features associated with the cluster center output by this iteration update, so as to obtain the cluster center output by another iteration update, until the distance between the cluster center output by another iteration update and the corresponding cluster center output by the last iteration update is less than or equal to the first preset value, so as to obtain the final target cluster center. By analogy, the cluster center output by the last iteration update is used as the target cluster center, so as to obtain the final K target cluster centers.

[0214] After obtaining the final K target cluster centers, the distance between the first sample feature and each of the K target cluster centers can be determined, so as to obtain the target cluster center closest to the first sample feature, and the target cluster center is determined as the target cluster center corresponding to the first sample feature, and the category of the target cluster center is determined as the true category corresponding to the first sample feature.

[0215] For example, see Figure 7 , is an example diagram of a process of determining cluster centers based on the K-means method provided in an embodiment of the present application, Figure 7 In (1), the coordinate points of each sample feature mapping are distributed, as follows: Figure 7 There are 11 points in total, A, B, C, D, E, F, G, H, J, M and N, as shown in the circles in (1). The first server determines 4 cluster centers from these 11 points, specifically, four candidate cluster centers, namely, B, D, H and N.

[0216] The first server determines the distances between A, C, E, F, H, J and M and the four candidate cluster centers B, D, G and N respectively. The first server determines that A, C and E are closest to point B, G and F are closest to H, and J and M are closest to N, so these 11 points are clustered respectively. The distance results are shown in the figure below. Figure 7 As shown in (2), the points belonging to a cluster are located in an ellipse.

[0217] The first server determines the mean of each point belonging to the same cluster, and determines the mean between A, C, H and B as follows: Figure 7 K1 shown in (3) above, the mean of D is still point D, and the mean of G, F and H is determined as follows: Figure 7 K2 shown in (3) determines the mean value between J, N and M as follows Figure 7 K3 shown in (3) in the figure, thereby updating the candidate cluster center corresponding to B to K1, the candidate cluster center corresponding to H to K2, and the candidate cluster center corresponding to N to K3, thereby obtaining each updated candidate cluster center.

[0218] The first server determines that the distance between K1 and B is less than the first preset value, determines K1 as the final target cluster center, determines D as a target cluster center, determines that the distance between K2 and H is less than the first preset value, determines K2 as the final target cluster center, and determines that the distance between K3 and N is less than the first preset value, thereby obtaining the target cluster center.

[0219] In the embodiment of the present application, the real category of the sample image is determined by using the K-means clustering method, and there is no need to manually annotate the sample image, which can relatively improve the efficiency of image annotation, and the image processing model can be unsupervisedly trained based on the real category. In addition, when determining the real category of the sample image, the sample features output by the feature extraction module are used for clustering, and the feature extraction module can extract the features required for classifying the sample image at different clustering granularities, which is equivalent to determining the real category based on various features of the sample image, thereby improving the accuracy of clustering the sample image, and further improving the accuracy of determining the real category of the sample image.

[0220] As discussed above, the value of the number K of candidate cluster centers is actually the same as the number of categories corresponding to the cluster granularity. In an embodiment of the present application, the number of categories at any two different granularity levels may be the same or different. In order to enable the image processing model to learn the features at each granularity level in a more hierarchical manner, in an embodiment of the present application, the first server may set the number of categories at each granularity level according to the size of the granularity level, for example: the larger the granularity level, the fewer the number of categories at the corresponding granularity level.

[0221] Specifically, the larger the granularity level, the coarser the classification level, so the image processing model can output the category at that granularity level based on the overall features of the image. If the granularity level is smaller, the classification level is finer, so the image processing model is required to learn more detailed features to output the category at that granularity level. Therefore, if the granularity level is smaller, the number of corresponding categories is greater, and the image processing model can learn the features corresponding to various categories. In other words, the image processing model can learn more comprehensive and rich features to facilitate more accurate classification of images.

[0222] Furthermore, in order to make the image processing model learn the distribution of features at different levels more widely, in the embodiment of the present application, the number of categories corresponding to two different granularity levels is different in magnitude, so that the image processing model can have richer features at different granularity levels, so that the image processing model can classify the image more accurately later. For example, the image processing model can output the number of categories at the first granularity level, the second granularity level, and the third granularity level, the number of categories corresponding to the first granularity level can be set to (N / 10), N is a positive integer, the number of categories corresponding to the second granularity level can be set to N, and the number of categories corresponding to the third granularity level can be set to (N*10).

[0223] As an embodiment, when classifying the first sample feature, the classification can be performed based on the high-level abstract features of the first sample feature points, and the high-level abstract features can be obtained by performing various operations on the underlying features. For an image classification, no matter which clustering granularity is used to classify the image, the underlying features of the image are relatively fixed. Therefore, in the embodiment of the present application, please refer to Figure 6 , which is a structural schematic diagram 2 of an image processing model provided for the implementation of the present application, wherein the feature extraction module in the image processing model includes a first feature extraction submodule for extracting underlying features and a second feature extraction submodule for extracting high-level abstract features, and when adjusting the parameters of the image processing model, only the parameters of the second feature extraction submodule need to be adjusted.

[0224] Specifically, the first server inputs the first sample image into the first feature extraction submodule to obtain the underlying features of the first sample image, and the underlying features are input into the second feature extraction submodule to obtain high-level abstract features of the first sample image, where the high-level abstract features are an example of the first sample features discussed above. The first feature extraction submodule may be pre-trained, for example, pre-trained based on ImageNet. When adjusting the parameters of the image processing model based on the training loss value later, it is only necessary to adjust the parameters of the second feature extraction submodule and the category recognition module in the image processing model, without adjusting the parameters of the first feature extraction submodule.

[0225] In a specific implementation, the first feature extraction submodule may use ResNet-101, wherein the specific structure of the first feature extraction submodule is shown in Table 4 below:

[0226] Table 4

[0227]

[0228] Referring to the first feature extraction module shown in Table 4, it includes five convolution groups, namely Conv1, Conv2_x, Conv3_x, Conv4_x and Conv5_x in Table 4, respectively. The first convolution group includes one convolution layer; the second convolution group includes a maximum pooling layer with a step size of 2, and 3 sub-modules, each of which includes three convolution layers; the third convolution group includes 4 sub-modules, each of which includes three convolution layers; the fourth convolution group includes 23 sub-modules, each of which includes three convolution layers; the fifth convolution group includes 3 sub-modules, each of which includes three convolution layers.

[0229] It should be noted that the above Table 1 is an example of the structure of the first feature extraction submodule, and does not actually limit the structure of the first feature extraction submodule.

[0230] As an embodiment, the second feature extraction submodule may adopt an embedding structure. The specific structure of the first feature extraction submodule is shown in Table 5 below:

[0231] Table 5

[0232] Layer name Output size Layer Pool_cr 1x2048 Max pool map 1x1024 Fully connected layer Fc_map 1x128 Fully connected layer

[0233] The embedding structure includes a maximum pooling layer, a first fully connected layer, and a second fully connected layer in sequence. In order to facilitate adjustment of the parameters of the second feature extraction submodule, the parameters of the second feature extraction submodule can be initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0234] It should be noted that the above Table 1 is an example of the structure of the second feature extraction submodule, and does not actually limit the structure of the second feature extraction submodule. For example, the second feature extraction submodule can add a deep neural network layer or a residual network structure, etc. The present application does not limit the specific structure of the second feature extraction submodule.

[0235] In an embodiment of the present application, the feature extraction module is divided into a first feature extraction submodule for extracting underlying features and a second feature extraction submodule for extracting high-level abstract features. Since the underlying features of the image are relatively fixed, even if the parameters of the first feature extraction submodule are not adjusted during subsequent parameter adjustment, the extraction results of the feature extraction module will not be affected. In this way, the accuracy of the output of the feature extraction module can be guaranteed while reducing the amount of parameter adjustment.

[0236] In another possible embodiment, when adjusting the parameters of the image processing model based on the training loss value, the parameters of the first feature extraction submodule may also be adjusted, but this application does not limit this.

[0237] In order to more clearly illustrate the image processing model training parameters in the embodiment of the present application, the image processing models shown in Tables 1 to 5 above are used as examples to introduce the process of training the image processing model:

[0238] 1. Determine the true category of the first sample image, which specifically includes the following process:

[0239] ①, cluster the sample image set into (N / 10) categories using the K-means clustering method, and obtain (N / 10) cluster centers, which are numbered 1, 2…(N / 10), which is equivalent to obtaining the true categories included at this level of granularity;

[0240] ② For each first sample image in the sample image set, perform the following process:

[0241] ③. Calculate the distance between the first sample image and all cluster centers;

[0242] ④. Select the cluster center number i (a value among 1, 2…N / 10 numbers) corresponding to the cluster center with the nearest distance as the true category of the sample cluster.

[0243] ⑤. Record the true category of each first sample image.

[0244] 2. Training the second feature extraction submodule and the first category identification branch module includes the following steps:

[0245] The first sample image is sequentially input into the first feature extraction submodule, the second feature extraction submodule and the first category recognition branch module to obtain the predicted category of the first sample image, and the loss value (classification loss) corresponding to the second feature extraction submodule and the first category recognition branch module is calculated based on the predicted category and the true category. Based on the loss value, the updated value of the parameters of the second feature extraction submodule and the first category recognition branch module is calculated, and based on the updated value, the parameters of the second feature extraction submodule and the first category recognition branch module are adjusted, specifically adjusting the parameters of the structures shown in Table 1, Table 2 and Table 5.

[0246] 3. Training the second feature extraction submodule, the first category recognition branch module, the second category recognition branch module and the third category recognition branch module specifically includes the following process:

[0247] Based on the trained second feature extraction submodule and first category recognition branch module, the parameters of the second feature extraction submodule, the first category recognition branch module, the second category recognition branch module, and the third category recognition branch module are adjusted. The learning process is similar to the coarse-grained pre-training, but two clustering tasks are added during clustering task generation and loss calculation (specifically, clustering tasks at the second granularity level and clustering tasks at the third granularity level):

[0248] ① Cluster the sample image set into N categories using the K-means clustering method to determine the category corresponding to each sample image at the medium granularity level;

[0249] ②, cluster the sample image set into (N*10) categories using the K-means clustering method to determine the category corresponding to each sample image at a fine-grained level;

[0250] ③. Calculate the loss value corresponding to the first granularity level, the loss value corresponding to the second granularity level, and the loss value corresponding to the third granularity level respectively, determine the weighted sum of the loss value corresponding to the first granularity level, the loss value corresponding to the second granularity level, and the loss value corresponding to the third granularity level, so as to obtain the corresponding training loss value.

[0251] Based on the training loss value, the update values ​​of the parameters corresponding to the second feature extraction submodule, the first category identification branch module, the second category identification branch module and the third category identification branch module are calculated; based on the update values, the parameters of the second feature extraction submodule, the first category identification branch module, the second category identification branch module and the third category identification branch module are adjusted until the second feature extraction submodule, the first category identification branch module, the second category identification branch module and the third category identification branch module meet the first convergence condition, thereby obtaining a trained image processing model.

[0252] Based on the same inventive concept, the embodiment of the present application provides an image processing method. The following takes the second server executing the image processing method as an example to introduce the image processing method involved in the embodiment of the present application. Please refer to Figure 8 The image processing method shown is a flowchart of the image processing method, the image processing method comprising:

[0253] S81, obtaining a trained image processing model.

[0254] The second server can obtain the trained image processing model through any of the image processing model training methods discussed above, or the second server can obtain the trained image processing model from the first server. The content of obtaining the trained image processing model can refer to the method for training the image processing model discussed above, which will not be repeated here.

[0255] S82, inputting the image to be processed into a feature extraction module to obtain target features corresponding to the image to be processed.

[0256] The second server can obtain the image to be processed from the terminal, or can obtain the image to be processed based on the user's input operation. After the second server obtains the image to be processed, the second server can input the image to be processed into the feature extraction module. The specific implementation of the feature extraction module can refer to the content discussed in the previous article, which will not be repeated here. After the image to be processed passes through the feature extraction module, the target feature corresponding to the image to be processed can be obtained. The implementation form of the target feature of the image to be processed can refer to the implementation form of the first sample feature discussed in the previous article, which will not be repeated here.

[0257] S83, inputting the target features into a category recognition module to obtain the target category of the image to be processed.

[0258] The second server can input the target features into the category recognition module in the image processing module to obtain the target category of the image to be processed. As discussed above, the category recognition module can output the corresponding prediction categories at least three clustering granularities. Therefore, the target category of the image to be processed can be at least one of the corresponding prediction categories at at least three clustering granularities.

[0259] When outputting any category based on the target feature, the category recognition module can directly output the category to which it belongs at the clustering granularity, or can output the probability that the image to be processed belongs to each category at the clustering granularity. The second server determines the category of the image to be processed at the clustering granularity according to the category corresponding to the probability that meets the preset condition. The preset condition is, for example, the maximum probability or the probability is greater than the preset probability.

[0260] exist Figure 8 In the illustrated embodiment, since the feature extraction module in the image processing model is capable of learning the features required for the category recognition task at at least three levels of granularity, the category recognition module classifies the image to be processed based on the features extracted by the feature extraction module, thereby improving the accuracy of the category of the output image to be processed.

[0261] As an embodiment, when the category identification module includes a first category identification branch module, a second category identification branch module and a third category identification branch module, the structures and functions of the first category identification branch module, the second category identification branch module and the third category identification branch module can refer to the contents discussed above and will not be repeated here. The target category is the category corresponding to the middle clustering granularity output by the second category identification branch module.

[0262] In an embodiment of the present application, the second category branch identification module in the category identification module is used as the output. Since the feature extraction module learns the features at a clustering granularity coarser than the medium clustering granularity and the features at a clustering granularity finer than the medium clustering granularity during the training process, the second category identification branch module can combine the features at the coarse, medium and fine clustering granularities to determine the category corresponding to the image to be processed at the medium clustering granularity, thereby improving the accuracy of the output target category.

[0263] For example, see Fig. 9 As shown in the example diagram of the process of determining the target category of the image to be processed, the second server obtains the target feature of the image to be processed through the feature extraction module, and outputs the corresponding classification of the image to be processed at the coarse clustering granularity, the medium clustering granularity and the fine clustering granularity based on the target feature, as shown in the specific Fig. 9 Things, vehicles and trucks shown.

[0264] Furthermore, after determining the target category of the image to be processed, the image to be processed may be clustered.

[0265] Specifically, the second server can divide the parameters of the category recognition module in the category recognition module into multiple cluster centers, determine the distance between the target feature of the image to be processed after passing through the feature extraction module and each cluster center, and select the cluster center closest to the target feature as its corresponding cluster center, which is equivalent to inputting the target feature into the category recognition branch module to obtain the target category corresponding to the image to be processed, thereby realizing image classification.

[0266] Continuing with the second category recognition branch module corresponding to Table 2 as an example, the parameters learned by Fc_cluster in Table 2 are specifically a (128*N) matrix, which can be used as N vectors, where N is a positive integer. The second server can use these N vectors as cluster centers to obtain N cluster centers and perform bucket search and recall on each image, specifically including:

[0267] 1) Database bucketing: For all images in the database, select the nearest cluster center as its bucket;

[0268] 2) Bucket recall: Determine the target feature of the image to be processed and the distance between it and N cluster centers to obtain N distances. The first M cluster centers closest to the image to be processed are determined as candidate cluster centers of the image to be processed. The data sets corresponding to these candidate cluster centers are determined as candidate data sets corresponding to the image to be processed. M is a positive integer less than N.

[0269] However, the candidate data set determined in this way is not necessarily accurate. Therefore, in an embodiment of the present application, the second server can further determine the data set corresponding to the image to be processed based on the local features of the image to be processed.

[0270] Specifically, the second server can determine the local features of the image to be processed, for example, the scale-invariant feature transform (SIFT), and determine the local features of the reference image in the candidate data set. The reference image in the candidate data set can be an image in the candidate data set that meets preset conditions, or an image randomly selected from the candidate data set. The preset conditions, for example, are that the image clarity is greater than the preset clarity or the image color saturation is greater than the preset saturation, etc., and then determine the similarity between the local features of the image to be processed and the reference image. If the determined similarity is greater than or equal to the second preset value, it means that the image to be processed has a high similarity with the reference image in the candidate data set. Therefore, the image to be processed is clustered into the candidate data set, and the target data set to which the image to be processed belongs is determined.

[0271] In the embodiment of the present application, since the global features and detail features of the image to be processed are taken into consideration when determining the initial clustering of the image to be processed, the bucket recall result is more accurate.

[0272] Furthermore, if the above image processing method is applied to a video retrieval scenario, the image to be processed is a video frame in the video, and specifically, for example, it can be a plurality of video frames collected from the video according to a preset sampling time. By determining the target data set to which the image to be processed belongs through any of the above image processing methods, the bucket to which the image to be processed belongs is correspondingly determined, and the bucket to which the image to be processed belongs is used as the bucket to which the video belongs. Subsequently, the corresponding video can be directly retrieved from each bucket to achieve fast and accurate retrieval of the video.

[0273] Based on the same inventive concept, the present application embodiment provides an image processing model training device, which can realize the function of the first server discussed above, please refer to Fig.10 The structural diagram of the image processing model training device shown in FIG. 1 includes:

[0274] An image acquisition module 1001 is used to acquire a sample image set;

[0275] The model training module 1002 is used to iteratively train the image processing model to be trained based on the sample image set until a first convergence condition is met, wherein in one iterative training process, the following operations are performed:

[0276] Inputting a first sample image in the sample image set into a feature extraction module in the image processing model to obtain a first sample feature corresponding to the first sample image;

[0277] Inputting the first sample feature into a category recognition module in the image processing model to obtain predicted categories corresponding to the first sample image at at least three clustering granularities, wherein the at least three clustering granularities include clustering granularities of at least three different granularity levels;

[0278] Based on the obtained predicted categories and the real categories corresponding to the first sample image at at least three clustering granularities, a training loss value is determined, and the parameters of the feature extraction module and the category recognition module are adjusted based on the training loss value, wherein the training loss value is used to represent the error between the obtained predicted categories and the corresponding real categories.

[0279] In a possible embodiment, the category identification module includes a plurality of category identification branch modules; the model training module 1002 is specifically used for:

[0280] The first sample features are respectively input into each category recognition branch module in the category recognition module to obtain the predicted categories output by each category recognition branch module, wherein each predicted category output by each category recognition branch module includes at least corresponding categories under at least three clustering granularities of different granularity levels.

[0281] In a possible embodiment, the multiple category recognition branch modules include at least a first category recognition branch module, a second category recognition branch module and a third category recognition branch module; the model training module 1002 is specifically used for:

[0282] Inputting the first sample features into the first category recognition branch module respectively to obtain the predicted category corresponding to the first sample image at the first clustering granularity; and

[0283] Inputting the first sample features into the second category recognition branch module respectively to obtain the predicted category corresponding to the first sample image at the second clustering granularity, wherein the granularity level of the second clustering granularity is smaller than the granularity level of the first clustering granularity; and

[0284] The first sample features are respectively input into the third category recognition branch module to obtain the predicted category corresponding to the first sample image at the third clustering granularity, wherein the granularity level of the third clustering granularity is smaller than the granularity level of the second clustering granularity.

[0285] In a possible embodiment, the model training module 1002 is specifically used for:

[0286] Before adjusting the parameters of the feature extraction module and the category recognition module based on the training loss value, the feature extraction module and the first category recognition branch module are iteratively trained based on the sample image set until the feature extraction module and the first category recognition branch module meet the second convergence condition, wherein, in one iterative training process, the following operations are performed:

[0287] Determine a loss value corresponding to the first clustering granularity based on a predicted category corresponding to a second sample image in the sample image set at the first clustering granularity and a true category corresponding to the second sample image at the first clustering granularity;

[0288] Based on the loss value corresponding to the first clustering granularity, the parameters of the feature extraction module and the first category identification branch module are adjusted.

[0289] In a possible embodiment, the feature extraction module includes a pre-trained first feature extraction submodule and a second feature extraction submodule; the model training module 1002 is specifically used for:

[0290] Inputting the first sample image into a first feature extraction submodule to obtain underlying features of the first sample image, wherein the underlying features are used to describe features presented by the first sample image on a visual layer;

[0291] Inputting the bottom-level features into the second feature extraction submodule to obtain high-level abstract features of the first sample image, wherein the high-level abstract features are used to describe the features presented by the first sample image at the conceptual level;

[0292] Based on the training loss value, adjust the parameters of the second feature extraction submodule.

[0293] In a possible embodiment, the model training module 1002 is specifically used for:

[0294] In a possible embodiment, the model training module is specifically used for:

[0295] Before determining the training loss value based on the obtained predicted categories and the real categories corresponding to the first sample images at at least three clustering granularities, the real categories corresponding to the first sample images at at least three clustering granularities are obtained in the following manner:

[0296] From each sample feature, K sample features are determined as K candidate cluster centers, wherein each sample feature is output by each sample image in the sample image set through a feature extraction module, and K is a positive integer;

[0297] For the K candidate cluster centers, the following method is used for iterative updating until the distance between the same candidate cluster centers outputted from two iterative updates is not greater than a first preset value: clustering each sample feature to the candidate cluster center with the closest distance to the corresponding sample feature among the K candidate cluster centers, and updating the K candidate cluster centers based on the mean values ​​of the sample features corresponding to the K candidate cluster centers;

[0298] The K candidate cluster centers output by the last iterative update are used as K target candidate cluster centers, and the first sample feature is clustered to the target cluster center that is closest to the first sample feature among the K target cluster centers to obtain the true category corresponding to the first sample feature.

[0299] It should be noted that Fig.10 The device shown can also implement any of the image processing methods discussed above, which will not be described in detail here.

[0300] Fig.10 The image processing model training device shown can be set in the first server discussed above, or in other devices with image processing capabilities. The present application is not limited to this. The image processing model training device can train the image processing model in combination with category recognition tasks at at least three granularity levels, so that the image processing model can be classified based on category recognition and classification tasks at more granularity levels to improve the accuracy of the image processing model output.

[0301] Based on the same inventive concept, the embodiment of the present application provides an image processing device, which can realize the function of the second server discussed above, please refer to Fig.11 The structural schematic diagram of the image processing device shown in FIG. 1 includes:

[0302] A model acquisition module 1101 is used to obtain a trained image processing model;

[0303] A feature acquisition module 1102 is used to input the image to be processed into a feature extraction module in the trained image processing model to obtain target features corresponding to the image to be processed;

[0304] The category acquisition module 1103 is used to input the target features into the category recognition module in the trained image processing model to obtain the target category of the image to be processed, wherein the target category belongs to the predicted category corresponding to the image to be processed output by the category recognition module at at least three clustering granularities, and the at least three clustering granularities include at least three clustering granularities of different granularity levels.

[0305] In a possible embodiment, the apparatus further includes an image clustering module 1104, wherein the image clustering module 1104 is configured to:

[0306] Based on the target category, determine the candidate data set to which the image to be processed belongs;

[0307] Determine the similarity between the local features of the image to be processed and the local features of the reference images in the candidate data set;

[0308] If the determined similarity is greater than or equal to the second preset value, the image to be processed is clustered into a candidate data set.

[0309] As an example, Fig.11 The image clustering module 1104 in is an optional part.

[0310] It should be noted that Fig.11 The device shown can also implement any of the image processing methods discussed above, which will not be described in detail here.

[0311] Fig.11The image processing device shown can be set in the second server discussed above, or in other devices with image processing capabilities. This application is not limited to this. The image processing device can combine the features required for the category recognition tasks at at least three granularity levels to output the target category of the image to be processed. Since the classification is based on the features required for the category recognition classification tasks at more granularity levels, the accuracy of the output target category is improved. Moreover, since the image processing model can output categories at multiple granularity levels at one time, the accuracy of the target category of the output image to be processed is improved. Moreover, if the image processing device needs to output categories at multiple granularity levels, it only needs to deploy one image processing model, which relatively reduces the resource overhead required to deploy the model.

[0312] Based on the same inventive concept, the embodiment of the present application provides a computer device, which can implement the functions of the first server mentioned above, please refer to Fig.12 , is a structural diagram of a computer device provided in an embodiment of the present application, wherein the computer device includes a processor 1201 and a memory 1202.

[0313] The processor 1201 may be a central processing unit (CPU) or a digital processing unit, etc. The specific connection medium between the memory 1202 and the processor 1201 is not limited in the embodiment of the present application. Fig.12 In the embodiment, the memory 1202 and the processor 1201 are connected via a bus 1203. The bus 1203 is connected to the processor 1201 via a bus 1203. Fig.12 The connections between other components are shown in bold lines, and are not intended to be limiting. The bus 1203 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0314] The memory 1202 may be a volatile memory, such as a random-access memory (RAM); the memory 1202 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 1202 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1202 may be a combination of the above memories.

[0315] Processor 1201 is used to execute any of the image processing model training methods discussed above when calling the computer program stored in memory 1202.

[0316] Fig.12 The computer device shown may be, for example, the first server discussed above, or other devices with image processing capabilities, and the present application is not limited to this.

[0317] Based on the same inventive concept, the embodiment of the present application provides a computer device, which can implement the functions of the second server discussed above. Please refer to Fig.13 , is a structural diagram of a computer device provided in an embodiment of the present application, wherein the computer device includes a processor 1301 and a memory 1302.

[0318] The implementation forms of the processor 1301, the memory 1302 and the bus 1303 can refer to the contents discussed above, and will not be repeated here. The processor 1301 can be used to execute any image processing method discussed above when calling the computer program stored in the memory 1302.

[0319] Fig.13 The computer device shown may be, for example, the first server discussed above, or other devices with image processing capabilities, and the present application is not limited to this.

[0320] Based on the same inventive concept, an embodiment of the present application provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes any of the image processing model training methods or image processing methods discussed above.

[0321] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0322] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes any of the above-mentioned image processing model training methods or image processing methods.

[0323] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, etc., various media that can store program codes.

[0324] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0325] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for training an image processing model, characterized in that: include: Based on the sample image set, the image processing model to be trained is iteratively trained until a first convergence condition is met, wherein in one iterative training process, the following operations are performed: Inputting a first sample image in the sample image set into a feature extraction module in the image processing model to obtain a first sample feature corresponding to the first sample image, wherein the feature extraction module includes a pre-trained first feature extraction submodule and a second feature extraction submodule; Inputting the first sample feature into a category recognition module in the image processing model to obtain predicted categories corresponding to the first sample image at at least three clustering granularities, wherein the at least three clustering granularities include clustering granularities of at least three different granularity levels; Determine a training loss value based on each of the obtained predicted categories and the real categories corresponding to the first sample image at the at least three clustering granularities, and adjust the parameters of the feature extraction module and the category recognition module based on the training loss value, wherein the training loss value is used to represent the error between each of the obtained predicted categories and the corresponding real category; The step of inputting the first sample image in the sample image set into the feature extraction module in the image processing model to obtain the first sample feature corresponding to the first sample image comprises: inputting the first sample image into the first feature extraction submodule to obtain the underlying features of the first sample image, wherein the underlying features are used to describe the features presented by the first sample image on the visual layer, and inputting the underlying features into the second feature extraction submodule to obtain the high-level abstract features of the first sample image, wherein the high-level abstract features are used to describe the features presented by the first sample image on the conceptual layer; The adjusting the parameters of the feature extraction module based on the training loss value includes: adjusting the parameters of the second feature extraction submodule based on the training loss value.

2. The method according to claim 1, characterized in that The category identification module includes a plurality of category identification branch modules; The step of inputting the first sample feature into a category recognition module in the image processing model to obtain the predicted categories corresponding to the first sample image at at least three clustering granularities respectively includes: The first sample features are respectively input into each category recognition branch module in the category recognition module to obtain the predicted categories output by each category recognition branch module, wherein each predicted category output by each category recognition branch module includes at least corresponding categories under at least three clustering granularities of different granularity levels.

3. The method according to claim 2, characterized in that The plurality of category identification branch modules at least include a first category identification branch module, a second category identification branch module and a third category identification branch module; Inputting the first sample feature into each category recognition branch module in the category recognition module respectively, and obtaining the predicted category output by each category recognition branch module respectively, comprises: Inputting the first sample features into the first category recognition branch module respectively to obtain the predicted category corresponding to the first sample image at the first clustering granularity; and Inputting the first sample features into the second category recognition branch module respectively to obtain the predicted category corresponding to the first sample image at a second clustering granularity, wherein the granularity level of the second clustering granularity is smaller than the granularity level of the first clustering granularity; and The first sample features are respectively input into the third category recognition branch module to obtain the predicted category corresponding to the first sample image at the third clustering granularity, wherein the granularity level of the third clustering granularity is smaller than the granularity level of the second clustering granularity.

4. The method according to claim 3, characterized in that Before adjusting the parameters of the feature extraction module and the category identification module based on the training loss value, the method further includes: Based on the sample image set, the feature extraction module and the first category identification branch module are iteratively trained until the feature extraction module and the first category identification branch module meet a second convergence condition, wherein in one iterative training process, the following operations are performed: Determine a loss value corresponding to the first clustering granularity based on a predicted category corresponding to a second sample image in the sample image set at the first clustering granularity and a true category corresponding to the second sample image at the first clustering granularity; Based on the loss value corresponding to the first clustering granularity, the parameters of the feature extraction module and the first category identification branch module are adjusted.

5. The method according to any one of claims 1 to 4, characterized in that The determining of the training loss value based on the obtained predicted categories and the real categories respectively corresponding to the first sample images at the at least three clustering granularities includes: For each of the at least three clustering granularities, the following operations are performed respectively: based on the predicted category corresponding to the first sample image at the clustering granularity and the true category corresponding to the first sample image at the clustering granularity, determining the loss value corresponding to the clustering granularity; The obtained loss values ​​are weightedly summed to obtain the training loss value.

6. The method according to any one of claims 1 to 4, characterized in that Before determining the training loss value based on the obtained predicted categories and the real categories corresponding to the first sample images at the at least three clustering granularities, the method further includes: obtaining the real categories corresponding to the first sample images at the at least three clustering granularities in the following manner: Determine K sample features from each sample feature as K candidate cluster centers, wherein each sample feature is output by each sample image in the sample image set after being output by the feature extraction module, and K is a positive integer; For the K candidate cluster centers, iterative updating is performed in the following manner until the distance between the same candidate cluster centers output from two iterative updates is no greater than a first preset value: clustering each of the sample features to a candidate cluster center that is closest to the corresponding sample feature among the K candidate cluster centers, and updating the K candidate cluster centers based on the mean values ​​of the sample features corresponding to the K candidate cluster centers; The K candidate cluster centers output by the last iterative update are used as K target candidate cluster centers, and the first sample feature is clustered to the target candidate cluster center that is closest to the first sample feature among the K target candidate cluster centers to obtain the true category corresponding to the first sample feature.

7. An image processing method, characterized in that: include: Obtaining a trained image processing model, wherein the trained image processing model is obtained by the method of any one of claims 1 to 6, and the trained image processing model includes a feature extraction module and a category recognition module; Inputting the image to be processed into the feature extraction module to obtain the target features corresponding to the image to be processed; The target feature is input into the category recognition module to obtain the target category of the image to be processed, wherein the target category belongs to the predicted category corresponding to each of the image to be processed output by the category recognition module at at least three clustering granularities, and the at least three clustering granularities include clustering granularities of at least three different granularity levels.

8. The method according to claim 7, characterized in that After inputting the target feature into the category recognition module to obtain the target category of the image to be processed, the method further includes: Based on the target category, determining a candidate data set to which the image to be processed belongs; Determining the similarity between the local features of the image to be processed and the local features of the reference images in the candidate data set; If the determined similarity is greater than or equal to a second preset value, the image to be processed is clustered into the candidate data set.

9. An image processing module training device, characterized in that: include: An image acquisition module, used to acquire a sample image set; The model training module is used to iteratively train the image processing model to be trained based on the sample image set until a first convergence condition is met, wherein in one iterative training process, the following operations are performed: Inputting a first sample image in the sample image set into a feature extraction module in the image processing model to obtain a first sample feature corresponding to the first sample image, wherein the feature extraction module includes a pre-trained first feature extraction submodule and a second feature extraction submodule; Inputting the first sample feature into a category recognition module in the image processing model to obtain predicted categories corresponding to the first sample image at at least three clustering granularities, wherein the at least three clustering granularities include clustering granularities of at least three different granularity levels; Determine a training loss value based on each of the obtained predicted categories and the real categories corresponding to the first sample image at the at least three clustering granularities, and adjust the parameters of the feature extraction module and the category recognition module based on the training loss value, wherein the training loss value is used to represent the error between each of the obtained predicted categories and the corresponding real category; The step of inputting the first sample image in the sample image set into the feature extraction module in the image processing model to obtain the first sample feature corresponding to the first sample image comprises: inputting the first sample image into the first feature extraction submodule to obtain the underlying features of the first sample image, wherein the underlying features are used to describe the features presented by the first sample image on the visual layer, and inputting the underlying features into the second feature extraction submodule to obtain the high-level abstract features of the first sample image, wherein the high-level abstract features are used to describe the features presented by the first sample image on the conceptual layer; The adjusting the parameters of the feature extraction module based on the training loss value includes: adjusting the parameters of the second feature extraction submodule based on the training loss value.

10. The device according to claim 9, characterized in that The category recognition module includes a plurality of category recognition branch modules; the model training module is specifically used for: The first sample features are respectively input into each category recognition branch module in the category recognition module to obtain the predicted categories output by each category recognition branch module, wherein each predicted category output by each category recognition branch module includes at least corresponding categories under at least three clustering granularities of different granularity levels.

11. An image processing device, characterized in that: include: A model acquisition module, used to obtain a trained image processing model, wherein the trained image processing model is obtained by any method of claims 1 to 6; A feature acquisition module, used for inputting the image to be processed into the feature extraction module in the trained image processing model to obtain the target features corresponding to the image to be processed; A category acquisition module, used for inputting the target feature into a category recognition module in the trained image processing model to obtain a target category of the image to be processed, wherein the target category belongs to the predicted categories of the image to be processed output by the category recognition module at at least three clustering granularities, respectively, and the at least three clustering granularities include clustering granularities of at least three different granularity levels; The image clustering module is used to cluster the to-be-processed images based on the target categories.

12. A computer device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method according to any one of claims 1 to 6, or implements the method according to claim 7 or 8 by executing the instructions stored in the memory.

13. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the method according to any one of claims 1 to 6, or implements the method according to claim 7 or 8.

Images