Image classification model training method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image classification model training method and device, equipment and a storage medium, which can be applied to various scenes such as artificial intelligence, cloud technology, smart traffic, maps, vehicle-mounted vehicles and auxiliary driving, and the method comprises the following steps: in each round of training process, selecting a training image from a training sample set to carry out feature extraction, constructing a basic loss function by training image features and corresponding type labels; and acquiring auxiliary image features matched with the training image features from the auxiliary image-text feature set, performing supervised alignment on the auxiliary image features and auxiliary text features associated with the auxiliary image features to construct an auxiliary loss function, and performing parameter adjustment on the image classification model in combination with the basic loss function and the auxiliary loss function to obtain the image classification model. The fine-tuned model can be better in performance on unlearned data, and the zero sample learning ability and generalization ability of the model are improved, so that the classification accuracy of the image classification model is improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a training method, apparatus, device and storage medium for an image classification model. Background Art

[0002] Image-text pre-training models, such as CLIP (Contrastive Language-Image Pre-training) and ALIGN, align large numbers of images and text into a common space through contrastive learning, demonstrating excellent domain transfer and zero-shot learning capabilities. In practical applications, image-text pre-training models also require fine-tuning on training datasets corresponding to specific tasks, such as image classification and image-text retrieval, to achieve better performance for the corresponding tasks.

[0003] However, existing mainstream fine-tuning methods, such as robust fine-tuning, typically use category information for supervision. As a result, pre-trained image and text models can only learn features directly related to category labels during fine-tuning. However, over-reliance on features of specific categories and neglect of other useful information can cause the fine-tuned model to overfit to the specific category labels in the training dataset, resulting in poor performance on unlearned data and a significant decrease in the model's zero-shot learning ability. In real applications, both data distribution and classification tasks may vary. Models with weak zero-shot learning capabilities cannot effectively handle different data and tasks, resulting in reduced generalization and classification accuracy.

[0004] For example, the CLIP model was robustly fine-tuned on a training dataset to obtain an image classification model for animal classification tasks. This training dataset contained images of various animal categories, such as cats, dogs, and birds. The fine-tuned CLIP image classification model was able to achieve relatively accurate classification results for these animal categories. However, because only images of these animal categories were used during the fine-tuning process, the CLIP image classification model's generalization ability for image classification tasks of other categories, such as trains and bicycles, was neglected, resulting in a significant decrease in the CLIP image classification model's classification accuracy for these other categories. Summary of the Invention

[0005] The embodiments of the present application provide a training method, apparatus, device, and storage medium for an image classification model, which are used to improve the generalization ability and classification accuracy of the image classification model.

[0006] In one aspect, a method for training an image classification model is provided, the method comprising:

[0007] Based on multiple training sample sets and in combination with auxiliary image-text feature sets, the image classification model to be trained is trained for multiple rounds of iterative training to obtain a trained target image classification model; each training sample set includes: a training image and a type label for identifying the type of the training image; the auxiliary image-text feature set includes: multiple pairs of auxiliary image features and auxiliary text features; wherein each round of iteration includes:

[0008] Performing feature extraction on the training images in the selected training sample set to obtain corresponding training image features; constructing a basic loss function based on the training image features and corresponding type labels;

[0009] Based on the training image features, selecting a matching target auxiliary image feature from the auxiliary image-text feature set, wherein the similarity between the target auxiliary image feature and the training image feature is greater than a set threshold; performing supervised alignment processing on the target auxiliary image feature and the associated target auxiliary text feature, and constructing an auxiliary loss function;

[0010] In combination with the basic loss function and the auxiliary loss function, the model parameters of the image classification model are adjusted.

[0011] In one aspect, a training device for an image classification model is provided, the device comprising:

[0012] A model training unit is configured to perform multiple rounds of iterative training on the image classification model to be trained based on multiple training sample sets in combination with an auxiliary image-text feature set to obtain a trained target image classification model; each training sample set includes: a training image and a type label for identifying the type of the training image; the auxiliary image-text feature set includes: multiple pairs of auxiliary image features and auxiliary text features;

[0013] The model training unit includes a basic function sub-unit, an auxiliary function sub-unit and a parameter adjustment sub-unit, wherein:

[0014] The basic function subunit is used to extract features of the training images in the selected training sample set to obtain corresponding training image features; and construct a basic loss function based on the training image features and corresponding type labels;

[0015] The auxiliary function subunit is configured to select, based on the training image features, a matching target auxiliary image feature from the auxiliary image-text feature set, wherein the similarity between the target auxiliary image feature and the training image feature is greater than a set threshold; perform supervised alignment processing on the target auxiliary image feature and the associated target auxiliary text feature to construct an auxiliary loss function;

[0016] The parameter adjustment subunit is used to adjust the model parameters of the image classification model in combination with the basic loss function and the auxiliary loss function.

[0017] Optionally, the model parameters include preset embedded image features and embedded text features, and the auxiliary function subunit is further used to:

[0018] Based on the first embedded image feature corresponding to the current round model, perform feature aggregation processing on the target auxiliary image feature to obtain a first aggregated image feature; the first embedded image feature and the target auxiliary image feature have the same feature dimension;

[0019] Based on the first embedded text feature corresponding to the current round model, the target auxiliary text feature is subjected to feature aggregation to obtain a first aggregated text feature; the first embedded text feature and the target auxiliary text feature have the same feature dimension;

[0020] Perform supervised alignment processing on the first aggregated image features and the first aggregated text features to construct the auxiliary loss function.

[0021] Optionally, the parameter adjustment subunit is specifically used to:

[0022] Based on the respective weight values ​​of the basic loss function and the auxiliary loss function, weighted processing is performed on the basic loss function and the auxiliary loss function to obtain a target loss function;

[0023] When a loss value of the target loss function exceeds a preset loss threshold and the current number of iterations is less than a preset number of iterations, adjusting the model parameters of the image classification model based on the loss value;

[0024] When the loss value is less than the preset loss threshold, or the current number of iterations is not less than the preset number of iterations, the trained target image classification model is obtained.

[0025] Optionally, the auxiliary function subunit is specifically used to:

[0026] Normalizing each auxiliary text feature in the auxiliary image-text feature set with the training image feature;

[0027] Performing matrix multiplication operations on the normalized training image features and each auxiliary text feature to obtain similarities between the training image features and each auxiliary image feature;

[0028] Based on the obtained similarities, the target auxiliary image feature is selected from the auxiliary image feature set.

[0029] Optionally, the device further includes a model application unit, configured to:

[0030] Obtain a target image to be classified and multiple type labels, each type label indicating an image type;

[0031] Performing feature extraction on the target image and the multiple type labels respectively to obtain target image features corresponding to the target image and predicted text features corresponding to each of the multiple type labels;

[0032] From the first similarities between each predicted text feature and the target image feature, a target first similarity that meets a preset classification condition and a target predicted text feature corresponding to the target first similarity are obtained; and the image type associated with the target predicted text feature is used as the target image type of the target image.

[0033] Optionally, the model parameters of the target image classification model include trained second embedded image features and second embedded text features, and the model application unit is specifically used to:

[0034] performing feature aggregation processing on the target image feature based on the second embedded image feature to obtain a second aggregated image feature;

[0035] Based on the second embedded text feature, performing feature aggregation processing on each predicted text feature and the second embedded text feature to obtain second aggregated text features corresponding to each predicted text feature;

[0036] Based on the second similarities between the respective second aggregated text features and the second aggregated image features, the second aggregated text feature corresponding to the maximum second similarity is used as the target predicted text feature.

[0037] In one aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0038] In one aspect, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the steps of any of the above methods are implemented.

[0039] In one aspect, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the steps of any of the above methods.

[0040] The beneficial effects of the embodiments of the present application are as follows:

[0041] The present invention provides an iterative training method for an image classification model using a training sample set and an auxiliary image and text feature set. During each round of training, training images are selected from the training sample set for feature extraction. A basic loss function is constructed using the extracted training image features and the class labels of the training images. This helps the model better understand the relationship between image features and class labels, improving image classification accuracy.

[0042] Furthermore, the embodiment of the present application will also obtain auxiliary image features that match the training image features from the auxiliary image and text feature set. Since the auxiliary text features associated with the auxiliary image features have richer and more detailed image description information, the embodiment of the present application introduces an auxiliary loss function by performing supervised alignment processing on the auxiliary image and text features, which can enrich the information source of the model and avoid the fine-tuned model from overfitting to the specific type of label of the training image. At the same time, the embodiment of the present application combines the basic loss function and the auxiliary loss function to adjust the parameters of the image classification model, which can make the fine-tuned model perform better on unlearned data, improve the model's zero-sample learning ability and generalization ability, and thus improve the classification accuracy of the image classification model.

[0043] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 Schematic diagram of application scenarios provided by embodiments of the present application;

[0046] Figure 2 A schematic diagram of a training sample set and auxiliary image-text pairs provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of iterative training of an image classification model provided in an embodiment of the present application;

[0048] Figure 4 A flowchart of a method for training an image classification model provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a process for constructing a contrast loss function provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of a loss function acquisition process provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the aggregation of embedded graphic features and auxiliary graphic features provided in an embodiment of the present application;

[0052] Figure 8 A flowchart of an image classification method provided in an embodiment of the present application;

[0053] Figure 9 A schematic diagram of an application scenario of an image classification method provided in an embodiment of the present application;

[0054] Figure 10 A schematic diagram of an application scenario of another image classification method provided in an embodiment of the present application;

[0055] Figure 11 A schematic diagram of the structure of a training device for an image classification model provided in an embodiment of the present application;

[0056] Figure 12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Unless there is a conflict, the embodiments in the present application and the features in the embodiments can be combined with each other in any way. In addition, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that here.

[0058] The terms "first" and "second" in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of its variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in this application can mean at least two, for example, two, three or more, and the embodiments of this application are not limited thereto.

[0059] The term "and / or" in the embodiments of this application is simply a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0060] It is understood that in the following specific implementations of this application, data related to retention rates and the like are involved. When the various embodiments of this application are applied to specific products or technologies, relevant licenses or consents need to be obtained, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions. For example, when it is necessary to obtain training sample sets and auxiliary graphic feature sets, the data of these volunteers can be recruited and signed with relevant agreements for volunteer authorization data, and then implemented using these volunteers' data; or, by implementing within the scope of an authorized organization, data management can be performed by implementing the following implementation methods using the data of members within the organization; or, the relevant data used in the specific implementation are all simulated data, such as simulated data generated in a virtual scene.

[0061] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0062] Pretrained model: A neural network model pre-trained on a large dataset, typically using unsupervised or weakly supervised learning to learn general feature representations from large-scale data. With rich knowledge and feature representations, pretrained models can be applied to specific tasks through techniques such as fine-tuning or transfer learning to improve their performance.

[0063] CLIP: A contrastive learning model for images and text. By embedding images and text into the same feature space and using a contrastive loss function to train the model, similar images and text are represented more closely in the feature space, enabling the model to understand the semantic relationship between images and text.

[0064] ALIGN: In machine learning, it stands for the process of aligning or matching different data or features. This process can include adjusting the data representation, standardizing the data distribution, and making the features of different data more consistent in the same space, facilitating effective comparison, fusion, or integration.

[0065] Domain transfer is the process of migrating a machine learning model from one domain (the source domain) to a different but related domain (the target domain). In the target domain, the model may fail or perform poorly due to differences in the distribution or characteristics of the data from the source domain. Domain transfer techniques aim to improve the model's performance in the target domain by adapting the model or data representation to the target domain's data distribution.

[0066] Zero-shot learning: refers to the task of learning and classifying by leveraging additional auxiliary information (such as attributes, category descriptions, etc.) in the absence of task-related training samples. In zero-shot learning, the model needs to learn the relationship between categories from the auxiliary information and generalize it to new, unseen categories.

[0067] Generalization: This refers to the performance of a machine learning model when faced with unseen data. Good generalization means that the model can learn general patterns in the data during training and generalize these patterns to unseen data, rather than simply memorizing examples from the training set.

[0068] Fine-tuning: Based on a pre-trained model, further training the model on a task-specific dataset to adjust the model's parameters and adapt it to the specific task. Fine-tuning is usually performed on the top or middle layers of the pre-trained model so that the model can learn the characteristics or knowledge of the specific task.

[0069] Bidirectional Encoder Representations from Transformer (BERT): The goal of the BERT model is to use large-scale unlabeled corpus training to obtain a representation of text that contains rich semantic information, that is, the semantic representation of the text, and then fine-tune the semantic representation of the text in a specific natural language processing (NLP) task and finally apply it to the NLP task.

[0070] The embodiments of the present application relate to artificial intelligence (AI) and machine learning technology, and are designed based on computer vision technology and machine learning (ML) in artificial intelligence.

[0071] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in ways similar to human intelligence.

[0072] Artificial intelligence (AI) is the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technologies primarily encompass computer vision, natural language processing, and machine learning / deep learning. With the advancement of AI technology, research and application are expanding across a wide range of fields, including smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, driverless and autonomous driving, robotics, and smart healthcare. As technology advances, AI will be applied in even more areas and play an increasingly important role.

[0073] NLP is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0074] Machine learning is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Compared to data mining, which seeks to identify patterns within large datasets, machine learning focuses more on algorithmic design, enabling computers to automatically "learn" patterns from data and use these patterns to make predictions about unknown data.

[0075] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are widespread in all areas of artificial intelligence. Machine learning and deep learning generally include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. The image and text classification model in the embodiments of this application is trained using machine learning or deep learning techniques. The training method based on the image and text classification model in the embodiments of this application can improve the generalization ability and classification accuracy of the image and text classification model.

[0076] The method for training a graph and text classification model proposed in the embodiment of the present application can be divided into two parts, including a training part and an application part. Among them, the training part involves the technical field of machine learning. In the training part, the graph and text classification model is trained by machine learning. Specifically, the training sample set given in the embodiment of the present application is used to iteratively train the graph and text classification model. After the training sample passes through each round of the graph and text classification model, the output result of the graph and text classification task is obtained. Combined with the output result, the model parameters are continuously adjusted to output the trained graph and text classification model. The application part is used to use the graph and text classification model trained in the training part to perform various real-world graph and text classification tasks.

[0077] The following is a brief introduction to the design concept of the embodiment of this application:

[0078] Pre-trained models based on contrastive text-image pairs, such as CLIP and ALIGN, are gradually becoming mainstream image-text pre-training models, attracting widespread attention and adoption, thanks to their ability to align large amounts of images and text into a common space through contrastive learning. These models offer excellent domain transfer and zero-shot learning capabilities. However, in practical applications, these pre-trained models still require fine-tuning on training datasets corresponding to downstream tasks, such as image classification and image-text retrieval, to ensure they are more suitable for the tasks.

[0079] Current mainstream fine-tuning methods, such as robust fine-tuning, typically use category information to supervise the model. As a result, during the fine-tuning process, pre-trained image and text models can only learn features directly related to the category label. This over-reliance on features of specific categories and ignoring other useful information can cause the fine-tuned model to overfit to the specific category labels in the training dataset, resulting in poor performance on unlearned data and a significant decrease in the model's zero-shot learning ability. However, in real-world applications, both the data distribution and the classification task may vary. Models with weak zero-shot learning capabilities cannot effectively handle different data and tasks, resulting in reduced generalization and classification accuracy.

[0080] In light of this, embodiments of the present application provide a method, apparatus, electronic device, and medium for training an image classification model. The model is iteratively trained using a training sample set and an auxiliary image and text feature set. During each round of training, training images are selected from the training sample set for feature extraction. A basic loss function is constructed using the extracted training image features and the type labels of the training images. This helps the image classification model better understand the relationship between image features and type labels, thereby improving image classification accuracy.

[0081] Furthermore, the embodiment of the present application will also obtain auxiliary image features that match the training image features from the auxiliary image and text feature set. Since the auxiliary text features associated with the auxiliary image features have richer and more detailed image description information, the embodiment of the present application introduces an auxiliary loss function by performing supervised alignment processing on the auxiliary image and text features, which can enrich the information source of the model and avoid the fine-tuned model from overfitting to the specific type of label of the training image. At the same time, the embodiment of the present application combines the basic loss function and the auxiliary loss function to adjust the parameters of the image classification model, which can make the fine-tuned model perform better on unlearned data, improve the model's zero-sample learning ability and generalization ability, and thus improve the classification accuracy of the image classification model.

[0082] Furthermore, in order to improve the domain migration capability of the image classification model, the auxiliary graphic feature set of the embodiment of the present application may also include auxiliary image features of multiple image domains, which are different from the image domain where the training image is located. By performing supervised alignment processing on the auxiliary graphic features of multiple image domains, an auxiliary loss function is obtained and used to fine-tune the image classification model, the domain migration capability of the image classification model can be improved, the generalization performance of the model can be ensured, and it can be applied to a wide range of image classification tasks.

[0083] After introducing the design concepts of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limiting. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0084] The solution provided by the embodiment of the present application can be applied to any scenario involving training of image and text pre-training models, for example, training of image and text pre-training models for image classification scenarios, image retrieval scenarios, environmental detection scenarios in assisted driving, face recognition scenarios, medical image recognition scenarios, security video detection scenarios, etc. Figure 1 , which is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes a client 101 and a server 102.

[0085] The client 101 generally refers to a device that can provide a training sample set, an auxiliary graphic feature set, or a trained target image classification model to the server 102, such as a terminal device, a third-party application that the terminal device can access, or a web page that the terminal device can access. Among them, the terminal device includes but is not limited to mobile phones, tablet computers (PADs), laptops, desktop computers, smart TVs, smart car devices, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, and smart wearable devices. A client related to image classification can be installed on the terminal device, and the client can be software, a web page, a small program, etc., which is not specifically limited in the embodiments of the present application.

[0086] The server 102 generally refers to a device that can train or use the target image classification model, such as a terminal device or a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Both the client 101 and the server 102 can use cloud computing to reduce the occupancy of local computing resources; cloud storage can also be used to reduce the occupancy of local storage resources. The server 102 may include one or more processors 1021, a memory 1022, and an I / O interface 1023 for interacting with the terminal. In addition, the server 102 can also be configured with a database 1024, which can be used to store training sample sets, auxiliary graphic feature sets, model parameters, etc. In which, the memory 1022 of the server 102 can also store the training method of the image classification model provided in the embodiment of the present application. When these program instructions are executed by the processor 1021, they can be used to implement the steps of the training method of the image classification model provided in the embodiment of the present application to obtain a trained target image classification model.

[0087] The client 101 and the server 102 can be directly or indirectly connected to each other through one or more networks 103. The network 103 can be a wired network, for example, communicating via a network cable or a serial cable; or a wireless network, for example, a mobile cellular network or a Wireless Fidelity (WIFI) network. Of course, other possible networks are also possible, and the embodiments of the present invention are not limited thereto.

[0088] In one possible implementation, the client 101 and the server 102 can be the same device, that is, the method can be performed jointly by the client 101 and the server 102, or can be performed independently by the client 101 or the server 102, and this application does not impose any specific restrictions on this. In the embodiments of this application, the client 101 and the server 102 are different devices as an example for description.

[0089] It should be noted that, in the embodiment of the present application, the number of the client 101 can be one or more, and similarly, the number of the server 102 can be one or more, that is, for Figure 1 The examples shown are just for illustration. In fact, the number and communication mode of the client and server are not limited and are not specifically limited in the embodiments of this application.

[0090] In one possible implementation, the image classification model of the embodiment of the present application can be applied to an image retrieval scenario, that is, an image retrieval application can be installed on the client 101. For each image to be retrieved input by the user into the image retrieval application, the server 102 can use the target image classification model trained by the training method of the image classification model of the embodiment of the present application to output the category of the object contained in the image to be retrieved, thereby retrieving similar images from the image library based on the category of the object.

[0091] In another possible implementation, the image classification model of the embodiment of the present application can be applied to the environmental detection scenario in assisted driving, that is, an environmental detection application can be installed on the client 101, and the target image classification model trained by the training method of the image classification model of the embodiment of the present application is used to classify the detected environmental images and output the categories of objects contained in the environmental images, thereby making assisted driving decisions based on the categories of the objects.

[0092] Of course, the method provided in the embodiment of the present application is not limited to the above application scenarios, and can also be used in other possible application scenarios, which are not limited by the embodiment of the present application. Figure 1 The functions that can be implemented by each device in the application scenario shown will be described in subsequent method embodiments and will not be described in detail here.

[0093] The following will combine the application scenarios described above and the reference drawings to explain the overall implementation method of the embodiment of the present application from the model training process and the model application process respectively.

[0094] First, in an embodiment of the present application, the training process is a process of performing multiple iterative training on the image classification graph model to be trained using a training sample set and an auxiliary graphic feature set. It can mainly include a data preparation stage and an iterative training stage. Each stage will be introduced below.

[0095] 1. Data Preparation Phase

[0096] Data collection is of paramount importance in machine learning, and can be said to be the most important link. The receipt preparation stage of the embodiment of the present application mainly includes: the preparation process of the training sample set and the preparation process of the auxiliary graphic feature set.

[0097] 1) Preparation process of training sample set:

[0098] In the embodiment of the present application, each training sample set includes: one or more training images, and a type label for identifying the type of each training image. Specifically, the sample image can be a real-life image, or an image produced by a device, etc., which is not limited in the embodiment of the present application. Taking the training images in the training sample set as animal images as an example, each type label indicating the type of animal can be associated with multiple animal images, and the multiple animal images can include frontal face images, side face images, top-down animal images, or top-down animal images of the corresponding type of animal, etc., which are not specifically limited. The type of the sample image is set according to the application scenario.

[0099] In one possible implementation, relatively mature open-source datasets can be used as training sample sets, including but not limited to: the MNIST dataset, which is used for handwritten digit recognition and contains 28x28 pixel grayscale images in 10 categories, from 0 to 9; the ImageNet dataset, which contains over 15 million labeled high-resolution images in approximately 22,000 categories and is commonly used to train large-scale image classification models; and the CIFAR-10 / 100 dataset, which contains 10 categories, with 6,000 color images per category. Alternatively, training sample sets can be obtained by collecting training images for specific application scenarios or classification tasks and annotating the training images with their corresponding category labels.

[0100] Specifically, for example, in the animal recognition scenario, a large number of images of different animal species are collected in advance. These images can come from various sources, such as animal websites, books, and social media. Objects in the training images are annotated, and type labels are added to indicate the animal type in the image, such as dog, cat, or bird. This allows the correspondence between the training images and the type labels to be determined. In one possible implementation, object annotations can be drawn manually or generated using an automatic annotation algorithm.

[0101] 2) Preparation process of auxiliary image and text feature set:

[0102] The embodiment of the present application collects multiple auxiliary image-text pairs in advance and extracts their features to obtain multiple pairs of auxiliary image features and auxiliary text features, i.e., the auxiliary image-text feature set required for model training. In the auxiliary image-text pairs, the auxiliary text associated with each auxiliary image is no longer just a simple type label that identifies the image category, but contains richer and more detailed image description information. Therefore, the embodiment of the present application can enrich the information source of the model by performing supervised alignment processing on the auxiliary image features and auxiliary text features to obtain an auxiliary loss function, thereby preventing the fine-tuned model from overfitting to the specific type labels of the training images.

[0103] For details, please refer to Figure 2 As shown, compared to the type label of the training image which simply indicates "Samoyed", the auxiliary text associated with the auxiliary image no longer contains only simple animal species information, but indicates the image domain, breed, characteristics, color and other information about the animal. Figure 2 As shown, for the Samoyed auxiliary image 1, its associated auxiliary text 1 can be: "A white Samoyed with its mouth open on a leash". In this way, by introducing auxiliary text with richer semantic information, the information source of the image classification model can be enriched and the generalization ability of the model can be improved.

[0104] In one possible implementation, a multi-threaded crawler strategy can be used to obtain a large-scale open source image-text pair dataset as an auxiliary image-text pair. Multiple threads executed concurrently can increase the acquisition speed of the image-text pair dataset, thereby improving the training efficiency of the image classification model. Specifically, the image-text pair dataset can be selected from the CC3M dataset containing 3 million images and corresponding text descriptions, or CC12M, LAION5B, and any other image-text pair dataset that has more comprehensive and rich semantic information than the training sample set and is closer to the data used for pre-training of the image-text pre-training model. The embodiment of the present application does not specifically limit this. Then, each text data in the acquired image-text pair dataset is input into the text encoder, and the natural language texts such as sentences, paragraphs, and words are converted into vector representations through the text encoder. Similarly, each image data in the image-text pair dataset is input into the image encoder, and the image is converted into a vector representation through the image encoder. Specifically, the text encoder can use but is not limited to text Transformer models such as BERT and GPT, and the image encoder can use but is not limited to models such as convolutional neural network (CNN) or visual Transformer (ViT). These models can extract features from images and generate vector representations through convolution or self-attention mechanisms.

[0105] In one possible implementation, in order to improve the domain migration capability of the image classification model, the embodiment of the present application may use auxiliary images whose image domain is different from the image domain of the training image to obtain auxiliary image features, thereby performing supervised alignment processing on the auxiliary graphic features of multiple image domains, obtaining an auxiliary loss function and using it to fine-tune the image classification model to improve the domain migration capability of the image classification model, ensure the generalization performance of the model, and enable it to be applied to a wide range of image classification tasks.

[0106] For details, please refer to Figure 2 As shown, taking the animal classification scenario as an example, in this scenario, the traditional training sample set mainly contains a variety of animal images in the natural image domain. Although fine-tuning the image classification model with this training sample set can enable the model to obtain good classification performance for animal images in the natural image domain, the model's classification ability for animal images in other domains such as sketches and stick figures will decline. In view of this, the embodiment of the present application introduces an image-text pair dataset containing auxiliary images from various image domains such as sketches and stick figures, as shown above. Figure 2 As shown in the example, for auxiliary image 2, auxiliary text 2 could be "A sketch of a Samoyed with its mouth closed." For auxiliary image N, auxiliary text N could be "A full-body sketch of a Samoyed with its head held high." In this way, by introducing auxiliary images from more image domains, we can further enrich the information sources of the image classification model, improve its domain transfer capabilities, and ensure the model's generalization performance, enabling its application to image classification tasks in a variety of image domains, including natural images, sketches, and stick figures.

[0107] 2. Iterative training phase

[0108] Please refer to Figure 3 As shown, in an embodiment of the present application, during the model training process, the image classification model to be trained is subjected to multiple rounds of iterative training using a training sample set and an auxiliary graphic feature set. Each round of iterative training will adjust the model parameters based on the difference between the predicted image type and the actual type label, and ultimately obtain a trained target image classification model. The training sample set is trained once in the image classification model to be trained, which is called a round of iteration. In each round of iteration, due to reasons such as limited video memory resources of the training machine, all training samples cannot be input into the model for training at one time. Therefore, all training samples need to be trained in batches, and each batch of training samples is generated by methods such as random partitioning, and each batch of training samples is input into the model separately for training such as forward calculation, backward calculation, and model parameter update.

[0109] Since the operations performed in each model iteration training are the same, the following uses one iteration training as an example to illustrate the training process of the image classification model. Figure 4As shown, it is a flow chart of a training method of an image classification model provided in an embodiment of the present application. The execution subject of this method can be Figure 1 The server 102 shown in FIG. 1 includes the following steps:

[0110] Step S401: extracting features from the training images in the selected training sample set to obtain training image features.

[0111] In an embodiment of the present application, during each round of training of the image classification model, one or a group of training images can be selected from the training sample set and input into the image classification model of this round, and the feature representation of the training image, i.e., the training image feature, can be obtained through the image encoder of the model.

[0112] In one possible implementation, a graphic and text pre-training model such as CLIP or ALIGN can be used as the image classification model to be trained to perform feature extraction on the training image, or a pre-trained convolutional neural network (such as ResNet, VGG, etc.) or Transformer model can be used to forward propagate the training image, and the feature vector obtained by the last layer of the convolutional network or the global average pooling layer is used as the feature representation of the image.

[0113] Step S402: Construct a basic loss function based on the training image features and corresponding type labels.

[0114] Please refer to Figure 5 As shown, in the embodiment of the present application, after the corresponding training image features and training text features are obtained by the encoder for the training image and text label, a cross entropy loss function or a contrastive learning loss function can be used to construct a basic loss function L according to the difference between the predicted image type calculated by the training image features and the corresponding training text features. cl Of course, other possible loss functions may also be used, such as the 0-1 loss function (zero-one loss), etc., and this embodiment of the present application does not limit this.

[0115] Step S403: Based on the training image features, a matching target auxiliary image feature is selected from the auxiliary image and text feature set.

[0116] In the embodiment of the present application, each training image feature is compared with each image feature in the auxiliary image feature set to find a matching target auxiliary image feature. For example, this can be achieved by calculating the cosine similarity between the features. Based on the similarity calculation results between the features, the target auxiliary image feature whose similarity with the training image feature exceeds a set threshold is obtained from the auxiliary image feature set.

[0117] Specifically, each auxiliary text feature in the auxiliary image-text feature set can be used as a memory, and the training image features to be matched can be used as the query for retrieval. The CLIP model's own similarity calculation capability can be used to calculate the similarity between the query and the memory. Based on the similarity calculation results, the auxiliary text features whose similarity with the training image features is greater than a preset threshold are selected, and the auxiliary image features associated with the auxiliary text features are used as target auxiliary image features. In this way, compared with the unimodal matching method between images, the multimodal matching method can utilize the rich semantic information between images and texts to more comprehensively express the image content and capture the correlation between images and texts. The target auxiliary image features obtained in this way can help the model learn feature representations with more generalization capabilities and provide richer training information for model training. In addition, by utilizing the original similarity calculation capability of the CLIP model, the training images and the auxiliary image-text feature set can be matched more accurately, thereby constructing an auxiliary loss function through the target auxiliary image features, providing additional supervision signals, helping the model learn better image classification representations and improving classification performance.

[0118] In one possible implementation, before matching the training image features with each auxiliary text feature, each auxiliary text feature and the training image feature can be normalized. A matrix multiplication operation is then performed on the normalized training image feature and each auxiliary text feature to obtain the similarity between the training image feature and each auxiliary image feature. Based on the obtained similarities, target auxiliary image features with a similarity greater than a preset threshold are selected from the auxiliary image and text feature set. Normalizing the features in this manner ensures that the feature vectors have the same scale, facilitating better comparison of similarities between features.

[0119] Specifically, L2 normalization can be used to train image features (F v ) and each auxiliary text feature (F t ) is normalized and each training image feature F v and F t Divide by its L2 norm so that the length (modulus) of each feature becomes 1.

[0120] The specific normalization formula is as follows:

[0121]

[0122]

[0123] Among them, F v ′ represents the normalized training image features, F v Represents the training image features;

[0124] F t ′ represents the normalized auxiliary text features, F t Represents auxiliary text features.

[0125] Next, the normalized training image features (F v ′ ) and each auxiliary text feature (F t ′ ) performs matrix multiplication to obtain the similarity between the training image feature and each auxiliary image feature, thereby selecting the auxiliary text feature with the greatest similarity to the training image feature as the retrieval target. The specific formula is as follows:

[0126] argmax c∈C (F v ′ ,F t ′ )

[0127] Among them, (F v ′ ,F t ′ ) represents the similarity scalar value obtained by performing matrix multiplication operation on the training image features and each auxiliary text feature.

[0128] argmax c∈C It means selecting the maximum value from the obtained similarity scalar values, that is, selecting the auxiliary text feature that is most similar to the training image feature, and taking the auxiliary image feature associated with the auxiliary text feature as the target auxiliary image feature.

[0129] Step S404: Perform supervised alignment processing on the target auxiliary image features and the associated target auxiliary text features to construct an auxiliary loss function.

[0130] Please refer to the above Figure 5 As shown in the embodiment of the present application, the target auxiliary image features and the target auxiliary text features are obtained by extracting the target auxiliary image and the target auxiliary text through the image encoder and the text encoder respectively. After obtaining the target auxiliary image features and the target auxiliary text features, the cross entropy or contrastive learning method can be used to perform supervised alignment processing on the two to construct the auxiliary loss function L ret Of course, other possible loss functions can also be used, such as the 0-1 loss function (zero-one loss), etc., which are not limited in this embodiment of the present application. Auxiliary loss function L retIt plays the role of a regularization term in the subsequent model fine-tuning process. Since it uses auxiliary image and text features containing richer semantic information to calculate the loss function, it enriches the training information source of the model, avoids the fine-tuned model from overfitting to the specific type of labels of the training image, and enables the fine-tuned model to perform better on unlearned data, thereby improving the model's zero-sample learning ability and generalization ability, thereby improving the classification accuracy of the image classification model.

[0131] Specifically, the contrastive learning loss function is the same as the loss function used in CLIP model pre-training. Therefore, the auxiliary loss function L is obtained by aligning the auxiliary image and text features with the contrastive learning loss function. Ret , which can simulate the pre-training process of the CLIP model, making the fine-tuned model closer to the CLIP model.

[0132] In one possible implementation, reference Figure 6 As shown, the embodiment of the present application adopts a contrastive learning method to construct an auxiliary loss function L Ret The specific process can be shown as follows:

[0133] S601: Multimodal Embedding.

[0134] The target auxiliary image features and target auxiliary text features are linearly projected to obtain embedded target auxiliary image features and target auxiliary text features. The purpose is to project the original features into a lower-dimensional space for subsequent processing. In addition, the projected features are L2 normalized. The L2 normalization process is the same as the above-mentioned step S403 and will not be further described here.

[0135] S602: Calculate similarity.

[0136] The cosine similarity between the target auxiliary image features and the target auxiliary text features can be calculated using matrix multiplication.

[0137] S603: Construct a loss function.

[0138] Create labels labels=np.arange(n) with the same number of samples for the calculation of the comparative learning loss function.

[0139] Calculate the contrastive learning loss loss_i between the target auxiliary image features and the target auxiliary text features. The specific calculation formula is as follows:

[0140] loss_i=cross_entropy_loss(logits,labels,axis=0)

[0141] loss_t=cross_entropy_loss(logits,labels,axis=1)

[0142] The cross-entropy loss function is used here. Axis = 0 indicates that the loss is calculated in the image direction, while axis = 1 indicates that the loss is calculated in the text direction. Logits represents the similarity score matrix, with dimension [n, n], which represents the similarity score between each target auxiliary image feature and the target auxiliary text feature.

[0143] The average loss of the contrast learning loss between image features and text features is taken as loss = (loss_i + loss_t) / 2, which is used as the final auxiliary loss function L ret .

[0144] In this way, by using the above-mentioned contrastive learning method, the distance between the target auxiliary image features and the corresponding target auxiliary text features can be made closer, and the distance between irrelevant features can be made farther, thereby maintaining the image semantic alignment relationship in the feature space and preventing the fine-tuned model from overfitting to the training sample set.

[0145] Step 405: Combine the basic loss function and the auxiliary loss function to adjust the model parameters of the image classification model.

[0146] In the embodiment of the present application, the basic loss function L is obtained by using the training sample set and the auxiliary image and text feature set. CL And the auxiliary loss function L Ret After that, the auxiliary loss function L Ret As a regularization term and the basic loss function L CL Combined and used together to fine-tune the model.

[0147] In one possible implementation, the basic loss function and the auxiliary loss function can be weighted by their respective weight values ​​to obtain a target loss function, which is used to adjust the model parameters of the image classification model.

[0148] Specifically, we can use the regularization form of weighted combination to adjust the basic loss function L CL And the auxiliary loss function L Ret Combine them to obtain the final target loss function L for model fine-tuning total , thereby integrating the category information and the supervision information of the auxiliary graphics and text on the data, improving the model's zero-shot learning ability and generalization ability. The specific formula is as follows:

[0149] L total =L CL +λ·L Ret

[0150] Among them, λ is a hyperparameter used to control the loss function L Ret The weight in the fine-tuning process. By adjusting λ, the influence of the supervision information on the fine-tuning process can be controlled.

[0151] In one possible implementation, learnable embedded image features and embedded text features can be added to the initialization model parameters of the image classification model. The embedded image features and embedded text features have the same feature dimensions as the output of the image classification model and are respectively aggregated with auxiliary image features and auxiliary text features. The aggregated features are then used to construct an auxiliary loss function. The embedded image features and embedded text features are updated during the iterative training process as the auxiliary loss function is optimized. Therefore, the embedded image features and embedded text features are feature representations with rich semantic information obtained through training and learning. They can better capture the essential characteristics of the data and have better generalization capabilities.

[0152] For details, please refer to Figure 7 As shown in the figure, during each round of iterative training, the target auxiliary image features are aggregated based on the first embedded image features corresponding to the current model, resulting in the first aggregated image features. Simultaneously, the target auxiliary text features are aggregated based on the first embedded text features corresponding to the current model, resulting in the first aggregated text features. The first aggregated image features and the first aggregated text features are then supervisedly aligned to construct an auxiliary loss function. By aggregating embedded features, the model can learn a more comprehensive and integrated feature representation, achieving good classification performance even on unseen data, thereby improving the model's zero-shot learning and generalization capabilities.

[0153] In one possible implementation, in the process of adjusting the parameters of the image classification model through the target loss function, it can be used to determine whether the target loss value of the target loss function converges to a preset target value to determine whether the image classification model meets the convergence conditions. If the target loss value is less than or equal to the preset target value, it indicates that the classification accuracy of the image classification model meets the requirements, and thus it can be determined that the image classification model has converged. On the contrary, the image splitting model has not converged. Alternatively, when the change amplitude of the target loss value obtained after N consecutive training times is less than or equal to the preset target value, it is determined that the model loss value has converged to the preset target value, indicating that the image classification model meets the convergence conditions; otherwise, it indicates that the image classification model does not meet the convergence conditions. Alternatively, when the number of model training iterations meets the preset value, the model can also be considered to have converged.

[0154] Specifically, if the target loss value of the target loss function exceeds the preset loss threshold and the current number of iterations is less than the preset number of iterations, that is, if the target loss value has not converged, an optimization algorithm can be used to minimize the target loss value and adjust the model parameters. During each iterative training process, the model will update the model parameters to reduce the target loss value.

[0155] In one possible implementation, the optimization algorithm includes but is not limited to the gradient descent method, the quasi-Newton method, and the like. Take the gradient descent method as an example. To initialize the model parameters, random initialization or other methods may be used. In the first round of iterations, the model calculates the corresponding target loss value, such as the mean square error or other suitable loss function value, for the initialized model parameters. And calculates the gradient of the loss function with respect to the model parameters. The gradient is the rate of change of the target loss value with respect to the parameter, indicating in which direction the model parameters should be updated to reduce the loss. According to the calculated gradient, the model updates the model parameters, and the amplitude and direction of the update are affected by the gradient. The above steps are repeated iteratively multiple times until the model reaches the convergence condition. In this way, the model gradually learns the model parameters that minimize the loss. When the target loss value is less than the preset loss threshold, or the current number of iterations is not less than the preset number of iterations, it proves that the model has converged, and the trained target image classification model can be obtained.

[0156] The following will explain the overall implementation of the embodiment of this application from the model application process. Figure 8 A flow chart of an image classification method provided in an embodiment of the present application, after obtaining a trained target image classification model, Figure 1 The client 101 can execute the image classification method through the target image classification model. The process of the method is described as follows

[0157] Step 801: Obtain a target image to be classified and multiple type labels.

[0158] In the embodiments of the present application, the trained target image classification model can be used to perform various tasks requiring image classification, including but not limited to image retrieval scenarios, face recognition scenarios, medical image recognition scenarios, assisted driving scenarios, security scenarios, etc. When the model performs an image classification task, it needs to obtain the target image to be classified as input to the model, and pre-set multiple possible type labels corresponding to the image, where each type label indicates a different image type.

[0159] Specifically, type labels can be set according to different classification tasks and classification scenarios. For example, in the animal classification scenario, type labels can include the class names of animal categories such as lion, elephant, giraffe, and zebra; in the vehicle classification scenario, type labels can include the class names of vehicle categories such as cars, trucks, bicycles, and motorcycles.

[0160] Step 802: Feature extraction is performed on the target image and the multiple type tags respectively to obtain target image features corresponding to the target image and predicted text features corresponding to the multiple type tags.

[0161] In an embodiment of the present application, the target image and type label are input into a trained target image classification model. The module can perform feature extraction on the target image and type label respectively through an image encoder and a text encoder to obtain target image features and multiple predicted text features.

[0162] Specifically, a target image to be classified is input. This target image is processed by the target image classification model's image encoder to obtain target image features. For a given type label to be classified, the text data corresponding to each type label is input into the text encoder to obtain predicted text features. The model then processes the class name represented by each type label, specifically combining the class name into the form "a photo of a + class name." For example, for an N-classification problem, the class name data corresponding to N type labels is input into the text encoder to obtain N predicted text features.

[0163] Step 803: Obtain a target first similarity that meets a preset classification condition and a target predicted text feature corresponding to the target first similarity from the first similarities between each predicted text feature and the target image feature.

[0164] In an embodiment of the present application, the first similarity between the target image feature and each predicted text feature can be calculated by cosine similarity or the like, and according to preset classification conditions, the target first similarity that meets the conditions and its corresponding target predicted text feature are screened out.

[0165] Specifically, the preset classification condition can be achieved by setting an appropriate similarity threshold, and when the first similarity is greater than the similarity threshold, it is determined to be the target first similarity that meets the condition, or the maximum value of the first similarities is directly taken as the target first similarity. The embodiment of the present application does not impose any specific restrictions on this.

[0166] In one possible embodiment, to further improve the model's zero-shot learning and generalization capabilities, learnable embedded image features and embedded text features are introduced into the image classification model to be trained during the aforementioned model training process. Therefore, the model parameters of the target image classification model may also include trained second embedded image features and second embedded text features. During model application, the target image features are feature aggregated based on the second embedded image features to obtain second aggregated image features. Simultaneously, each predicted text feature is feature aggregated based on the second embedded text features to obtain second aggregated text features corresponding to each predicted text feature. Based on this, the second similarity between each second aggregated text feature and the second aggregated image feature is calculated, and the second aggregated text feature corresponding to the maximum second similarity is selected as the target predicted text feature. Because the second embedded image features and the second embedded text features are feature representations rich in semantic information obtained through training, they can better capture the essential characteristics of the data and have better generalization capabilities. By aggregating the embedded features, the model can learn a more comprehensive and integrated feature representation, thereby achieving good classification performance even when faced with unseen data, thereby improving the model's zero-shot learning and generalization capabilities.

[0167] Step 804: The image type associated with the target predicted text feature is used as the target image type of the target image.

[0168] In the embodiments of the present application, after determining the target predicted text features corresponding to the target image, the image type associated with the target predicted text features is the target image type calculated by the image classification model. Depending on the classification tasks and application scenarios employed by the image classification model, different subsequent processing can be performed on this calculated target image type.

[0169] In a possible implementation, the application process of the target image classification model is introduced by taking the image retrieval task as an example. Figure 9 As shown, an image containing a certain animal is input into the image retrieval application of a terminal device. The terminal device sends the image to be retrieved to the backend server of the image retrieval application. The backend server can use the image classification method provided in the embodiment of the present application to input the image to be retrieved into the target image classification model and output the classification result as follows: the probability of cat is 0.7, the probability of dog is 0.2, and the probability of bird is 0.1, determining that the animal type in the image to be retrieved is a cat. The backend server can then retrieve multiple images containing cats from the image library and send the retrieved multiple images to the terminal device, so that the terminal device displays the retrieved multiple images in the image retrieval application.

[0170] In another possible implementation, taking the environmental monitoring task of assisted driving as an example, the application process of the target image classification model is introduced. Figure 10 As shown, the image classification model of the embodiment of the present application can be applied to the environment detection scenario in assisted driving, that is, an environment detection application can be installed on the vehicle terminal, and the vehicle terminal captures road images in real time through the camera mounted on the vehicle as images to be classified. In this scenario, the type label can include common object categories on the road, such as vehicles, pedestrians, and traffic signs. The vehicle terminal can use the target image classification model trained by the training method of the image classification model of the embodiment of the present application to classify the collected road images, identify the object categories contained in the road images, and use them as the output of the environment detection system for assisted driving decision-making. For example, while the vehicle is driving, the camera captures a road image. After classification by the target image classification model, it is recognized that there is a high probability that pedestrians are contained in the image. This information can be used by the assisted driving system to judge the safety status of the surrounding environment and adopt corresponding driving strategies, such as slowing down or issuing an alarm.

[0171] See Figure 11 Based on the same inventive concept, the present embodiment further provides a training device 110 for an image classification model, the device comprising:

[0172] The model training unit 1101 is configured to perform multiple rounds of iterative training on the image classification model to be trained based on multiple training sample sets in combination with the auxiliary image-text feature set to obtain a trained target image classification model; each training sample set includes: a training image and a type label for identifying the type of the training image; the auxiliary image-text feature set includes: multiple pairs of auxiliary image features and auxiliary text features;

[0173] The model training unit includes a basic function subunit 1102, an auxiliary function subunit 1103 and a parameter adjustment subunit 1104, wherein:

[0174] The basic function subunit 1102 is used to extract features of the training images in the selected training sample set to obtain corresponding training image features; and construct a basic loss function based on the training image features and corresponding type labels;

[0175] An auxiliary function subunit 1103 is configured to select, based on the training image features, matching target auxiliary image features from the auxiliary image-text feature set, where the similarity between the target auxiliary image features and the training image features is greater than a set threshold; perform supervised alignment processing on the target auxiliary image features and the associated target auxiliary text features, and construct an auxiliary loss function;

[0176] The parameter adjustment subunit 1104 is used to adjust the model parameters of the image classification model by combining the basic loss function and the auxiliary loss function.

[0177] Optionally, the model parameters include preset embedded image features and embedded text features, and the auxiliary function subunit 1103 is further used to:

[0178] Based on the first embedded image feature corresponding to the current model, the target auxiliary image feature is subjected to feature aggregation processing to obtain a first aggregated image feature; the first embedded image feature and the target auxiliary image feature have the same feature dimension;

[0179] Based on the first embedded text feature corresponding to the current model, the target auxiliary text feature is aggregated to obtain a first aggregated text feature; the first embedded text feature and the target auxiliary text feature have the same feature dimension;

[0180] The first aggregated image features and the first aggregated text features are supervisedly aligned to construct an auxiliary loss function.

[0181] Optionally, the parameter adjustment subunit 1104 is specifically configured to:

[0182] Based on the respective weight values ​​of the basic loss function and the auxiliary loss function, the basic loss function and the auxiliary loss function are weighted to obtain the target loss function;

[0183] When the loss value of the target loss function exceeds a preset loss threshold and the current number of iterations is less than the preset number of iterations, the model parameters of the image classification model are adjusted based on the loss value;

[0184] When the loss value is less than a preset loss threshold, or the current number of iterations is not less than a preset number of iterations, the trained target image classification model is obtained.

[0185] Optionally, the auxiliary function subunit 1103 is specifically configured to:

[0186] Normalize each auxiliary text feature in the auxiliary image-text feature set with the training image features;

[0187] Perform matrix multiplication on the normalized training image features and each auxiliary text feature to obtain the similarity between the training image features and each auxiliary image feature;

[0188] Based on the obtained similarities, target auxiliary image features are selected from the auxiliary image feature set.

[0189] Optionally, the apparatus further includes a model application unit 1105, configured to:

[0190] Obtain a target image to be classified and multiple type labels, each type label indicating an image type;

[0191] Perform feature extraction on the target image and multiple type labels respectively to obtain target image features corresponding to the target image and predicted text features corresponding to each of the multiple type labels;

[0192] From the first similarities between each predicted text feature and the target image feature, a target first similarity that meets the preset classification condition and a target predicted text feature corresponding to the target first similarity are obtained; and the image type associated with the target predicted text feature is used as the target image type of the target image.

[0193] Optionally, the model parameters of the target image classification model include the trained second embedded image features and second embedded text features, and the model application unit 1105 is specifically configured to:

[0194] Based on the second embedded image feature, performing feature aggregation processing on the target image feature to obtain a second aggregated image feature;

[0195] Based on the second embedded text feature, feature aggregation processing is performed on each predicted text feature and the second embedded text feature to obtain a second aggregated text feature corresponding to each predicted text feature;

[0196] Based on the second similarities between the respective second aggregated text features and the second aggregated image features, the second aggregated text feature corresponding to the maximum second similarity is used as the target predicted text feature.

[0197] The device can be used to execute the training method of the image classification model provided in each embodiment of the present application. Therefore, for the functions that can be achieved by each functional module of the device, please refer to the description of the aforementioned embodiments and no further details will be given.

[0198] See Figure 12 Based on the same technical concept, the embodiment of the present application further provides a computer device 120, which may include a memory 1201, a communication module 1203 and one or more processors 1202. The computer device 120 may be Figure 1 As shown in the client 121 or the server 122 , when the computer device 120 is the server 122 , the memory 1201 and the processor 1202 may correspond to the memory 1222 and the processor 1221 of the server 122 , respectively.

[0199] Memory 1201 is used to store computer programs executed by processor 1202. Memory 1201 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and programs required for running instant messaging functions, while the data storage area may store various instant messaging messages and operating instruction sets.

[0200] Memory 1201 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1201 may be a combination of the aforementioned memories.

[0201] The processor 1202 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1202 is configured to implement the above-mentioned image classification model training method when calling the computer program stored in the memory 1201 .

[0202] The communication module 1203 is used to communicate with the client or other service ends.

[0203] The specific connection medium between the memory 1201, the communication module 1203 and the processor 1202 is not limited in the embodiment of the present application. Figure 12 In the embodiment, the memory 1201 and the processor 1202 are connected via a bus 1204. The bus 1204 is connected to the processor 1202 via a bus 1204. Figure 12 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus 1204 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 12 The diagram shows a single thick line, but this does not indicate that there is only one bus or one type of bus.

[0204] The memory 1201 stores a computer storage medium, which stores computer executable instructions. The computer executable instructions are used to implement the training method of the image classification model of the embodiment of the present application. The processor 1202 is used to execute the training method of the image classification model of the above-mentioned embodiments.

[0205] Based on the same inventive concept, an embodiment of the present application also provides a storage medium, which stores a computer program. When the computer program runs on a computer, it enables the computer to execute the steps of the training method of the image classification model according to various exemplary embodiments of the present application described above in this specification.

[0206] In some possible implementations, various aspects of the risk assessment method provided in the present application may also be implemented in the form of a computer program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to enable the computer device to execute the steps of the training method of the image classification model according to various exemplary embodiments of the present application described above in this specification. For example, the computer device can execute the steps of each embodiment.

[0207] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0208] The program product of the embodiment of the present application may be a portable compact disc read-only memory (CD-ROM) and include a computer program, and can be run on a computer device. However, the program product of the present application is not limited thereto. In the present application, a readable storage medium may be any tangible medium containing or storing a program, and the computer program included therein may be used by or in conjunction with a command execution system, apparatus, or device.

[0209] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with a command execution system, apparatus, or device.

[0210] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0211] Computer programs for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages.

[0212] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0213] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0214] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0215] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0216] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A training method for an image classification model, characterized in that: The method comprises: Based on multiple training sample sets and in combination with auxiliary image-text feature sets, the image classification model to be trained is trained for multiple rounds of iterative training to obtain a trained target image classification model; each training sample set includes: a training image and a type label for identifying the type of the training image; the auxiliary image-text feature set includes: multiple pairs of auxiliary image features and auxiliary text features; wherein each round of iteration includes: Performing feature extraction on the training images in the selected training sample set to obtain corresponding training image features; constructing a basic loss function based on the training image features and corresponding type labels; Based on the training image features, selecting a matching target auxiliary image feature from the auxiliary image-text feature set, wherein the similarity between the target auxiliary image feature and the training image feature is greater than a set threshold; performing supervised alignment processing on the target auxiliary image feature and the associated target auxiliary text feature, and constructing an auxiliary loss function; In combination with the basic loss function and the auxiliary loss function, the model parameters of the image classification model are adjusted.

2. The method according to claim 1, wherein The model parameters include preset embedded image features and embedded text features, and the supervised alignment processing of the target auxiliary image features and the associated target auxiliary text features is performed to construct an auxiliary loss function, including: Based on the first embedded image feature corresponding to the current round model, perform feature aggregation processing on the target auxiliary image feature to obtain a first aggregated image feature; the first embedded image feature and the target auxiliary image feature have the same feature dimension; Based on the first embedded text feature corresponding to the current round model, the target auxiliary text feature is subjected to feature aggregation to obtain a first aggregated text feature; the first embedded text feature and the target auxiliary text feature have the same feature dimension; Perform supervised alignment processing on the first aggregated image features and the first aggregated text features to construct the auxiliary loss function.

3. The method according to claim 1, wherein The adjusting the model parameters of the image classification model by combining the basic loss function and the auxiliary loss function includes: Based on the respective weight values ​​of the basic loss function and the auxiliary loss function, weighted processing is performed on the basic loss function and the auxiliary loss function to obtain a target loss function; When a loss value of the target loss function exceeds a preset loss threshold and the current number of iterations is less than a preset number of iterations, adjusting the model parameters of the image classification model based on the loss value; When the loss value is less than the preset loss threshold, or the current number of iterations is not less than the preset number of iterations, the trained target image classification model is obtained.

4. The method according to claim 1, wherein The selecting, based on the training image features, matching target auxiliary image features from the auxiliary image and text feature set includes: Normalizing each auxiliary text feature in the auxiliary image-text feature set with the training image feature; Performing matrix multiplication operations on the normalized training image features and each auxiliary text feature to obtain similarities between the training image features and each auxiliary image feature; Based on the obtained similarities, the target auxiliary image feature is selected from the auxiliary image feature set.

5. The method according to any one of claims 1 to 4, characterized in that: After obtaining the trained target image classification model, the method further includes: Obtain a target image to be classified and multiple type labels, each type label indicating an image type; Performing feature extraction on the target image and the multiple type labels respectively to obtain target image features corresponding to the target image and predicted text features corresponding to each of the multiple type labels; From the first similarities between each predicted text feature and the target image feature, a target first similarity that meets a preset classification condition and a target predicted text feature corresponding to the target first similarity are obtained; and the image type associated with the target predicted text feature is used as the target image type of the target image.

6. The method according to claim 5, wherein The model parameters of the target image classification model include the trained second embedded image features and the second embedded text features. Then, obtaining a target first similarity that satisfies a preset classification condition and a target predicted text feature corresponding to the target first similarity from the first similarities between each predicted text feature and the target image feature includes: performing feature aggregation processing on the target image feature based on the second embedded image feature to obtain a second aggregated image feature; Based on the second embedded text feature, performing feature aggregation processing on each predicted text feature and the second embedded text feature to obtain second aggregated text features corresponding to each predicted text feature; Based on the second similarities between the respective second aggregated text features and the second aggregated image features, the second aggregated text feature corresponding to the maximum second similarity is used as the target predicted text feature.

7. A training device for an image classification model, characterized in that: include: A model training unit is used to perform multiple rounds of iterative training on the image classification model to be trained based on multiple training sample sets and in combination with auxiliary image and text feature sets to obtain a trained target image classification model; Each training sample set includes: a training image and a type label for identifying the type of the training image; the auxiliary image-text feature set includes: multiple pairs of auxiliary image features and auxiliary text features; The model training unit includes a basic function sub-unit, an auxiliary function sub-unit and a parameter adjustment sub-unit, wherein: The basic function subunit is used to extract features of the training images in the selected training sample set to obtain corresponding training image features; and construct a basic loss function based on the training image features and corresponding type labels; The auxiliary function subunit is configured to select, based on the training image features, a matching target auxiliary image feature from the auxiliary image-text feature set, wherein the similarity between the target auxiliary image feature and the training image feature is greater than a set threshold; perform supervised alignment processing on the target auxiliary image feature and the associated target auxiliary text feature to construct an auxiliary loss function; The parameter adjustment subunit is used to adjust the model parameters of the image classification model in combination with the basic loss function and the auxiliary loss function.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images