Image classification method and device, model training method and device, electronic equipment and medium

Abstract

The invention provides an image classification method and device, a model training method and device, electronic equipment and a medium, and relates to the field of data processing, in particular to the field of image processing and artificial intelligence. The method comprises the steps that an image classification model is obtained, the image classification model comprises a shared feature extraction network and at least two expert full-connection layers arranged in parallel, and the at least two expert full-connection layers comprise at least one judgment expert layer used for judging whether an image is suitable for being displayed as a full-size message cover picture or not, the problem expert layer is used for identifying whether the image has defect characteristics which are not suitable for being displayed as a full-size message cover picture or not; cross-task common features are obtained; performing task-specific feature mapping through the at least two expert full-connection layers to obtain a prediction vector; and determining whether the to-be-classified image is suitable for being displayed as a full-size message cover image based on the prediction vector.

Description

Image classification methods, model training methods, devices, electronic devices, media Technical Field

[0001] This disclosure relates to the field of data processing technology, and more particularly to image processing and artificial intelligence, specifically to an image classification method, a model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a training method for a multimodal image classification model, an image classification method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of this disclosure, a method for training a multimodal image classification model is provided, comprising: obtaining training samples, the training samples including sample images, prompt text, and classification labels, the classification labels including a first label for indicating whether an image is suitable as a full-size message cover image; obtaining an image classification model to be trained, the image classification model including a shared feature extraction network and at least two expert fully connected layers configured in parallel, wherein the at least two expert fully connected layers include at least one judgment expert layer for determining whether an image is suitable for display as a full-size message cover image, and at least one question expert layer for identifying whether an image has defective features that make it unsuitable for display as a full-size message cover image; obtaining cross-task common features based on the sample images and prompt text, through the shared feature extraction network; performing task-specific feature mapping on the cross-task common features through the at least two expert fully connected layers to obtain prediction vectors; and determining a loss value based on the prediction vectors and the classification labels, and updating model parameters based on the determined loss value to obtain a trained multimodal image classification model.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a training method or an image classification method for a multimodal image classification model according to one or more embodiments of this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform a training method or an image classification method for a multimodal image classification model according to one or more embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, implements a training method or an image classification method for a multimodal image classification model according to one or more embodiments of this disclosure.

[0009] According to one or more embodiments of this disclosure, multimodal information can be used to better understand the semantics and quality standards of image content, while different expert layers focus on specific discrimination dimensions, thereby balancing training interference between different tasks in a multi-task learning framework and improving overall classification performance.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0012] Figure 1 illustrates a schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure;

[0013] Figure 2 shows a flowchart of an image classification method according to an embodiment of the present disclosure; Figure 3 shows a training method 300 for a multimodal image classification model according to an embodiment of the present disclosure; Figure 4 shows a schematic diagram of the data flow for training and application of a multimodal image classification model according to an embodiment of the present disclosure; Figure 5 shows a schematic diagram of a model to which the image classification method according to an embodiment of the present disclosure can be applied; Figure 6 shows a structural block diagram of an image classification apparatus according to an embodiment of the present disclosure; Figure 7 shows a structural block diagram of a training apparatus for a multimodal image classification model according to an embodiment of the present disclosure; Figure 8 shows a structural block diagram of an exemplary electronic device that can be used to implement an embodiment of the present disclosure. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0016] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0017] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0018] Figure 1 illustrates a schematic diagram of an exemplary system 100 in which the various methods and apparatuses described herein may be implemented according to embodiments of the present disclosure. Referring to Figure 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0019] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of a training method or image classification method for a multimodal image classification model according to this disclosure.

[0020] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0021] In the configuration shown in Figure 1, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 is an example of a system for implementing the various methods described herein and is not intended to be limiting.

[0022] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to train multimodal image classification models or perform image classification, etc. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface. Although Figure 1 depicts only six client devices, those skilled in the art will understand that this disclosure can support any number of client devices.

[0023] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0024] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0025] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0026] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0027] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0028] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0029] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0030] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or general stores supported by a file system. The system 100 of FIG1 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0031] The image classification method 200 according to an exemplary embodiment of the present disclosure is described below with reference to FIG2.

[0032] In step S201, the image to be classified and the prompt text are obtained.

[0033] At step S202, a multimodal image classification model is obtained. The multimodal image classification model includes a shared feature extraction network and at least two expert fully connected layers set in parallel. The at least two expert fully connected layers include at least one judgment expert layer for determining whether an image is suitable for display as a full-size message cover image, and at least one problem expert layer for identifying whether an image has defective features that are unsuitable for display as a full-size message cover image.

[0034] In step S203, based on the image to be classified and the prompt text, cross-task common features are obtained through the shared feature extraction network.

[0035] In step S204, task-specific feature mapping is performed on the common features across tasks through at least two expert fully connected layers to obtain prediction vectors.

[0036] In step S205, it is determined whether the image to be classified is suitable for display as a full-size message cover image based on the prediction vector.

[0037] The method described in the embodiments of this disclosure can better understand the semantics and quality standards of image content by utilizing multimodal information, while focusing on specific discriminative dimensions by different expert layers, thereby balancing the interference between different tasks in a multi-task learning framework and improving the overall classification performance.

[0038] According to this embodiment, by combining multimodal inputs of images and prompt text, the model can more accurately understand the semantic content and quality standards of images based on structured text descriptions. Simultaneously, by employing an architecture combining a shared feature extraction network and parallel expert layers, it can effectively separate general visual feature extraction from task-specific discrimination (overall applicability determination and specific defect identification), thereby obtaining a comprehensive evaluation result on image quality in one go during the inference stage, significantly improving classification accuracy and inference efficiency.

[0039] Those skilled in the art will understand that multimodal image classification models can include deep learning models capable of processing multiple modalities of data (such as images and text), such as, but not limited to, vision-language models based on the Transformer architecture. An expert fully connected layer refers to a neural network layer in a multi-task learning architecture (such as MMoE) that is specifically responsible for a particular subtask. It is understood that the above are merely examples, and this disclosure is not limited thereto. Additionally and / or alternatively, the method can be deployed in a cloud server cluster to handle high concurrency requests, or it can be deployed on edge devices to reduce latency.

[0040] In some embodiments, the shared feature extraction network may include a backbone network and a shared fully connected layer. In such an embodiment, obtaining cross-task common features may include: obtaining a hidden state vector through the backbone network; and obtaining cross-task common features based on the hidden state vector through the shared fully connected layer.

[0041] This implementation leverages the powerful feature extraction capabilities of the backbone network to capture high-dimensional hidden state vectors from original images and text, encompassing rich semantic and visual information. Subsequently, these hidden states are further mapped to a common feature space suitable for subsequent multi-task expert processing through shared fully connected layers. This achieves effective feature dimensionality reduction while ensuring that different expert layers share basic representations, avoiding redundant computation and improving the overall computational efficiency of the model.

[0042] It is understood that the backbone network can be a large, pre-trained model with frozen or fine-tuned parameters. The hidden state vectors are feature representations of the backbone network's output. Shared fully connected layers can include one or more linear transformations and activation functions. It is understood that the above are merely examples, and this disclosure is not limited thereto.

[0043] In some embodiments, the notification text may include a structured description of the evaluation criteria for the image serving as a full-size message cover image.

[0044] According to this embodiment, by inputting a structured description containing evaluation criteria, the model is essentially provided with explicit prior knowledge and task instructions. This structured text guides the model to focus on features in the image that are closely related to the evaluation criteria (such as sharpness, composition, content security, etc.), thereby reducing interference from irrelevant information. This allows the model to have a clearer semantic alignment regarding what constitutes a suitable cover image and what constitutes a defect, improving the robustness of classification.

[0045] Those skilled in the art will understand that a structured description can refer to text organized according to a specific logic or format (such as key-value pairs, a list of specific delimiters), and may include explicitly listed indicators for judging an image. For example: clarity: high; content: no inappropriate information; composition: subject centered. It is understood that the above are merely examples, and this disclosure is not limited thereto. In some additional embodiments, the content of the prompt text can be dynamically adjusted according to changes in business rules without retraining the core of the model.

[0046] In some embodiments, obtaining a prediction vector may include concatenating the outputs of the at least two fully connected expert layers to generate a multidimensional prediction vector.

[0047] In this example, the task-specific results from different expert layers are integrated into a unified vector representation through a concatenation operation. This approach allows the model to simultaneously present the overall applicability assessment and the identification results of various specific defect features at a single output level. This facilitates the subsequent logic's comprehensive processing of this information and simplifies the design of the model's output interface.

[0048] It is understood that concatenation involves joining multiple vectors along a specific dimension. For example, the decision expert layer outputs a 1-dimensional confidence score, and the problem expert layer outputs an N-dimensional defect probability; concatenation yields a 1+N-dimensional prediction vector. It is understood that the above is merely an example, and this disclosure is not limited thereto.

[0049] In some embodiments, the at least one problem expert layer may include multiple problem expert layers corresponding to multiple defect feature categories respectively.

[0050] According to this embodiment, a dedicated expert network is set up to process defect features of different natures (such as technical quality issues and content semantic issues) independently. This design avoids mutual interference (negative transfer) caused by excessive differences in the distribution of different types of defect features, thereby significantly improving the model's sensitivity and accuracy in identifying various fine-grained defects.

[0051] Those skilled in the art will understand that defect feature categories are broad classifications of reasons why an image is unsuitable for display, such as technical, content, and design-related categories. Each expert layer focuses on learning one type of feature. It is understood that the above is merely an example, and this disclosure is not limited thereto.

[0052] In some embodiments, one or more of the at least one problem expert layer may correspond to more than one defect feature subcategory and have more than one corresponding output dimension.

[0053] This implementation allows for the processing of multiple related sub-tasks within the same expert layer. For example, different manifestations of the same type of defect (such as design flaws) (poor composition, chaotic layout) often share certain underlying visual features. Unified modeling by the same expert layer can promote collaborative learning of features, while multi-dimensional outputs ensure independent identification of each sub-defect, improving the model's parameter utilization.

[0054] It is understandable that, for example, a technical defect expert layer can simultaneously output the probabilities of three sub-dimensions: blur, overexposure, and blackness. In this case, the output dimension of the expert layer is 3. It is also understood that the above is merely an example, and this disclosure is not limited to this.

[0055] In some embodiments, the plurality of problem expert layers may include: a technical defect problem expert layer, used to identify quality problems of the image at the visual technology level; a content inappropriateness problem expert layer, used to identify inappropriateness problems at the level of the subject or meaning of the image content; a design defect problem expert layer, used to identify design problems of the image at the level of composition or layout; and a value deficiency problem expert layer, used to identify problems of the image lacking in information display value.

[0056] Based on this implementation, a comprehensive quality assessment system covering four dimensions—visual, content, design, and value—was constructed. This refined division of labor ensures that the model can not only identify explicit image quality issues but also understand implicit content meanings and aesthetic design problems. This allows it to meet the extremely high quality requirements for cover images in content distribution scenarios and avoids the risk of missed detections caused by judging from a single dimension.

[0057] Those skilled in the art will understand that these classifications correspond to common reasons for low-quality images in actual business operations. For example, lack of value may refer to images with empty content and no informational value. It is understood that the above are merely examples, and this disclosure is not limited thereto. In some other embodiments, other types of expert layers, such as those for copyright risk assessment, can be added according to business needs.

[0058] In some embodiments, the prediction vector may include a first dimension indicating confidence in the prediction that the image to be classified is suitable as a full-size message cover image, and the prediction vector may include at least one other dimension indicating that the image to be classified has at least one defective feature that makes it unsuitable for display as a full-size message cover image.

[0059] In this example, the prediction vector exhibits a total-to-part structure. The first dimension provides macro-level decision-making recommendations, while the other dimensions provide micro-level defect diagnostic information. This rich information output offers significant flexibility for downstream business decisions, supporting both rapid screening and in-depth degradation strategy analysis or user feedback generation.

[0060] Understandably, the first dimension typically represents the output of the expert layer, with values ​​usually ranging from 0 to 1. The other dimensions correspond to the outputs of the expert layer for each specific issue, indicating the likelihood of a particular defect existing. It is understood that the above is merely an example, and this disclosure is not limited thereto.

[0061] In some embodiments, determining whether the image to be classified is suitable as a full-size message cover image based on the prediction vector may include: determining that the image to be classified is not suitable as a full-size message cover image in response to determining that the confidence level of the first dimension is lower than a confidence threshold.

[0062] This implementation provides a highly efficient veto mechanism. When the model deems an image of insufficient quality in its global assessment, it can directly classify it as unsuitable without consuming computational resources to analyze the specific type of defect. This strategy can quickly filter out a large amount of low-quality content, ensuring the system's processing efficiency and maintaining a basic quality standard for the content pool.

[0063] It is understood that the confidence threshold is a hyperparameter that can be adjusted according to the business's needs for accuracy and recall, for example, set to 0.6. It is understood that the above is merely an example, and this disclosure is not limited thereto.

[0064] In some embodiments, determining whether the image to be classified is suitable as a full-size message cover image based on the prediction vector may include: in response to determining that the confidence level of the first dimension is not lower than a confidence threshold, determining whether the image to be classified is suitable as a full-size message cover image based on at least one other dimension.

[0065] This implementation introduces a dual verification mechanism, significantly enhancing the security of the judgment. Even if an image performs well in the overall score (high in the first dimension), the system will still check for specific serious defects (such as pornography or political content). If other dimensions indicate the presence of such defects, the system can still reject the image. This effectively prevents high-scoring but low-quality images from being overlooked, ensuring the absolute safety and high quality of the cover image.

[0066] It is understood that this step may involve applying specific threshold judgments to the values ​​of other dimensions. For example, even if the overall score is high, if the predicted value of the inappropriate content dimension exceeds the safety threshold, it will still be judged as inappropriate. It is understood that the above is only an example, and this disclosure is not limited thereto.

[0067] For example, the multimodal image classification model used in step S202 may be a multimodal image classification model trained according to the method of the embodiments of this disclosure, or it may be an image classification model obtained by other methods that can determine whether the image to be classified is suitable as a full-size message cover image based on the image to be classified and the prompt text.

[0068] The image classification method provided in this disclosure can utilize a trained multimodal image classification model to simultaneously output a confidence score indicating overall applicability and a feature vector indicating specific defects in multiple dimensions (such as blurriness, truncation, poor layout, etc.) in a single forward inference. This multi-dimensional output structure supports more flexible and refined business decision-making logic: on the one hand, clearly unqualified images can be quickly filtered out using the confidence score of the first dimension, achieving a global "one-vote veto" to ensure the bottom line of content; on the other hand, when the global confidence score is at an edge, fine-grained verification can be further performed based on the predicted values ​​of other defect dimensions to accurately identify the specific reasons for the decline in image quality. Compared with the existing technology that serially or in parallel calls multiple independent small models (such as calling the clarity model, border recognition model, etc. separately), this method significantly reduces the latency and system resource consumption of online inference, while avoiding overall decision-making errors caused by the failure of a single sub-model, effectively improving the display quality of cover images, screen efficiency ratio, and user browsing experience in the content distribution platform.

[0069] The training method 300 for a multimodal image classification model according to an exemplary embodiment of the present disclosure is described below with reference to FIG3.

[0070] In step S301, training samples are obtained. The training samples include sample images, prompt text, and classification labels. The classification labels include a first label indicating whether the image is suitable as a full-size message cover image.

[0071] In step S302, an image classification model to be trained is obtained. The image classification model includes a shared feature extraction network and at least two expert fully connected layers set in parallel. The at least two expert fully connected layers include at least one judgment expert layer for determining whether an image is suitable for display as a full-size message cover image, and at least one problem expert layer for identifying whether an image has defective features that are unsuitable for display as a full-size message cover image.

[0072] In step S303, based on the sample images and prompt text, cross-task common features are obtained through a shared feature extraction network.

[0073] In step S304, task-specific feature mapping is performed on the common features across tasks through at least two expert fully connected layers to obtain prediction vectors.

[0074] In step S305, the loss value is determined based on the predicted vector and the classification label, and the model parameters are updated based on the determined loss value to obtain a trained multimodal image classification model.

[0075] The method described in the embodiments of this disclosure can better understand the semantics and quality standards of image content by utilizing multimodal information, while focusing on specific discrimination dimensions by different expert layers, thereby balancing training interference between different tasks in a multi-task learning framework and improving overall classification performance.

[0076] According to this embodiment, through end-to-end training, the model can simultaneously learn how to extract general features across tasks and how to make classification decisions for specific tasks. Introducing cue text as training input allows the model to establish a correlation between visual features and structured semantic standards during the training phase. Furthermore, by employing a multi-task learning framework, the model parameters are jointly optimized using the supervision signals from the first label and defect features. This enables the model to possess both overall discriminative ability and fine-grained defect recognition capabilities, resulting in more robust and comprehensive classification performance.

[0077] Those skilled in the art will understand that the training process can utilize back propagation and gradient descent optimizers (such as Adam and SGD). Training samples can be derived from cleaned and labeled historical business data. It is understood that the above are merely examples, and this disclosure is not limited thereto. Additionally and / or alternatively, a pre-training-fine-tuning strategy can be employed, where the backbone network is first pre-trained on a large-scale dataset, and then fine-tuned using samples from this method.

[0078] The following describes some variations of training method 300.

[0079] In some embodiments, the at least one problem expert layer may include multiple problem expert layers corresponding to multiple defect feature categories, respectively. In such embodiments, the classification label may further include a second label for indicating whether an image has features of a corresponding category among the multiple defect feature categories.

[0080] In such examples, by introducing a second label, explicit supervision information is provided to the expert layer for each specific task. This forces the model to learn not just an abstract concept of good / bad, but to learn to distinguish different defect patterns. This fine-grained supervision helps the model better distinguish between positive and negative samples in the feature space, especially when dealing with ambiguous boundary samples, where explicit defect category labels provide stronger gradient guidance.

[0081] It is understood that the second label can be multi-dimensional, with each dimension corresponding to a specific defect category. For example, the label vector could be [1, 0, 1, 0...], corresponding to whether it is blurry, whether it is truncated, etc. It is understood that the above is only an example, and this disclosure is not limited thereto.

[0082] In some embodiments, one or more of the at least one problem expert layer may correspond to more than one defect feature subcategory and have more than one corresponding output dimension.

[0083] This implementation allows for joint optimization of related subtasks during training. By having a single expert layer simultaneously predict multiple related sub-category labels, the model can learn common features among these sub-categories, thus achieving better generalization ability with limited sample data. Furthermore, the multi-output dimension design enables the training process to fully utilize the rich, fine-grained label information contained within the data.

[0084] Understandably, this corresponds to the subtask grouping strategy in multi-label classification problems. For example, all sub-labels related to image composition are grouped into the same expert layer for training. Understandably, the above is merely an example, and this disclosure is not limited thereto.

[0085] In some embodiments, the plurality of problem expert layers may include: a technical defect problem expert layer, used to identify quality problems of the image at the visual technology level; a content inappropriateness problem expert layer, used to identify inappropriateness problems at the level of the subject or meaning of the image content; a design defect problem expert layer, used to identify design problems of the image at the level of composition or layout; and a value deficiency problem expert layer, used to identify problems of the image lacking in information display value.

[0086] This implementation establishes multi-dimensional quality control objectives during the training phase. This forces the model to retain sufficiently rich information during feature extraction to meet the subsequent judgment needs of different expert layers regarding visual, semantic, aesthetic, and value information. This comprehensive training objective setting ensures that the final model can cope with complex and ever-changing real-world business scenarios, reducing the tendency for models to specialize in only one area.

[0087] Those skilled in the art will understand that the division of these expert layers is based on the business-side definition of cover image quality standards. During training, each expert layer calculates the loss value for its corresponding task. It is understood that the above is merely an example, and this disclosure is not limited thereto.

[0088] In some embodiments, the loss value may be determined based on a weighted multi-label binary cross-entropy loss.

[0089] According to this embodiment, by employing a weighted loss function, the contributions of different tasks in the model optimization process can be effectively balanced. Given that the distribution of various defects in real-world samples is often uneven (e.g., blurriness is common, while vulgarity is rare), and that different tasks have different importance (e.g., content security is more important than aesthetics), by adjusting the weight coefficients, the model can focus more on long-tail samples or key tasks, thereby training a model that performs more balancedly and superiorly across various metrics.

[0090] It is understandable that the loss function formula can be... ,in Here, represents the weight of the i-th task, and BCE is the binary cross-entropy loss. It is understood that the above is merely an example, and this disclosure is not limited thereto.

[0091] The training method for the multimodal image classification model provided in this disclosure effectively utilizes the semantic understanding capabilities of the multimodal large model by introducing prompt text along with the image as input. This allows the model to deeply understand the underlying reasons why an image is unsuitable as a full-size cover image (such as content meaning or design flaws) based on structured evaluation criteria, rather than being limited to pixel-level visual features. Simultaneously, this method employs a multi-task learning architecture (such as the MMoE architecture) that includes a shared feature extraction network and multiple task-specific expert layers. The shared layer extracts common features across tasks, while the expert layers capture specific features related to technical defects, inappropriate content, design flaws, and value loss, effectively addressing the potential negative transfer problem between different defect identification tasks. Furthermore, by employing a weighted multi-label binary cross-entropy loss function, the learning weight of different tasks can be dynamically balanced, significantly improving the difficulty of model training caused by the uneven distribution of sample classes. This enables the training of a high-precision, highly generalizable multimodal model that can simultaneously output overall discrimination and fine-grained defect classification in an end-to-end manner, greatly reducing the manpower and computing power costs of maintaining multiple independent sub-models.

[0092] The data flow for training and application of a multimodal image classification model according to an embodiment of the present disclosure will now be described with reference to FIG4. As shown in FIG4, an embodiment of the present disclosure provides a complete training method, system, and apparatus for a large multimodal image template style classification model, including offline sample construction, model training framework, and online application.

[0093] As shown in Figure 4, an exemplary system apparatus is divided into offline and online parts. The offline (training side) may include standard and prompt formulation, training sample construction, training and encapsulation of a multimodal large model for image template style classification. The online (application side) may include batch requesting images to generate large and small image features from the model and storing them in a database; during the online service phase, reading the large and small image features of the content cover image for further style optimization strategies and low-quality large image degradation strategies, etc. It is understood that the various implementation details described herein with reference to Figure 4 can be applied to appropriate steps of the various methods described with reference to Figures 2 and 3, or as variations of the various methods described with reference to Figures 2 and 3, and this disclosure is not limited thereto.

[0094] On the offline training side, standards and prompts can be established first. To reduce the difficulty of learning the latent features of cover images for multimodal large models, several categories of problems unsuitable for large images can be systematically defined: 1) Technical defects: blurry image, subject truncated, key area occlusion; 2) Inappropriate content: unbalanced proportion of close-ups of people, content that is offensive; 3) Design defects: unattractive borders, dense text, poor composition, poor layout; 4) Lack of value: monotonous elements lacking necessity for large image display. Based on the above standards and their descriptions, a structured prompt template can be constructed to drive model learning.

[0095] Next, we can construct the training samples. In order to classify the cover image as suitable for large-formatting and dozens of issues that make it unsuitable for large-formatting, we can label each sample image with multiple labels: 1) Whether it is suitable for large-formatting: {Yes, No}; 2) Reasons for unsuitability for large-formatting: for example, label up to three issues that make it unsuitable for large-formatting.

[0096] Next, we can proceed with model training.

[0097] The training framework for the multimodal large model for image template style classification is shown in Figure 5. For example, the model may include: a multimodal large model: the input cover image and structured prompt words are processed by the multimodal large model to obtain the final output hidden state vector H.

[0098] Shared FC Layer: The output hidden state vector in a multimodal large model can be input into a shared linear layer to extract common features F across tasks.

[0099] Multiple expert FC layers: Adding multiple (e.g., five) expert modules allows the model to learn characteristic representations of different problems. Compared to models using only shared layers, this further enhances the model's fine-grained classification ability. The common features F output from the shared FC layer are input into the five expert FC layers to capture task-specific features. Based on criteria such as those mentioned above, problems unsuitable for large images can be divided into four categories, thus requiring four problem expert FC layers; in addition, a decision expert FC layer for whether a problem is suitable for large images is also needed, for a total of five expert FC layers. As shown in Figure 5, the output dimensions of the five expert layers from top to bottom are 1, 3, 3, 4, and 1, which are concatenated into a 12-dimensional output [p big ,p tech ,p content , p design ,p value ].

[0100] For model training, a multi-label binary cross-entropy loss with task weights is used, with total loss = α. Is it suitable to output large image loss + β? Technical defect type loss + γ Content inappropriateness loss + δ Design defect type losses + The missing value class loss is a binary cross-entropy loss (BCE with Logits Loss) for each part. The weighted loss function adjusts the importance of different tasks by assigning weights to each expert task, effectively addressing the problem of class imbalance. An example loss function could be as follows:

[0101] in, y represents the label, and p represents the prediction. The prediction vector can be [p big , p tech , p content , p design , p value The label vector is [y]. big , y tech , y content ,y design , y value ] ∈ {0,1} 12 .

[0102] Next, referring to Figure 4, we will describe an exemplary model for online inference and application.

[0103] During online inference, multi-dimensional (12-dimensional in the example above) prediction vectors can be generated through end-to-end training. Each dimension, after sigmoid activation, generates a confidence score for the corresponding category [p]. big ,p tech , p content ,p design ,p value For the last 11 dimensions (excluding p) big For classification tasks that are not suitable for large image problems, a class-adaptive threshold optimization strategy is adopted: based on the validation set PR curve, the optimal discrimination threshold is independently calculated for each problem category, maximizing the micro-F1 score to achieve a balance between precision and recall in fine-grained problem detection. For the first dimension (p big To determine whether an image template style classification task is suitable for large-scale image prediction, a threshold correction mechanism is implemented: when the confidence level of this dimension is lower than the threshold, a global unsuitability for large-scale image determination is forcibly triggered. This two-level decision framework significantly improves the precision and recall of image template style classification tasks by weightedly integrating fine-grained problem features and global adaptation features, while ensuring multi-task joint optimization.

[0104] In online applications, for example, users can request images to be posted in batches in real time, and the size and image features of the images are then stored in the database. During the online service phase, the size and image features of the content cover image are read to make further template style decisions and low-quality large image degradation strategies.

[0105] Referring now to FIG. 6, an image classification apparatus 600 according to an embodiment of the present disclosure is described. The image classification apparatus 600 may include an input acquisition unit 601, a model acquisition unit 602, a common feature acquisition unit 603, a prediction vector acquisition unit 604, and a classification unit 605. The input acquisition unit 601 may be used to acquire an image to be classified and prompt text. The model acquisition unit 602 may be used to acquire a multimodal image classification model, the multimodal image classification model including a shared feature extraction network and at least two expert fully connected layers configured in parallel, wherein the at least two expert fully connected layers include at least one judgment expert layer for determining whether an image is suitable for display as a full-size message cover image, and at least one problem expert layer for identifying whether an image has defective features that make it unsuitable for display as a full-size message cover image. The common feature acquisition unit 603 may be used to acquire cross-task common features based on the image to be classified and the prompt text, through the shared feature extraction network. The prediction vector acquisition unit 604 may be used to perform task-specific feature mapping on the cross-task common features through the at least two expert fully connected layers to obtain prediction vectors. The classification unit 605 can be used to determine, based on the prediction vector, whether the image to be classified is suitable for display as a full-size message cover image.

[0106] The training apparatus 700 for a multimodal image classification model according to an embodiment of the present disclosure will now be described with reference to FIG. 7. The training apparatus 700 for the multimodal image classification model may include a training sample acquisition unit 701, a model acquisition unit 702, a common feature acquisition unit 703, a prediction vector acquisition unit 704, and a parameter update unit 705. The training sample acquisition unit 701 may be used to acquire training samples, which include sample images, prompt text, and classification labels, the classification labels including a first label indicating whether an image is suitable as a full-size message cover image. The model acquisition unit 702 may be used to acquire a multimodal image classification model to be trained, which includes a shared feature extraction network and at least two expert fully connected layers configured in parallel, wherein the at least two expert fully connected layers include at least one judgment expert layer for determining whether an image is suitable for display as a full-size message cover image, and at least one problem expert layer for identifying whether an image has defective features that make it unsuitable for display as a full-size message cover image. The common feature acquisition unit 703 can be used to obtain cross-task common features based on the sample image and the prompt text through the shared feature extraction network. The prediction vector acquisition unit 704 can be used to perform task-specific feature mapping on the cross-task common features through the at least two expert fully connected layers to obtain prediction vectors. The parameter update unit 705 can be used to determine the loss value based on the prediction vector and the classification label, and update the model parameters based on the determined loss value to obtain a trained multimodal image classification model.

[0107] The apparatus described in the embodiments of this disclosure can better understand the semantics and quality standards of image content by utilizing multimodal information, while focusing on specific discrimination dimensions through different expert layers, thereby balancing training interference between different tasks in a multi-task learning framework and improving overall classification performance.

[0108] It is understood that the details, variations, and benefits described in relation to the methods of the various embodiments of this disclosure are also applicable to apparatuses according to the various embodiments of this disclosure, and will not be repeated here.

[0109] The collection, acquisition, storage, use, processing, transmission, provision, and public application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0110] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0111] Referring to Figure 8, a structural block diagram of an electronic device 800 that can serve as a server or client of this disclosure is now described, which is an example of hardware devices that can be applied to various aspects of this disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the disclosure described and / or claimed herein.

[0112] As shown in Figure 8, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 can also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0113] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, output unit 807, storage unit 808, and communication unit 809. Input unit 806 can be any type of device capable of inputting information to electronic device 800. Input unit 806 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 807 can be any type of device capable of presenting information, and can include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 808 can include, but is not limited to, disk and optical disk. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0114] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as methods 200 and / or 300 and their variations. For example, in some embodiments, methods 200 and / or 300 and their variations may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of methods 200 and / or 300 and their variations described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute method 200 and / or 300 and its variations by any other suitable means (e.g., by means of firmware).

[0115] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0117] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0119] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0120] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0121] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0122] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. An image classification method, comprising: The process involves: obtaining an image to be classified and a prompt text; obtaining a multimodal image classification model, which includes a shared feature extraction network and at least two expert fully connected layers configured in parallel, wherein the at least two expert fully connected layers include at least one judgment expert layer for determining whether an image is suitable for display as a full-size message cover image, and at least one question expert layer for identifying whether an image has defective features that make it unsuitable for display as a full-size message cover image; obtaining cross-task common features based on the image to be classified and the prompt text through the shared feature extraction network; performing task-specific feature mapping on the cross-task common features through the at least two expert fully connected layers to obtain prediction vectors; and determining whether the image to be classified is suitable for display as a full-size message cover image based on the prediction vectors.

2. The method according to claim 1, wherein, The shared feature extraction network includes a backbone network and a shared fully connected layer, wherein obtaining cross-task common features includes: obtaining a hidden state vector through the backbone network; and obtaining cross-task common features based on the hidden state vector through the shared fully connected layer.

3. The method according to claim 1 or 2, wherein, The prompt text includes a structured description of the evaluation criteria for the image used as the full-size message cover image.

4. The method according to any one of claims 1-3, wherein, Obtaining the prediction vector involves concatenating the outputs of the at least two fully connected expert layers to generate a multidimensional prediction vector.

5. The method according to any one of claims 1-4, wherein, The at least one problem expert layer includes multiple problem expert layers corresponding to multiple defect feature categories respectively.

6. The method according to claim 5, wherein, One or more of the at least one problem expert layers correspond to more than one defect feature sub-category and have more than one corresponding output dimension.

7. The method according to any one of claims 5-6, wherein, The multiple problem expert layers include: a technical defect problem expert layer, used to identify quality problems of images at the visual technology level; a content inappropriateness problem expert layer, used to identify inappropriateness problems at the level of the main content or meaning of the image; a design defect problem expert layer, used to identify design problems of images at the level of composition or layout; and a value deficiency problem expert layer, used to identify problems of images lacking information display value.

8. The method according to any one of claims 1-7, wherein, The prediction vector includes a first dimension indicating the confidence level of the prediction that the image to be classified is suitable as a full-size message cover image, and the prediction vector includes at least one other dimension indicating that the image to be classified has at least one defective feature that makes it unsuitable for display as a full-size message cover image.

9. The method according to claim 8, wherein, Determining whether the image to be classified is suitable as a full-size message cover image based on the prediction vector includes: in response to determining that the confidence level of the first dimension is lower than a confidence threshold, determining that the image to be classified is not suitable as a full-size message cover image.

10. The method according to claim 9, wherein, Determining whether the image to be classified is suitable as a full-size message cover image based on the prediction vector includes: in response to determining that the confidence level of the first dimension is not lower than a confidence threshold, determining whether the image to be classified is suitable as a full-size message cover image based on at least one other dimension.

11. A training method for a multimodal image classification model, comprising: Obtain training samples, which include sample images, prompt text, and classification labels, the classification labels including a first label for indicating whether the image is suitable as a full-size message cover image; A training image classification model is obtained, comprising a shared feature extraction network and at least two expert fully connected layers configured in parallel. The at least two expert fully connected layers include at least one judgment expert layer for determining whether an image is suitable as a full-size message cover image, and at least one question expert layer for identifying whether an image has defective features that would make it unsuitable as a full-size message cover image. Based on the sample image and the prompt text, cross-task common features are obtained through the shared feature extraction network. Task-specific feature mapping is then performed on the cross-task common features through the at least two expert fully connected layers to obtain prediction vectors. Finally, a loss value is determined based on the prediction vectors and the classification label, and the model parameters are updated based on the determined loss value to obtain a trained multimodal image classification model.

12. The method according to claim 11, wherein, The at least one problem expert layer includes multiple problem expert layers corresponding to multiple defect feature categories, and wherein the classification label further includes a second label for indicating whether the image has features of the corresponding category among the multiple defect feature categories.

13. The method according to claim 12, wherein, One or more of the at least one problem expert layers correspond to more than one defect feature sub-category and have more than one corresponding output dimension.

14. The method according to any one of claims 12-13, wherein, The multiple problem expert layers include: a technical defect problem expert layer, used to identify quality problems of images at the visual technology level; a content inappropriateness problem expert layer, used to identify inappropriateness problems at the level of the main content or meaning of the image; a design defect problem expert layer, used to identify design problems of images at the level of composition or layout; and a value deficiency problem expert layer, used to identify problems of images lacking information display value.

15. The method according to any one of claims 11-14, wherein, The loss value is determined based on the weighted multi-label binary cross-entropy loss.

16. An image classification device, comprising: The input acquisition unit is used to acquire the image to be classified and the prompt text; A model acquisition unit is used to acquire a multimodal image classification model, wherein the multimodal image classification model includes a shared feature extraction network and at least two expert fully connected layers configured in parallel, wherein the at least two expert fully connected layers include at least one judgment expert layer for determining whether an image is suitable for display as a full-size message cover image, and at least one question expert layer for identifying whether an image has defective features that make it unsuitable for display as a full-size message cover image; a common feature acquisition unit is used to acquire cross-task common features based on the image to be classified and the prompt text, through the shared feature extraction network; a prediction vector acquisition unit is used to perform task-specific feature mapping on the cross-task common features through the at least two expert fully connected layers to obtain prediction vectors; and a classification unit is used to determine whether the image to be classified is suitable for display as a full-size message cover image based on the prediction vectors.

17. A training device for a multimodal image classification model, comprising: A training sample acquisition unit is used to acquire training samples, which include sample images, prompt text, and classification labels. The classification labels include a first label indicating whether the image is suitable as a full-size message cover image. A model acquisition unit is used to acquire an image classification model to be trained. The image classification model includes a shared feature extraction network and at least two expert fully connected layers set in parallel. The at least two expert fully connected layers include at least one judgment expert layer for determining whether an image is suitable for display as a full-size message cover image, and at least one question expert layer for identifying whether an image has defective features that make it unsuitable for display as a full-size message cover image. A common feature acquisition unit is used to acquire cross-task common features based on the sample image and the prompt text through the shared feature extraction network. A prediction vector acquisition unit is used to perform task-specific feature mapping on the cross-task common features through the at least two expert fully connected layers to obtain prediction vectors. A parameter update unit is used to determine a loss value based on the prediction vector and the classification label, and update the model parameters based on the determined loss value to obtain a trained multimodal image classification model.

18. An electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-15.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-15.

20. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-15.

Images