Model training method, device and equipment, computer readable storage medium and computer program product

Abstract

The invention provides a model training method, device and equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring an image sample corresponding to a label type, and classifying the image sample through an image classification model to obtain a first classification probability of the image sample; determining a loss gradient corresponding to the label type based on the first classification probability of the image sample corresponding to the label type; determining a first loss based on the loss gradient corresponding to the label type, and determining a second loss based on the loss gradient corresponding to the target label type and the loss gradient corresponding to the label type; and performing fusion processing on the first loss and the second loss to obtain third loss, and updating the image classification model based on the third loss to obtain an updated image classification model. According to the method and the device, the model can equally learn all the label types when the data of the image samples corresponding to the label types are not uniformly distributed, and the situation that the high-frequency label types dominate the optimization direction of the model is avoided.

Description

Technical Field

[0001] The present application relates to model training technology, and in particular to a model training method, apparatus, device, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of short video platforms, the number of short videos generated by users has exploded, making short video review a critical component of platform operations. Current short video review methods rely primarily on manual review, requiring reviewers to watch each frame of the video to determine whether it meets the requirements. However, this manual approach is inefficient and prone to misjudgments or omissions due to human error.

[0003] In addition, the related technology uses automated review technology to review short videos, which is essentially image classification processing. However, the processing effect is poor when the label types corresponding to the training data are unevenly distributed, which can easily lead to label type skew problems, causing the model to recognize high-frequency label types with a higher accuracy, while the recognition accuracy of low-frequency label types is lower. Summary of the Invention

[0004] The embodiments of the present application provide a model training method, apparatus, device, computer-readable storage medium, and computer program product, which enable the model to learn all label types equally when the data distribution of image samples corresponding to the label types is uneven, thereby avoiding high-frequency label types dominating the optimization direction of the model.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a model training method, which includes:

[0007] Obtaining an image sample corresponding to each label type, and classifying each image sample using an image classification model to obtain a first classification probability for each image sample;

[0008] For each of the label types, determining a loss gradient corresponding to the label type based on a first classification probability of image samples belonging to the label type;

[0009] Determining a first loss based on loss gradients corresponding to the plurality of label types;

[0010] Determining a second loss based on the loss gradient corresponding to the target label type and the loss gradient corresponding to each of the label types;

[0011] The target label type is a label type whose corresponding number of image samples is less than a quantity threshold among the label types;

[0012] The first loss and the second loss are fused to obtain a third loss, and the image classification model is updated based on the third loss to obtain an updated image classification model.

[0013] The present invention provides a model training device, comprising:

[0014] A classification module is used to obtain image samples corresponding to each label type, and classify each image sample using an image classification model to obtain a first classification probability for each image sample;

[0015] a loss gradient determination module, configured to determine, for each label type, a loss gradient corresponding to the label type based on a first classification probability of image samples belonging to the label type;

[0016] A first loss determination module, configured to determine a first loss based on loss gradients corresponding to the plurality of label types;

[0017] A second loss determination module is configured to determine a second loss based on a loss gradient corresponding to a target label type and a loss gradient corresponding to each of the label types; wherein the target label type is a label type for which the number of corresponding image samples is less than a quantity threshold;

[0018] A third loss determination module is used to fuse the first loss and the second loss to obtain a third loss, and to update the image classification model based on the third loss to obtain an updated image classification model.

[0019] In the above scheme, the second loss determination module is also used to average the loss gradients corresponding to multiple target label types to obtain an average loss gradient; sum the loss gradients corresponding to multiple label types to obtain the sum of loss gradients; and generate the second loss based on the average loss gradient and the sum of loss gradients.

[0020] In the above scheme, the second loss determination module is also used to map the average loss gradient based on the sign function to obtain the sign value of the average loss gradient; determine the first difference between the sum of the loss gradients and the sign value, and determine the second loss that is positively correlated with the first difference.

[0021] In the above scheme, the first loss determination module is also used to determine the weight of the label type for each label type based on the number of image samples of the label type; based on the weight of each label type, the loss gradients corresponding to multiple label types are fused to obtain the first loss.

[0022] In the above scheme, the first loss determination module is also used to obtain the historical prediction accuracy of the label type corresponding to the nth training round for each value of n, wherein n is a positive integer not greater than N, the Nth round is the current round, and N is a positive integer; determine a first ratio between the number of image samples of the label type and the total number of image samples; based on the historical prediction accuracy of the nth training round and the first ratio, determine the weight of the label type corresponding to the nth training round; wherein the historical prediction accuracy of the nth training round is positively correlated with the weight of the label type of the nth training round, and the first ratio is negatively correlated with the weight of the label type of the nth training round; the weights of the label types of N training rounds are fused to obtain the weight of the label type.

[0023] In the above scheme, the first loss determination module is also used to, when the value of n is 1, use the prediction accuracy of the nth training round as the historical prediction accuracy of the nth training round; when the value of n is greater than 1, average the prediction accuracy of the nth training round and the prediction accuracy of the K training rounds before the nth training round to obtain the historical prediction accuracy of the nth training round.

[0024] In the above scheme, the loss gradient determination module is also used to determine the classification loss corresponding to the label type based on the first classification probability of the image samples belonging to the label type; and determine the loss gradient corresponding to the label type based on the classification loss corresponding to the label type.

[0025] In the above scheme, the loss gradient determination module is also used to perform the following processing for each image sample belonging to the label type: based on the label type, generate a probability distribution label of the image sample corresponding to multiple label types; based on the first classification probability of the image sample corresponding to multiple label types and the probability distribution label corresponding to multiple label types, determine the classification loss corresponding to the image sample; and fuse the classification losses corresponding to the multiple image samples belonging to the label type to obtain the classification loss corresponding to the label type.

[0026] In the above scheme, the classification module is also used to obtain the original image sample corresponding to each of the label types, and perform the following processing on each of the original image samples: based on the number of pixels of the original image sample, determine the number of tokens of the image token assigned to the original image sample; based on the number of tokens, resize the original image sample to obtain an image sample corresponding to the original image sample.

[0027] In the above scheme, the classification module is also used to determine a second ratio that is positively correlated with the number of pixels of the original image sample and negatively correlated with the total number of pixels of all the original image samples; and multiply the second ratio by the total number of image tokens of all the original image samples to obtain the number of image tokens assigned to the original image sample.

[0028] In the above scheme, the classification module is also used to multiply the number of pixels of the image token by the number of tokens to obtain the new number of pixels of the original image sample; based on the ratio between the new number of pixels of the original image sample and the number of pixels of the original image sample, the width and height of the original image sample are proportionally reduced to obtain an image sample corresponding to the original image sample.

[0029] An embodiment of the present application provides an electronic device, comprising:

[0030] a memory for storing computer-executable instructions or computer programs;

[0031] The processor is used to implement the model training method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0032] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the model training method provided in the embodiment of the present application when executed by a processor.

[0033] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the model training method provided in the embodiment of the present application is implemented.

[0034] The embodiments of the present application have the following beneficial effects:

[0035] Through the embodiments of the present application, image samples corresponding to each label type are obtained and classified using an image classification model to obtain a first classification probability for each image sample. The first classification probability of each image sample belonging to a label type can be obtained, providing data support for subsequent loss calculations. For each label type, a loss gradient corresponding to the label type is determined based on the first classification probability of the image samples belonging to the label type. Determining the loss gradient for the label type can reflect the overall fit of the model to that label type. The gradient direction indicates the direction of parameter update, and the magnitude of the loss gradient reflects the optimization difficulty. Based on the loss gradients corresponding to multiple label types, a first loss is determined, which can be used to evaluate the classification performance of the model on multiple label types. A second loss is determined based on the loss gradient corresponding to the target label type and the loss gradients corresponding to each label type. The target label type is a label type for which the number of corresponding image samples is less than a threshold. By increasing the second loss for the target label type, i.e., the low-frequency label type, the model is forced to focus more on learning the target label type, alleviating the data imbalance problem of image samples corresponding to multiple label types. This can improve the model's sensitivity to the target label type, i.e., the low-frequency label type. The first loss and the second loss are fused to obtain a third loss, and the image classification model is updated based on the third loss to obtain an updated image classification model. The third loss obtained by fusing the first loss and the second loss can balance the performance of the model on low-frequency label types and high-frequency label types, so that the model can maintain the recognition accuracy of high-frequency label types after the update, and improve the recognition ability of low-frequency label types. Through the embodiments of the present application, the model can learn all label types equally when the data distribution of image samples corresponding to the label types is uneven, avoiding the high-frequency label types dominating the optimization direction of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of the structure of the model training system architecture provided in an embodiment of the present application;

[0037] Figure 2 Schematic diagram of the structure of the model training device provided in the embodiment of the present application;

[0038] Figures 3A-3H This is a flow chart of the model training method provided in the embodiment of the present application;

[0039] Figure 4 This is the short video review interface provided in the embodiment of this application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0041] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0043] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0044] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0045] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0046] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0047] (1) Supervised fine-tuning: Based on the pre-trained model, supervised fine-tuning is performed using labeled data to make the pre-trained model better suited to specific tasks.

[0048] (2) Video memory exhaustion: When processing short videos of long duration, the graphics card's video memory is exhausted, causing the program to crash or report an error.

[0049] (3) Tokens: Each video frame (original image sample) will be divided into small blocks (patches) of fixed size. Each small block is regarded as a token. The total number of tokens corresponding to each video frame (original image sample) is the number of tokens allocated to the video frame (original image sample).

[0050] (4) Low-frequency categories (target label types): categories (label types) whose number of corresponding video frames (original image samples) in multiple categories is less than the number threshold.

[0051] (5) High-frequency categories: categories (label types) whose number of corresponding video frames (original image samples) in multiple categories is greater than or equal to the quantity threshold.

[0052] (6) Regularization: By introducing additional constraints or penalties during model training, the complexity of the model is limited so that it performs well on the training data and can also generalize better to unseen data.

[0053] (7) Early Stopping: During model training, the performance of the validation set is continuously tested. When the performance of the model on the validation set no longer improves, the training is terminated early. The validation set is part of the training data in the entire training dataset.

[0054] With the rapid development of short video platforms, the number of short videos generated by users has exploded, making short video review a critical component of platform operations. Current short video review methods rely primarily on manual review, requiring reviewers to watch each frame of the video to determine whether it meets the requirements. However, this manual approach is inefficient and prone to misjudgments or omissions due to human error.

[0055] Furthermore, related technologies use automated review techniques to audit short videos, but this is less effective when the distribution of label types corresponding to the training data is uneven. This can easily lead to label type skew, causing the model to have a higher accuracy rate for identifying high-frequency label types and a lower accuracy rate for identifying low-frequency label types. Furthermore, related technologies often face insufficient video memory when processing long short videos, leading to memory exhaustion.

[0056] The embodiments of the present application provide a model training method, apparatus, device, computer-readable storage medium, and computer program product, which enable the model to learn all label types equally when the data distribution of image samples corresponding to the label types is uneven, thereby avoiding the high-frequency label types dominating the optimization direction of the model. The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The device provided by the embodiment of the present application can be implemented as various types of terminals such as laptops, tablet computers, and desktop computers, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a server.

[0057] See also Figure 1 , Figure 1 It is a structural diagram of the model training system architecture provided in an embodiment of the present application. In order to support a model training application, in the model training system 100, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The server 200 uploads the data to the database 500.

[0058] Server 200 obtains image samples corresponding to each label type from database 500, and classifies each image sample using an image classification model to obtain a first classification probability for each image sample. For each label type, server 200 determines a loss gradient corresponding to the label type based on the first classification probability of image samples belonging to the label type. A first loss is determined based on the loss gradients corresponding to multiple label types. A second loss is determined based on the loss gradient corresponding to a target label type and the loss gradient corresponding to each label type. The target label type is a label type for which the number of corresponding image samples is less than a threshold value. The first and second losses are fused to obtain a third loss, and the image classification model is updated based on the third loss to obtain an updated image classification model. Server 100 deploys the updated image classification model in terminal 400 to perform classification processing on the image samples.

[0059] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0060] See also Figure 2 , Figure 2 2 is a schematic diagram of the structure of the model training device provided in the embodiment of the present application. The server 200 includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the server 200 are coupled together through a bus system 240. It can be understood that the bus system 240 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 240 .

[0061] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0064] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0065] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0066] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0067] A network communication module 252 for reaching other electronic devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0068] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., a display screen, a speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0069] The input processing module 254 is configured to detect one or more user inputs or interactions from one of the one or more input devices 232 and to translate the detected inputs or interactions.

[0070] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 The model training device 255 stored in the memory 250 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a classification module 2551, a loss gradient determination module 2552, a first loss determination module 2553, a second loss determination module 2554, and a third loss determination module 2555. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0071] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the model training method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0072] In some embodiments, the terminal or server can implement the model training method provided by the embodiment of the present application by running various computer executable instructions or computer programs.For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level.The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run; it can also be a small program that can be embedded in any APP, that is, a program that only needs to be downloaded to a browser environment and can be run.In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0073] The model training method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0074] Below, the model training method provided by the embodiment of the present application is described. As mentioned above, the electronic device that implements the model training method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0075] See also Figure 3A , Figure 3A This is a flow chart of the model training method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained, Figure 3A The body of the step is the server.

[0076] In step 101, an image sample corresponding to each label type is obtained.

[0077] In some embodiments, see Figure 3B , Figure 3A Step 101 shown can be implemented by steps 1011 to 1013 below. Figure 3B It should be noted that steps 1012 to 1013 are performed for each original image sample.

[0078] In step 1011, an original image sample corresponding to each of the label types is obtained.

[0079] In step 1012 , the number of image tokens allocated to the original image sample is determined based on the number of pixels of the original image sample.

[0080] In some embodiments, Figure 3B Step 1012 shown can be implemented by the following technical solution: determine a second ratio that is positively correlated with the number of pixels of the original image sample and negatively correlated with the total number of pixels of all the original image samples; multiply the second ratio by the total number of image tokens of all the original image samples to obtain the number of image tokens assigned to the original image sample.

[0081] As an example, the number of image tokens assigned to an original image sample is calculated as shown in the following formula (1).

[0082]

[0083] Among them, T i is the number of image tokens assigned to the original image sample, P i is the number of pixels of the original image sample, P total is the total number of pixels of all original image samples, T is the total number of image tokens of all original image samples, the total number of image tokens of all original image samples can be set freely, and there is no restriction here. is the second ratio.

[0084] Through the embodiments of the present application, the allocation of image token resources can be adaptively adjusted based on the number of pixels in the original image sample, preventing original image samples with fewer pixels from occupying too many image token resources or original image samples with more pixels from being insufficiently allocated image token resources. This ensures that original image samples with different pixel numbers can all obtain reasonable image token resources.

[0085] In step 1013, the original image sample is resized based on the number of tokens to obtain an image sample corresponding to the original image sample.

[0086] In some embodiments, see Figure 3C , Figure 3B Step 1013 shown can be implemented by the following steps 1013A to 1013B, as shown below. Figure 3C Provide explanation.

[0087] In step 1013A, the number of pixels of the image token is multiplied by the number of tokens to obtain a new number of pixels of the original image sample.

[0088] As an example, the number of pixels in an image token is positively correlated with the number of local blocks into which the original image sample is segmented. The number of pixels in an image token is calculated as shown in the following formula (2). The number of new pixels in an original image sample is positively correlated with the product of the number of pixels in an image token and the number of tokens. The number of new pixels in an original image sample is calculated as shown in the following formula (3):

[0089] num=patch×patch (2)

[0090] P′ i =T i ×num (3)

[0091] Among them, num is the number of pixels of the image token, patch is the number of local patches into which the original image sample is divided, and is the basic unit for the model to process visual information; T i The number of image tokens assigned to the original image sample is also the number of tokens, P′ i is the new number of pixels in the original image sample.

[0092] In step 1013B, based on the ratio between the number of new pixels of the original image sample and the number of pixels of the original image sample, the width and height of the original image sample are scaled down to obtain an image sample corresponding to the original image sample.

[0093] As an example, after determining the number of new pixels and the number of pixels of the original image sample, the original image sample can be geometrically reduced to obtain an image sample of the original image sample. The calculation method for obtaining the image sample of the original image sample is shown in the following formulas (4) and (5).

[0094]

[0095] Among them, P′ i is the new number of pixels of the original image sample, P iis the number of pixels of the original image sample, sqrt() is the square root function, which means taking the square root of a number, that is, finding its square root. W is the width of the original image sample, H is the height of the original image sample, W′ is the width of the image sample of the original image sample, and H′ is the height of the image sample of the original image sample. The proportional reduction process is to reduce the original image sample with width W and height H to an image sample with width W′ and height H′.

[0096] Through the embodiments of the present application, the size of the original image sample can be dynamically adjusted. The equal ratio scaling based on the ratio between the new number of pixels of the original image sample and the number of pixels of the original image sample can intelligently retain the key visual features of the original image sample, and the image sample of the original image sample after compression can occupy less video memory resources, so that there will be no video memory overflow problem when the image classification model classifies the image sample, and it can process a larger number of image samples, accelerating model convergence, and is applicable to deployment environments with limited video memory resources.

[0097] Through the embodiments of the present application, dynamically allocating the number of image tokens and combining adaptive size adjustment significantly reduces the video memory occupancy and computational load of the image sample. It effectively alleviates the video memory overflow problem caused by processing a large number of image samples, enabling the model to process larger-scale image samples.

[0098] Continue to refer to Figure 3A , in step 102, each of the image samples is classified by the image classification model to obtain the first classification probability of each of the image samples.

[0099] As an example, when each image sample is classified by the image classification model, the first classification probability of each image sample can be achieved by calculating the similarity between the image sample and the features corresponding to the label type. The result after normalizing the similarity can be used as the first classification probability, and the first classification probability represents the probability that a certain image sample belongs to each label type. Assume that the feature vector of the image sample is A = {a1, a2,..., a n}, and the feature vector of the features corresponding to the label type is B = {b1, b2,..., b n}. The calculation method of the similarity between the image sample and the features corresponding to the label type is shown in the following formula (6). [[ID=Z19]]

[0100]

[0101] where D is the Euclidean distance between the feature vector of the image sample and the feature vector of the features corresponding to the label type. The Euclidean distance D is normalized and mapped to D ∈ (0, 1], that is, making the value of D be 0 < D ≤ 1. The specific calculation method is shown in the following formula (7).

[0102]

[0103] Where D is the Euclidean distance between the feature vector of the image sample and the feature vector corresponding to the label type, and R is the normalized result of the Euclidean distance D. The normalized result R is used as the second classification probability that the image sample belongs to the label type. For each image sample, the second classification probabilities of the image sample and multiple label types need to be calculated, and the union of the multiple second classification probabilities is used as the first classification probability of each image sample. The label type corresponding to the maximum second classification probability is used as the predicted label type for the image sample using the image classification model.

[0104] In step 103, for each of the label types, a loss gradient corresponding to the label type is determined based on a first classification probability of image samples belonging to the label type.

[0105] In some embodiments, see Figure 3D , Figure 3A Step 103 shown can be implemented by the following steps 1031 to 1032, as shown below. Figure 3D Provide explanation.

[0106] In step 1031 , based on the first classification probability of the image samples belonging to the label type, the classification loss corresponding to the label type is determined.

[0107] In some embodiments, see Figure 3E , Figure 3D Step 1031 shown can be implemented by the following steps 1031A to 1031C, as shown below. Figure 3E It should be noted that steps 1031A to 1031B are performed for each image sample belonging to the label type.

[0108] In step 1031A, based on the label type, a probability distribution label corresponding to a plurality of label types of the image sample is generated.

[0109] As an example, the label type to which an image sample belongs refers to the actual label type of the image sample. For example, the label type to which the image sample belongs is the first label type. The probability distribution label of the label type is represented using one-hot encoding. For example, if the label type to which the image sample belongs is the first label type, and the first label type corresponds to the data at the first position in the probability distribution label, then the probability distribution labels corresponding to multiple label types should be [1, 0, …, 0]. That is, based on the label type to which the image sample belongs and the data at the i-th position in the probability distribution label corresponding to the label type to which the image sample belongs, the data at the i-th position in the probability distribution label is set to 1, and the data at other positions are set to 0.

[0110] In step 1031B, the classification loss corresponding to the image sample is determined based on the first classification probabilities of the image sample corresponding to the multiple label types and the probability distribution labels corresponding to the multiple label types.

[0111] As an example, the classification loss corresponding to the image sample adopts the cross entropy loss. The calculation method of the classification loss corresponding to the image sample is shown in the following formula (8).

[0112]

[0113] in, is the classification loss corresponding to the image sample, c is the label type to which the image sample belongs, that is, the actual label type of the image sample, is the probability distribution label of multiple label types, is the first classification probability, that is Figure 3A In the example of step 102 shown, the union of multiple second classification probabilities is shown, where i is an image sample corresponding to the label type c.

[0114] In step 1031C, the classification losses corresponding to the plurality of image samples belonging to the label type are fused to obtain the classification loss corresponding to the label type.

[0115] As an example, the calculation method of the classification loss corresponding to the label type is shown in the following formula (9).

[0116]

[0117] in, is the classification loss corresponding to the label type, c is the label type to which the image sample belongs, that is, the actual label type of the image sample, and N is the number of training data, that is, image samples, corresponding to the label type c in all training data, that is, all image samples. In other words, formula (9) is the fusion process of the classification losses corresponding to all image samples with label type c, and the classification loss corresponding to label type c is obtained.

[0118] Through the embodiments of the present application, the first classification probability of each image sample is combined with the probability distribution label of the label type, so that the model not only learns the label type of the image sample itself, but also perceives the correlation between different label types and the data distribution characteristics. The classification loss corresponding to the label type effectively quantifies the difference between the first classification probability and the probability distribution label of the label type. By fusing the classification losses corresponding to multiple image samples, a more representative label type-level loss, that is, the classification loss corresponding to the label type, is ultimately obtained, which significantly improves the model's adaptability to imbalanced data distribution of image samples corresponding to the label type.

[0119] Continue to see Figure 3D In step 1032, based on the classification loss corresponding to the label type, the loss gradient corresponding to the label type is determined.

[0120] As an example, the calculation method for determining the loss gradient corresponding to the label type is shown in the following formula (10).

[0121]

[0122] in, is the loss gradient corresponding to the label type, c is the label type to which the image sample belongs, that is, the actual label type of the image sample. N is the number of training data, that is, image samples, corresponding to the label type c in all training data, that is, all image samples; is the classification loss corresponding to the label type.

[0123] Through the embodiments of this application, the model can perform differentiated learning based on the characteristics of different label types. By refining the loss calculation granularity to the label type level, it can more sensitively capture the learning difficulty and data distribution characteristics of each label type, which is particularly helpful for addressing the data distribution imbalance problem of image samples corresponding to the label type.

[0124] Continue to see Figure 3A In step 104, a first loss is determined based on the loss gradients corresponding to the plurality of label types.

[0125] In some embodiments, see Figure 3F , Figure 3A Step 104 shown can be implemented by the following steps 1041 to 1042, as shown below. Figure 3F Provide explanation.

[0126] In step 1041 , for each of the label types, a weight of the label type is determined based on the number of image samples of the label type.

[0127] In some embodiments, see Figure 3G , Figure 3F Step 1041 shown can be implemented by the following steps 1041A to 1041D, as shown below. Figure 3G Provide explanation.

[0128] In step 1041A, for each value of n, the historical prediction accuracy of the label type corresponding to the nth training round is obtained.

[0129] As an example, n is a positive integer not greater than N, the Nth round is the current round, and N is a positive integer. In each nth training round, the training data, i.e., image samples, used in training are part of all the training data, and the number of training data, i.e., image samples, used in each nth training round can be the same or different. When the number of training data, i.e., image samples, used in each nth training round is the same, the total number of all training data is divided by the number of training data, i.e., image samples, used in each nth training round, and the result is the total number of training rounds. When the number of training data, i.e., image samples, used in each nth training round is different, the number of training data, i.e., image samples, used in each nth training round is summed until the sum is equal to the total number of all training data, and the nth training round corresponding to the sum equal to the total number of all training data is taken as the last training round, thereby obtaining the total number of training rounds. The Nth round is the current round, and can be any training round from the first round to the last training round, 1≤n <N。

[0130] In some embodiments, Figure 3G Step 1041A shown can also be implemented by the following technical solution: when the value of n is 1, the prediction accuracy of the nth training round is used as the prediction accuracy of the nth training round; when the value of n is greater than 1, the prediction accuracy of the nth training round and the prediction accuracy of K training rounds before the nth training round are averaged to obtain the prediction accuracy of the nth training round, where K is a positive integer.

[0131] As an example, the historical prediction accuracy of the nth training round smoothly records the prediction accuracy of the most recent K training rounds. The calculation formula for the historical prediction accuracy of the nth training round is shown in the following formula (11).

[0132]

[0133] in, is the historical prediction accuracy of the nth training round, accuracy j is the prediction accuracy of each j-th training round in the K training rounds before the n-th training round, 1≤j≤n. When n is 1, the above formula (11) can be simplified. The simplified formula (11) can be found in the following formula (12).

[0134]

[0135] As an example, the prediction accuracy of the nth training round can be determined by the following formula (13).

[0136]

[0137] Among them, accuracy n is the prediction accuracy of the nth training round, a is the predicted label type of the image sample predicted by the image classification model, and c is the actual label type of the image sample. When the predicted label type and the actual label type are the same, the prediction accuracy is 100%; when the predicted label type and the actual label type are different, the prediction accuracy is 0.

[0138] In the embodiment of the present application, when n=1, the prediction accuracy of the nth training round is directly used as the prediction accuracy of the nth training round. As the number of training rounds increases, that is, when n>1, the prediction accuracy of the first training round to the prediction accuracy of the nth training round is smoothed. This design not only retains the sensitivity of the early training stage, but also effectively eliminates random fluctuations in the later training stage, and can obtain a more stable prediction accuracy of the nth training round.

[0139] In step 1041B, a first ratio between the number of image samples of the label type and the total number of image samples is determined.

[0140] As an example, the first ratio is the ratio of the number of training data corresponding to the nth training round, when the number of training data, i.e., image samples, is b and the label type of the image samples is c. The ratio of the number of training data, i.e., the first ratio, can suppress the deviation of the weight of the determined label type due to the fluctuation of the minimum number of training data, avoid the overfitting problem caused by the accidental distribution of the target label type in the current training data, and thus improve the generalization ability of the model for the target label type. The target label type is the label type for which the number of corresponding image samples in the label type is less than the quantity threshold. The calculation method of the first ratio is shown in the following formula (14).

[0141]

[0142] in, is the first ratio, num1 is all the training data used in model training, that is, the amount of data with label type c in the image samples, and num2 is the nth training round, and the corresponding training data, that is, the amount of data with label type c in the image samples when the number of training data, that is, image samples, is b.

[0143] In step 1041C, based on the historical prediction accuracy of the nth training round and the first ratio, a weight of the label type corresponding to the nth training round is determined.

[0144] As an example, the historical prediction accuracy of the nth training round is positively correlated with the weight of the label type of the nth training round, and the first ratio is negatively correlated with the weight of the label type of the nth training round. The weight of the label type of the nth training round is calculated as shown in the following formula (15).

[0145]

[0146] in, is the weight of the label type in the nth training round, where the label type is c. is the historical prediction accuracy of the nth training round, is the first ratio, ∈ is the minimum value, for example 1×10 -8 , by in the denominator Adding a minimum value ∈ in can ensure that the denominator It is not zero and meets the requirements of the fraction.

[0147] In step 1041D, the weights of the label types of N training rounds are fused to obtain the weights of the label types.

[0148] As an example, since the current training round is the Nth, it is necessary to fuse the weights of label type c corresponding to each training round from the 1st training round to the Nth training round to obtain the label type weight for label type c. The calculation method of the label type weight is shown in the following formula (16).

[0149]

[0150] Among them, w c is the weight of the label type for label type c, is the weight of the label type in the nth training round, and N is the current training round.

[0151] The embodiments of this application effectively improve the balance of model training and prediction accuracy. By calculating the weights of label types, label types with higher historical prediction accuracy are given greater weights, and the weights of label types with fewer image samples, i.e., target label types, are appropriately suppressed, avoiding overfitting of the model to label types with fewer image samples and alleviating the deviation caused by the imbalanced distribution of the number of image samples corresponding to the label types.

[0152] Continue to see Figure 3F In step 1042, based on the weight of each of the label types, the loss gradients corresponding to the multiple label types are fused to obtain the first loss.

[0153] As an example, the first loss considers that under each label type, the product of the weight of the label type and the loss gradient corresponding to the label type is positively correlated with the first loss. The calculation method of the first loss is shown in the following formula (17).

[0154]

[0155] Among them, M is the set of label types corresponding to all training data, i.e., image samples, c is the label type in the set M of label types corresponding to all training data, i.e., image samples, w c is the weight of the label type for label type c, It is the loss gradient corresponding to the label type when the label type is c, and L1 is the first loss.

[0156] The present invention effectively mitigates the impact of data imbalance of image samples with certain label types on model training, improving the generalization ability of the model. Label types with more image samples are given higher weights, preventing the model from overfitting to label types with fewer image samples, and improving the classification accuracy of the model.

[0157] Continue to see Figure 3A In step 105, a second loss is determined based on the loss gradient corresponding to the target label type and the loss gradient corresponding to each of the label types.

[0158] As an example, the target label type is a label type whose corresponding number of image samples is less than a number threshold.

[0159] In some embodiments, see Figure 3H , Figure 3A Step 105 shown can be implemented by steps 1051 to 1053 below. Figure 3H Provide explanation.

[0160] In step 1051, the loss gradients corresponding to the multiple target label types are averaged to obtain an average loss gradient.

[0161] As an example, the average gradient loss is the average of the loss gradients corresponding to multiple target label types. The calculation formula of the average loss gradient is shown in the following formula (18).

[0162]

[0163] in, is the average loss gradient, B minority is a set of target label types, including the identifier of each target label type. The target label type is the label type whose corresponding number of image samples is less than the number threshold. minorityis the number of target label types. Average loss gradient The loss gradient corresponding to all target label types The mean of minority , c is the set B of target label types minority The label type in .

[0164] In step 1052, the loss gradients corresponding to the multiple label types are summed to obtain the total loss gradient.

[0165] As an example, the sum of the loss gradients is obtained by summing the loss gradients corresponding to all label types. The calculation method of the sum of the loss gradients is shown in the following formula (19).

[0166]

[0167] Among them, c is the category, is the loss gradient corresponding to the label type when the label type is c, M is the set of label types corresponding to all training data, that is, image samples, and c is the label type in the set M of label types corresponding to all training data, that is, image samples. is the sum of loss gradients.

[0168] In step 1053, the second loss is generated based on the average loss gradient and the sum of the loss gradients.

[0169] In some embodiments, Figure 3H Step 1053 shown can be implemented by the following technical solution: mapping the average loss gradient based on a sign function to obtain a sign value of the average loss gradient, determining a first difference between the sum of the loss gradients and the sign value, and determining a second loss positively correlated with the first difference.

[0170] As an example, the second loss is determined by referring to the following formula (20).

[0171]

[0172] Among them, L adv For the second loss, is the total loss gradient, is the average loss gradient, is the first difference, ‖‖ 2 It is the square of the L2 norm, which is usually used to represent the square of the Euclidean distance of a vector. Assume that a vector is x={x1,x2,…,x n}, the specific calculation method of the square of the L2 norm of vector x is shown in the following formula (21).

[0173]

[0174] is the sign value of the average loss gradient, sign() is the sign function, and the sign function returns a discrete value representing the sign based on the positive or negative value of the input data. The mathematical definition of the sign function is shown in the following formula (22).

[0175]

[0176] Among them, z is the input data of the sign function sign(). Indicates that when hour, when hour, when hour,

[0177] Through the embodiments of the present application, the model's learning ability for label types with fewer image samples is effectively enhanced, so that the model not only pays attention to the numerical size of the loss gradient, but also emphasizes the directional consistency of the loss gradient, thereby reducing the optimization direction of label types with fewer image samples from being affected by the gradient of label types with more image samples.

[0178] The embodiments of this application utilize a gradient optimization strategy for the target label type, effectively alleviating the data imbalance problem of image samples corresponding to the label type and improving the model's ability to recognize label types with fewer image samples. During backpropagation, the gradient of the target label type can be strengthened to prevent it from being dominated by the gradient of the label type with more image samples.

[0179] Continue to see Figure 3A In step 106, the first loss and the second loss are fused to obtain a third loss.

[0180] As an example, the second loss needs to be adjusted and then fused with the first loss to obtain the third loss. The calculation formula of the adjusted second loss is shown in the following formula (23), and the calculation formula of the third loss is shown in the following formula (24).

[0181] L2=λL adv (twenty three)

[0182] L=L1+L2 (24)

[0183] Among them, L2 is the adjusted second loss, L advis the second loss, λ is a tuning parameter, a hyperparameter that controls the weight of the second loss, λ∈[0,1]; L is the third loss, and L1 is the first loss. By explicitly reducing the gradient direction conflict between high-frequency and low-frequency label types, the gradient of the high-frequency label type is prevented from dominating the model optimization direction, alleviating the model's bias towards high-frequency label types. Here, low-frequency label types refer to target label types, i.e., label types for which the number of corresponding image samples is less than the threshold. High-frequency label types refer to label types for which the number of corresponding image samples is greater than or equal to the threshold.

[0184] In step 107, the image classification model is updated based on the third loss to obtain an updated image classification model.

[0185] After obtaining the third loss, the model is typically updated using an optimization algorithm such as gradient descent. Its core goal is to adjust the model parameters to minimize the loss function, also known as the third loss. First, the gradient of the loss function with respect to the model parameters is calculated. The direction of the gradient indicates the direction of the fastest increase in the loss function, so parameter updates need to be performed in the opposite direction of the gradient. With batch gradient descent, each update is based on the average gradient of all training data, i.e., image samples. Stochastic gradient descent uses the gradient of a single training data point, resulting in more frequent updates but greater volatility. Mini-batch gradient descent offers a compromise between batch gradient descent and stochastic gradient descent, calculating the gradient using a small batch of training data, such as a preset number of training data points from all training data. This balances efficiency and stability. The learning rate, a hyperparameter, controls the step size of each update. Excessively large values ​​can lead to oscillation or divergence, while too small values ​​can slow convergence. Advanced optimizers such as Adaptive Moment Estimation (Adam) and Root Mean Square Propagation (RMSProp) dynamically adjust the learning rate or introduce a momentum term to accelerate convergence and avoid local minima. In each iteration, parameters are updated according to the rule "new parameters = old parameters - learning rate × gradient" until the third loss converges or the preset stopping condition is reached. This process is repeated repeatedly, ultimately allowing the model to approach the optimal solution for the training data, while relying on techniques such as regularization and early stopping to prevent overfitting.

[0186] Here, the Adaptive Moment Estimation (Adam) optimizer dynamically adjusts the learning rate of each parameter by calculating the first-order moment (mean) and the second-order moment (uncentered variance) of the gradient. The Root Mean Square Propagation (RMSProp) optimizer adaptively adjusts the learning rate by taking an exponentially weighted moving average of the squared gradients.

[0187] When training a model, regularization and early stopping are key techniques to prevent overfitting. The specific steps are as follows:

[0188] (1) Regularization: It can be L1 regularization or L2 regularization, and a penalty term for the parameter weight is added to the third loss. For example, L1 regularization uses the absolute value of the penalty term, and L2 regularization uses the sum of the squares of the penalty term. The intensity is controlled by hyperparameters to force the model to learn smaller weights and reduce sensitivity to noise.

[0189] (2) Early Stopping: Continuously test the performance of the validation set during the training process. The validation set is part of the training data in all the training data, which is used to verify the performance of the model training. For the embodiment of the present application, the validation set is used to verify whether the classification result of the model is accurate. When the validation index does not improve after several consecutive iterations, for example 10 times, that is, the classification effect does not improve, the training is stopped and rolled back to the optimal parameter state. Here, the classification effect corresponding to the next parameter state of the optimal parameter state is the same as or worse than the classification effect corresponding to the optimal parameter state. Early stopping is essentially an implicit regularization that prevents the model from overfitting the training data by limiting the number of training rounds. When applied in combination, regularization is usually configured first, and then training is dynamically terminated by early stopping.

[0190] Through the embodiment of the present application, image samples corresponding to each label type are obtained, and each of the image samples is classified through an image classification model to obtain a first classification probability of each of the image samples; the first classification probability of each image sample belonging to the label type can be obtained, providing data support for subsequent loss calculations. For each of the label types, the loss gradient corresponding to the label type is determined based on the first classification probability of the image samples belonging to the label type; determining the loss gradient corresponding to the label type can reflect the overall fitting of the model to the label type. The gradient direction indicates the parameter update direction, and the size of the loss gradient reflects the optimization difficulty. Through the loss gradient corresponding to the label type, it can be identified that the model has insufficient learning of certain label types, such as a small loss gradient, or overfitting, such as an abnormally large loss gradient, so as to adjust the optimization strategy in a targeted manner. Based on the loss gradients corresponding to multiple label types, the first loss is determined; it can be used to evaluate the classification performance of the model on multiple label types. Based on the loss gradient corresponding to the target label type and the loss gradient corresponding to each of the label types, a second loss is determined; wherein the target label type is a label type for which the number of corresponding image samples in the label type is less than a quantity threshold; by adding a second loss for the target label type, i.e., the low-frequency label type, the model is forced to pay more attention to the learning of the target label type, thereby alleviating the problem of data imbalance in the distribution of image samples corresponding to multiple label types. The sensitivity of the model to the target label type, i.e., the low-frequency label type, can be improved. The first loss and the second loss are fused to obtain a third loss, and the image classification model is updated based on the third loss to obtain an updated image classification model. The third loss obtained by fusing the first loss and the second loss can balance the performance of the model on low-frequency label types and high-frequency label types, so that after the update, the model can maintain the recognition accuracy of high-frequency label types while improving the recognition ability of low-frequency label types. Through the embodiments of the present application, the model can learn all label types equally when the data distribution of image samples corresponding to the label types is uneven, avoiding the high-frequency label types dominating the optimization direction of the model.

[0191] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0192] The audit business based on short videos requires a large amount of manual review. Due to the temporal characteristics of videos, auditors need to watch short videos for a long time to determine whether the short videos are compliant and classified. The classification includes 12 first-level categories and more than 60 second-level categories, resulting in low audit efficiency. The temporal nature of the video here refers to the temporal correlation characteristics between frames in the video data, that is, the dynamic information of the video content changing over time. It is the core feature that distinguishes video from static images. The embodiment of the present application can utilize the image understanding ability of the multimodal large model to identify the frames in the short video and obtain non-compliant video frames (original image samples) (video frames (original image samples) are image samples), and can directly and explicitly feedback the non-compliant video frames (original image samples) to the auditor, thereby improving the audit efficiency. The multimodal large model refers to a large-scale artificial intelligence model that can simultaneously understand and process multiple types of data such as text, images, audio, video, etc. The multimodal large model can achieve deep fusion and reasoning of cross-modal information such as text, images, audio, video, etc. through a unified architecture or joint training.

[0193] In the process of manual review of short videos, the embodiment of the present application uses a dedicated data set to supervise and fine-tune the multimodal large model, and modifies the loss function to prevent label tilt and reduced accuracy when the data distribution is unbalanced. At the same time, an adjustment strategy for dynamic video frames (original image samples) is proposed to ensure that longer short videos can be trained or inferred as much as possible under limited graphics card resources. Ultimately, it is possible to quickly identify risky video frames (original image samples) in short videos and quickly feed back to the reviewers, thereby improving review efficiency. Here, the dedicated data set is a labeled data set, and supervised fine-tuning is an optimization technology in machine learning for pre-trained models such as large language models, visual models, and the models provided by the embodiments of the present application. The model is further trained with labeled data so that the model can adapt to specific tasks or enhance specific capabilities. It is one of the key steps in transfer learning, and is widely used in the fields of natural language processing and computer vision.

[0194] With the rapid development of short video platforms, the amount of object-generated content has exploded, making short video review a critical component of platform operations. Traditional review methods rely primarily on manual review, requiring reviewers to watch short video content frame by frame to determine whether each frame (raw image sample) meets regulatory requirements. However, this method is inefficient and prone to misjudgments or omissions due to human error, making it impossible to achieve automated short video review.

[0195] In recent years, automated auditing technology has gradually emerged, using technologies such as artificial intelligence, machine learning, and natural language processing to perform image recognition, speech recognition, and natural language processing to automatically detect, filter, and review whether specific content complies with specific rules or standards. The model provided in the embodiments of this application can be used to perform preliminary screening of short video content. For example, image recognition technology can detect non-compliant video frames (original image samples) in short videos, and speech recognition technology is used to analyze sensitive words in the audio of short videos.

[0196] Traditional manual review methods rely on human reviewers to watch short videos frame by frame, which is not only time-consuming and labor-intensive, but also struggles to cope with the massive amount of content generated daily by short video platforms. With the exponential growth in the number of short videos uploaded by users, relying solely on manual review leads to inefficiencies, high costs, and technical issues. The accuracy of review conclusions can also be affected by human fatigue or subjective factors.

[0197] Related technologies do not perform well when faced with an unbalanced data distribution, which significantly affects the training effect and generalization ability of the model and easily leads to label skew, that is, the model has a higher recognition accuracy for high-frequency categories, but a poor recognition effect for low-frequency categories (target label types). In the short video content review scenario, non-compliant video frames (original image samples) usually only account for a very small proportion of all data, perhaps less than 1%, while normal video frames (original image samples) account for the vast majority. This extremely unbalanced data distribution will cause the model to be overly biased towards high-frequency categories, that is, normal video frames (original image samples), which is manifested as a significant drop in the recognition rate, that is, the recall rate, of low-frequency categories, that is, non-compliant video frames (original image samples). In addition, when processing short videos with longer durations, there will be a problem of insufficient video memory, which makes it impossible to train and infer the model efficiently.

[0198] The method provided in the embodiment of the present application includes the following contents:

[0199] 1. Dynamic video frame (raw image sample) adjustment strategy: By dynamically adjusting the number of image tokens for each video frame (raw image sample), the problem of video memory overflow can be effectively avoided, thereby processing long short videos with limited graphics card resources.

[0200] 2. Dynamic Adversarial Weight Loss: By dynamically adjusting category weights (label type weights) and introducing adversarial gradient directions (second loss), this approach alleviates label skew caused by imbalanced data distribution and improves the model's ability to recognize low-frequency categories (target label types). Low-frequency categories refer to categories whose corresponding video frames (raw image samples) are less than a threshold number across multiple categories. High-frequency categories refer to categories whose corresponding video frames (raw image samples) are greater than or equal to a threshold number across multiple categories.

[0201] 3. Application of multimodal large models: Utilize the image understanding capabilities of multimodal large models to quickly identify non-compliant video frames (original image samples) in short videos and directly provide feedback to reviewers, significantly improving review efficiency.

[0202] Through the method provided in the embodiment of the present application, the accuracy rate in the security review task of short videos can reach 90%, which increases the efficiency of reviewers by 25%, greatly reduces the workload of reviewers, and improves review efficiency.

[0203] See also Figure 4 , Figure 4 This is the short video review interface provided by the embodiment of the present application. When the reviewer opens the short video review interface, if there is a video frame (original image sample) in the short video that is judged to be non-compliant, the video frame (original image sample) will be highlighted and returned directly to the object. The reviewer can click on the large image to view the details of each image. At the same time, the category involved in the video frame (original image sample) is displayed to help the reviewer quickly read and locate the reason.

[0204] The method provided in the embodiment of the present application can solve the problem of video memory overflow when processing short videos of long duration in limited training resources. Video memory overflow means that when processing short videos of long duration, the video memory of the graphics card is exhausted, which causes the program to crash or report an error. The embodiment of the present application provides a dynamic image adjustment strategy, specifically: setting the total number of tokens of all video frames (original image samples) in the model training process to T. It is necessary to calculate the number of image tokens assigned to each video frame (original image sample), and calculate the number of image tokens assigned to each video frame (original image sample) based on the pixel ratio of each video frame (original image sample) to all video frames (original image samples). Assume that the pixel of the i-th video frame (original image sample) is P i (the number of pixels of the original image sample), the total number of pixels of all video frames (original image samples) is P total , the number of image tokens T assigned to the i-th video frame (original image sample) i The calculation method of is shown in the following formula (25).

[0205]

[0206] Among them, P i is the pixel of the i-th video frame (original image sample), P total (the total number of pixels of all original image samples) is the total number of pixels of all video frames (original image samples), T (the total number of image tokens of original image samples) is the total number of tokens of all video frames (original image samples) during model training, and T i(The number of image tokens assigned to the original image sample) is the number of image tokens assigned to the i-th video frame (original image sample).

[0207] Then the number of pixels of each token needs to be calculated. The calculation method of the number of pixels of each token is shown in the following formula (26).

[0208] num=patch×patch (26)

[0209] Among them, num is the number of pixels of each token (the number of pixels of the image token), patch is the number of local blocks into which the input video frame (original image sample) is divided, and is the basic unit for the model to process visual information.

[0210] Then calculate the new pixels of each video frame (original image sample), which needs to be based on the number of image tokens T assigned to the i-th video frame (original image sample) i The specific calculation method is shown in the following formula (27).

[0211] P′ i =T i ×num (27)

[0212] Where num is the number of pixels per token (the number of pixels in the image token), T i is the number of image tokens assigned to the i-th video frame (original image sample), P′ i (number of new pixels for original image samples) is the number of new pixels for each video frame (original image samples).

[0213] Finally, we need to calculate the new pixel P′ for each video frame (original image sample) i and pixel P i , the width W and height H of the video frame (original image sample) are scaled down in pixels to obtain a new width W′ and a new height H′ of the video frame (original image sample) (the new width W′ and the new height H′ correspond to the image samples of the original image sample). The calculation method of the new width W′ is shown in the following formula (28), and the calculation method of the new height H′ is shown in the following formula (29).

[0214]

[0215] Among them, P′ i is the number of new pixels per video frame (the number of new pixels in the original image sample), P i is the pixel of the i-th video frame (the number of pixels of the original image sample), sqrt() is the square root function, which means performing a square root operation on a number, that is, finding its quadratic root.

[0216] Through the above dynamic image adjustment strategy, the number of image tokens allocated to each video frame (original image sample) can be dynamically adjusted under limited video memory resources, and the width and height of the video frame (original image sample) can be proportionally reduced in pixels. The resulting reduced video frame (original image sample) can effectively avoid the problem of video memory overflow.

[0217] To address the label skew problem under imbalanced data distribution, this embodiment of the application also proposes a loss function design method based on dynamic adversarial weight loss adjustment, and improves the robustness of the model through the following two core mechanisms:

[0218] (1) Dynamically adjust the class weights (label type weights): Instead of using fixed global class weights, class weights should be dynamically adjusted based on the ratio of the current training data to the amount of training data used in each training round and the historical prediction accuracy. The class weights (label type weights) are calculated as shown in Formulas (30) and (31).

[0219]

[0220] Where c is the category of the video frame (original image sample) identified by the model, t is the training round, which is a basic unit in the training process, indicating that the model has completed one forward propagation and back propagation, and updated the model parameters based on the calculated loss function; ∈ is the minimum value, for example 1×10 -8 , by in the denominator Adding a minimum value ∈ in can ensure that the denominator It is not zero and meets the requirements of the fraction. is the historical prediction accuracy corresponding to the category of the current video frame (original image sample) identified by the model in the tth training round. is the ratio of the number of training data corresponding to the category c of the current video frame (original image sample) recognized by the model in the tth training round when the number of training data is b. In the tth training round, when the category of the current video frame (original image sample) recognized by the model is c, the category weight of the tth training round corresponding to category c. When the category of the current video frame (original image sample) recognized by the model is c, the category weight of the tth training round corresponding to the category of the current video frame (original image sample) from the 1st training round to the Tth training round is Perform summation to obtain the category weight w corresponding to the category c of the current video frame (original image sample) c .

[0221] The number of training rounds of the above model is positively correlated with the total number of training data used in model training, and negatively correlated with the number of training data corresponding to each training round in model training. The specific method for determining the number of training rounds is shown in the following formula (32).

[0222]

[0223] Where B is the total number of training data used in model training, b is the number of training data corresponding to each training round in model training, T is the total number of model training rounds, that is, the model needs to undergo T training rounds, the tth training round is the tth training round, and t is any value in [1, 2, …, t, …, T].

[0224] above is the historical prediction accuracy corresponding to the category c of the current video frame (original image sample) identified by the model in the tth training round. The historical prediction accuracy smoothly records the prediction accuracy of the current video frame (original image sample) identified by the model as category c in the most recent training rounds. The calculation formula of the historical prediction accuracy is shown in the following formula (33).

[0225]

[0226] in, is the historical prediction accuracy corresponding to the category c of the current video frame (original image sample) recognized by the model in the tth training round, N is the number of recent training rounds, and accuracy j It is the prediction accuracy when the category of the current video frame (original image sample) is c in each training round. When t is 1, that is, the current is the 1st training round, the historical prediction accuracy when the category of the current video frame (original image sample) recognized by the model is c is the prediction accuracy when the category of the current video frame (original image sample) is c. When t is any value greater than 1 and less than or equal to T, T is the total number of model training rounds, when the category of the current video frame (original image sample) recognized by the model in the tth training round is c, the corresponding historical prediction accuracy is the average of the prediction accuracy when the category of the current video frame (original image sample) recognized by the model in each training round from the 1st training round to the tth training round. The prediction accuracy when the category of the current video frame (original image sample) is c in the tth training round is accuracy t The calculation method of (the prediction accuracy of the tth training round) is shown in the following formula (34).

[0227]

[0228] Among them, a is the predicted category (predicted label type) of the current video frame (original image sample) when the category recognized by the model is c, and c is the actual category (actual label type) of the current video frame (original image sample). When the predicted category and the actual category are the same, the prediction accuracy is 100%; when the predicted category and the actual category are different, the prediction accuracy is 0.

[0229] above is the ratio of the number of training data points corresponding to the category c of the current video frame (original image sample) identified by the model in the tth training round, when the number of training data points is b. The ratio of the number of training data points can suppress the deviation of the determined category weights due to fluctuations in the number of minimum training data points, avoid overfitting caused by the accidental distribution of video frames (original image samples) of low-frequency categories in the current training data, and thus improve the model's generalization ability for low-frequency categories. The calculation method of the ratio of the number of training data points (first ratio) is shown in the following formula (35).

[0230]

[0231] in, is the ratio of the number of training data corresponding to the category c of the current video frame (original image sample) recognized by the model in the tth training round and the number of training data is b. num1 is the amount of data of category c in all the training data used in model training. num2 is the amount of data of category c in the training data corresponding to the tth training round and the number of training data is b.

[0232] (2) The adversarial gradient direction (second loss) is introduced and combined with the dynamic adjustment of the class weight (label type weight) to determine the dynamic adversarial weight loss (third loss). This can force the model to reduce the gradient conflict between low-frequency and high-frequency classes while optimizing. The first step is to calculate the standard cross entropy loss gradient for each class in all the training data used during model training. The formula for calculating the standard cross entropy loss gradient for each class (the loss gradient corresponding to the label type) is shown in the following formula (36).

[0233]

[0234] Among them, c (the label type to which the image sample belongs) is the category, (loss gradient corresponding to label type) is the standard cross entropy loss gradient of category c, N is the number of training data corresponding to category c in all training data, The probability distribution of the current video frame (original image sample) identified by the model when the category is c, that is, (First classification probability) is the probability distribution of the predicted category when the actual category of the video frame (original image sample) is c, (Probability distribution label of label type) is the probability distribution when the actual category of the current video frame (original image sample) is c. Here The probability distribution represented can be represented by one-hot encoding. One-hot encoding is a method for converting categorical data into numerical data. The core idea is to use a set of binary vectors with only 0 and 1 to represent the existence or non-existence of each category, 0 means the category does not exist, and 1 means the category exists. For example, [0,0,1,0] represents the third category, which means the actual category is c. is the classification loss corresponding to the label type.

[0235] The calculation formula for the standard cross entropy loss gradient (the sum of loss gradients) for all categories is shown in the following formula (37).

[0236]

[0237] Among them, c is the category, (loss gradient corresponding to label type) is the standard cross entropy loss gradient of category c, M is the set of categories corresponding to all training data, c is the category in the set M of categories corresponding to all training data, and the standard cross entropy loss gradient of all categories is (sum of loss gradients) is the standard cross entropy loss gradient for each category The harmony.

[0238] Secondly, it is necessary to calculate the mean cross entropy loss gradient (average loss gradient) of the low-frequency categories. The specific calculation method is shown in the following formula (38).

[0239]

[0240] in, is the mean gradient of the cross entropy loss of the low-frequency category, B minority is a collection of low-frequency categories, including the identifier of each low-frequency category, num minority is the number of low-frequency categories. The mean gradient of the cross entropy loss of low-frequency categories is the gradient of the standard cross entropy loss corresponding to all low-frequency categories The mean of minority , c is the set B of low-frequency categories minority Categories in .

[0241] Next, it is necessary to construct an adversarial gradient direction to alleviate the label tilt problem caused by data imbalance. The specific construction method of the adversarial gradient direction is shown in the following formula (39).

[0242]

[0243] Among them, L adv is the adversarial gradient direction (second loss), is the gradient of the standard cross entropy loss over all categories, is the mean gradient of the cross entropy loss for low-frequency categories, ‖‖ 2 It is the square of the L2 norm, which is usually used to represent the square of the Euclidean distance of a vector. Assume that a vector is x={x1,x2,…,x n}, the specific calculation method of the square of the L2 norm of vector x is shown in the following formula (40).

[0244]

[0245] is the sign value of the mean cross entropy loss gradient, that is, the average loss gradient. sign() is the sign function, which returns a discrete value representing the sign according to the positive or negative nature of the input data.

[0246] The mathematical definition of the sign function is given by the following formula (41).

[0247]

[0248] Among them, z is the input data of the sign function sign(). Indicates that when hour, when hour, when hour,

[0249] Finally, the dynamic adversarial weight loss of the model is determined based on the adversarial gradient direction and class weight. The calculation method of the dynamic adversarial weight loss is shown in the following formula (42).

[0250]

[0251] Among them, M is the set of categories corresponding to all training data, c is the category in the set M of categories corresponding to all training data, and w c is the category weight of category c, is the standard cross entropy loss gradient for category c, L adv is the adversarial gradient direction, λ is a tuning parameter, a hyperparameter that controls the weight of the adversarial gradient direction, λ∈[0,1]; and L is the dynamic adversarial weight loss. By explicitly reducing the gradient direction conflicts between high-frequency and low-frequency categories, we prevent the gradient of high-frequency categories from dominating the model's optimization direction, alleviating the model's bias toward high-frequency categories. The first loss.

[0252] After obtaining the dynamic adversarial weight loss, the model is typically updated using an optimization algorithm such as gradient descent. Its core goal is to adjust the model parameters to minimize the loss function, also known as the dynamic adversarial weight loss. First, the gradient of the loss function with respect to the model parameters is calculated. The direction of the gradient indicates the direction of the fastest increase in the loss function, so parameter updates need to be performed in the opposite direction of the gradient. With batch gradient descent, each update is based on the average gradient of all training data. Stochastic gradient descent uses the gradient of a single training data point, resulting in more frequent updates but greater volatility. Mini-batch gradient descent offers a compromise between batch gradient descent and stochastic gradient descent, calculating the gradient using a small batch of training data, such as a preset number of training data points from all training data. This balances efficiency and stability. The learning rate, a hyperparameter, controls the step size of each update. Excessively large values ​​can lead to oscillation or divergence, while too small values ​​can slow convergence. Advanced optimizers such as Adaptive Moment Estimation (Adam) and Root Mean Square Propagation (RMSProp) dynamically adjust the learning rate or introduce a momentum term to accelerate convergence and avoid local minima. In each iteration, parameters are updated according to the rule "new parameters = old parameters - learning rate × gradient" until the dynamic adversarial weight loss converges or reaches a preset stopping condition. This iterative process ultimately allows the model to approach the optimal solution for the training data, while relying on techniques such as regularization and early stopping to prevent overfitting.

[0253] Here, the Adaptive Moment Estimation (Adam) optimizer dynamically adjusts the learning rate of each parameter by calculating the first-order moment (mean) and the second-order moment (uncentered variance) of the gradient. The Root Mean Square Propagation (RMSProp) optimizer adaptively adjusts the learning rate by taking an exponentially weighted moving average of the squared gradients.

[0254] When training a model, regularization and early stopping are key techniques to prevent overfitting. The specific steps are as follows:

[0255] (1) Regularization: It can be L1 regularization or L2 regularization, and a penalty term for parameter weights is added to the dynamic adversarial weight loss. For example, L1 regularization uses the absolute value of the penalty term, and L2 regularization uses the sum of squares of the penalty term. The intensity is controlled by hyperparameters to force the model to learn smaller weights and reduce sensitivity to noise.

[0256] (2) Early Stopping: Continuously test the performance of the validation set during the training process. The validation set is part of the training data in all the training data, which is used to verify the performance of the model training. For the embodiment of the present application, the validation set is used to verify whether the classification result of the model is accurate. When the validation index does not improve after several consecutive iterations, for example 10 times, that is, the classification effect does not improve, the training is stopped and rolled back to the optimal parameter state. The classification effect corresponding to the next parameter state of the optimal parameter state is the same as or worse than the classification effect corresponding to the optimal parameter state. Early stopping is essentially an implicit regularization, which prevents the model from overfitting the training data by limiting the number of training rounds. When applied in combination, regularization is usually configured first, and then the training is dynamically terminated by early stopping.

[0257] The model obtained by the model training method provided in the embodiment of the present application can use dynamic video frame (original image sample) adjustment strategies to train and apply the model to videos corresponding to more than a thousand video frames (original image samples) within limited graphics card resources, such as 4*910B. Furthermore, the dynamic adversarial weight loss can enable the model to appropriately fit training data corresponding to low-frequency categories. Compared to using only cross-entropy loss, as shown in Table 1 below, the model obtained by the model training method provided in the embodiment of the present application can significantly improve the number of classifications when processing training data corresponding to low-frequency categories.

[0258] Table 1. Comparison of the number of classifications using only cross entropy loss and the number of classifications using dynamic adversarial weight loss.

[0259]

[0260] The following continues to describe the exemplary structure of the model training device 255 provided in the embodiment of the present application as a software module. In some embodiments, see Figure 2 , the software modules stored in the model training device 255 of the memory 250 may include:

[0261] The classification module 2551 is used to obtain image samples corresponding to each label type, and classify each image sample using an image classification model to obtain a first classification probability for each image sample;

[0262] a loss gradient determining module 2552 for determining, for each label type, a loss gradient corresponding to the label type based on a first classification probability of image samples belonging to the label type;

[0263] A first loss determination module 2553 is configured to determine a first loss based on loss gradients corresponding to the plurality of label types;

[0264] A second loss determination module 2554 is configured to determine a second loss based on a loss gradient corresponding to a target label type and a loss gradient corresponding to each of the label types; wherein the target label type is a label type for which the number of corresponding image samples is less than a quantity threshold.

[0265] The third loss determination module 2555 is used to fuse the first loss and the second loss to obtain a third loss, and update the image classification model based on the third loss to obtain an updated image classification model.

[0266] In some embodiments, the second loss determination module 2554 is further used to average the loss gradients corresponding to multiple target label types to obtain an average loss gradient; sum the loss gradients corresponding to multiple label types to obtain a total loss gradient; and generate the second loss based on the average loss gradient and the total loss gradient.

[0267] In some embodiments, the second loss determination module 2554 is also used to map the average loss gradient based on a sign function to obtain a sign value of the average loss gradient; determine a first difference between the sum of the loss gradients and the sign value, and determine a second loss that is positively correlated with the first difference.

[0268] In some embodiments, the first loss determination module 2553 is further used to determine the weight of each label type based on the number of image samples of the label type; based on the weight of each label type, the loss gradients corresponding to multiple label types are fused to obtain the first loss.

[0269] In some embodiments, the first loss determination module 2553 is further used to obtain, for each value of n, the historical prediction accuracy of the label type corresponding to the nth training round, wherein n is a positive integer not greater than N, the Nth round is the current round, and N is a positive integer; determine a first ratio between the number of image samples of the label type and the total number of image samples; determine the weight of the label type corresponding to the nth training round based on the historical prediction accuracy of the nth training round and the first ratio; wherein the historical prediction accuracy of the nth training round is positively correlated with the weight of the label type of the nth training round, and the first ratio is negatively correlated with the weight of the label type of the nth training round; and fuse the weights of the label types of N training rounds to obtain the weight of the label type.

[0270] In some embodiments, the first loss determination module 2553 is also used to, when the value of n is 1, use the prediction accuracy of the nth training round as the historical prediction accuracy of the nth training round; when the value of n is greater than 1, average the prediction accuracy of the nth training round and the prediction accuracy of the K training rounds before the nth training round to obtain the historical prediction accuracy of the nth training round.

[0271] In some embodiments, the loss gradient determination module 2552 is further used to determine the classification loss corresponding to the label type based on the first classification probability of the image samples belonging to the label type; and determine the loss gradient corresponding to the label type based on the classification loss corresponding to the label type.

[0272] In some embodiments, the loss gradient determination module 2552 is further used to perform the following processing for each image sample belonging to the label type: based on the label type, generate a probability distribution label for the image sample corresponding to multiple label types; based on the first classification probability of the image sample corresponding to multiple label types and the probability distribution label corresponding to multiple label types, determine the classification loss corresponding to the image sample; and fuse the classification losses corresponding to multiple image samples belonging to the label type to obtain the classification loss corresponding to the label type.

[0273] In some embodiments, the classification module 2551 is also used to obtain the original image sample corresponding to each of the label types, and perform the following processing on each of the original image samples: based on the number of pixels of the original image sample, determine the number of tokens of the image token assigned to the original image sample; based on the number of tokens, resize the original image sample to obtain an image sample corresponding to the original image sample.

[0274] In some embodiments, the classification module 2551 is also used to determine a second ratio that is positively correlated with the number of pixels of the original image sample and negatively correlated with the total number of pixels of all the original image samples; and multiply the second ratio by the total number of image tokens of all the original image samples to obtain the number of image tokens assigned to the original image sample.

[0275] In some embodiments, the classification module 2551 is also used to multiply the number of pixels of the image token by the number of tokens to obtain the new number of pixels of the original image sample; based on the ratio between the new number of pixels of the original image sample and the number of pixels of the original image sample, the width and height of the original image sample are proportionally reduced to obtain an image sample corresponding to the original image sample.

[0276] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the model training method described in the present invention.

[0277] An embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the model training method provided by the embodiment of the present application.

[0278] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0279] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0280] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0281] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0282] In summary, through the embodiments of the present application, combined with the dynamic allocation of the number of image tokens and adaptive size adjustment, the original image samples are reduced to image samples, reducing the video memory usage and computing load of the image samples. Image samples corresponding to each label type are obtained, and each of the image samples is classified through the image classification model to obtain the first classification probability of each image sample; the first classification probability of each image sample belonging to the label type can be obtained, providing data support for subsequent loss calculations. For each label type, based on the first classification probability of the image samples belonging to the label type, the loss gradient corresponding to the label type is determined; determining the loss gradient corresponding to the label type can reflect the overall fit of the model to the label type. The gradient direction indicates the parameter update direction, and the loss gradient size reflects the optimization difficulty. Through the loss gradient corresponding to the label type, it is possible to identify the model's insufficient learning of certain label types, such as a small loss gradient, or overfitting, such as an abnormally large loss gradient, so as to adjust the optimization strategy in a targeted manner. Refining the loss calculation granularity to the label type level can more sensitively capture the learning difficulty and data distribution characteristics of each label type, which is particularly beneficial for dealing with the problem of unbalanced data distribution of image samples corresponding to the label type. A first loss is determined based on the loss gradients corresponding to the multiple label types. This can be used to evaluate the model's classification performance on multiple label types, with label types with more image samples receiving higher weights. This prevents the model from overfitting to label types with fewer image samples, thereby improving the model's classification accuracy. A second loss is determined based on the loss gradient corresponding to the target label type and the loss gradients corresponding to each of the label types. The target label type is a label type for which the number of image samples is less than a threshold. By adding a second loss for the target label type, i.e., the low-frequency label type, the model is forced to focus more on learning the target label type, alleviating the imbalance in the data distribution of image samples corresponding to multiple label types. This can improve the model's sensitivity to the target label type, i.e., the low-frequency label type. The first and second losses are fused to obtain a third loss, and the image classification model is updated based on the third loss to obtain an updated image classification model. The third loss, obtained by fusing the first and second losses, balances the model's performance on low-frequency and high-frequency label types, allowing the updated model to maintain recognition accuracy for high-frequency label types while improving its recognition of low-frequency label types. Through the embodiments of the present application, the model can learn all label types equally when the data distribution of image samples corresponding to the label types is uneven, thereby avoiding high-frequency label types dominating the optimization direction of the model.

[0283] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Obtaining an image sample corresponding to each label type, and classifying each image sample using an image classification model to obtain a first classification probability for each image sample; For each of the label types, determining a loss gradient corresponding to the label type based on a first classification probability of image samples belonging to the label type; Determining a first loss based on loss gradients corresponding to the plurality of label types; Determining a second loss based on the loss gradient corresponding to the target label type and the loss gradient corresponding to each of the label types; The target label type is a label type whose corresponding number of image samples is less than a quantity threshold among the label types; The first loss and the second loss are fused to obtain a third loss, and the image classification model is updated based on the third loss to obtain an updated image classification model.

2. The method according to claim 1, characterized in that The determining the second loss based on the loss gradient corresponding to the target label type and the loss gradient corresponding to each of the label types includes: Averaging the loss gradients corresponding to the multiple target label types to obtain an average loss gradient; Summing the loss gradients corresponding to the multiple label types to obtain the sum of the loss gradients; The second loss is generated based on the average loss gradient and the sum of the loss gradients.

3. The method according to claim 2, characterized in that Generating the second loss based on the average loss gradient and the sum of the loss gradients includes: Mapping the average loss gradient based on a sign function to obtain a sign value of the average loss gradient; A first difference between the loss gradient sum and the symbol value is determined, and a second loss positively correlated with the first difference is determined.

4. The method according to claim 1, wherein The determining the first loss based on the loss gradients corresponding to the plurality of label types includes: For each of the label types, determining a weight of the label type based on the number of image samples of the label type; Based on the weight of each of the label types, the loss gradients corresponding to the multiple label types are fused to obtain the first loss.

5. The method according to claim 4, characterized in that The determining the weight of the label type based on the number of image samples of the label type includes: For each value of n, obtain the historical prediction accuracy of the tag type corresponding to the nth training round, where n is a positive integer not greater than N, the Nth round is the current round, and N is a positive integer; determining a first ratio between the number of image samples of the label type and the total number of image samples; Determining a weight of the label type corresponding to the nth training round based on the historical prediction accuracy of the nth training round and the first ratio; The historical prediction accuracy of the nth training round is positively correlated with the weight of the label type of the nth training round, and the first ratio is negatively correlated with the weight of the label type of the nth training round; The weights of the label types of N training rounds are fused to obtain the weights of the label types.

6. The method according to claim 5, characterized in that The obtaining of the historical prediction accuracy of the tag type corresponding to the nth training round includes: When the value of n is 1, the prediction accuracy of the nth training round is used as the historical prediction accuracy of the nth training round; When the value of n is greater than 1, the prediction accuracy of the nth training round and the prediction accuracy of the K training rounds before the nth training round are averaged to obtain the historical prediction accuracy of the nth training round, where K is a positive integer.

7. The method according to claim 1, characterized in that The determining, based on the first classification probability of the image samples belonging to the label type, a loss gradient corresponding to the label type, includes: Determining a classification loss corresponding to the label type based on a first classification probability of the image sample belonging to the label type; Based on the classification loss corresponding to the label type, a loss gradient corresponding to the label type is determined.

8. The method according to claim 7, characterized in that The determining, based on the first classification probability of the image samples belonging to the label type, the classification loss corresponding to the label type includes: For each image sample belonging to the label type, the following processing is performed: Based on the label type, generating a probability distribution label corresponding to a plurality of label types of the image sample; Determining a classification loss corresponding to the image sample based on first classification probabilities of the image sample corresponding to the plurality of label types and probability distribution labels corresponding to the plurality of label types; The classification losses corresponding to the plurality of image samples belonging to the label type are fused to obtain the classification loss corresponding to the label type.

9. The method according to claim 1, characterized in that The obtaining of image samples corresponding to each label type includes: Obtain an original image sample corresponding to each of the label types, and perform the following processing on each of the original image samples: Determining the number of image tokens to be allocated to the original image sample based on the number of pixels of the original image sample; Based on the number of tokens, the original image sample is resized to obtain an image sample corresponding to the original image sample.

10. The method according to claim 9, characterized in that The determining, based on the number of pixels of the original image sample, the number of image tokens allocated to the original image sample comprises: determining a second ratio that is positively correlated with the number of pixels of the original image sample and negatively correlated with the total number of pixels of all the original image samples; The second ratio is multiplied by the total number of image tokens of all the original image samples to obtain the number of image tokens allocated to the original image samples.

11. The method according to claim 9, characterized in that The resizing process of the original image sample based on the number of tokens to obtain an image sample corresponding to the original image sample includes: multiplying the number of pixels of the image token by the number of tokens to obtain a new number of pixels of the original image sample; Based on the ratio between the number of new pixels of the original image sample and the number of pixels of the original image sample, the width and height of the original image sample are proportionally reduced to obtain an image sample corresponding to the original image sample.

12. A model training device, characterized in that: The device comprises: A classification module is used to obtain image samples corresponding to each label type, and classify each image sample using an image classification model to obtain a first classification probability for each image sample; a loss gradient determination module, configured to determine, for each label type, a loss gradient corresponding to the label type based on a first classification probability of image samples belonging to the label type; A first loss determination module, configured to determine a first loss based on loss gradients corresponding to the plurality of label types; A second loss determination module is configured to determine a second loss based on a loss gradient corresponding to a target label type and a loss gradient corresponding to each of the label types; wherein the target label type is a label type for which the number of corresponding image samples is less than a quantity threshold; A third loss determination module is used to fuse the first loss and the second loss to obtain a third loss, and to update the image classification model based on the third loss to obtain an updated image classification model.

13. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the method according to any one of claims 1 to 11 when executing the computer-executable instructions or computer program stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images