Methods, apparatus, and computer program products for training image classification models

Abstract

Embodiments of this disclosure relate to methods, apparatus, and computer program products for training image classification models. The method includes determining a first classification result of the image classification model for pixels of a sample image in a first training iteration, wherein the first classification result indicates that each pixel belongs to one of a plurality of object categories. The method also includes determining a second classification result of the image classification model for pixels of a sample image in a second training iteration. The method further includes determining a first set of recall and a second set of recall for the plurality of object categories based on the first and second classification results. The method also includes adjusting the image classification model based on the first and second sets of recall to obtain a trained image classification model. This method enables dynamic adjustment of the weights of minor classes in image categories, better balancing the accuracy of minor and major classes, and reducing computational costs.

Description

Technical Field

[0001] Embodiments of this disclosure generally relate to the field of image processing, and more specifically to methods, apparatus, and computer program products for training image classification models. Background Technology

[0002] In recent years, with the development of computer and network technologies, people often transfer some computing tasks to edge devices on the network in order to provide services to users more quickly.

[0003] Furthermore, with the development of artificial intelligence, computer vision technology is also evolving rapidly. Computer vision tasks involve processing acquired images or videos to identify information within them. These tasks primarily include image classification, object detection, object tracking, and image segmentation. As edge devices become capable of handling more tasks, some computer vision processing tasks are being moved to them. While significant progress has been made in utilizing edge devices for computer vision tasks, many challenges remain to be addressed. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, and computer program product for training an image classification model.

[0005] According to a first aspect of this disclosure, a method for training an image classification model is provided. The method includes determining a first classification result of the image classification model for pixels of a sample image in a first training iteration, the first classification result indicating that each pixel belongs to one of a plurality of object categories. The method also includes determining a second classification result of the image classification model for pixels of the sample image in a second training iteration. The method further includes determining a first set of recall and a second set of recall for the plurality of object categories based on the first and second classification results. The method also includes adjusting the image classification model based on the first and second sets of recall to obtain a trained image classification model.

[0006] According to a second aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processor; and a memory coupled to the at least one processor and having instructions stored thereon, the instructions causing the device to perform actions when executed by the at least one processor, the actions including: determining a first classification result of an image classification model for pixels of a sample image in a first training iteration, the first classification result indicating that each pixel belongs to one of a plurality of object categories; determining a second classification result of the image classification model for pixels of the sample image in a second training iteration; determining a first set of recall and a second set of recall for the plurality of object categories based on the first and second classification results; and adjusting the image classification model based on the first and second set of recall to obtain a trained image classification model.

[0007] According to a third aspect of this disclosure, a computer program product is provided, which is tangibly stored on a non-volatile computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform the steps of the methods in the first or second aspect of this disclosure. Attached Figure Description

[0008] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0009] Figure 1 The illustration shows a schematic diagram of an example environment 100 in which the devices and / or methods of embodiments of the present disclosure may be implemented;

[0010] Figure 2 The illustration shows a flowchart of a method 200 for training an image classification model according to an embodiment of the present disclosure;

[0011] Figure 3 A flowchart illustrating a method 300 for adjusting a loss function according to an embodiment of the present disclosure is shown;

[0012] Figure 4 The illustration shows a schematic diagram of an example 400 for training an image classification model according to an embodiment of the present disclosure;

[0013] Figure 5 A schematic block diagram of an example device 500 suitable for implementing embodiments of the present disclosure is shown.

[0014] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] As mentioned above, computer vision tasks include semantic segmentation and image classification. These tasks are typically implemented using convolutional neural networks (CNNs) from the deep learning domain. In a CNN, a feature extractor with a set of convolutional layers is usually configured, followed by pooling layers to make the image less sensitive to small transformations and to preserve its integrity. The final stage of the CNN model is the classifier, which classifies each pixel into one of several categories. However, using CNNs to handle computer vision tasks requires significant computational resources.

[0018] For example, in image processing scenarios such as autonomous driving, it is sometimes necessary to utilize edge devices for inference of image classification models. Therefore, some collaborative computing frameworks are used to realize computation on edge devices. However, these solutions suffer from problems such as bandwidth limitations and latency.

[0019] Furthermore, imbalanced datasets are a significant problem in many computer vision tasks. In semantic segmentation, for example, imbalance is naturally occurring, with different classes exhibiting varying degrees of importance. For instance, in an outdoor driving segmentation dataset, lampposts and pedestrians are often considered minor classes compared to larger classes like buildings, sky, and roads, which typically occupy a larger proportion of the image. These minor classes are often considered more important than the larger classes for safety reasons. When presented with imbalanced datasets, standard loss functions, such as cross-entropy loss, often produce unsatisfactory results because the training process naturally favors larger classes, leading to lower accuracy and precision for minor classes.

[0020] To address at least the aforementioned and other potential problems, embodiments of this disclosure propose a method for training an image classification model. In this method, a computing device uses the image classification results from previous training iterations to determine the recall corresponding to each of a plurality of object categories. The two sets of recall from the previous two iterations are then used to adjust the image classification model to be used currently. This method allows for dynamic adjustment of small weights to better balance the accuracy of minor and major classes, and reduces computational costs, enabling deployment on edge devices.

[0021] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings, wherein... Figure 1 An example environment 100 is shown in which the devices and / or methods of embodiments of this disclosure may be implemented.

[0022] The environment 100 includes a computing device 104, which is used to train an image classification model 106. Figure 1 A computing device is shown in the illustration, but it is merely an example and not a specific limitation of this disclosure. Those skilled in the art can set the number of computing devices for training image classification models as needed.

[0023] Examples of computing device 104 include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0024] Computing device 104 is configured to receive one or more images 102. In some embodiments, image 102 may be a sample image used to train image classification model 106. In some embodiments, image 102 may be a test image used to test image classification model 106. In some embodiments, image 102 includes both sample images used to train image classification model 106 and test images used to test image classification model 106. The above examples are merely illustrative of this disclosure and are not intended to limit its specific scope.

[0025] Image classification model 106 is used to classify each pixel in image 102 into different object categories. For example, if the image is an image of the environment surrounding a vehicle in autonomous driving, each pixel in the image can be classified into one of different object categories, such as a lamppost, a pedestrian, or a building. In some embodiments, the image classification model is a machine model. Further, the image classification model is a neural network model, such as a convolutional neural network model. The above examples are merely for describing this disclosure and are not intended to limit the specific scope of the disclosure.

[0026] When training the image classification model 106, the computing device 104 initializes a loss function for that model, including the weights for each object category. Then, iteratively trains the image classification model using image 102. In each iteration, the parameters of the image classification model are adjusted using the loss function.

[0027] During the training of the image classification model, the weights of the loss function are dynamically adjusted after at least one iteration. The computing device 104 calculates the recall 110 for each object category using the classification results 108 from the previous iteration. The recall is then used to determine the candidate object category corresponding to the weight coefficients that need adjustment. The weights of the loss function corresponding to that candidate object category are then adjusted. The weighted loss function is then applied to subsequent iterations of training the image classification model.

[0028] This method allows for dynamic adjustment of the weights of subcategories to better balance the accuracy of subcategories and major categories, while also reducing computational costs, thus enabling deployment on edge devices.

[0029] The above combination Figure 1 A block diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is described. The following is in conjunction with… Figure 2 A flowchart describing a method 200 for training an image classification model according to embodiments of the present disclosure is provided. Method 200 can be performed in... Figure 1 The computing device at location 104 executes.

[0030] At box 202, computing device 104 determines a first classification result of the image classification model for the pixels of the sample image in the first training iteration. The first classification result indicates that each pixel belongs to one of multiple object categories.

[0031] In some embodiments, before running the image classification model 106, the computing device 104 initializes the weights of the loss function of the image classification model. The loss function for the image classification model is formed by combining the loss function value for each category with its corresponding weight. For example, the loss function of an image classification model that classifies two object categories A and B is shown in equation (1) below:

[0032] L=W A L(A)+W B L(B) (1)

[0033] Among them W A W represents the weight for object category A. BLet L(A) and L(B) represent the weights for object class B, where L(A) and L(B) represent the loss values ​​for class A and object class B, respectively. In one example, the loss function is the cross-entropy loss function. In another example, the loss function is the KL divergence loss function. The above examples are merely for describing this disclosure and are not intended to limit the specific scope of this disclosure. Those skilled in the art can set the required loss function as needed.

[0034] When initializing the loss function, the weights for each object category can be determined based on the number of object categories. For example, if an image classification model can classify an image into N categories, the initial weight corresponding to each category can be set to 1 / N. This method allows for quick initialization of the loss function.

[0035] After setting the initial loss function weights, the image classification model can be used to iteratively process the sample images. In each training iteration, the image classification model calculates the classification result for each pixel in the image.

[0036] At box 204, computing device 104 determines a second classification result for the pixels of the sample image in the second training iteration of the image classification model. In one example, the first and second training iterations are two consecutive training iterations. In another example, the first and second training iterations are iterations separated by a predetermined number of iterations. The above examples are merely for describing this disclosure and are not intended to specifically limit this disclosure.

[0037] At box 206, computing device 104 determines a first set of recall rates and a second set of recall rates for multiple object categories based on the first classification result and the second classification result, respectively. The first set of recall rates is determined using the first classification result, and the second set of recall rates is determined using the second classification result. To ensure that minor classes are handled well during training, computing device 104 dynamically adjusts the weight coefficients in the loss function using the classification results obtained during training iterations. During this adjustment process, computing device 104 performs a predetermined number of iterations on the image classification model and then obtains the classification results from the previous two iterations. In one example, the predetermined number of iterations is 5. In another example, the predetermined number of iterations is 8. The above examples are merely for describing this disclosure and are not intended to limit the specific scope of this disclosure. Those skilled in the art can set the number of iterations required before adjusting the weights of the loss function as needed.

[0038] After a predetermined number of training iterations, the recall rate corresponding to each of the multiple object categories is obtained using the results from the first iteration in the two adjacent previous iterations to form a first set of recall rates. Simultaneously, the classification results from the second iteration in the two previous iterations are also used to obtain the recall rate corresponding to each of the multiple object categories to form a second set of recall rates.

[0039] At box 208, computing device 104 adjusts the image classification model based on a first set of recall rates and a second set of recall rates to obtain a trained image classification model. The computing device uses the obtained two sets of recall rates to adjust the image classification model.

[0040] In some embodiments, the computing device uses a first set of recall rates and a second set of recall rates to adjust the loss function of an image classification model. In one example, the computing device determines multiple recall distances for multiple object categories based on the first set of recall rates and the second set of recall rates, where the recall distance refers to the difference between two recall rates. For example, for each of the multiple categories, the recall distance corresponding to that category is obtained by subtracting the recall rate corresponding to that category from the recall rate corresponding to that category in the first set of recall rates.

[0041] Then, the computing device 104 adjusts the loss function using multiple recall distances. The process of adjusting the loss function using multiple recall distances will be explained below. Figure 3 The following describes the process. After adjusting the loss function data, the adjusted loss function is used to continue training the image classification model. This method allows for rapid adjustment of the loss function and a small increase in the weights.

[0042] In some embodiments, the computing device 104 may also select at least one recall distance with a smaller recall distance from a plurality of recall distances, for example, less than a certain distance threshold. Then, the object category corresponding to the at least one recall distance is determined. Subsequently, image data corresponding to the at least one object category is removed from the sample images. At this point, training is performed using the sample images with the image data removed. This method reduces the amount of data processed by the image classification model and improves the processing speed.

[0043] Specifically, in one example, the computing device 104 determines a second recall distance that is less than a second threshold distance from a plurality of recall distances. For example, the computing device 104 selects the smallest recall distance from the plurality of recall distances and can determine the object category corresponding to the smallest recall distance. The computing device then determines the second object category corresponding to the second recall distance. The computing device removes pixels from the sample image that correspond to the second object category to adjust the sample image. The adjusted sample image is then input into the image classification model.

[0044] In another example, the computing device determines multiple recall distances below a certain threshold from multiple recall rates, and determines multiple categories corresponding to these multiple recall distances. The computing device then determines multiple object categories corresponding to the multiple recall distances. The computing device adjusts the sample image by removing pixels from the sample image that correspond to the multiple object categories. The computing device then inputs the adjusted sample image into an image classification model. The above examples are merely illustrative of this disclosure and not intended to limit its specific scope.

[0045] This method allows for dynamic adjustment of the weights of subcategories to better balance the accuracy of subcategories and major categories, while also reducing computational costs, thus enabling deployment on edge devices.

[0046] The above combination Figure 2 A flowchart of a method 200 for training an image classification model according to embodiments of the present disclosure is described below. Figure 3 A flowchart describing a method 300 for adjusting a loss function according to embodiments of the present disclosure is provided. Method 300 can be performed in... Figure 1 The computing device at location 104 executes.

[0047] At box 302, computing device 104 determines a first recall distance that is greater than a first threshold distance from a plurality of recall distances. For example, computing device 104 determines the largest recall distance from a plurality of recall distances.

[0048] At box 304, computing device 104 determines a first object category corresponding to a first recall distance. This first object category typically corresponds to a minor class in the image. Then, at box 306, computing device 104 adjusts the weights in the loss function associated with the first object category.

[0049] In one example, the weights of the loss function need to be adjusted during the i-th training iteration. If the object category u corresponding to the first recall distance is determined using the calculation results of the i-1 and i-2 iterations, the computing device uses the following equation (2) to adjust the weights corresponding to the object category u to be used in the i-th iteration.

[0050] W i，u =W i-1，u +β (2)

[0051] Among them W i-1，u Let β represent the weight for object category u in the (i-1)th iteration, and let β be an adjustment factor that can be set to any suitable range. In one example, β is 0.2 or 0.5. The above example is for illustrative purposes only and is not intended to limit the scope of this disclosure.

[0052] The above combination Figure 3 A flowchart is described for a method 300 for adjusting a loss function according to an embodiment of the present disclosure.

[0053] The following example details the process of determining the object categories whose weights need to be adjusted and the object categories whose image data needs to be removed. For instance, before the i-th training iteration, the recall rates from the (i-1)-th and (i-2)-th iterations are used, combined with the following equation (3), to determine the set D, which includes the maximum recall distance and the minimum recall distance:

[0054] D = h j∈[1：n] (M i-1，j M i-2，j (3)

[0055] Where n is the number of object categories, M i-1，j M represents the recall rate in the (i-1)th iteration corresponding to the j-th class of objects. i-2，j Let h represent the recall rate in the (i-2)th iteration corresponding to the j-th class of objects, and let h be expressed as equations (4) and (5):

[0056] Candidate(class) u =Argmax j∈[1：n] ((M i-1，1 -M i-2，1 ), ..., (M i-1，n -M i-2，n (4)

[0057] Candidate(class) v =Argmin j∈[1：n] ((M i-1，1 -M i-2，1 ), ..., (M i-1，n -M i-2，n (5)

[0058] In this function, `Argmax()` selects the maximum value, `Argmin()` selects the minimum value, `i` represents the current iteration number, `n` represents the number of object categories, `u` represents the class corresponding to the maximum value, `v` represents the class corresponding to the minimum value, and `Candidate(class)`... u And Candidate(class) v This represents the maximum and minimum values ​​of the recall distance obtained.

[0059] After determining categories u and v, the computing device adjusts the weights of the object category u corresponding to the maximum value obtained in the loss function, and removes the image data of the object category v corresponding to the minimum value from the training images for subsequent calculations.

[0060] After each training iteration of the image classification model, it is also necessary to determine whether to stop the training process to complete the training based on the loss function value or the number of iterations.

[0061] In some embodiments, the computing device 104 may determine whether to stop training based on the loss function value. For example, in each training iteration, the computing device 104 applies the image classification model to a sample image to determine a first loss function value for that sample image. Furthermore, after at least one training iteration, the computing device 104 also applies the image classification model at that time to a test image to determine a second loss function value for that test image. The computing device 104 then determines the difference between the first and second loss function values. If the difference is determined to be less than a threshold, for example, 0.1, training stops. If the difference is greater than or equal to the threshold, training continues.

[0062] In some embodiments, the computing device 104 may also determine whether to stop training by the number of consecutive training iterations in which the second loss function value for the test image remains unchanged. For example, the computing device 104 determines the number of consecutive training iterations in which the second loss function value for the test image remains unchanged. This number is then compared to a first threshold number, for example, 20 times. If the first threshold number is exceeded, training stops. If it is not exceeded, training continues.

[0063] In some embodiments, the computing device 104 may also determine whether to stop training based on the total number of training iterations. Specifically, the computing device 104 first determines the total number of training iterations for the classification model. Then, the computing device 104 compares the total number of iterations with a second threshold number, for example, the second threshold number is 250. Additionally, the second threshold number is greater than a first threshold number. If it is determined that the total number of iterations exceeds the second threshold number, training stops. If it does not exceed the second threshold number, training continues.

[0064] In some embodiments, the computing device 104 may combine the above three methods to determine whether to stop training. For example, after each iteration, the computing device 104 may determine whether the difference between the first loss function value and the second loss function value is less than a threshold. If it is less than the threshold, training is stopped. If it is not less than the threshold, and the number of consecutive training iterations in which the second loss function value for the test image has not changed exceeds the first threshold number, training is stopped. If the difference between the first loss function value and the second loss function value is not less than the threshold, and the number of consecutive training iterations in which the second loss function value for the test image has not changed does not exceed the first threshold number, the total number of training iterations is checked to determine whether to stop training. The above examples are merely for describing this disclosure and are not intended to specifically limit this disclosure.

[0065] The following is combined Figure 4 Example 400 for training an image classification model according to embodiments of the present disclosure is described.

[0066] In the example of training an image classification model, there are two phases: a training phase and a testing phase. In one example, a test is performed after each training iteration. In another example, a test is performed after several training iterations. The above examples are merely for describing this disclosure and are not intended to limit it.

[0067] At the start of training, the initial weights of the loss function of the image classification model are set based on the number of object categories the model needs to classify. Then, sample image 402 is input into the trained model 404 to classify each pixel in the image to determine its object category. The loss function is then calculated to further adjust the parameters of the classification model. After a predetermined number of training iterations, the weights of the loss function need to be adjusted, for example, once every five iterations. If weight adjustment is needed in the i-th iteration, the recall rate for multiple object categories after each iteration is determined based on the classification results of the two nearest iterations i-1 and i-2. Then, for each object category, the recall rate distance is determined by subtracting the recall rate of the i-2th iteration from the recall rate of the i-1th iteration. The recall rate distance is then used to determine the candidate category 406.

[0068] Candidate category 406 includes the first object category corresponding to the maximum recall distance among multiple recall distances. The weights 408 corresponding to this first object category in the loss function are then adjusted. At this point, an updated weighted loss function 410 is formed. The model is then trained further using the updated weighted loss function.

[0069] Alternatively, candidate categories also include other object categories. Since categories with large amounts of image data tend to have relatively accurate recall, the recall distance for these categories usually doesn't vary much. Therefore, to reduce the computational cost for these categories, the smallest recall distance can be selected from multiple recall distances. The second object category corresponding to the smallest recall distance is then determined. The corresponding data is then removed from the training images to form adjusted training images, which are then used for further training. Simultaneously, the recall distance for this second object category is saved. If, in subsequent calculations, an object category with a smaller recall distance than this second object category exists, the model is trained again using training images containing image data corresponding to the second object category in the new training process.

[0070] In some embodiments, a testing phase is provided to appropriately stop model training. The trained model 404 is tested using test image 412 to obtain classification result 416. Alternatively or additionally, a test can be performed after at least one training iteration. The termination of training is then determined using the loss function value from the testing phase, the loss function value from the training phase, and the number of training iterations. The termination conditions are as described in the example above.

[0071] This method allows for dynamic adjustment of the weights of subcategories to better balance the accuracy of subcategories and major categories, while also reducing computational costs, thus enabling deployment on edge devices.

[0072] Figure 5 A schematic block diagram of an example device 500 that can be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device 104 can be implemented using device 500. As shown, device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 502 or loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 can also store various programs and data required for the operation of device 500. CPU 501, ROM 502, and RAM 503 are interconnected via bus 6504. Input / output (I / O) interface 505 is also connected to bus 504.

[0073] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0074] The various processes and handling described above, such as methods 200 and 300, can be executed by processing unit 501. For example, in some embodiments, methods 200 and 300 can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by CPU 501, one or more actions of methods 200 and 300 described above can be performed.

[0075] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0076] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0077] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0078] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0079] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0080] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0081] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0083] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training an image classification model, comprising: Determine the first classification result of the image classification model for the pixels of the sample image in the first training iteration, wherein the first classification result indicates that each pixel in the pixel belongs to one of multiple object categories; Determine the second classification result of the image classification model for the pixels of the sample image in the second training iteration; Based on the first classification result and the second classification result, a first group of recall rates and a second group of recall rates are determined for the multiple object categories; as well as The image classification model is adjusted based on the first set of recall rates and the second set of recall rates to obtain a trained image classification model; The adjustment of the image classification model includes: Based on the first set of recall rates and the second set of recall rates, determine multiple recall rate distances for the multiple object categories; as well as One or more weights used in training the image classification model are adjusted based on the multiple recall distances.

2. The method according to claim 1, wherein adjusting the image classification model comprises: The loss function of the image classification model is adjusted based on the first set of recall rates and the second set of recall rates; The image classification model is trained using the adjusted loss function.

3. The method of claim 2, wherein adjusting the loss function of the image classification model comprises: The loss function is adjusted based on the multiple recall distances.

4. The method of claim 3, wherein adjusting the loss function based on the plurality of recall distances comprises: A first recall distance that is greater than a first threshold distance is determined from the plurality of recall distances; Determine the first object category corresponding to the first recall distance; as well as Adjust the weights in the loss function that are related to the first object category.

5. The method according to claim 3, further comprising: A second recall distance that is less than the second threshold distance is determined from the plurality of recall distances; Determine the second object category corresponding to the second recall distance; The sample image is adjusted by removing pixels in the sample image that correspond to the second object category; as well as The adjusted sample images are input into the image classification model.

6. The method according to claim 1, further comprising: Based on the number of the multiple object categories, initialize multiple weights for the loss function used in the image classification model.

7. The method according to claim 1, further comprising: The image classification model is applied to the sample image to determine a first loss function value for the sample image; The image classification model is applied to a test image to determine a second loss function value for the test image; Determine the difference between the first loss function value and the second loss function value; as well as If the difference is determined to be less than the threshold, the training is stopped.

8. The method of claim 7, further comprising: Determine the number of consecutive training iterations in which the value of the second loss function for the test image remains unchanged; as well as If the number of attempts exceeds the first threshold, the training is stopped.

9. The method according to claim 1, further comprising: Determine the total number of training iterations for the classification model; as well as If the total number of iterations exceeds the second threshold, the training is stopped.

10. An electronic device, comprising: At least one processor; as well as A memory, coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the device to perform actions for training an image classification model, the actions including: Determine the first classification result of the image classification model for the pixels of the sample image in the first training iteration, wherein the first classification result indicates that each pixel in the pixel belongs to one of multiple object categories; Determine the second classification result of the image classification model for the pixels of the sample image in the second training iteration; Based on the first classification result and the second classification result, a first set of recall rates and a second set of recall rates are determined for the multiple object categories; and The image classification model is adjusted based on the first set of recall rates and the second set of recall rates to obtain a trained image classification model; The adjustment of the image classification model includes: Based on the first set of recall rates and the second set of recall rates, determine multiple recall rate distances for the multiple object categories; and One or more weights used in training the image classification model are adjusted based on the multiple recall distances.

11. The electronic device of claim 10, wherein adjusting the image classification model comprises: The loss function of the image classification model is adjusted based on the first set of recall rates and the second set of recall rates; The image classification model is trained using the adjusted loss function.

12. The electronic device of claim 11, wherein adjusting the loss function of the image classification model comprises: The loss function is adjusted based on the multiple recall distances.

13. The electronic device of claim 12, wherein adjusting the loss function based on the plurality of recall distances comprises: A first recall distance that is greater than a first threshold distance is determined from the plurality of recall distances; Determine the first object category corresponding to the first recall distance; as well as Adjust the weights in the loss function that are related to the first object category.

14. The electronic device according to claim 12, wherein the action further includes A second recall distance that is less than the second threshold distance is determined from the plurality of recall distances; Determine the second object category corresponding to the second recall distance; The sample image is adjusted by removing pixels in the sample image that correspond to the second object category; as well as The adjusted sample images are input into the image classification model.

15. The electronic device according to claim 10, further comprising: Based on the number of the multiple object categories, initialize multiple weights for the loss function used in the image classification model.

16. The electronic device according to claim 10, further comprising: The image classification model is applied to the sample image to determine a first loss function value for the sample image; The image classification model is applied to a test image to determine a second loss function value for the test image; Determine the difference between the first loss function value and the second loss function value; as well as If the difference is determined to be less than the threshold, the training is stopped.

17. The electronic device of claim 16, further comprising: Determine the number of consecutive training iterations in which the value of the second loss function for the test image remains unchanged; as well as If the number of attempts exceeds the first threshold, the training is stopped.

18. The electronic device of claim 10, further comprising: Determine the total number of training iterations for the classification model; as well as If the total number of iterations exceeds the second threshold, the training is stopped.

19. A computer program product tangibly stored on a non-volatile computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the steps of the method according to any one of claims 1 to 9.

Images