Image classification model training method and device, electronic equipment and storage medium

Abstract

The application provides an image classification model training method and device, electronic equipment and a computer readable storage medium. The image classification model training method comprises: obtaining a target sample image; extracting heat information from the target sample image by using a preset heat map mapping parameter to obtain a sample heat map of the target sample image, wherein the heat map mapping parameter is used to reflect a constraint relationship between the sample heat map and category label information, and the heat map mapping parameter is learned based on a preset heat map extraction module; determining category label information of the target sample image based on the sample heat map; and training an image classification model to be trained based on the category label information and the target sample image until a preset first stop training condition is met, thereby obtaining a trained image classification model. The application can improve the classification accuracy of the image classification model.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to an image classification model training method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the continuous development of computer vision technology, it is being applied to more and more fields. Among them, image classification is an important task in computer vision.

[0003] Existing image classification models require training based on a large amount of labeled data as supervision information in order to ensure the classification accuracy of the image classification model.

[0004] However, in certain special scenarios, due to the small differences between categories, the human eye cannot accurately distinguish image categories, which will lead to labeling errors. The problem of the trained image classification model being confused due to the use of incorrect manual labeling will ultimately result in low classification accuracy of the trained image classification model. Summary of the Invention

[0005] This application provides an image classification model training method, apparatus, electronic device, and computer-readable storage medium, aiming to solve the problem that image classification models have low classification accuracy due to the tendency to use incorrect manual annotations to train them when the differences between categories are too small.

[0006] Firstly, this application provides a method for training an image classification model, the method comprising:

[0007] Acquire the target sample image;

[0008] By using preset heatmap mapping parameters, thermal information is extracted from the target sample image to obtain a sample heatmap of the target sample image. The heatmap mapping parameters are used to reflect the constraint relationship between the sample heatmap and the category labeling information. The heatmap mapping parameters are learned based on the preset heatmap extraction module.

[0009] Based on the sample heatmap, the category labeling information of the target sample image is determined;

[0010] Based on the category labeling information and the target sample image, the image classification model to be trained is trained until a preset first stop training condition is met, and a trained image classification model is obtained.

[0011] Secondly, this application provides an image classification model training device, the image classification model training device comprising:

[0012] The acquisition unit is used to acquire the target sample image;

[0013] The extraction unit is used to extract thermal information from the target sample image using preset heatmap mapping parameters to obtain a sample heatmap of the target sample image. The heatmap mapping parameters are used to reflect the constraint relationship between the sample heatmap and the category labeling information. The heatmap mapping parameters are learned based on the preset heatmap extraction module.

[0014] The determining unit is used to determine the category labeling information of the target sample image based on the sample heatmap;

[0015] The training unit is used to train the image classification model to be trained based on the category labeling information and the target sample image until a preset first stop training condition is met, thus obtaining the trained image classification model.

[0016] In some embodiments of this application, the determining unit is specifically used for:

[0017] A preset mapping relationship between thermal location information and category information is obtained, wherein the preset mapping relationship is used to indicate the relationship between the thermal location information of the sample heatmap and the category labeling information of the target sample image;

[0018] Obtain the thermal location information of the sample heatmap;

[0019] Based on the thermal location information of the sample heatmap and the preset mapping relationship, the category labeling information of the target sample image is determined.

[0020] In some embodiments of this application, before training the image classification model to be trained based on the category labeling information and the target sample image until a preset first stopping training condition is met, and before obtaining the trained image classification model, the training unit is specifically used for:

[0021] Obtain an image sample set and preset annotation data, wherein the image sample set includes multiple initial sample images, and the preset annotation data includes preset annotation categories of the initial sample images;

[0022] Based on a preset classification model, the predicted category of each initial sample image in the image sample set is obtained.

[0023] From the image sample set, obtain the target initial sample image whose preset labeled category is different from the predicted category;

[0024] Using the predicted category of the target initial sample image as supervision information, the preset classification model is trained based on the target initial sample image until a preset second stopping training condition is met, thus obtaining the image classification model to be trained.

[0025] In some embodiments of this application, before determining the category labeling information of the target sample image based on the sample heatmap, the training unit is specifically used for:

[0026] By using the heatmap mapping parameters, thermal information is extracted from the initial target sample image to obtain the target heatmap of the initial target sample image;

[0027] The target category of the initial sample image is obtained by predicting the target category based on the target initial sample image using the image classification model to be trained.

[0028] Obtain the thermal location information of the target heatmap;

[0029] Based on the thermal location information of the target heat map and the target category, a preset mapping relationship between thermal location information and category information is obtained.

[0030] In some embodiments of this application, the extraction unit is specifically used for:

[0031] By using the heatmap mapping parameters, feature extraction is performed on the target sample image to obtain the target feature map of the target sample image;

[0032] Based on the target feature map, a sample heatmap of the target sample image is obtained.

[0033] In some embodiments of this application, the image classification model training device further includes a classification unit, which is specifically used for:

[0034] Acquire the target scene image of the target scene region;

[0035] The trained image classification model is used to classify the target scene image to obtain the target scene state of the target scene region.

[0036] In some embodiments of this application, the target scene image includes a prior scene image and a subsequent scene image of the target scene region, and the classification unit is specifically used for:

[0037] Based on the prior sample image and the subsequent sample image, determine the sample optical flow information of the target sample image;

[0038] Using the category labeling information as supervision information, the image classification model to be trained is trained based on the prior sample images and the sample optical flow information until the preset first stop training condition is met, and the trained image classification model is obtained.

[0039] In some embodiments of this application, the target sample image includes prior sample images and subsequent sample images of the sample scene region, and the training unit is specifically used for:

[0040] Based on the prior scene image and the subsequent scene image, determine the scene optical flow information of the target scene image;

[0041] The target scene image is classified using the trained image classification model based on the prior scene image and the scene optical flow information to obtain the target scene state of the target scene region.

[0042] Thirdly, this application also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it executes the steps in any of the image classification model training methods provided in this application.

[0043] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the steps in the image classification model training method.

[0044] In this application, since the heatmap mapping parameters are learned based on a preset heatmap extraction module, they can also reflect the relationship between the indicator sample heatmap and image feature information. Furthermore, the image feature information extracted by the feature extraction layer is directly related to the image category; therefore, the heatmap mapping parameters also reflect, to some extent, the constraint relationship between the sample heatmap and the category labeling information. Thus, by using sample heatmaps to indicate the category labeling information of the target sample image, and determining the category labeling information of the target sample image based on the sample heatmap, the image classification model to be trained is obtained. Firstly, this improves the classification accuracy of the image classification model. Secondly, especially when the differences between categories are too small for the human eye to accurately distinguish image categories, determining the category labeling information of the target sample image based on the sample heatmap avoids the problem of misclassification caused by training the image classification model with incorrect manual labeling, thereby also improving the classification accuracy of the trained image classification model to some extent. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of a scene for the image classification model training system provided in an embodiment of this application;

[0047] Figure 2 This is a schematic flowchart of an image classification model training method provided in an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of a network structure of an image classification model to be trained provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of another network structure of the image classification model to be trained provided in the embodiments of this application;

[0050] Figure 5 This is a schematic diagram of a network structure of the heatmap extraction module provided in the embodiments of this application;

[0051] Figure 6 This is a schematic diagram of an embodiment of the image classification model training device provided in this application.

[0052] Figure 7 This is a schematic diagram of an embodiment of the electronic device provided in this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0055] To enable any person skilled in the art to implement and use this application, the following description is provided. In this description, details are set forth for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known processes will not be described in detail to avoid obscuring the description of the embodiments of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0056] This application provides an image classification model training method, apparatus, electronic device, and computer-readable storage medium. The image classification model training apparatus can be integrated into an electronic device, which may be a server or a terminal, etc.

[0057] The execution subject of the image classification model training method in this application embodiment can be the image classification model training device provided in this application embodiment, or different types of electronic devices such as server equipment, physical host, or user equipment (UE) that integrate the image classification model training device. The image classification model training device can be implemented in hardware or software. The UE can specifically be a terminal device such as a smartphone, tablet computer, laptop computer, handheld computer, desktop computer, or personal digital assistant (PDA).

[0058] The electronic device can operate independently or in a cluster.

[0059] See Figure 1 , Figure 1 This is a schematic diagram of a scenario for an image classification model training system provided in an embodiment of this application. The image classification model training system may include an electronic device 100, which integrates an image classification model training device. For example, the electronic device can acquire a target sample image; extract heat information from the target sample image using preset heatmap mapping parameters to obtain a sample heatmap of the target sample image, wherein the heatmap mapping parameters reflect the constraint relationship between the sample heatmap and category labeling information, and the heatmap mapping parameters are learned based on a preset heatmap extraction module; determine the category labeling information of the target sample image based on the sample heatmap; and train the image classification model to be trained based on the category labeling information and the target sample image until a preset first stop training condition is met, thus obtaining a trained image classification model.

[0060] In addition, such as Figure 1As shown, the image classification model training system may also include a memory 200 for storing data, such as image data and video data.

[0061] It should be noted that, Figure 1 The schematic diagram of the image classification model training system shown is merely an example. The image classification model training system and scenario described in this application are for the purpose of more clearly illustrating the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of image classification model training systems and the emergence of new business scenarios, the technical solutions provided in this invention are also applicable to similar technical problems.

[0062] The image classification model training method provided in the embodiments of this application will now be introduced. In the embodiments of this application, an electronic device is used as the execution subject. For the sake of simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments.

[0063] Reference Figure 2 , Figure 2 This is a flowchart illustrating an image classification model training method provided in an embodiment of this application. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here. The image classification model training method includes steps 201-204, wherein:

[0064] 201. Obtain the target sample image.

[0065] Here, the target sample image refers to the sample image used for training. The target sample image can be an RGB image or an image of other formats; there are no restrictions on the format. Depending on the different business scenarios, the target sample image can have various representations. For example, when training an image classification model for classifying dogs, the target sample image refers to a sample image of a dog. As another example, when training an image classification model for classifying the states of a target scene area (such as a cargo loading / unloading port), the target sample image refers to sample images of the target scene area in various states (e.g., the loading / unloading port in seven states: no vehicle, rest, arrival, off-duty, loading, unloading, and working).

[0066] In this embodiment of the application, the training of an image classification model for classifying the state of a target scene region is taken as an example.

[0067] There are several ways to obtain the target sample image in step 201, including, for example:

[0068] (1) In practical applications, electronic devices can integrate cameras in hardware to capture video frames or images of the target scene area in real time, which can be used as target sample images.

[0069] (2) Alternatively, video frames or images of the target scene area can be captured in real time by cameras on other terminals. For example, the camera of the target scene area monitoring device captures video frames or images of the target scene area, and the electronic device establishes a network connection with cameras on other terminals. Based on this network connection, video frames or images of the target scene area captured by cameras on other terminals can be obtained online as target sample images.

[0070] (3) Electronic devices can also read the target scene area image captured by the camera from the relevant storage medium that stores the target scene area image captured by the camera (including the camera integrated in the electronic device or the camera of other terminals) and use it as the target sample image.

[0071] (4) Read the video frames or images of the target scene area that have been pre-acquired and stored inside the electronic device as target sample images.

[0072] The method of obtaining the target sample image here is only an example and is not limited to this.

[0073] 202. By using preset heatmap mapping parameters, thermal information is extracted from the target sample image to obtain a sample heatmap of the target sample image.

[0074] Among them, the thermal location information of the sample heatmap refers to the object or logo corresponding to the highlighted position in the sample heatmap.

[0075] The heatmap mapping parameters reflect the constraint relationship between the sample heatmap and the category labeling information. These parameters are learned based on a pre-defined heatmap extraction module.

[0076] The heatmap mapping parameters can be learned through the heatmap extraction module. The heatmap extraction module can be integrated into the image classification model to be trained in step 203, or it can stand alone as a model. The two cases are described below:

[0077] 1) The heatmap extraction module can be integrated into the image classification model to be trained in step 203.

[0078] When the heatmap extraction module can be integrated into the image classification model to be trained in step 203, the image classification model to be trained can be obtained by training the preset classification model using labeled data. At this time, the preset heatmap mapping parameters can be extracted from the heatmap extraction module in the image classification model to be trained, and used to determine the sample heatmap of the target sample image in step 202.

[0079] For example, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a network structure for an image classification model to be trained provided in an embodiment of this application. In some embodiments, the image classification model to be trained includes a feature extraction layer, a classification result output layer, and a heatmap output layer. Wherein:

[0080] The feature extraction layer is used to output image feature information of the target sample image based on the target sample image. The feature extraction layer takes the target sample image as input and performs one or more operations on the target sample image, including but not limited to convolution and pooling, to extract the image feature information of the target sample image. For example, for an RGB format target sample image, each channel (R, G, and B channels) will undergo convolution and pooling operations to form feature maps for the R, G, and B channels. These three feature maps constitute the image feature information of the target sample image. The feature extraction layer can be a Convolutional Neural Network (CNN), such as a ResNet50 classification network with its fully connected layers removed (denoted as FC) as the feature extraction layer.

[0081] The classification result output layer is used to output the classification result of the target sample image based on its image feature information. The feature extraction layer takes the image feature information of the target sample image as input, predicts the probability of each pixel in the target sample image belonging to each category based on this information, and determines and outputs the classification result of the target sample image based on the probability of each pixel belonging to each category. The classification result of the target sample image indicates its category. The classification result output layer can be a fully connected layer structure; for example, the fully connected layers in the ResNet50 classification network can be directly used as the classification result output layer.

[0082] The heatmap output layer outputs a sample heatmap of the target sample image based on its image feature information. The feature extraction layer takes the image feature information of the target sample image as input, extracts heatmap information based on this information, and then outputs the sample heatmap. For example, for an RGB format target sample image, the feature extraction layer outputs feature maps for the R, G, and B channels as its image feature information. The pixel values ​​at corresponding pixel positions in these feature maps are then summed to form the sample heatmap. Since the heatmap output layer is directly connected to the feature extraction layer, the heatmap mapping parameters also reflect the relationship between heatmap location information and image feature information. Furthermore, image feature information is directly related to image categories; therefore, the heatmap mapping parameters also reflect, to some extent, the constraint relationship between the sample heatmap and category labeling information.

[0083] At this point, the heatmap extraction module specifically includes a feature extraction layer and a heatmap output layer. In some embodiments, the weight parameters of the feature extraction layer and the heatmap output layer in the image classification model to be trained can be directly extracted as preset heatmap mapping parameters for obtaining the sample heatmap of the target sample image in step 202. Before step 202, the method further includes: extracting the weight parameters of the feature extraction layer and the heatmap output layer in the image classification model to be trained as preset heatmap mapping parameters; step 202 specifically includes the following steps 2021A to 2022A:

[0084] 2021A. By using the weight parameters of the corresponding feature extraction layer in the heatmap mapping parameters, feature extraction is performed on the target sample image to obtain the target feature map of the target sample image.

[0085] 2022A. Based on the target feature map, obtain the sample heatmap of the target sample image by using the weight parameters of the corresponding heatmap output layer in the heatmap mapping parameters.

[0086] Alternatively, in some embodiments, step 202 may directly use the weight parameters of the feature extraction layer and the heatmap output layer in the image classification model to be trained as preset heatmap mapping parameters. Step 202 directly obtains the sample heatmap of the target sample image through the feature extraction layer and the heatmap output layer in the image classification model to be trained. In this case, step 202 may specifically include the following steps 2021B to 2022B:

[0087] 2021B. Using the weight parameters of the feature extraction layer in the image classification model to be trained, feature extraction is performed on the target sample image to obtain the target feature map of the target sample image.

[0088] 2022B. Using the weight parameters of the heatmap output layer in the image classification model to be trained, obtain the sample heatmap of the target sample image based on the target feature map.

[0089] 2) The heatmap extraction module is treated as a standalone model. In this case, such as Figure 4 and Figure 5 As shown, the heatmap extraction module and the image classification model to be trained are two independent models. Figure 4 This is a schematic diagram of another network structure of the image classification model to be trained provided in the embodiments of this application. Figure 5 This is a schematic diagram of a network structure of the heatmap extraction module provided in the embodiments of this application.

[0090] When the heatmap extraction module is used as a standalone model (referred to as the "heatmap extraction model"), it can be trained using labeled data to obtain a trained heatmap extraction model. Then, preset heatmap mapping parameters can be extracted from the trained heatmap extraction model for use in step 202 to determine the sample heatmap of the target sample image.

[0091] like Figure 4 As shown, the image classification model to be trained includes a first feature extraction layer and a classification result output layer. The functional architecture and working principle of the first feature extraction layer are similar to those of the feature extraction layer described above, and can be found in the description of the feature extraction layer in section 1) above. The functional architecture and working principle of the classification result output layer are similar to those of the classification result output layer described above, and can be found in the description of the classification result output layer in section 1) above. For the sake of simplicity, they will not be described again here.

[0092] like Figure 5 As shown, the heatmap extraction module, as an independent model, includes a second feature extraction layer and a heatmap output layer. The functional architecture and working principle of the second feature extraction layer are similar to those of the feature extraction layer described above; please refer to the description of the feature extraction layer in section 1) above. The functional architecture and working principle of the heatmap output layer are also similar to those of the heatmap output layer described above; please refer to the description of the heatmap output layer in section 1) above. For simplicity, these details will not be elaborated upon here. Specifically, to ensure that the heatmap mapping parameters can learn the constraint relationship between the sample heatmap and the category labeling information, Figure 5 The second feature extraction layer in Figure 4 The first feature extraction layer in the model shares weight parameters.

[0093] Similar to 1), the heatmap extraction module here specifically includes a second feature extraction layer and a heatmap output layer. In some embodiments, the weight parameters of the second feature extraction layer and the heatmap output layer in the heatmap extraction module can be directly extracted as preset heatmap mapping parameters for obtaining the sample heatmap of the target sample image in step 202. In this case, before step 202, the module further includes: extracting the weight parameters of the second feature extraction layer and the heatmap output layer in the heatmap extraction module as preset heatmap mapping parameters; step 202 specifically includes the following steps 2021C to 2022C:

[0094] 2021C. By using the weight parameters corresponding to the second feature extraction layer in the heatmap mapping parameters, feature extraction is performed on the target sample image to obtain the target feature map of the target sample image.

[0095] 2022C. Based on the target feature map, obtain the sample heatmap of the target sample image by using the weight parameters of the corresponding heatmap output layer in the heatmap mapping parameters.

[0096] Alternatively, in some other embodiments, step 202 may directly use the weight parameters of the second feature extraction layer and the heatmap output layer in the heatmap extraction module as preset heatmap mapping parameters. Step 202 then directly obtains the sample heatmap of the target sample image through the second feature extraction layer and the heatmap output layer in the heatmap extraction module. In this case, step 202 may specifically include the following steps 2021D to 2022D:

[0097] 2021D. By using the weight parameters of the second feature extraction layer in the heatmap extraction module, feature extraction is performed on the target sample image to obtain the target feature map of the target sample image.

[0098] 2022D. Based on the target feature map, obtain the sample heatmap of the target sample image by using the weight parameters of the heatmap output layer in the heatmap extraction module.

[0099] As can be seen from the above steps 2021A to 2022A, 2021B to 2022B, 2021C to 2022C, and 2021D to 2022D, step 202 may specifically include: extracting features from the target sample image using the heatmap mapping parameters to obtain a target feature map of the target sample image; and obtaining a sample heatmap of the target sample image based on the target feature map.

[0100] Since the heatmap mapping parameters reflect the constraint relationship between the sample heatmap and the category labeling information, obtaining the sample heatmap of the target sample image based on the heatmap mapping parameters can provide accurate data for determining the category labeling information of the target sample image, thereby improving the classification accuracy of the trained image classification model.

[0101] 203. Based on the sample heatmap, determine the category labeling information of the target sample image.

[0102] Among them, the category labeling information refers to the labeling category of the target sample image. For example, if the loading and unloading port in the target sample image is in a state of no vehicle, then the target sample image will be labeled as the "no vehicle" category.

[0103] In some embodiments, in step 203, the category labeling information of the target sample image can be determined manually based on the sample heat map and the preset mapping relationship between heat location information and category information.

[0104] In another embodiment, the electronic device in this application embodiment can also determine the category labeling information of the target sample image based on the sample heat map and the preset mapping relationship between heat location information and category information.

[0105] For example, when annotation is performed via an electronic device, step 203 may specifically include the following steps 2031 to 2033:

[0106] 2031. Obtain the preset mapping relationship between thermal location information and category information.

[0107] The preset mapping relationship is used to indicate the relationship between the thermal location information of the sample heatmap and the category labeling information of the target sample image.

[0108] Among them, thermal location information refers to the object or icon corresponding to the highlighted position in the thermal map. For example, if the highlighted position in a thermal map containing a person is at the hair, then the thermal location information is hair.

[0109] For example, in the target scene area, the cargo loading and unloading port has seven status categories: no vehicle, rest, arrival, off-duty, loading, unloading, and working. The preset mapping relationship is shown in Table 1 below. Table 1 indicates that when the thermal location information is "vehicle", the category label is "loading"; when the thermal location information is "person", the category label is "unloading"; when the thermal location information is "trailer", the category label is "off-duty", and so on.

[0110] Table 1

[0111] Thermal location information Category labeling information vehicle Loading people discharge trailer Leave of post … …

[0112] There are several ways to obtain the preset mapping relationship in step 2031. For example, they include:

[0113] (1) The preset database stores a preset mapping relationship between thermal location information and category information. In this case, the preset mapping relationship can be directly obtained from the preset database in step 2031.

[0114] (2) Determine the preset mapping relationship in real time.

[0115] For example, the preset mapping relationship can be obtained through the following steps A1 to A4:

[0116] A1. Using the heatmap mapping parameters, thermal information is extracted from the reference sample image to obtain the target heatmap of the reference sample image.

[0117] The reference sample image is a sample image used to summarize the relationship between thermal location information and category labeling information. In some embodiments, the reference sample image may be a randomly acquired image; in another embodiment, in order to improve the accuracy of the preset mapping relationship, thereby improving the accuracy of the category labeling information of the target sample image and thus improving the classification accuracy of the trained image classification model, the reference sample image may be the "target initial sample image" in steps B1 to B4 below.

[0118] Reference Figure 3 In some embodiments, the image classification model to be trained in step 1) above can be used to classify and predict the reference categories of some reference sample images; thermal information can be extracted to obtain reference heatmaps of the reference sample images; and the correspondence between thermal location information in the reference heatmaps and the predicted reference categories can be summarized as a preset mapping relationship between thermal location information and category information. In this case, step A1 may specifically include: extracting features from the reference sample images using the weight parameters of the feature extraction layer in the image classification model to be trained, obtaining a feature map of the reference sample images; and obtaining a target heatmap of the reference sample images based on the feature map of the reference sample images using the weight parameters of the heatmap output layer in the image classification model to be trained.

[0119] Or, refer to Figure 4 and Figure 5In other embodiments, the image classification model to be trained in step 2) above can be used to classify and predict the reference categories of some reference sample images; the heatmap extraction module can extract thermal information to obtain a reference heatmap of the reference sample images; and the correspondence between the thermal location information in the reference heatmap and the predicted reference categories can be summarized as a preset mapping relationship between thermal location information and category information. In this case, step A1 may specifically include: extracting features from the reference sample images using the weight parameters of the second feature extraction layer in the heatmap extraction module to obtain a feature map of the reference sample images; and obtaining a target heatmap of the reference sample images based on the feature map of the reference sample images using the weight parameters of the heatmap output layer in the heatmap extraction module.

[0120] A2. Using the image classification model to be trained, predict the target category of the reference sample image based on the reference sample image.

[0121] Reference Figure 3 In some embodiments, the image classification model to be trained in step 1) above can be used to classify and predict the reference categories of some reference sample images; thermal information can be extracted to obtain reference heatmaps of the reference sample images; and the correspondence between thermal location information in the reference heatmaps and the predicted reference categories can be summarized as a preset mapping relationship between thermal location information and category information. In this case, step A2 may specifically include: extracting features from the reference sample images using the weight parameters of the feature extraction layer in the image classification model to be trained, obtaining feature maps of the reference sample images; and performing classification prediction based on the feature maps of the reference sample images using the weight parameters of the classification result output layer in the image classification model to be trained, obtaining the target category of the reference sample images.

[0122] Or, refer to Figure 4 and Figure 5 In other embodiments, the image classification model to be trained in step 2) above can be used to classify and predict the reference categories of some reference sample images; the heat map extraction module can extract thermal information to obtain reference heat maps of the reference sample images; and the correspondence between thermal location information in the reference heat map and the predicted reference categories can be summarized as a preset mapping relationship between thermal location information and category information. In this case, step A2 may specifically include: extracting features from the reference sample images using the weight parameters of the second feature extraction layer in the image classification model to be trained to obtain feature maps of the reference sample images; and performing classification prediction based on the feature maps of the reference sample images using the weight parameters of the classification result output layer in the image classification model to be trained to obtain the target category of the reference sample images.

[0123] A3. Obtain the thermal location information of the target heat map.

[0124] Among them, the thermal location information of the target heatmap refers to the object or label corresponding to the highlighted position in the target heatmap.

[0125] A4. Based on the thermal location information of the target heat map and the target category, obtain the preset mapping relationship between the thermal location information and the category information.

[0126] As shown in Table 1, for example, the target scene area, the cargo loading and unloading port, has seven status categories: no vehicle, resting, arriving at port, off duty, loading, unloading, and working. If the heat map's location information is "vehicle" and the target category is "loading," a mapping relationship can be formed between the heat map's location information "vehicle" and the category label information "loading." If the heat map's location information is "person" and the target category is "unloading," a mapping relationship can be formed between the heat map's location information "person" and the category label information "unloading." If the heat map's location information is "trailer" and the target category is "off duty," a mapping relationship can be formed between the heat map's location information "trailer" and the category label information "off duty," and so on, various preset mapping relationships between heat map location information and category information can be formed.

[0127] By establishing a pre-defined mapping relationship between thermal location information and category information, the category labeling information of the target sample image can be quickly determined directly based on the pre-defined mapping relationship. This improves the labeling speed of the target sample image to a certain extent, thereby increasing the training speed of the image classification model.

[0128] 2032. Obtain the thermal location information of the sample heatmap.

[0129] 2033. Based on the thermal location information of the sample heatmap and the preset mapping relationship, determine the category labeling information of the target sample image.

[0130] For ease of understanding, let's continue with the preset mapping relationship shown in Table 1 as an example. If the thermal location information of the sample heat map is "vehicle", then the category labeling information of the target sample image can be determined as "loading".

[0131] Since the pre-defined mapping relationship between thermal location information and category information can accurately reflect the category labeling information and the category labeling information of the target sample image, the category labeling information of the target sample image can be determined based on the thermal location information of the sample heatmap and the pre-defined mapping relationship between thermal location information and category information. This can improve the accuracy of the category labeling information of the target sample image, thereby improving the classification accuracy of the trained image classification model.

[0132] 204. Based on the category labeling information and the target sample image, train the image classification model to be trained until the preset first stop training condition is met, and obtain the trained image classification model.

[0133] In this embodiment, since the heatmap mapping parameters are learned based on a preset heatmap extraction module, they can also reflect the relationship between the indicator sample heatmap and image feature information. Furthermore, the image feature information extracted by the feature extraction layer is directly related to the image category; therefore, the heatmap mapping parameters also reflect the constraint relationship between the sample heatmap and category labeling information to a certain extent. Thus, by using sample heatmaps to indicate the category labeling information of the target sample image, and determining the category labeling information of the target sample image based on the sample heatmap, the image classification model to be trained is trained to obtain a trained image classification model. Firstly, this improves the classification accuracy of the image classification model. Secondly, especially when the differences between categories are too small for the human eye to accurately distinguish image categories, determining the category labeling information of the target sample image based on the sample heatmap avoids the problem of misclassification caused by training the image classification model with incorrect manual labeling, thereby also improving the classification accuracy of the trained image classification model to a certain extent.

[0134] Furthermore, since the human eye cannot accurately distinguish image categories in some cases, incorrect annotations may be used to train the classification model, resulting in low image classification accuracy of the trained image classification model. In some embodiments, to minimize such situations and improve the image classification accuracy of the trained image classification model, the manually labeled category can be compared with the predicted category of the preset classification model for the same image. If the manually labeled category is different from the predicted category of the image classification model, the predicted category of the image classification model is used as the category labeling information for that image to train the image classification model. That is, in some embodiments, the image classification model training method further includes the following steps B1 to B4:

[0135] B1. Obtain the image sample set and preset annotation data.

[0136] The image sample set includes multiple initial sample images, and the preset annotation data includes preset annotation categories for the initial sample images.

[0137] Initial sample images refer to the sample images used to train a pre-defined classification model in order to form the image classification model to be trained.

[0138] Preset annotation categories refer to the categories of the initial sample images that are manually annotated.

[0139] B2. Based on a preset classification model, predict the predicted category of each initial sample image in the image sample set.

[0140] The predicted category refers to the category of the initial sample image obtained by predicting it through a preset classification model.

[0141] The functional architecture of the preset classification model is similar to that of the classification model to be trained. For details, please refer to the relevant description of the classification model to be trained. It will not be repeated here.

[0142] The method for determining the predicted category in step B2 can be found in the relevant explanations on determining the classification result using the image classification model described above, and will not be repeated here.

[0143] B3. Obtain target initial sample images from the image sample set whose preset labeled categories are different from the predicted categories.

[0144] The target initial sample image refers to the initial sample image whose preset labeled category is different from the predicted category.

[0145] For example, if the image sample set includes sample images 1, 2, and 3, and the preset label categories of sample images 1, 2, and 3 are "loading", "unloading", and "loading" respectively, and the predicted categories of sample images 1, 2, and 3 are "loading", "loading", and "unloading" respectively in step B2, then the target initial sample images can be determined to be sample images 2 and 3.

[0146] B4. Using the predicted category of the target initial sample image as supervision information, train the preset classification model based on the target initial sample image until the preset second stop training condition is met, and obtain the image classification model to be trained.

[0147] The second stopping condition can be set according to actual needs. For example, it could be when all the target initial sample images in the image sample set have been used to train the preset classification model, or when the number of iterations using the target initial sample images to train the preset classification model reaches the maximum number of iterations.

[0148] Specifically, each iteration of the training process of the preset classification model is as follows: feature extraction is performed on the target initial sample image through the preset classification model to obtain the feature map of the target initial sample image; classification prediction is performed on the feature map of the target initial sample image to determine the category of the target initial sample image; based on the category of the target initial sample image, the predicted category of the target initial sample image, and the preset second loss function, the loss value of the preset classification model is determined; the model parameters of the preset classification model (e.g., the weight parameters of the feature extraction layer and the weight parameters of the classification result output layer) are adjusted based on the loss value of the preset classification model.

[0149] By determining the loss value of a pre-defined classification model using the category of the initial target sample image, the predicted category of the initial target sample image, and a pre-defined second loss function, the model parameters of the pre-defined classification model are adjusted based on this loss value. This allows the predicted category of the initial target sample image to be used as supervision information for training the pre-defined classification model, thus avoiding the problem of incorrect labeling during training and subsequent misclassification. Therefore, this approach can improve the image classification accuracy of previously trained image classification models, which may have been previously low.

[0150] Furthermore, to improve the classification accuracy of the trained image classification model in step 204, and to enhance the classification accuracy of image categories with very small differences between categories that cannot be accurately distinguished by the human eye, a complete training cycle can be defined as follows: "Steps B1 to B4 first train a preset classification model to obtain an image classification model to be trained, and then execute the training process of steps 201 to 204 on the image classification model to be trained obtained in steps B1 to B4 to obtain a trained image classification model." After each complete training cycle, when a trained image classification model is obtained, the trained image classification model is used as the preset classification model in steps B1 to B4, and the above steps B1 to B4 and steps 201 to 204 are repeated until multiple complete training cycles are completed or the preset accuracy requirement is met, thus obtaining the final trained image classification model.

[0151] After training the image classification model, the trained model can be used for classification prediction. For example, in some embodiments, using the trained image classification model for classification prediction may specifically include the following steps C1 to C2:

[0152] C1. Obtain the target scene image of the target scene area.

[0153] The target scene area can include loading and unloading points for goods, etc.

[0154] The acquisition of the target scene image is similar to the acquisition of the target sample image described above. For details, please refer to the relevant instructions on acquiring the target sample image, which will not be repeated here.

[0155] C2. Classify the target scene image using the trained image classification model to obtain the target scene state of the target scene region.

[0156] The scenario state refers to the state of the target scenario area. For example, if the target scenario area is a cargo loading and unloading port, its scenario state includes seven states: no vehicle at the cargo loading and unloading port, resting, arriving at the port, off duty, loading, unloading, and working.

[0157] The target scene state refers to the scene state of the target scene area at the time the target scene image is acquired. For example, the target scene state is "no vehicles at the loading and unloading port".

[0158] In some embodiments, the trained image classification model can be Figure 3 In the network structure shown, step C2 may specifically include: extracting features from the target scene image using the weight parameters of the feature extraction layer in the trained image classification model to obtain the feature map of the target scene image; and performing classification prediction based on the feature map of the target scene image using the weight parameters of the classification result output layer in the trained image classification model to obtain the image category of the target scene image, which serves as the target scene state of the target scene region.

[0159] In some embodiments, the trained image classification model can be Figure 4 In the network structure shown, step C2 may specifically include: extracting features from the target scene image using the weight parameters of the first feature extraction layer in the trained image classification model to obtain a feature map of the target scene image; and performing classification prediction based on the feature map of the target scene image using the weight parameters of the classification result output layer in the trained image classification model to obtain the image category of the target scene image, which serves as the target scene state of the target scene region.

[0160] Since scene states typically involve changes in motion, predicting scene states based on a single image will significantly reduce the accuracy of scene state prediction. To improve the accuracy of scene state prediction, in this embodiment, optical flow information can be further incorporated into the scene state prediction. In this case, the target scene image in steps C1 to C2 includes both the preceding and subsequent scene images of the target scene region. Step C2 specifically includes steps C21 to C22:

[0161] C21. Based on the prior scene image and the subsequent scene image, determine the scene optical flow information of the target scene image.

[0162] Scene optical flow information refers to the optical flow information of the target scene region determined based on the prior scene image and the subsequent scene image. Specifically, it can include the motion speed and motion direction of objects in the target scene region.

[0163] Optical flow refers to the instantaneous velocity of pixels moving on an image when an object is in motion. Optical flow matching is a method for obtaining the motion state of an object, and its principle is as follows:

[0164] By utilizing the grayscale value changes of a single pixel in an image, the optical flow vector, or motion vector, of that pixel is obtained. This motion vector contains the pixel's velocity and direction of motion. Then, the motion vectors of all pixels are obtained using the same method. Finally, all pixels of an object in the image are acquired, and the object's motion state is determined using the motion vectors of all pixels. Therefore, optical flow matching can determine not only the object's velocity but also its direction of motion.

[0165] Specifically, based on existing optical flow matching algorithms, such as the Lucas-Kanade optical flow matching method, the motion speed and direction of objects in the target scene region can be calculated based on the preceding and subsequent scene images, thereby obtaining the scene optical flow information of the target scene image.

[0166] C22. Using the trained image classification model, classify the target scene image according to the prior scene image and the scene optical flow information to obtain the scene state of the target scene region.

[0167] Correspondingly, step 204, the target sample image, includes a prior sample image and a subsequent sample image of the sample scene region. Step 204 may specifically include: determining the sample optical flow information of the target sample image based on the prior sample image and the subsequent sample image; using the category labeling information as supervision information, training the image classification model to be trained based on the prior sample image and the sample optical flow information until a preset first stopping training condition is met, thereby obtaining a trained image classification model.

[0168] The step of "determining the sample optical flow information of the target sample image based on the prior sample image and the subsequent sample image" is similar to step C21, and you can refer to the description of step C21 for details.

[0169] "Using the category labeling information as supervision information, the image classification model to be trained is trained based on the prior sample images and the sample optical flow information until a preset first stopping training condition is met, thus obtaining a trained image classification model." This is similar to step 204 above, and you can refer to the relevant description of step 204 for details. The difference is that here, the feature maps of the prior sample images and the sample optical flow information are combined to perform classification prediction and determine the category of the target sample image.

[0170] To better implement the image classification model training method in the embodiments of this application, based on the image classification model training method, the embodiments of this application also provide an image classification model training device, such as... Figure 6 The diagram shown is a structural schematic of an embodiment of the image classification model training device 600 in this application. The image classification model training device 600 includes:

[0171] Acquisition unit 601 is used to acquire target sample image;

[0172] Extraction unit 602 is used to extract thermal information from the target sample image using preset heatmap mapping parameters to obtain a sample heatmap of the target sample image. The heatmap mapping parameters are used to reflect the constraint relationship between the sample heatmap and the category labeling information. The heatmap mapping parameters are learned based on the preset heatmap extraction module.

[0173] The determining unit 603 is used to determine the category labeling information of the target sample image based on the sample heatmap;

[0174] The training unit 604 is used to train the image classification model to be trained based on the category labeling information and the target sample image until a preset first stop training condition is met, thereby obtaining a trained image classification model.

[0175] In some embodiments of this application, the determining unit 603 is specifically used for:

[0176] A preset mapping relationship between thermal location information and category information is obtained, wherein the preset mapping relationship is used to indicate the relationship between the thermal location information of the sample heatmap and the category labeling information of the target sample image;

[0177] Obtain the thermal location information of the sample heatmap;

[0178] Based on the thermal location information of the sample heatmap and the preset mapping relationship, the category labeling information of the target sample image is determined.

[0179] In some embodiments of this application, before training the image classification model to be trained based on the category labeling information and the target sample image until a preset first stop training condition is met, and before obtaining the trained image classification model, the training unit 604 is specifically used for:

[0180] Obtain an image sample set and preset annotation data, wherein the image sample set includes multiple initial sample images, and the preset annotation data includes preset annotation categories of the initial sample images;

[0181] Based on a preset classification model, the predicted category of each initial sample image in the image sample set is obtained.

[0182] From the image sample set, obtain the target initial sample image whose preset labeled category is different from the predicted category;

[0183] Using the predicted category of the target initial sample image as supervision information, the preset classification model is trained based on the target initial sample image until a preset second stopping training condition is met, thus obtaining the image classification model to be trained.

[0184] In some embodiments of this application, before determining the category labeling information of the target sample image based on the sample heatmap, the training unit 604 is specifically used for:

[0185] By using the heatmap mapping parameters, thermal information is extracted from the initial target sample image to obtain the target heatmap of the initial target sample image;

[0186] The target category of the initial sample image is obtained by predicting the target category based on the target initial sample image using the image classification model to be trained.

[0187] Obtain the thermal location information of the target heatmap;

[0188] Based on the thermal location information of the target heat map and the target category, a preset mapping relationship between thermal location information and category information is obtained.

[0189] In some embodiments of this application, the extraction unit 602 is specifically used for:

[0190] By using the heatmap mapping parameters, feature extraction is performed on the target sample image to obtain the target feature map of the target sample image;

[0191] Based on the target feature map, a sample heatmap of the target sample image is obtained.

[0192] In some embodiments of this application, the image classification model training device 600 further includes a classification unit (not shown in the figure), which is specifically used for:

[0193] Acquire the target scene image of the target scene region;

[0194] The trained image classification model is used to classify the target scene image to obtain the target scene state of the target scene region.

[0195] In some embodiments of this application, the target scene image includes a prior scene image and a subsequent scene image of the target scene region, and the classification unit is specifically used for:

[0196] Based on the prior sample image and the subsequent sample image, determine the sample optical flow information of the target sample image;

[0197] Using the category labeling information as supervision information, the image classification model to be trained is trained based on the prior sample images and the sample optical flow information until the preset first stop training condition is met, and the trained image classification model is obtained.

[0198] In some embodiments of this application, the target sample image includes a prior sample image and a subsequent sample image of the sample scene region, and the training unit 604 is specifically used for:

[0199] Based on the prior scene image and the subsequent scene image, determine the scene optical flow information of the target scene image;

[0200] The target scene image is classified using the trained image classification model based on the prior scene image and the scene optical flow information to obtain the target scene state of the target scene region.

[0201] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0202] Because this image classification model training device can perform the functions described in this application, Figures 1 to 5 Corresponding to the steps in the image classification model training method in any embodiment, the present application can be implemented as described above. Figures 1 to 5 For details on the beneficial effects that the image classification model training method can achieve in any embodiment, please refer to the preceding description, which will not be repeated here.

[0203] Furthermore, to better implement the image classification model training method in the embodiments of this application, based on the image classification model training method, the embodiments of this application also provide an electronic device, see below. Figure 7 , Figure 7 This illustration shows a structural diagram of an electronic device according to an embodiment of this application. Specifically, the electronic device provided in this embodiment includes a processor 701, which executes a computer program stored in a memory 702 to implement, for example... Figures 1 to 5 Corresponding to each step of the image classification model training method in any embodiment; or, when the processor 701 executes the computer program stored in the memory 702, it implements as follows: Figure 6 The functions of each unit in the corresponding embodiment.

[0204] For example, a computer program can be divided into one or more modules / units, one or more of which are stored in memory 702 and executed by processor 701 to complete the embodiments of this application. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device.

[0205] The electronic device may include, but is not limited to, processor 701 and memory 702. Those skilled in the art will understand that the illustrations are merely examples of an electronic device and do not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc., and processor 701, memory 702, input / output devices, and network access devices are connected via a bus.

[0206] The processor 701 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.

[0207] The memory 702 can be used to store computer programs and / or modules. The processor 701 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 702 and by calling data stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, video data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0208] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the image classification model training device, electronic equipment, and their corresponding units described above can be found in, for example... Figures 1 to 5 The specific details of the image classification model training method in any embodiment will not be repeated here.

[0209] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0210] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the present application. Figures 1 to 5 For the steps in the image classification model training method corresponding to any embodiment, please refer to the following for specific operations: Figures 1 to 5 The description of the image classification model training method corresponding to any embodiment will not be repeated here.

[0211] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0212] Because of the instructions stored in the computer-readable storage medium, the present application can be executed as described above. Figures 1 to 5 Corresponding to the steps in the image classification model training method in any embodiment, the present application can be implemented as described above. Figures 1 to 5For details on the beneficial effects that the image classification model training method can achieve in any embodiment, please refer to the preceding description, which will not be repeated here.

[0213] The foregoing has provided a detailed description of an image classification model training method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for training an image classification model, characterized in that, The method includes: Acquire the target sample image; By using preset heatmap mapping parameters, thermal information is extracted from the target sample image to obtain a sample heatmap of the target sample image. The heatmap mapping parameters are used to reflect the constraint relationship between the sample heatmap and the category labeling information. The heatmap mapping parameters are learned based on the preset heatmap extraction module. Based on the sample heatmap, a preset mapping relationship between heatmap location information and category information is obtained. The preset mapping relationship is used to indicate the relationship between the heatmap location information and the category labeling information of the target sample image. The heatmap location information is obtained. Based on the heatmap location information and the preset mapping relationship, the category labeling information of the target sample image is determined. The heatmap location information refers to the object or logo corresponding to the highlighted position in the heatmap, and the category labeling information refers to the labeled category of the target sample image. Based on the category labeling information and the target sample image, the image classification model to be trained is trained until a preset first stop training condition is met, and a trained image classification model is obtained.

2. The image classification model training method according to claim 1, characterized in that, Before training the image classification model based on the category labeling information and the target sample image until a preset first stopping training condition is met, and before obtaining the trained image classification model, the process further includes: Obtain an image sample set and preset annotation data, wherein the image sample set includes multiple initial sample images, and the preset annotation data includes preset annotation categories of the initial sample images; Based on a preset classification model, the predicted category of each initial sample image in the image sample set is obtained. From the image sample set, obtain the target initial sample image whose preset labeled category is different from the predicted category; Using the predicted category of the target initial sample image as supervision information, the preset classification model is trained based on the target initial sample image until a preset second stopping training condition is met, thus obtaining the image classification model to be trained.

3. The image classification model training method according to claim 2, characterized in that, Before determining the category labeling information of the target sample image based on the sample heatmap, the method further includes: By using the heatmap mapping parameters, thermal information is extracted from the initial target sample image to obtain the target heatmap of the initial target sample image; The target category of the initial sample image is obtained by predicting the target category based on the target initial sample image using the image classification model to be trained. Obtain the thermal location information of the target heatmap; Based on the thermal location information of the target heat map and the target category, a preset mapping relationship between thermal location information and category information is obtained.

4. The image classification model training method according to claim 1, characterized in that, The step of extracting thermal information from the target sample image using preset heatmap mapping parameters to obtain a sample heatmap of the target sample image includes: By using the heatmap mapping parameters, feature extraction is performed on the target sample image to obtain the target feature map of the target sample image; Based on the target feature map, a sample heatmap of the target sample image is obtained.

5. The image classification model training method according to any one of claims 1-4, characterized in that, The method further includes: Acquire the target scene image of the target scene region; The trained image classification model is used to classify the target scene image to obtain the target scene state of the target scene region.

6. The image classification model training method according to claim 5, characterized in that, The target sample image includes a prior sample image and a subsequent sample image of the sample scene region, and the target scene image includes a prior scene image and a subsequent scene image of the target scene region; The step of training the image classification model to be trained based on the category labeling information and the target sample image until a preset first stopping training condition is met, thereby obtaining a trained image classification model, includes: Based on the prior sample image and the subsequent sample image, determine the sample optical flow information of the target sample image; Using the category labeling information as supervision information, the image classification model to be trained is trained based on the prior sample images and the sample optical flow information until the preset first stop training condition is met, and the trained image classification model is obtained. The step of classifying the target scene image using the trained image classification model to obtain the target scene state of the target scene region includes: Based on the prior scene image and the subsequent scene image, determine the scene optical flow information of the target scene image; The target scene image is classified using the trained image classification model based on the prior scene image and the scene optical flow information to obtain the target scene state of the target scene region.

7. An image classification model training device, characterized in that, The image classification model training device includes: The acquisition unit is used to acquire the target sample image; The extraction unit is used to extract thermal information from the target sample image using preset heatmap mapping parameters to obtain a sample heatmap of the target sample image. The heatmap mapping parameters are used to reflect the constraint relationship between the sample heatmap and the category labeling information. The heatmap mapping parameters are learned based on the preset heatmap extraction module. The determining unit is configured to: obtain a preset mapping relationship between thermal location information and category information based on the sample heatmap; the preset mapping relationship is used to indicate the relationship between the thermal location information of the sample heatmap and the category labeling information of the target sample image; obtain the thermal location information of the sample heatmap; and determine the category labeling information of the target sample image based on the thermal location information of the sample heatmap and the preset mapping relationship, wherein the thermal location information refers to the object or label corresponding to the highlighted position in the heatmap, and the category labeling information refers to the labeled category of the target sample image; The training unit is used to train the image classification model to be trained based on the category labeling information and the target sample image until a preset first stop training condition is met, thus obtaining the trained image classification model.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the image classification model training method as described in any one of claims 1 to 6 when it invokes the computer program in the memory.

9. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to execute the steps in the image classification model training method according to any one of claims 1 to 6.

Images