Training method of image classification model, electronic device and readable storage medium

Abstract

The application discloses a kind of training method of image classification model, electronic equipment and readable storage medium, belong to terminal technical field.The method includes: obtaining positive sample image training set, multiple sample images in positive sample image training set and multiple first text information one-to-one correspondence;According to positive sample image training set, determine negative sample image training set, including multiple sample images in the negative sample image training set, the text information of each sample image in negative sample image training set is second text information, the second text information of any one sample image in negative sample image training set is one first text information in multiple first text information except the first text information corresponding to any one sample image;Based on positive sample image training set and negative sample image training set, initial classification model is iteratively trained, and target image classification model picture is obtained.So, reduce the tediousness of data labeling, improve the accuracy of training image classification model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of terminals, and in particular to a training method of an image classification model, an electronic device and a readable storage medium. BACKGROUND

[0002] With the development of terminal technology, the application of image processing technology is also more and more widely, such as applied in the field of intelligent transportation, can be used for road recognition, and so on, such as applied in the field of medicine, can be used for medical diagnosis. Among them, the recognition and classification of images are the basis of image processing technology, and in order to realize the recognition and classification of images, the electronic device can train the image classification model.

[0003] At present, the user can mark the sample image through the text information, and the text information is used to describe the scene category to which the sample image belongs, such as in the field of intelligent transportation, the sample image can be marked as belonging to the viaduct scene, the urban road scene, the rain and fog scene and the like through the text information. The electronic device can train the model based on the sample images of different scene categories marked by the user to obtain the image classification model, and the trained image classification model can be used to identify the scene category corresponding to any image.

[0004] However, in the process of training the image classification model, in order to obtain an image classification model with better classification effect, it is required that the sample image can be marked more detailed when the data of the sample image is marked, and the work of marking data is relatively complex and tedious, which is difficult to achieve. Therefore, there is an urgent need for a training method of an image classification model for improving the classification effect. SUMMARY

[0005] The present application provides a training method of an image classification model, an electronic device and a readable storage medium, which can be used to reduce the tediousness of sample data marking and improve the accuracy of image classification model training. The technical solution is as follows:

[0006] In a first aspect, a training method of an image classification model is provided, applied in an electronic device, and the method comprises:

[0007] obtain a positive sample image training set, the positive sample image training set comprising a plurality of sample images and a plurality of first text information, each sample image belonging to at least one of a plurality of scene categories, the plurality of sample images corresponding to the plurality of first text information one by one, and one first text information being used to describe a scene category to which a corresponding sample image belongs; determine a negative sample image training set according to the positive sample image training set, the negative sample image training set comprising a plurality of sample images, and text information of each sample image in the negative sample image training set being second text information, the second text information of any sample image in the negative sample image training set being one of the plurality of first text information except for the first text information corresponding to the any sample image; and iteratively train an initial classification model based on the positive sample image training set and the negative sample image training set to obtain a target image classification model, the target image classification model being capable of identifying an image belonging to at least one of the plurality of scene categories.

[0008] It should be noted that each sample image belongs to at least one of a plurality of scene categories means that any sample image can involve multiple scene categories or only one scene category.

[0009] As an example, the image content of each sample image in the negative sample image training set does not match the scene category described by the corresponding second text information.

[0010] It should be noted that the plurality of sample images in the negative sample image training set are the same as the plurality of sample images in the positive sample image training set, that is, the electronic device can determine the plurality of sample images in the positive sample image training set as the plurality of sample images in the negative sample image training set, and for the same sample image, the electronic device can adjust the text information corresponding to the sample image in the negative sample training set according to the plurality of first text information. In other words, for any sample image in the negative sample image, the text information corresponding to the sample image in the positive sample image training set is different from the sample information corresponding to the sample image in the negative sample image training set.

[0011] Since in the process of constructing the negative sample image training set, the plurality of sample images in the negative sample image training set are the plurality of sample images in the positive sample image training set, and the negative sample image training set also includes a plurality of first text information. However, for the same sample image, the text information corresponding to the sample image in the positive sample image training set is different from the sample information corresponding to the sample image in the negative sample image training set, so that in the process of constructing the positive sample image training set and the negative sample image training set, only one data labeling is required, thereby reducing the tediousness of data labeling and improving the efficiency of data labeling. In addition, since the same thing is used in the process of training the image classification model, different modal information is used, so that the trained image classification model is more accurate.

[0012] As an example of the present application, the operation of determining the negative sample image training set according to the positive sample image training set by the electronic device includes:

[0013] determining the similarity between the target sample image and the first text information corresponding to each target scene category, the target sample image being any one of the sample images in the positive sample image training set, and each target scene category being each scene category in the plurality of scene categories except the scene category to which the target sample image belongs; determining the first text information corresponding to the target scene category with the largest similarity as the second text information corresponding to the target sample image; and determining the plurality of sample images and the second text information corresponding to each sample image as the negative sample image training set.

[0014] As an example, the electronic device can also determine the similarity between the target sample image and the first text information of each scene category to which the target sample image does not belong, and determine the first text information of the scene category to which the target sample image does not belong with the largest similarity as the second text information.

[0015] It is worth noting that by determining the first text information corresponding to the target scene category with the largest similarity as the second text information corresponding to the target sample image, not only can the construction of the negative sample be successfully completed, but also since the similarity between the second text information and the corresponding target sample image is the highest, by training the image classification model to be trained by the target sample image and the corresponding second text information, the accuracy of the image classification model in identifying images with high similarity can be improved.

[0016] As an example of the present application, the operation of determining the similarity between the target sample image and the first text information corresponding to each target scene category by the electronic device includes:

[0017] The target sample image is processed by the pre-trained target image encoder to obtain a visual feature vector of the target sample image; the first text information corresponding to each target scene category is processed by the pre-trained target text encoder to obtain a text feature vector of the first text information corresponding to each target scene category; and a similarity between the visual feature vector of the target sample image and the text feature vector corresponding to each target scene category is determined.

[0018] It should be noted that the target text encoder and the target image encoder are trained by mutual cooperation in the pre-training process.

[0019] It should be noted that the consistency of the feature vectors can be ensured by processing each first text information by the pre-trained target text encoder and processing each sample image by the target image encoder respectively.

[0020] As an example of the present application, before determining the similarity between the target sample image and the first text information corresponding to each target scene category, the electronic device can also obtain a plurality of sample text information and a plurality of sample training images, the plurality of sample text information and the plurality of sample training images correspond one-to-one, and each sample text information in the plurality of sample text information is used to describe a scene category to which a corresponding sample training image belongs; the initial text encoder is iteratively trained based on the plurality of sample text information, and the initial image encoder is iteratively trained based on the plurality of sample training images; during the iterative training process, a loss value of a first loss function between the text encoder obtained after each training and the image encoder obtained after each training is determined; in the case of loss value convergence, the text encoder obtained at the time of convergence is determined as the target text encoder, and the image encoder obtained at the time of convergence is determined as the target image encoder, the target text encoder is used to determine the text feature vector of the first text information corresponding to each target scene category, and the target image encoder is used to determine the visual feature vector of the target sample image.

[0021] It should be noted that the plurality of sample text information and the plurality of sample training images correspond one-to-one, which means that for any one sample text information D, there is only one sample training image in the plurality of sample training images whose image content is the same as the content described by the any one sample text information D. In addition, during the iterative training process, the corresponding sample text information and sample training image can be trained in pairs.

[0022] Thus, by training the text encoder and the image encoder using the data matched by the text and the image (i.e., the sample text information and the corresponding sample training image), the sample training image and the corresponding sample text information in two modalities can be mapped to the same feature space, so that the text encoder and the image encoder with good training effect can be obtained.

[0023] As an example of the present application, the electronic device trains the initial classification model based on the positive sample image training set and the negative sample image training set to obtain the target image classification model, and the operation includes:

[0024] The visual feature vector of each sample image in the positive sample image training set is spliced with the text feature vector of the corresponding first text information to obtain a plurality of positive sample mixed feature vectors. The visual feature vector of each sample image in the negative sample image training set is spliced with the text feature vector of the corresponding second text information to obtain a plurality of negative sample mixed feature vectors. The initial classification model is iteratively trained according to the plurality of positive sample mixed feature vectors and the plurality of negative sample mixed feature vectors. In the iterative training process, the loss value of the second loss function between the classification result of the image classification model obtained after each training and the preset result is determined. In the case where the loss value of the second loss function converges, the image classification model obtained at the time of convergence is determined as the target image classification model.

[0025] It should be noted that the preset result is the classification label corresponding to each sample image in the positive sample image training set and the classification label corresponding to each sample image in the negative sample image training set. That is, in the case of training according to the sample images in the positive sample image training set, the preset result is the classification label corresponding to each sample image in the positive sample image training set, and in the case of training according to the sample images in the negative sample image training set, the preset result is the classification label corresponding to each sample image in the negative sample image training set.

[0026] Since the same thing in different modalities is used in the process of training the image classification model, the trained image classification model is more accurate.

[0027] As an example of the present application, after the electronic device obtains the target image classification model by iteratively training the initial classification model based on the positive sample image training set and the negative sample image training set, the electronic device can further obtain a to-be-classified image, determine a visual feature vector of the to-be-classified image, determine a similarity between the visual feature vector of the to-be-classified image and each of the plurality of first text information, obtain a plurality of similarities, splice the visual feature vector of the to-be-classified image and the text feature vector corresponding to each of the N first text information respectively, obtain N mixed feature vectors, the N first text information is the first text information corresponding to each of the N similarities after the plurality of similarities are arranged in descending order, and the N is a positive integer greater than or equal to 1, and process the N mixed feature vectors through the target image classification model to obtain a classification result of the to-be-classified image.

[0028] Since one image classification model can recognize all scene categories to which the to-be-classified image belongs through the target image classification model, the scene category to which the to-be-classified image belongs can be obtained at one time as much as possible, the efficiency of image classification of the target image classification model is improved, and the electronic device can use different display schemes according to the scene category to which the to-be-classified image belongs, thereby improving the display quality and display effect of the to-be-classified image.

[0029] As an example of the present application, the operation of the electronic device obtaining the to-be-classified image includes:

[0030] In the case where the camera is turned on, the preview image captured by the camera is determined as the to-be-classified image; or,

[0031] In the case where the image selection operation is received, the image selected by the image selection operation is determined as the to-be-classified image.

[0032] It should be noted that the case where the camera is turned on includes the case where the camera is turned on to scan two-dimensional codes, bar codes and the like, or the case where the camera is turned on to perform character recognition, object recognition, or the case where the camera is turned on to take pictures, etc.

[0033] Therefore, since the to-be-classified image can be a preview image or an image selected by a user, the electronic device can realize scene recognition of any image, thereby increasing the application scenarios of the image classification model and improving the practicality of the image classification model.

[0034] As an example of the present application, when the camera is turned on, the operation of the electronic device determining the preview image collected by the camera as the image to be classified includes: when the camera is turned on, displaying a scene recognition control in the shooting interface, and collecting an image through the camera to obtain a preview image, the scene recognition control is used to control whether to perform scene recognition; in response to the opening operation of the scene recognition control, the preview image is determined as the image to be classified.

[0035] As an example, when the scene recognition control is in the closed state, that is, the user does not perform the opening operation on the scene recognition control, the electronic device will not perform the operation of the scene recognition (or image recognition) behind.

[0036] In this way, by setting the scene recognition control, the user can autonomously select whether to perform scene recognition, thereby increasing the interactivity with the user, and saving the running resources of the electronic device without performing scene recognition.

[0037] As an example of the present application, the image to be classified is the preview image; in this way, after the electronic device determines the scene category to which the image to be classified belongs according to the classification result output by the target image classification model, it can also receive a shooting operation; in response to the shooting operation, the preview image after exposure is stored in the image folder corresponding to the scene category to which the preview image belongs.

[0038] As an example, in order to save storage space, the electronic device can also store the image to be classified in any one of the multiple scene categories to which the image to be classified belongs. Alternatively, the electronic device stores an image to be classified, and stores the image identifier of the image to be classified and the corresponding scene category. In the case of displaying images according to the classification mode, the electronic device can acquire the image to be classified according to the image identifier of the image to be classified, and display the image to be classified in the classification display interface.

[0039] In this way, by storing the preview image after exposure in the image folder corresponding to the scene category to which the preview image belongs, it is convenient for the user to find images according to the scene category, thereby improving the interactivity with the user and the user's stickiness.

[0040] As an example, when the image to be classified is the preview image, the electronic device can also display a scene label in the preview image after determining the scene category to which the image to be classified belongs, and the scene label is used to describe the scene category to which the image to be classified belongs.

[0041] As an example, in a case where the image to be classified is a preview image, after determining the scene category to which the image to be classified belongs, the electronic device can further determine an imaging scheme of the image to be classified according to the scene category of the image to be classified, such as determining an exposure parameter, a filter scheme, a shooting mode, a display resolution, and the like of the image to be classified.

[0042] In a second aspect, an electronic device is provided, which includes a processor and a memory in its structure. The memory is configured to store a program supporting the electronic device to execute the training method of the image classification model provided in the first aspect, and store data involved in the training method of the image classification model provided in the first aspect. The processor is configured to execute the program stored in the memory. The electronic device can further include a communication bus configured to establish a connection between the processor and the memory.

[0043] In a third aspect, a computer readable storage medium is provided, which stores instructions, when executed on a computer, cause the computer to execute the training method of the image classification model provided in the first aspect.

[0044] In a fourth aspect, a computer program product is provided, which includes instructions, when executed on a computer, cause the computer to execute the training method of the image classification model provided in the first aspect.

[0045] The technical effects obtained by the second aspect, the third aspect and the fourth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0048] Figure 3 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0049] Figure 4 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0050] Figure 5 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0051] Figure 6 is a flowchart of a training method of an image classification model provided by an embodiment of the present application;

[0052] Figure 7is a training process schematic diagram of an image classification model provided by an embodiment of the present application;

[0053] Figure 8 is a method flow schematic diagram of image classification provided by an embodiment of the present application;

[0054] Figure 9 is a process schematic diagram of an image classification method provided by an embodiment of the present application;

[0055] Figure 10 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0056] Figure 11 is a block diagram of a software system of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0058] It should be understood that the "multiple" mentioned in the present application refers to two or more than two. In the description of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. In addition, in order to clearly describe the technical solutions of the present application, the same items or similar items with basically the same functions and roles are distinguished by using "first", "second", etc. The skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. also do not necessarily mean different.

[0059] In the present application, the reference "one embodiment" or "some embodiments" means that in one or more embodiments of the present application, the specific features, structures or characteristics described in connection with the embodiment are included. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in other some embodiments" and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "include but not limited to", unless otherwise specifically emphasized.

[0060] With the development of terminal technology, the application of image processing technology is also more and more widely. Among them, the recognition and classification of images is the basis of image processing technology, and under normal circumstances, electronic devices can recognize images or scene recognition through image classification models. In some scenarios, in order to enable users to quickly find the images they need, the phone can classify the images in the gallery according to the scene categories shown, such as Figure 1 In order to facilitate the classification of images, a pre-trained image classification model can be built into the phone. That is, before the phone is shipped, a pre-trained image classification model can be built into the phone. Among them, the image classification model can be obtained by other electronic devices based on the model training of sample images of different scene categories marked. The sample images of different scene categories marked refer to that the workers mark different sample images through text information, and the text information is used to describe the scene category to which the sample image belongs, for example, the sample image can be marked through text information that it belongs to the viaduct scene, urban road scene, rain and fog scene, building scene, food scene, night scene, certificate scene, text scene, etc.

[0061] However, in the process of training the image classification model, in order to obtain an image classification model with better classification effect, it is required that the sample images can be marked more detailed when the data is marked, and the work of marking data is relatively complex and tedious, which is difficult to achieve. Of course, there are also some image classification models that only use single information (such as only use image information) as training data, which leads to that the trained image classification model cannot recognize some complex scenes.

[0062] In order to improve the classification accuracy of the image classification model and reduce the difficulty of the labeled data, an embodiment of the present application provides a training method of an image classification model. In the method, an electronic device can obtain a positive sample image training set, the positive sample image training set including a plurality of sample images and a plurality of first text information, each sample image belonging to at least one of a plurality of scene categories, and each first text information describing a scene category to which a corresponding sample image in the plurality of sample images belongs. A negative sample image training set is determined according to the positive sample image training set, the negative sample image training set including a plurality of sample images, and the text information corresponding to each sample image in the plurality of sample images in the negative sample image training set being second text information, the second text information being one of the plurality of first text information except for the first text information corresponding to any one sample image. Based on the positive sample image training set and the negative sample image training set, an initial image classification model is iteratively trained to obtain a target image classification model capable of recognizing images belonging to at least one of a plurality of scene categories. Since in the process of constructing the negative sample image training set, the plurality of sample images in the negative sample image training set are the plurality of sample images in the positive sample image training set, and the negative sample image training set also includes the plurality of first text information. However, for the same sample image, the text information corresponding to the sample image in the positive sample image training set is different from the sample information corresponding to the sample image in the negative sample image training set. Therefore, in the process of constructing the positive sample image training set and the negative sample image training set, only one data labeling is required, thereby reducing the complexity of data labeling and improving the efficiency of data labeling. In addition, since the same thing in different modalities is used in the process of training the image classification model, the trained image classification model is more accurate.

[0063] For ease of understanding, before the method provided by the embodiments of the present application is described in detail, the application scenarios related to the embodiments of the present application are introduced.

[0064] Please refer to Figure 2 , Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present application is shown in the following figure. In an application scenario, a user may take a picture through the camera of a mobile phone during use of the mobile phone. After starting the camera, the mobile phone can display a picture taken by the camera on the display screen. Figure 2The shooting interface shown in (a) of FIG. 1, in which a preview image A is displayed, the preview image A including a blue sky, grass, and trees. The phone, upon capturing the preview image A, can input the preview image A into a target image classification model that can identify images belonging to at least one of a plurality of scene categories, such as images belonging to a building scene category, images belonging to a tree scene category, images belonging to a grass scene category, images belonging to a blue sky scene category, images belonging to a flower scene category, images belonging to a night scene category, images belonging to a beach scene category, images belonging to a sea scene category, images belonging to an amusement park scene category, images belonging to a school scene category, and the like. The target image classification model can perform scene recognition (or image recognition, image classification) on the preview image A, and the target image classification model can output a classification result for the preview image A. See Figure 2 (b) of FIG. 1, the phone can display scene labels for the scene categories to which the preview image A belongs on the preview image A based on the classification result. For example, if the phone determines that the preview image A belongs to a sky scene category, a grass scene category, and a tree scene category, the phone can display scene labels “sky,” “grass,” and “tree” on the preview image A at locations where the sky, the grass, and the trees are located.

[0065] In another application scenario, the phone can also adjust an image display scheme based on the scene categories to which the preview image A belongs, such as adjusting a shooting mode, adjusting an exposure parameter, adjusting a display resolution, and the like. For example, if the preview image A belongs to an overexposure scene category, such as a moon scene category, a fire scene category, a sun scene category, a light scene category, and the like, that is, if the preview image A includes a light source such as a moon, a light source, or a flame, the scene categories to which the preview image A belongs usually include an overexposure scene category. In this case, the phone can adjust the shooting mode or reduce the exposure parameter of the preview image A. See Figure 3 (a) of FIG. 1, the preview image A includes a moon, and the phone identifies, through the target image classification model, that the scene categories to which the preview image A belongs include a moon scene category. The phone can then adjust the shooting mode to a moon mode. In the moon mode, the exposure parameter of the preview image A is changed. That is, the phone can reduce the exposure of the preview image A and shorten the exposure time of the preview image A, and display a preview image B as shown in (b) of FIG. 1. Figure 3

[0066] In yet another application scenario, see Figure 4 ​In Figure (a), if the user is satisfied with the current preview image A, the user can click the shooting control P1 on the shooting interface. In response to the click on the shooting control P1, the phone can expose the image based on the preview image A, obtaining the exposed preview image A. Furthermore, the phone can store the exposed preview image A in the image folder corresponding to the scene category to which the preview image A belongs. That is, the phone can store the exposed preview image A in at least one of the image folders corresponding to the sky scene, the grass scene, and the tree scene, such as storing it in the image folder corresponding to the sky scene category. Afterwards, if the user needs to find an image of a specific scene category, please refer to... Figure 4 In Figure (b), the user can click the application icon of the Gallery app on the desktop. In response to the click on the Gallery app's application icon, the phone can display something like... Figure 4 In Figure (c), the image preview interface P2 allows users to preview the "Discover" control P3 displayed in P2. In response to a click on the "Discover" control P3, the phone can display... Figure 4 The category view interface P4 shown in Figure (d) can display multiple image folders, such as "People", "Location" and "Things". The "Things" folder can include folders such as "Buildings", "Sky", "Portraits", "Grass", "Trees" and "Night Scenes". In this way, users can continue to search for more detailed scene categories from the "Things" folder.

[0067] Please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating one application scenario provided by an embodiment of this application. In another application scenario, after the mobile phone activates the camera, it displays as shown in the diagram. Figure 5 The shooting interface shown in Figure (a) displays not only the preview image A captured by the camera, but also a scene recognition control. If the user needs to identify the scene category of the captured image, the user can click the scene recognition control P5; the phone responds to the click operation of the scene recognition control P5, see [reference needed]. Figure 5 In Figure (b), the mobile phone can change the display mode of the scene recognition control P5 (in this embodiment, the display color of the scene recognition control P5 is changed from white background with black text to black background with white text as an example), and input the preview image A into the target image classification model. The target image classification model processes the preview image A and outputs the classification result of the preview image A.

[0068] It should be noted that the embodiments in this application are based only on the above.Figures 2-5 The illustrated scenario is used for illustration only and does not limit the embodiments of the present application.

[0069] Based on the application scenarios provided in the above embodiments, the training method of the image classification model provided by the embodiments of the present application will be introduced next. Please refer to Figure 6 Figure 6 is a flowchart of a training method of an image classification model according to an example, which is used for illustration only and not as a limitation of the embodiments of the present application. The method is used as an example for illustration of application in an electronic device, and the method can include the following parts or all of the contents:

[0070] Step 601: Obtain a positive sample image training set.

[0071] It should be noted that the positive sample image training set includes a plurality of sample images and a plurality of first text information. Each sample image belongs to at least one of a plurality of scene categories. The plurality of sample images correspond one-to-one to the plurality of first text information. One first text information is used to describe the scene category to which the corresponding sample image belongs.

[0072] As an example, the one-to-one correspondence between the plurality of sample images and the plurality of first text information means that for any one of the plurality of first text information C, there is only one sample image in the plurality of sample images whose image content is the same as the content described by the any one of the plurality of first text information C.

[0073] For example, one of the plurality of first text information can be "flower, a photo of flower sea" or "flower, an image of flower sea", and there is a sample image in the plurality of sample images whose image content is flower or flower sea. And the first text information describes the scene category to which the sample image belongs as the flower scene category.

[0074] It should be noted that each sample image belongs to at least one of a plurality of scene categories, which means that any one sample image can involve multiple scene categories, or it can only involve one scene category.

[0075] ​Exemplarily, the plurality of scene categories can include a blue sky scene category, a tree scene category, a flower scene category, a cat scene category, a dog scene category, a sea scene category, a building scene category, a portrait scene category, a night scene category, a snow scene category, and the like. If the image content of a sample image includes objects such as blue sky, flowers, and buildings, then the scene category to which the sample image belongs includes the blue sky scene category, the flower scene category, and the building scene category, and the first text information corresponding to the sample image is “an image of blue sky, flowers, and buildings”, which describes three scene categories. If the image content of a sample image includes a cat, then the scene category to which the sample image belongs is the cat scene category, and the first text information corresponding to the sample image is “an image of a cat”.

[0076] As an example, for any one of the plurality of sample images, the electronic device can receive an input operation for the any one of the plurality of sample images, and determine the text information carried by the input operation as the first text information corresponding to the any one of the plurality of sample images. Alternatively, the electronic device can perform a text grabbing operation in a specific page, and determine the grabbed text information as the first text information for the any one of the plurality of sample images. The specific page is a page describing the scene category to which the any one of the plurality of sample images belongs, and the scene category to which the any one of the plurality of sample images belongs is described by text information in the specific page.

[0077] Step 602: determining a negative sample image training set according to the positive sample image training set.

[0078] It should be noted that the negative sample image training set includes a plurality of sample images, the text information of each sample image in the negative sample image training set is second text information, and the second text information of any one of the sample images in the negative sample image training set is one of the plurality of first text information except the first text information corresponding to any one of the sample images.

[0079] As an example, the image content of each sample image in the negative sample image training set does not match the scene category described by the corresponding second text information.

[0080] In some embodiments, the operation of determining, by the electronic device, the negative sample image training set according to the positive sample image training set comprises: determining a similarity between a target sample image and first text information corresponding to each target scene category, the target sample image being any one of the sample images in the positive sample image training set, and the each target scene category being each scene category in the plurality of scene categories except for a scene category to which the target sample image belongs; determining the first text information corresponding to a target scene category with the largest similarity as second text information corresponding to the target sample image; and determining the plurality of sample images and the second text information corresponding to each sample image as the negative sample image training set.

[0081] It should be noted that the plurality of sample images in the negative sample image training set are the same as the plurality of sample images in the positive sample image training set, that is, the electronic device can determine the plurality of sample images in the positive sample image training set as the plurality of sample images in the negative sample image training set, and for the same sample image, the electronic device can adjust the text information corresponding to the sample image in the negative sample training set according to the plurality of first text information. In other words, for any one of the sample images in the negative sample image, the text information corresponding to the sample image in the positive sample image training set is different from the sample information corresponding to the sample image in the negative sample image training set.

[0082] In some embodiments, for a target sample image (any one of the sample images in the negative sample image set, or any one of the plurality of sample images), the electronic device can determine any one of the first text information other than the first text information corresponding to the target sample image (in the positive sample image training set) as the second text information corresponding to the target sample image in the negative sample image training set. Of course, in order to improve the accuracy of the trained image classification model in classifying similar images, the electronic device can determine the first text information corresponding to the target scene category with the largest similarity as the second text information corresponding to the target sample image.

[0083] Exemplarily, the image content of the target sample image is a dog, and the corresponding first text information of the target sample image in the positive sample image training set is "an image of a dog", i.e., the corresponding scene category of the target sample image is a dog scene category, then the electronic device can determine the first text information corresponding to other scene categories in the plurality of scene categories except the dog scene category (wherein, if the first text information describes the dog scene category and a grassland scene category, the electronic device can also obtain the text information describing the grassland scene category in the first text information, but will not obtain the text information describing the dog scene category in the first text information). Then, the electronic device can determine the similarity between the target sample image and the first text information corresponding to the determined other scene categories, and determine the first text information corresponding to the target scene category with the highest similarity as the second text information corresponding to the target sample image.

[0084] As an example, the electronic device can also determine the similarity between the target sample image and each first text information not describing the scene category to which the target sample image belongs, and determine the first text information of the scene category not describing the target sample image as the second text information.

[0085] Exemplarily, the image content of the target sample image is a dog, and the corresponding first text information of the target sample image in the positive sample image training set is "an image of a dog", i.e., the corresponding scene category of the target sample image is a dog scene category, then the electronic device can determine the first text information not including the description of the dog scene category, and determine the similarity between the target sample image and each first text information not describing the dog scene category, and determine the first text information not describing the dog scene category with the highest similarity as the second text information.

[0086] It is worth noting that by determining the first text information corresponding to the target scene category with the highest similarity as the second text information corresponding to the target sample image, not only can the construction of the negative sample be successfully completed, but also since the similarity between the second text information and the corresponding target sample image is the highest, in this way, by training the image classification model to be trained by the target sample image and the corresponding second text information, the accuracy of the image classification model in identifying images with high similarity can be improved.

[0087] In some embodiments, the operation of determining the similarity between the target sample image and the first text information corresponding to each of the target scene categories includes: processing the target sample image by using a pre-trained target image encoder to obtain a visual feature vector of the target sample image; processing the first text information corresponding to each of the target scene categories by using a pre-trained target text encoder to obtain a text feature vector of the first text information corresponding to each of the target scene categories; and determining the similarity between the visual feature vector of the target sample image and the text feature vector corresponding to each of the target scene categories.

[0088] As an example, the electronic device can determine at least one of the Euclidean distance, the cosine distance, the Jaccard distance, etc. between the visual feature vector of the target sample image and the text feature vector corresponding to each of the target scene categories, and in the case of determining one of the distances, determine the obtained distance as the similarity between the visual feature vector of the target sample image and the text feature vector corresponding to each of the target scene categories; in the case of determining multiple distances, determine the average of the multiple distances as the similarity between the visual feature vector of the target sample image and the text feature vector corresponding to each of the target scene categories; or in the case of determining multiple distances, assign different weights to each distance and then add them, and determine the obtained sum as the similarity between the visual feature vector of the target sample image and the text feature vector corresponding to each of the target scene categories. The embodiments of the present application do not make specific limitations on this.

[0089] It should be noted that the target text encoder and the target image encoder are trained by mutual cooperation in the pre-training process.

[0090] It should be noted that processing each of the first text information by using the pre-trained target text encoder and processing each of the sample images by using the target image encoder respectively can ensure the consistency of the feature vectors.

[0091] In some embodiments, before the electronic device determines the similarity between the target sample image and the first text information corresponding to each of the target scene categories, the target text encoder and the target image encoder can also be pre-trained.

[0092] As an example, the electronic device can obtain a plurality of sample text information and a plurality of sample training images, the plurality of sample text information corresponds to the plurality of sample training images one by one, and each sample text information in the plurality of sample text information is used to describe a scene category to which a corresponding sample training image belongs; iteratively train an initial text encoder based on the plurality of sample text information, and iteratively train an initial image encoder based on the plurality of sample training images; in the iterative training process, determine a loss value of a first loss function between the text encoder obtained after each training and the image encoder obtained after each training; in the case of loss value convergence, the text encoder obtained at the time of convergence is determined as a target text encoder, and the image encoder obtained at the time of convergence is determined as a target image encoder, the target text encoder is used to determine a text feature vector of first text information corresponding to each target scene category, and the target image encoder is used to determine a visual feature vector of a target sample image.

[0093] It should be noted that the plurality of sample text information corresponds to the plurality of sample training images one by one means that for any one sample text information D in the plurality of sample text information, there is only one sample training image in the plurality of sample training images whose image content is the same as the content described by the any one sample text information D.

[0094] It should also be noted that in the iterative training process, the corresponding sample text information and the sample training image can be trained in pairs. For example, the sample text information is "a photo of a cat" or "an image of a cat", and the corresponding sample training image is an image of a cat. The sample text information is input into the text encoder to train the text encoder, and the sample training image is input into the image encoder to train the image encoder.

[0095] It is worth noting that since the text encoder and the image encoder are trained using data matched by text and images (i.e., sample text information and corresponding sample training images), the sample training image and the corresponding sample text information can be mapped to the same feature space, and by calculating the similarity between the sample training image and the sample text information, the similarity between the sample training image and its corresponding sample text information is constrained to be the highest, and the similarity with other sample text information is low, so that the text encoder and the image encoder with good training effect can be obtained.

[0096] In some embodiments, the operation of determining, in the iterative training process, the loss value of the first loss function between the text encoder obtained after each training and the image encoder obtained after each training comprises: in the iterative training process, determining a sample text feature vector output by the text encoder obtained after each training and a sample visual feature output by the image encoder obtained after each training, and then determining the loss value of the first loss function according to the sample text feature vector and the sample visual feature vector.

[0097] For example, referring to Figure 7 In the process of iteratively training the text encoder and the image encoder, the electronic device can input multiple sample text information into the text encoder and input multiple sample training images into the image encoder. The text encoder can determine a sample text feature vector of the input sample text information, the image encoder can determine a sample visual feature vector of the input sample training image, and then the electronic device can determine the loss value of the first loss function according to the sample text feature vector and the sample visual feature vector.

[0098] As an example, the first loss function can be an image-text contrastive (ITC) loss function, and the first loss function can be as shown in the following first formula (1).

[0099]

[0100] It should be noted that in the above first formula (1), D w represents the Euclidean distance between two sample feature vectors (X1 and X2), P is the dimension of the feature, Y is the label indicating whether the two sample feature vectors match, where Y is 1 represents that the two samples are similar or match, Y is 0 represents that the two samples are not similar or do not match, m is a set threshold, and L(W, (Y, X1, X2)) is the loss value.

[0101] As an example, the first loss function can be represented as the above first formula (1), and of course can also be represented in other ways. For example, the first loss function can also be represented as the following second formula (2).

[0102]

[0103] It should be noted that in the above second formula (2), L itc is the loss value, s(I, T) and s(T, I) are the similarity between vectors, ω and v are network parameters, g v (v cls ) is a text feature vector, g’ w (w’ cls ) is a visual feature vector, and M is the number of scene categories. is a matching similarity between any one text feature vector and a corresponding visual feature vector. is a matching similarity between any one visual feature vector and a corresponding text feature vector.

[0104] As an example, the loss value converges in the case that the loss value is less than or equal to a first preset value, and / or the change of the loss value is less than or equal to a second preset value. The first preset value and the second preset value can be set in advance according to requirements.

[0105] It should be noted that by training the text encoder and the image encoder, the target text encoder and the target image encoder are obtained, so that the training accuracy is higher.

[0106] In some embodiments, the electronic device can not only determine the text encoder obtained when the loss value converges as the target text encoder and determine the image encoder obtained when the loss value converges as the target image encoder in the case that the loss value converges. The electronic device can also determine the number of iterations of the training, and in the case that the number of iterations is greater than or equal to a number threshold, the electronic device can determine the text encoder obtained when the number of iterations is greater than or equal to the number threshold as the target text encoder, and determine the image encoder obtained when the number of iterations is greater than or equal to the number threshold as the target image encoder.

[0107] It should be noted that the number threshold can also be set in advance according to requirements, for example, the number threshold can be 100 times, 150 times, etc.

[0108] In some embodiments, the electronic device can not only determine the text feature vector of each first text information and the visual feature vector of each sample image in the above manner, but also determine the text feature vector of each first text information and the visual feature vector of each sample image in other manners. For example, the electronic device can determine the visual feature vector of each sample image through a visual base network model (such as a Resent (Residual Neural Network) model, etc.), and determine the text feature vector of each first text information through a multi-language text model. Alternatively, the electronic device can perform vectorization processing on each first text information through a text vector model to obtain the text feature vector of each first text information, and the text vector model can be determined based on a pre-trained language representation model (Bidirectional Encoder Representation from Transformers, BETR). The embodiments of the present application do not make specific limitations in this regard.

[0109] Since the negative sample image training set is determined according to the positive sample image training set, in general, in order to ensure the balance of model training, the ratio between the number of sample images in the positive sample image training set and the number of negative sample images is usually 1:1. Of course, it can also be other ratios, such as taking part of the sample images from multiple sample images as sample images in the negative sample training set, and the ratio between the number of sample images in the positive sample image training set and the number of negative sample images can also be 2:1, 1.5:1, etc. The embodiments of the present application do not make specific limitations.

[0110] In some embodiments, each sample image in the positive sample image training set corresponds to a classification label, and each sample image in the negative sample training set also corresponds to a classification label, wherein the classification label is used to indicate the relationship between the sample image and the corresponding text information. The classification label can be [0, 1] and [1, 0], and the first element in the classification label represents the probability that the sample image does not match the corresponding text information, and the second element represents the probability that the sample image matches the corresponding text information. That is, the classification label of each sample image in the positive sample image training set can be [0, 1], the first element 0 represents that the probability that the sample image does not match the corresponding first text information is 0, and the second element 1 represents that the probability that the sample image matches the corresponding first text information is 1. The classification label of each sample image in the negative sample image training set can be [1, 0], the first element 1 represents that the probability that the sample image does not match the corresponding second text information is 1, and the second element 0 represents that the probability that the sample image matches the corresponding second text information is 0.

[0111] It should be noted that the classification label is only taken as an example in the embodiments of the present application, and the classification label can also be other styles of labels.

[0112] Step 603: based on the positive sample image training set and the negative sample image training set, iteratively training the initial classification model to obtain a target image classification model.

[0113] It should be noted that the target image classification model can identify images belonging to at least one of multiple scene categories. The initial classification model can be an image-text matching (ITM) module.

[0114] In some embodiments, the operation of training the initial classification model based on the positive sample image training set and the negative sample image training set to obtain the target image classification model includes: splicing the visual feature vector of each sample image in the positive sample image training set with the text feature vector of the corresponding first text information to obtain a plurality of positive sample mixed feature vectors; splicing the visual feature vector of each sample image in the negative sample image training set with the text feature vector of the corresponding second text information to obtain a plurality of negative sample mixed feature vectors; iteratively training the initial classification model according to the plurality of positive sample mixed feature vectors and the plurality of negative sample mixed feature vectors; determining a loss value of a second loss function between the classification result of the image classification model obtained after each training and the preset result in the process of iterative training; and determining the image classification model obtained when the loss value of the second loss function converges as the target image classification model. This process can be referred to as the training diagram shown in FIG. 8. Figure 7

[0115] It should be noted that the preset result is the classification label corresponding to each sample image in the positive sample image training set and the classification label corresponding to each sample image in the negative sample image training set. That is, in the case of training according to the sample images in the positive sample image training set, the preset result is the classification label corresponding to each sample image in the positive sample image training set, and in the case of training according to the sample images in the negative sample image training set, the preset result is the classification label corresponding to each sample image in the negative sample image training set.

[0116] In some embodiments, in the process of iteratively training the initial classification model according to the plurality of positive sample mixed feature vectors and the plurality of negative sample mixed feature vectors, the electronic device can also perform dimensionality increasing or dimensionality reducing processing on each mixed feature vector (including each positive sample mixed feature vector in the plurality of positive sample mixed feature vectors and each negative sample mixed feature vector in the plurality of negative sample mixed feature vectors) according to the dimension of the model parameters of the initial classification model, so that the dimension of each mixed feature is the same as the dimension of the network parameters of the target image classification model. Then, the initial classification model is iteratively trained according to the plurality of processed mixed feature vectors.

[0117] As an example, the second loss function can be an ITM loss function. Of course, it can also be other loss functions, and the embodiments of the present application do not make specific limitations thereto.

[0118] It should be noted that the second loss function can be represented by the following third formula (3).

[0119] L itm = E (I，T′) ~ DH(y itm , p itm ​(I, M)) (3)

[0120] In the embodiment of the present application, since in the process of constructing the negative sample image training set, the plurality of sample images in the negative sample image training set are the plurality of sample images in the positive sample image training set, and the negative sample image training set also includes a plurality of first text information. However, for the same sample image, the text information corresponding to the sample image in the positive sample image training set is different from the sample information corresponding to the sample image in the negative sample image training set. Therefore, in the process of constructing the positive sample image training set and the negative sample image training set, only one data labeling is required, thereby reducing the tediousness of data labeling and improving the efficiency of data labeling. In addition, since the information of the same thing in different modalities is used in the process of training the image classification model, the trained image classification model is more accurate.

[0121] It should be noted that, in the case that the electronic device obtains the target image classification model, the electronic device can perform scene recognition (or image recognition, or image classification) on images belonging to different scene categories through the target image classification model. In order to understand the embodiments of the present application, the way in which the electronic device identifies the scene category to which the image belongs through the target image classification model will be explained next. In addition, the inference process of the target image classification model is the same as the process of applying the target image classification model to image recognition, and the embodiments of the present application will not explain the inference process of the target image classification model.

[0122] Please refer to FIG. 8, Figure 8 is a flowchart of an exemplary image classification method, which is used as an example and is not limited, and the method is used as an example to illustrate the application of the electronic device, and the method can include all or part of the following content:

[0123] Step 801: obtaining an image to be classified.

[0124] As an example, the operation of the electronic device obtaining the image to be classified includes: in the case that the camera is turned on, determining the preview image collected by the camera as the image to be classified; or in the case that an image selection operation is received, determining the image selected by the image selection operation as the image to be classified. Therefore, the image to be classified can be any image, such as a network downloaded image, or a preview image collected by the camera of the electronic device, or any image stored by the electronic device.

[0125] It should be noted that the case where the camera is turned on includes the case where the camera is turned on to scan two-dimensional codes, bar codes and the like, or the case where the camera is turned on to perform character recognition and object recognition, or the case where the camera is turned on to take pictures, etc.

[0126] As an example, the image selection operation can refer to a selection operation on any one image in the gallery (consisting of stored images) of the electronic device, or a download operation, a save operation, etc. on a network image, and the embodiments of the present application do not make specific limitations thereto.

[0127] It is worth noting that since the image to be classified can be a preview image or a user-selected image, the electronic device can implement scene recognition on any one image, thereby increasing the application scenarios of the target image classification model and improving the practicality of the target image classification model.

[0128] In some embodiments, in the case that the camera is turned on, the operation of the electronic device determining the preview image captured by the camera as the image to be classified includes: in the case that the camera is turned on, displaying a scene recognition control in the shooting interface, and obtaining a preview image through image capture by the camera, the scene recognition control being used to control whether to perform scene recognition; in response to an opening operation on the scene recognition control, determining the preview image as the image to be classified. Illustratively, the scene can refer to the application scenarios shown in the above Figure 5

[0129] Since not all scenes need to be identified, in order to allow the user to selectively perform scene recognition, the electronic device can also display a scene recognition control in the shooting interface.

[0130] As an example, in the case that the scene recognition control is in a closed state, that is, in the case that the user does not perform an opening operation on the scene recognition control, the electronic device will not perform the operation of subsequent scene recognition (or image recognition).

[0131] It is worth noting that by setting the scene recognition control, the user can autonomously select whether to perform scene recognition, thereby increasing the interactivity with the user, and saving the running resources of the electronic device in the case that scene recognition is not needed.

[0132] Step 802: determining the visual feature vector of the image to be classified.

[0133] As an example, the electronic device can process the visual feature vector of the image to be classified through the above-mentioned target image encoder to obtain the visual feature vector of the image to be classified, as shown in Figure 9 Or, the electronic device can also determine the visual feature vector of the image to be classified in other ways, such as through the above-mentioned visual base network model (such as Resent (Residual Neural Network) model, etc.). The embodiments of the present application do not make specific limitations thereto.

[0134] ​Step 803: determining a similarity between the visual feature vector of the image to be classified and each of the plurality of first text information, to obtain a plurality of similarities.

[0135] It should be noted that the electronic device can store the first text feature information corresponding to each of the plurality of scene categories and / or the text feature vector of the first text feature information corresponding to each of the plurality of scene categories. Thus, as shown in Figure 7 The electronic device can determine a similarity between the visual feature vector of the image to be classified and each of the plurality of first text information, to obtain a plurality of similarities.

[0136] In some embodiments, the electronic device can determine at least one of the Euclidean distance, the cosine distance, and the Jaccard distance between the visual feature vector of the image to be classified and each of the text feature vectors, and in the case of determining one of the distances, the obtained distance is determined as the similarity between the visual feature vector of the image to be classified and each of the text feature vectors; in the case of determining a plurality of distances, the average of the plurality of distances is determined as the similarity between the visual feature vector of the image to be classified and each of the text feature vectors; or in the case of determining a plurality of distances, each distance is assigned a different weight and then added, and the obtained sum is determined as the similarity between the visual feature vector of the image to be classified and each of the text feature vectors. The embodiments of the present application do not make specific limitations in this regard.

[0137] Step 804: splicing the visual feature vector of the image to be classified and the text feature vector corresponding to each of the N first text information, respectively, to obtain N hybrid feature vectors.

[0138] It should be noted that the N first text information is the first text information corresponding to each of the similarities located in the front N after the plurality of similarities are arranged in descending order, and the N is a positive integer greater than or equal to 1.

[0139] As an example, in the case of obtaining a plurality of similarities, the electronic device can sort the plurality of similarities in descending order to obtain a first sorting result. The text feature vector of the first text information corresponding to each of the similarities located in the front N in the first sorting result is obtained; each of the N text feature vectors obtained is spliced with the visual feature vector to be classified to obtain N hybrid feature vectors.

[0140] As an example, the electronic device may, in the case of obtaining a plurality of similarities, further sort the plurality of similarities in ascending order to obtain a second sorting result. The electronic device obtains a first text detailed text feature vector corresponding to each of the similarities located in the last N in the second sorting result. The electronic device splices each of the obtained N text feature vectors with the visual feature vector to be classified to obtain N hybrid feature vectors.

[0141] As an example, the electronic device may, in the case of obtaining a plurality of similarities, further traverse the plurality of similarities, and obtain the maximum similarity in the traversed similarities each time, and then continue to traverse the remaining similarities except the traversed maximum similarity, and continue to obtain the maximum similarity in the remaining similarities; the electronic device repeats the traversal operation until the electronic device obtains N similarities; then, the electronic device obtains a first text detailed text feature vector corresponding to each of the N similarities. The electronic device splices each of the obtained N text feature vectors with the visual feature vector to be classified to obtain N hybrid feature vectors.

[0142] In some embodiments, the electronic device performing the above steps 803 and 804 may be implemented through a target image classification model, or may not be implemented through a target image classification model. That is, after the electronic device obtains the visual feature vector of the image to be classified, the electronic device may input the visual feature vector of the image to be classified into the target image classification model, the target image classification model may store a plurality of text feature vectors of first text information, and the electronic device may determine the similarity between the visual feature vector of the image to be classified and each first text feature vector through the target image classification model to obtain a plurality of similarities; then the electronic device splices the visual feature vector of the image to be classified with each of the N text feature vectors to obtain N hybrid feature vectors. Alternatively, referring to Figure 9 , the electronic device obtains N hybrid feature vectors without using the target image classification model, and then inputs the N hybrid feature vectors into the target image classification model, and then performs the operation of the following step 805.

[0143] Step 805: processing the N hybrid feature vectors through the target image classification model to obtain a classification result of the image to be classified.

[0144] In some embodiments, the electronic device may perform dimensionality increasing or dimensionality reducing processing on each of the N hybrid feature vectors according to the dimension of the network parameters of the target image classification model, so that the dimension of each of the N hybrid feature vectors is the same as the dimension of the network parameters of the target image classification model. Then, the electronic device performs relevant classification processing on the N hybrid feature vectors after the dimensionality increasing or dimensionality reducing processing to obtain a classification result of the image to be classified.

[0145] Since the target image classification model is capable of identifying images belonging to at least one of the plurality of scene categories, the electronic device can obtain a plurality of classification results for the image to be classified. The electronic device can determine the scene category to which the image to be classified belongs according to the plurality of classification results.

[0146] It should be noted that the electronic device can represent the classification result by the classification label mentioned in step 601, of course, it can also be represented by other ways, such as represented by at least one of letters, numbers, patterns, identifiers and the like, and the embodiments of the present application do not make specific limitations thereto.

[0147] Exemplarily, each of the plurality of classification results output by the target image classification model can be represented by a letter, in the case that the letter output by the target image classification model for one of the scene categories E that can be identified is "yes", it indicates that the scene category to which the image to be classified belongs is the scene category E. In the case that the letter output by the target image classification model for the scene category E is "no", it indicates that the scene category to which the image to be classified belongs is not the scene category E.

[0148] Since there can be multiple scene categories corresponding to the things in one image, the image to be classified can belong to at least one of the plurality of scene categories. In this case, the electronic device can determine the scene category to which the image to be classified belongs to be the at least one scene category.

[0149] Exemplarily, the target image classification model can identify an image belonging to a sky scene category, an image belonging to a grassland scene category, an image belonging to a night scene category, an image belonging to a cute pet scene category, an image belonging to a portrait scene category, an image belonging to a snow scene category, and the like, in a case where the image to be classified includes a dog, grass, a tree, and a blue sky. The electronic device can input the N mixed feature vectors corresponding to the image to be classified into the target image classification model. The target image classification model can perform scene recognition on the image to be classified, and output a plurality of classification results for each scene category, where each of the plurality of classification results can be represented by the classification label described above. For the sky scene category, the classification label output by the target image classification model is [0, 1]. For the grassland scene category, the classification label output by the target image classification model is [0, 1]. For the night scene category, the classification label output by the target image classification model is [1, 0]. For the cute pet scene category, the classification label output by the target image classification model is [0, 1]. For the portrait scene category, the classification label output by the target image classification model is [1, 0], and for the snow scene category, the classification label output by the target image classification model is [1, 0]. Based on the classification labels output by the target image classification model for each scene category, the electronic device can determine that the scene category to which the image to be classified belongs can be the cute pet scene category, the grassland scene category, and the sky scene category.

[0150] As an example, in a case where the image to be classified is a preview image, if the user is satisfied with the preview image, the user can trigger a shooting operation, and the electronic device can receive the shooting operation. In response to the shooting operation, the electronic device can store the preview image after exposure in the image folder corresponding to the scene category to which the image belongs. Figure 4

[0151] As can be seen from the above, the scene category to which the image to be classified belongs can be multiple, and in a case where the image to be classified needs to be stored, the electronic device can store the image to be classified in the image folder corresponding to each of the multiple scene categories to which the image to be classified belongs.

[0152] Of course, in order to save storage space, the electronic device can also store the image to be classified in any one of the multiple scene categories to which the image to be classified belongs. Alternatively, the electronic device stores one image to be classified, and stores the image identifier of the image to be classified and the corresponding scene category. In a case where the image needs to be displayed in a classified manner, the electronic device can acquire the image to be classified according to the image identifier of the image to be classified, and display the image to be classified in the classified display interface.

[0153] ​It is worth noting that by storing the preview image after exposure into the image folder corresponding to the scene category to which the preview image belongs, it is convenient for the user to find the image according to the scene category, thereby improving the interactivity with the user and the user stickiness.

[0154] As an example, in the case of a preview image as the image to be classified, the electronic device can also display a scene label in the preview image after determining the scene category to which the image to be classified belongs, the scene label being used to describe the scene category to which the image to be classified belongs. For example, if the scene category to which the image to be classified belongs includes the pet scene category, the grass scene category, and the sky scene category, the electronic device can display "pet", "grass", and "sky" in the preview image. This scene can refer to the application scenarios shown in the above Figure 2

[0155] As an example, in the case of a preview image as the image to be classified, the electronic device can also determine the imaging scheme of the image to be classified according to the scene category to which the image to be classified belongs, such as determining the exposure parameter, filter scheme, shooting mode, display resolution, etc. of the image to be classified. For example, this scene can refer to the scene shown in the above Figure 3

[0156] For example, in the case of the scene category to which the image to be classified belongs being the moon scene, the electronic device can reduce the exposure parameter of the image to be classified to obtain a clearer moon image. Alternatively, the electronic device can control the camera to enter the moon viewing mode (also known as the moon shooting mode) to change the imaging scheme of the image to be classified.

[0157] For example, in the case of the scene category to which the image to be classified belongs being the text scene category or the scene category rich in texture, the electronic device can perform picture super-resolution processing on the image to be classified, that is, magnify the image display resolution of the image to be displayed.

[0158] In the embodiments of the present application, since all scene categories to which the image to be classified belongs can be recognized by one image classification model of the target image classification model, the scene category to which the image to be classified belongs can be obtained as much as possible at one time, thereby improving the efficiency of image classification of the target image classification model, and the electronic device can use different display schemes according to the scene category to which the image to be classified belongs, thereby improving the display quality and display effect of the image to be classified.

[0159] After the training method of the image classification model provided in the embodiments of the present application is explained in detail, the electronic device related to the embodiments of the present application is described.

[0160] ​​As an example, the method can be applied in an electronic device capable of model training. As an example but not limitation, the electronic device can be, but not limited to, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an in-vehicle device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a mobile phone, and the like, and the present embodiments are not limited thereto.

[0161] In addition, the electronic device can also apply the trained target image classification model, and the electronic device for training the target image classification model and the electronic device for applying the target image classification model can be the same electronic device or different electronic devices, and the present embodiments are not specifically limited thereto.

[0162] Figure 10 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. Referring to FIG. 1, Figure 10 The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, and the like. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0163] It can be understood that the structure illustrated in the present embodiments does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than illustrated, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0164] The processor 110 can include one or more processing units such as: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0165] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0166] The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that the processor 110 has just used or repeatedly uses. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. Avoiding repeated access, reducing the waiting time of the processor 110, thus improving the efficiency of the system.

[0167] In some embodiments, the processor 110 can include one or more interfaces, such as: can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0168] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 100. In some other embodiments of the present application, the electronic device 100 can also use different interface connection modes or a combination of multiple interface connection modes in the above embodiments.

[0169] The electronic device 100 realizes the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0170] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 can include 1 or N display screens 194, and N is an integer greater than 1.

[0171] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0172] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the algorithm of the image noise, brightness, and skin color. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be arranged in the camera 193.

[0173] The camera 193 is configured to capture still images or videos. An object projects an optical image through a lens to a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard image signal in a format such as RGB, YUV, or the like. In some embodiments, the electronic device 100 can include one or N cameras 193, where N is an integer greater than 1.

[0174] The digital signal processor is configured to process digital signals, including digital image signals. For example, when the electronic device 100 is selecting a frequency, the digital signal processor is configured to perform a Fourier transform on the frequency energy, and the like.

[0175] The video codec is configured to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in a variety of encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and the like.

[0176] The NPU is a neural-network (NN) computing processor that is configured to process input information quickly by emulating a biological neural network structure, such as by emulating the transmission mode between human brain neurons, and is further configured to self-learn continuously. Through the NPU, the electronic device 100 can implement intelligent cognition applications, such as image recognition, face recognition, voice recognition, text understanding, and the like.

[0177] As an example, the NPU can include the target image classification model provided by the embodiments of the present disclosure.

[0178] The external memory interface 120 can be configured to connect an external memory card, such as a Micro SD card, to extend the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, music, video, and the like files can be saved in the external memory card.

[0179] The internal memory 121 can be used to store computer executable program codes including instructions. The processor 110 performs various functional applications of the electronic device 100 and data processing by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program (such as a sound play function, an image play function, etc.) required by a function, etc. The data storage area can store data (such as audio data, a phone book, etc.) created by the electronic device 100 during use, etc. In addition, the internal memory 121 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0180] The electronic device 100 can implement an audio function such as music play, recording, etc. through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc.

[0181] The pressure sensor 180A is used to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. The capacitive pressure sensor can include at least two parallel plates with conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation is applied to the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than a pressure threshold is applied to a short message application icon, an instruction to view a short message is executed. When a touch operation with a touch operation intensity greater than or equal to the pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.

[0182] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of shaking of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and lets the lens offset the shaking of the electronic device 100 by reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and motion sensing game scenarios.

[0183] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance by infrared or laser. In some embodiments, in a shooting scenario, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0184] The ambient light sensor 180L is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when shooting. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touch.

[0185] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application lock, fingerprint shooting, fingerprint answering, etc.

[0186] The touch sensor 180K, also known as a "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can pass the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0187] Next, the software system of the electronic device 100 is described.

[0188] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take the Android system with a layered architecture as an example to exemplarily describe the software system of the electronic device 100.

[0189] Figure 11is a block diagram of a software system of an electronic device 100 provided by an embodiment of the present application. Referring to Figure 11 , the layered architecture divides the software into several layers, each of which has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and the system layer, and the kernel layer.

[0190] The application layer can include a series of application packages. As shown in Figure 11 , the application packages can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0191] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions. As shown in Figure 11 , the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager, etc. The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, and intercept the screen, etc. The content provider is used to store and obtain data, and make the data accessible to the applications, which can include video, image, audio, dialed and received calls, browsing history and bookmarks, phonebook, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of the application, which can be composed of one or more views, such as a view for displaying a short message notification icon, a view for displaying text, and a view for displaying a picture. The phone manager is used to provide the communication function of the electronic device 100, such as the management of the call state (including connection, hang-up, etc.). The resource manager provides various resources for the applications, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the applications to display notification information in the status bar, which can be used to convey messages of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the completion of the download, message reminders, etc. The notification manager can also be a notification in the form of a chart or a scroll bar text in the top status bar of the system, such as a notification of an application running in the background. The notification manager can also be a notification in the form of a dialog window on the screen, such as a text message in the status bar, a prompt sound, a vibration of the electronic device, a flashing of the indicator light, etc.

[0192] The Android Runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core library contains two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java file of the application layer and the application framework layer into a binary file. The virtual machine is used to execute the management of the object life cycle, the management of the stack, the management of the thread, the management of the security and the exception, and the garbage collection and the like.

[0193] The system library can include a plurality of function modules, such as: a surface manager, media libraries, a three-dimensional graphics processing library (such as: OpenGL ES), a 2D graphics engine (such as: SGL) and the like. The surface manager is used for managing the display subsystem, and provides a plurality of applications with the fusion of 2D and 3D layers. The media library supports a plurality of commonly used audio, video format playback and recording, and static image files and the like. The media library can support a plurality of audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG and the like. The three-dimensional graphics processing library is used for realizing three-dimensional graphics drawing, image rendering, synthesis, and layer processing and the like. The 2D graphics engine is a drawing engine for 2D drawing.

[0194] The kernel layer is a layer between hardware and software. The kernel layer at least contains a display driver, a camera driver, an audio driver, and a sensor driver.

[0195] The working flow of the software and the hardware of the electronic device 100 is described below in combination with a capturing and photographing scene.

[0196] When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, a timestamp of the touch operation, and the like). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer, and identifies the control corresponding to the raw input event. Taking the touch operation as a single click operation, and the control corresponding to the single click operation as the control of the camera application icon as an example, the camera application calls the interface of the application framework layer, starts the camera application, and then calls the kernel layer to start the camera driver, and captures a still image or a video through the camera 193.

[0197] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, such as the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium accessible by a computer, or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as digital versatile disc (DVD)) or semiconductor media (such as solid state disk (SSD)) and the like.

[0198] The above is an optional embodiment provided by the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the technical scope disclosed by the present application shall be included in the protection scope of the present application.

Claims

1. A method for training an image classification model, the method comprising: The method is applied to an electronic device, and the method comprises: obtaining a positive sample image training set, the positive sample image training set comprising a plurality of sample images and a plurality of first text information, each sample image belonging to at least one of a plurality of scene categories, the plurality of sample images corresponding one-to-one to the plurality of first text information, and one first text information being used to describe the scene category to which the corresponding sample image belongs; determining a negative sample image training set according to the positive sample image training set, the negative sample image training set comprising the plurality of sample images, and the text information of each sample image in the negative sample image training set being second text information, and the second text information of any one sample image in the negative sample image training set being one first text information in the plurality of first text information other than the first text information corresponding to the any one sample image; iteratively training an initial classification model based on the positive sample image training set and the negative sample image training set to obtain a target image classification model, the target image classification model being capable of identifying an image belonging to at least one of the plurality of scene categories; the method further comprises: obtaining a to-be-classified image; determining a visual feature vector of the to-be-classified image; determining a similarity between the visual feature vector of the to-be-classified image and each first text information in the plurality of first text information to obtain a plurality of similarities; splicing the visual feature vector of the to-be-classified image and a text feature vector corresponding to each first text information in N first text information respectively to obtain N hybrid feature vectors, the N first text information being the first text information corresponding to each similarity in the first N similarities after the plurality of similarities are arranged in descending order, and the N being a positive integer greater than or equal to 1; processing the N hybrid feature vectors through the target image classification model to obtain a classification result of the to-be-classified image.

2. The method of claim 1, wherein, The determining of the negative sample image training set according to the positive sample image training set comprises: determining a similarity between a target sample image and first text information corresponding to each target scene category, the target sample image being any one sample image in the positive sample image training set, and each target scene category being each scene category in the plurality of scene categories other than the scene category to which the target sample image belongs; determining the first text information corresponding to the target scene category with the largest similarity as second text information corresponding to the target sample image; determining the plurality of sample images and the second text information corresponding to each sample image as the negative sample image training set.

3. The method of claim 2, wherein, The determining of the similarity between the target sample image and the first text information corresponding to each target scene category comprises: processing the target sample image through a pre-trained target image encoder to obtain a visual feature vector of the target sample image; processing the first text information corresponding to each target scene category through a pre-trained target text encoder to obtain a text feature vector of the first text information corresponding to each target scene category; determine a similarity between the visual feature vector of the target sample image and the text feature vector corresponding to each target scene category.

4. The method of claim 2 or 3, wherein, Before the determining the similarity between the target sample image and the first text information corresponding to each target scene category, the method further includes: obtaining a plurality of sample text information and a plurality of sample training images, wherein the plurality of sample text information correspond to the plurality of sample training images one by one, and each sample text information in the plurality of sample text information is used to describe a scene category to which a corresponding sample training image belongs; iteratively training an initial text encoder based on the plurality of sample text information, and iteratively training an initial image encoder based on the plurality of sample training images; during the iteratively training, determining a loss value of a first loss function between the text encoder obtained after each training and the image encoder obtained after each training; in a case where the loss value converges, determining the text encoder obtained at the time of convergence as a target text encoder, and determining the image encoder obtained at the time of convergence as a target image encoder, wherein the target text encoder is used to determine a text feature vector of the first text information corresponding to each target scene category, and the target image encoder is used to determine a visual feature vector of the target sample image.

5. The method of any one of claims 1-4, wherein, The iteratively training the initial classification model based on the positive sample image training set and the negative sample image training set to obtain the target image classification model includes: concatenating the visual feature vector of each sample image in the positive sample image training set with the text feature vector of the corresponding first text information to obtain a plurality of positive sample mixed feature vectors; concatenating the visual feature vector of each sample image in the negative sample image training set with the text feature vector of the corresponding second text information to obtain a plurality of negative sample mixed feature vectors; iteratively training the initial classification model according to the plurality of positive sample mixed feature vectors and the plurality of negative sample mixed feature vectors; during the iteratively training, determining a loss value of a second loss function between a classification result of the image classification model obtained after each training and a preset result; in a case where the loss value of the second loss function converges, determining the image classification model obtained at the time of convergence as the target image classification model.

6. The method of claim 1, wherein, The obtaining the image to be classified includes: in a case where a camera is turned on, determining a preview image captured by the camera as the image to be classified; or in a case where an image selection operation is received, determining an image selected by the image selection operation as the image to be classified.

7. The method of claim 6, wherein, The determining the preview image captured by the camera as the image to be classified in a case where the camera is turned on includes: in a case where the camera is turned on, displaying a scene recognition control in a shooting interface, and obtaining the preview image through image capturing by the camera, wherein the scene recognition control is used to control whether to perform scene recognition; in response to an opening operation on the scene recognition control, determining the preview image as the image to be classified.

8. An electronic device, comprising: The structure of the electronic device includes a processor and a memory; The memory is configured to store a program for supporting the electronic device to perform the method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored therein instructions, which when executed on a computer, cause the computer to perform the method of any one of claims 1-7.

Images