Image classification model training method, electronic device and readable storage medium

Abstract

The present application discloses a training method for an image classification model, an electronic device, and a readable storage medium, and belongs to the field of terminal technology. Applied in an electronic device, the method includes: determining a first text feature vector of a first text information corresponding to a first scene category and a visual feature vector of each sample image in a sample image training set corresponding to the first scene category; updating the weight of an initial image classification model corresponding to the first scene category based on the first text feature vector; iteratively training the updated initial image classification model based on the visual feature vector of each sample image to obtain a first image classification model corresponding to the first scene category, and the first image classification model can identify images belonging to the first scene category. The present application uses the first text feature vector as the prior information of the image classification model to be trained, thereby eliminating the need to fuse text information and sample images, reducing operations during the training process, and improving model training efficiency.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a training method for an image classification model, an electronic device, and a readable storage medium. Background Art

[0002] With the development of terminal technology, the application of image processing technology is becoming more and more extensive. For example, it can be used in the field of intelligent transportation to identify roads, and in the field of medicine to perform medical diagnosis. Among them, the recognition and classification of images are the basis of image processing technology. In order to achieve image recognition and classification, electronic devices can train image classification models.

[0003] Currently, users can annotate sample images with text information, which is used to describe the scene category to which the sample images belong. For example, in the field of intelligent transportation, sample images can be annotated with text information to indicate that they belong to viaduct scenes, urban road scenes, rain and fog scenes, etc. Electronic devices can perform model training based on annotated sample images of different scene categories to obtain an image classification model. The trained image classification model can be used to identify the scene category corresponding to any image.

[0004] However, in order to obtain an accurate image classification model, the electronic device needs to fuse text information and sample images during the training process, which makes the training process cumbersome. Summary of the invention

[0005] The present application provides a training method for an image classification model, an electronic device, and a readable storage medium, which can be used to improve the training accuracy and efficiency of the image classification model. The technical solution is as follows:

[0006] In a first aspect, a method for training an image classification model is provided, which is applied to an electronic device, and the method comprises:

[0007] Determine a first text feature vector of first text information corresponding to a first scene category and a visual feature vector of each sample image in a sample image training set corresponding to the first scene category; update the weight of an initial image classification model corresponding to the first scene category based on the first text feature vector; iteratively train the updated initial image classification model based on the visual feature vector of each sample image to obtain a first image classification model corresponding to the first scene category, wherein the first image classification model can recognize images belonging to the first scene category.

[0008] It should be noted that the sample image training set includes multiple positive sample images and multiple negative sample images, the image content of each positive sample image in the multiple positive sample images is consistent with the scene category described by the first text information, and the image content of each negative sample image in the multiple negative sample images is inconsistent with the scene category described by the first text information.

[0009] As an example, the ratio between the number of positive sample images and the number of negative sample images in the sample image training set is usually 1:1, thereby ensuring the balance of model training.

[0010] In this way, in the process of training the image classification model, the electronic device can fix the weight of the image classification model to be trained based on the first text feature vector, that is, the first text feature vector can be used as the prior information of the image classification model to be trained, and then the model is trained based on the sample image training set, so there is no need to fuse the text information and the sample image, which reduces the operations in the training process, reduces the cumbersome training procedures, and improves the efficiency of model training. In addition, in the process of training the image classification model, information of different modes of the same thing is used, so that the trained image classification model is more accurate.

[0011] As an example of the present application, the operation of the electronic device determining a first text feature vector of first text information corresponding to a first scene category and a visual feature vector of each sample image in a sample image training set corresponding to the first scene category includes:

[0012] Acquire first text information and a sample image training set corresponding to a first scene category; process the first text information through a pre-trained target text encoder to obtain a first text feature vector; and process each sample image separately through a pre-trained target image encoder to obtain a visual feature vector for each sample image.

[0013] It should be noted that the target text encoder and the target image encoder are obtained by cooperating with each other during the pre-training process.

[0014] In this way, the first text information is processed by the pre-trained target text encoder, and each sample image is processed by the target image encoder respectively, so that the consistency of the feature vector can be guaranteed.

[0015] As an example of the present application, before the electronic device determines the first text feature vector of the first text information corresponding to the first scene category and the visual feature vector of each sample image in the sample image training set corresponding to the first scene category, it can also obtain multiple sample text information and multiple sample training images, the multiple sample text information corresponds one-to-one to the multiple sample training images, and each sample text information in the multiple sample text information is used to describe the scene category to which the corresponding sample training image belongs; iteratively train the initial text encoder based on the multiple sample text information, and iteratively train the initial image encoder based on the multiple sample training images; during the iterative training process, determine the loss value of the first loss function between the text encoder obtained after each training and the image encoder obtained after each training; when the loss value converges, determine the text encoder obtained at the time of convergence as the target text encoder, and determine the image encoder obtained at the time of convergence as the target image encoder, wherein the target text encoder is used to determine the first text feature vector, and the target image encoder is used to determine the visual feature vector of each sample image.

[0016] It should be noted that the multiple sample text information corresponds to the multiple sample training images one-to-one, and the one-to-one correspondence between the multiple sample text information and the multiple sample training images means that for any sample text information C, there is only one sample training image among the multiple sample training images whose image content is the same as the content described by the any sample text information C. In addition, during the iterative training process, the corresponding sample text information and sample training images can be trained in pairs.

[0017] In this way, since the text encoder and image encoder are trained using image-text matching data (i.e., sample text information and corresponding sample training images), the two modal information of sample training images and corresponding sample text information can be mapped to the same feature space, thereby obtaining a text encoder and image encoder with better training effect.

[0018] As an example of the present application, the operation of the electronic device updating the weight of the initial image classification model corresponding to the first scene category based on the first text feature vector includes:

[0019] According to the dimension of the initial weight of the initial image classification model, the first text feature vector is reduced in dimension or increased in dimension to obtain a first vector; and the weight of the initial image classification model is updated to the first vector.

[0020] As an example, a weight learning module may be provided in the electronic device, and the electronic device may input the first text feature vector into the weight learning module. The weight learning module may reduce or increase the dimension of the first text feature vector according to the dimension of the initial weight of the initial image classification model to obtain a first vector, and send the first vector to the initial image classification model. The initial image classification model may update the weight to the first vector.

[0021] In some embodiments, the initial classification model may be a binary classifier.

[0022] In this way, by updating the weight of the initial classification model to the first vector, the first text information can be used as prior information of the initial classification model, thereby reducing the complexity of subsequent model training.

[0023] As an example of the present application, an electronic device includes multiple image classification models, the multiple image classification models include the first image classification model, and different image classification models in the multiple image classification models can identify different scene categories; in this way, the electronic device iteratively trains the updated initial image classification model based on the visual feature vector of each sample image, and after obtaining the first image classification model corresponding to the first scene category, it can also obtain the image to be classified; input the image to be classified into multiple image classification models respectively; and determine the scene category to which the image to be classified belongs based on the classification results output by each image classification model in the multiple image classification models.

[0024] In this way, since scene recognition can be performed separately on the images to be classified through multiple image classification models, the scene category to which the images to be classified belong can be obtained as much as possible. Then, the electronic device can adopt different display schemes according to the scene category to which the images to be classified belong, thereby improving the display quality and display effect of displaying the images to be classified.

[0025] As an example of the present application, the operation of the electronic device acquiring the image to be classified includes:

[0026] When the camera is turned on, determining that the preview image captured by the camera is the image to be classified; or,

[0027] In the case where an image selection operation is received, it is determined that the image selected by the image selection operation is an image to be classified.

[0028] It should be noted that the situation in which the camera is turned on includes the situation in which the camera is turned on to scan QR codes, bar codes and other information, or the situation in which the camera is turned on to perform text recognition, object recognition, or the situation in which the camera is turned on to take pictures.

[0029] In this way, since the image to be classified can be a preview image or an image selected by the user, the electronic device can realize scene recognition for any image, thereby increasing the application scenarios of the image classification model and improving the practicability of the image classification model.

[0030] As an example of the present application, when the camera of an electronic device is turned on, the operation of determining that a preview image captured by the camera is an image to be classified includes: when the camera is turned on, displaying a scene recognition control in a shooting interface, and performing image capture through the camera to obtain a preview image, wherein the scene recognition control is used to control whether to perform scene recognition; in response to the operation of turning on the scene recognition control, determining that the preview image is an image to be classified.

[0031] As an example, when the scene recognition control is in a closed state, that is, when the user has not turned on the scene recognition control, the electronic device will not perform post-scene recognition (or image recognition) operations.

[0032] In this way, by setting the scene recognition control, the user can independently choose whether to perform scene recognition, thereby increasing the interactivity with the user and saving the operating resources of the electronic device when there is no need to perform scene recognition.

[0033] As an example of the present application, the image to be classified is the preview image; in this way, after the electronic device determines the scene category to which the image to be classified belongs based on the classification results output by each image classification model in multiple image classification models, it can also receive a shooting operation; in response to the shooting operation, the exposed preview image is stored in an image folder corresponding to the scene category to which the preview image belongs.

[0034] As an example, in order to save storage space, the electronic device may also store the image to be classified in any one of the multiple scene categories to which the image to be classified belongs. Alternatively, the electronic device stores an image to be classified and stores the image identifier of the image to be classified and the corresponding scene category. When it is necessary to display the image in a classified manner, the electronic device may obtain the image to be classified according to the image identifier of the image to be classified, and display the image to be classified in the classification display interface.

[0035] In this way, by storing the exposed preview image in the image folder corresponding to the scene category to which the preview image belongs, it is convenient for users to search for images according to the scene category, thereby improving the interactivity with users and user stickiness.

[0036] As an example, when the image to be classified is a preview image, after determining the scene category to which the image to be classified belongs, the electronic device may further display a scene label in the preview image, where the scene label is used to describe the scene category to which the image to be classified belongs.

[0037] As an example, when the image to be classified is a preview image, after determining the scene category to which the image to be classified belongs, the electronic device can also determine the imaging scheme of the image to be classified according to the scene category described by the image to be classified, such as determining the exposure parameters, filter scheme, shooting mode, display resolution, etc. of the image to be classified.

[0038] In a second aspect, an electronic device is provided, wherein the structure of the electronic device includes a processor and a memory, wherein the memory is used to store a program that supports the electronic device to execute the training method of the image classification model provided in the first aspect, and to store data involved in implementing the training method of the image classification model described in the first aspect. The processor is configured to execute the program stored in the memory. The electronic device may also include a communication bus, which is used to establish a connection between the processor and the memory.

[0039] In a third aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the image classification model training method described in the first aspect.

[0040] In a fourth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the image classification model training method described in the first aspect.

[0041] The technical effects obtained by the above-mentioned second aspect, third aspect and fourth aspect are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0043] Figure 2 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0044] Figure 3 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0045] Figure 4 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0046] Figure 5is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0047] Figure 6 It is a flow chart of a training method of an image classification model provided in an embodiment of the present application;

[0048] Figure 7 It is a schematic diagram of the training process of an image classification model provided in an embodiment of the present application;

[0049] Figure 8 is a schematic diagram of the training process of another image classification model provided in an embodiment of the present application;

[0050] Fig. 9 This is a flow chart of an image classification method provided in an embodiment of the present application;

[0051] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0052] Fig.11 It is a block diagram of a software system of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0054] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0055] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0056] With the development of terminal technology, the application of image processing technology is becoming more and more extensive. Among them, the recognition and classification of images are the basis of image processing technology. Usually, electronic devices can perform image recognition or scene recognition through image classification models. In some scenarios, in order to enable users to quickly find the images they need, the mobile phone can classify the images in the gallery according to the scene categories shown, such as Figure 1 As shown. In order to facilitate the classification of images, a pre-trained image classification model can be built into the mobile phone. That is, before the mobile phone leaves the factory, a pre-trained image classification model can be built into the mobile phone. Among them, the image classification model can be obtained by model training based on sample images of different scene categories labeled by other electronic devices. The labeled sample images of different scene categories refer to the staff labeling different sample images through text information. The text information is used to describe the scene category to which the sample image belongs. For example, the sample image can be labeled as belonging to an elevated bridge scene, an urban road scene, a rain and fog scene, a building scene, a food scene, a night scene, a certificate scene, a text scene, etc. through text information.

[0057] However, in order to obtain an accurate image classification model, the electronic device needs to fuse text information and sample images during the training process. In other words, in each training process, the text information and the corresponding sample images need to be spliced ​​separately, and then the image classification model is trained based on the information obtained after splicing. If the number of sample images and corresponding text information is large, the electronic device needs to perform the fusion operation multiple times, which makes the training process cumbersome.

[0058] In order to streamline the process of training the image classification model and improve the classification accuracy of the image classification model. The embodiment of the present application provides a training method for an image classification model, in which the electronic device can determine any one of the multiple scene categories to be trained, that is, the text feature vector of the first text information corresponding to the first scene category and the visual feature vector of each sample image in the sample image training set corresponding to the first scene category; based on the first text feature vector, the weight of the initial image classification model corresponding to the first scene category is updated; based on the visual feature vector of each sample image, the updated initial image classification model is iteratively trained to obtain a first image classification model that can identify images belonging to the first scene category. Since in the process of training the model, the weight of the image classification model to be trained can be fixed based on the first text feature vector, that is, the first text feature vector can be used as the prior information of the image classification model to be trained, and then the image classification model is trained based on the sample image training set, so that there is no need to fuse the text information and the sample image, which reduces the operations in the training process, reduces the cumbersome training procedures, and improves the model training efficiency. In addition, since in the process of training the image classification model, information of different modalities of the same thing is used, the trained image classification model is more accurate.

[0059] For ease of understanding, before introducing the method provided in the embodiments of the present application in detail, the application scenarios involved in the embodiments of the present application are introduced below.

[0060] Please refer to Figure 2 , Figure 2 A schematic diagram of an application scenario provided in an embodiment of the present application. In an application scenario, a user may take a photo through the camera of the mobile phone while using the mobile phone. After the mobile phone starts the camera, the following may be displayed on the display screen: Figure 2 The shooting interface shown in Figure (a) in the figure displays a preview image A, which includes a blue sky, grass and trees. When the mobile phone captures the preview image A, it can input the preview image A into multiple image classification models, which include image classification models corresponding to building scene categories, image classification models corresponding to tree scene categories, image classification models corresponding to grass scene categories, image classification models corresponding to blue sky scene categories, image classification models corresponding to flower scene categories, image classification models corresponding to night scene categories, image classification models corresponding to beach scene categories, image classification models corresponding to ocean scene categories, image classification models corresponding to playground scene categories, image classification models corresponding to school scene categories, and the like. Multiple image classification models can perform scene recognition (or image recognition) on the preview image A respectively, and each image classification model can output the classification result of the preview image A. See. Figure 2In figure (b), the mobile phone can display the scene label of the scene category to which the preview image A belongs on the preview image A according to the classification result. For example, if it is determined that the scene category to which the preview image A belongs includes a sky scene category, a grass scene category and a tree scene category, then the mobile phone can display the scene labels "sky", "grass" and "tree" at the positions of the blue sky, grass and trees in the preview image A.

[0061] In another application scenario, the mobile phone can also adjust the image display scheme according to the scene category to which the preview image A belongs, for example, adjusting the shooting mode, adjusting the exposure parameters, adjusting the display resolution, etc. Exemplarily, if the scene category to which the preview image A belongs is an overexposed scene category, for example, the overexposed scene category includes a moon scene category, a flame scene category, a sun scene category, an electric light scene category, etc., that is, when the preview image A includes a light source such as the moon, a lamp source, or a flame, the scene category to which the preview image A belongs usually includes an overexposed scene category. In this case, the mobile phone can adjust the shooting mode, or reduce the exposure parameters of the preview image A. See, Figure 3 In Figure (a), the preview image A includes the moon. Then the mobile phone recognizes that the scene category to which the preview image A belongs includes the moon scene category through the image classification model corresponding to the moon scene category in multiple image classification models. Then the mobile phone can adjust the shooting mode to the moon-gazing mode. In the moon-gazing mode, the exposure parameters of the preview image A change. That is, the mobile phone can reduce the exposure of the preview image A and shorten the exposure time of the preview image A, and display the following. Figure 3 Preview image B shown in Figure (b) .

[0062] In another application scenario, see Figure 4 In Figure (a), if the user is satisfied with the current preview image A, the user can click the shooting control P1 in the shooting interface; in response to the click operation on the shooting control P1, the mobile phone can perform image exposure based on the preview image A to obtain the exposed preview image A, and the mobile phone can store the exposed preview image A in the image folder corresponding to the scene category to which the preview image A belongs, that is, the mobile phone can store the exposed preview image A in at least one of the image folders corresponding to the sky scene, the image folder corresponding to the grass scene, and the image folder corresponding to the tree scene, such as storing it in the image folder corresponding to the sky scene category. Afterwards, if the user needs to find an image of a certain scene category that has been shot, refer to Figure 4 In the figure (b), the user can click the application logo of the gallery application on the desktop. In response to the click operation on the application logo of the gallery application, the mobile phone can display the following Figure 4In the image preview interface P2 shown in FIG. (c), the user can click the "discover" control P3 displayed in the image preview interface P2. In response to the click operation on the "discover" control P3, the mobile phone can display the following Figure 4 The classification view interface P4 shown in Figure (d) may display multiple image folders, for example, a "people" folder, a "place" folder, and a "thing" folder, etc. The "thing" folder may include a "building" folder, a "sky" folder, a "portrait" folder, a "grass" folder, a "tree" folder, and a "night scene" folder, etc. In this way, the user may continue to search for more detailed scene categories from the "thing" folder.

[0063] Please refer to Figure 5 , Figure 5 A schematic diagram of an application scenario provided by an embodiment of the present application. In another application scenario, after the mobile phone starts the camera, the display is as follows Figure 5 The shooting interface shown in Figure (a) in the figure can not only display the preview image A captured by the camera, but also display the scene recognition control in the shooting interface. If the user needs to identify the scene category of the captured image, the user can click the scene recognition control P5; the mobile phone responds to the click operation of the scene recognition control P5, see Figure 5 In Figure (b), the mobile phone can change the display mode of the scene recognition control P5 (the embodiment of the present application takes the example of changing the display color of the scene recognition control P5 from black text on a white background to white text on a black background as an example), and inputs the preview image A into multiple image classification models, and the preview image A is processed by the multiple image classification models respectively, and each image classification model can output the classification result of the preview image A.

[0064] It should be noted that, in the embodiments of the present application, only the above Figure 2-Figure 5 The scenario shown is used as an example for illustration and does not constitute a limitation on the embodiments of the present application.

[0065] Based on the application scenarios provided in the above embodiments, the training method of the image classification model provided in the embodiments of the present application is introduced below. Figure 6 , Figure 6 The present invention is a flowchart of a training method for an image classification model according to an exemplary embodiment. As an example but not a limitation, the method is described by taking application in an electronic device as an example. The method may include some or all of the following contents:

[0066] Step 601: Determine a first text feature vector of first text information corresponding to a first scene category and a visual feature vector of each sample image in a sample image training set corresponding to the first scene category.

[0067] It should be noted that the first scene category is any one of the multiple scene categories to be trained. The sample image training set includes multiple positive sample images and multiple negative sample images, the image content of each positive sample image in the multiple positive sample images is consistent with the scene category described by the first text information, and the image content of each negative sample image in the multiple negative sample images is inconsistent with the scene category described by the first text information.

[0068] Exemplarily, if the first scene category is flower, the first text information may be "flower, a photo of flower sea" or "flower, an image of a sea of ​​flowers". Then, the image content of each positive sample image is a flower or a sea of ​​flowers, and the image content of each negative sample image does not contain a flower.

[0069] Generally, in order to ensure the balance of model training, the ratio between the number of positive sample images and the number of negative sample images in the sample image training set is usually 1: 1. Of course, it can also be other ratios, such as 2: 1, 1.5: 1, etc., and the embodiment of the present invention does not make specific restrictions on this.

[0070] In some embodiments, each sample image corresponds to a classification label, and the classification label is used to indicate the relationship between the sample image and the first text information. The classification label can be [0, 1] and [1, 0], and the first element in the classification label represents the probability that the sample image does not match the first text information, and the second element represents the probability that the sample image matches the first text information. That is, the classification label of each positive sample image can be [0, 1], the first element 0 represents that the probability that the sample image does not match the first text information is 0, and the second element 1 represents that the probability that the sample image matches the first text information is 1. The classification label of each negative sample image can be [1, 0], the first element 1 represents that the probability that the sample image does not match the first text information is 1, and the second element 0 represents that the probability that the sample image matches the first text information is 0.

[0071] It should be noted that the above-mentioned classification labels are only used as examples in the embodiments of the present application, and do not constitute a limitation on the embodiments of the present application. The classification labels may also be labels of other styles.

[0072] In one possible case, the operation of an electronic device determining a first text feature vector of first text information corresponding to a first scene category and a visual feature vector of each sample image in a sample image training set corresponding to the first scene category includes: acquiring the first text information and the sample image training set corresponding to the first scene category; processing the first text information through a pre-trained target text encoder to obtain a first text feature vector; and processing each sample image separately through a pre-trained target image encoder to obtain a visual feature vector of each sample image.

[0073] As an example, the electronic device may receive an input operation and determine the text information carried by the input operation as the first text information. Alternatively, the electronic device may perform a text capture operation in a specific page and determine the captured text information as the first text information, the specific page being a page describing a first scene category, and the first scene category is described in the specific page through text information.

[0074] It should be noted that the target text encoder and the target image encoder are obtained by cooperating with each other during the pre-training process.

[0075] It is worth noting that by processing the first text information through the pre-trained target text encoder and processing each sample image respectively through the target image encoder, the consistency of the feature vector can be ensured.

[0076] In some embodiments, before determining the first text feature vector of the first text information corresponding to the first scene category and the visual feature vector of each sample image in the sample image training set corresponding to the first scene category, the electronic device may also pre-train a target text encoder and a target image encoder.

[0077] As an example, an electronic device can obtain multiple sample text information and multiple sample training images, the multiple sample text information correspond one-to-one to the multiple sample training images, and each sample text information in the multiple sample text information is used to describe the scene category to which the corresponding sample training image belongs; iteratively train the initial text encoder based on the multiple sample text information, and iteratively train the initial image encoder based on the multiple sample training images; during the iterative training process, determine the loss value of the first loss function between the text encoder obtained after each training and the image encoder obtained after each training; when the loss value converges, determine the text encoder obtained at the time of convergence as the target text encoder, and determine the image encoder obtained at the time of convergence as the target image encoder, the target text encoder is used to determine the first text feature vector, and the target image encoder is used to determine the visual feature vector of each sample image.

[0078] It should be noted that the one-to-one correspondence between the multiple sample text information and the multiple sample training images means that for any one sample text information C among the multiple sample text information, there is only one sample training image among the multiple sample training images whose image content is the same as the content described by the any one sample text information C.

[0079] It should also be noted that, during the iterative training process, the corresponding sample text information and the sample training image can be paired for training. For example, the sample text information is "a photo of a cat" or "an image of a cat", and the corresponding sample training image is an image of a cat. The sample text information is input into the text encoder to train the text encoder, and the sample training image is input into the image encoder to train the image encoder.

[0080] It is worth noting that, since the text encoder and the image encoder are trained using image-text matching data (i.e., sample text information and corresponding sample training images), the two modal information of sample training images and corresponding sample text information can be mapped to the same feature space. By calculating the similarity between the sample training images and the sample text information, the similarity between the sample training images and their corresponding sample text information is constrained to be the highest, and the similarity with other sample text information is low, thereby obtaining a text encoder and an image encoder with better training effect.

[0081] In some embodiments, during the iterative training process, the operation of determining the loss value of the first loss function between the text encoder obtained after each training and the image encoder obtained after each training includes: during the iterative training process, determining the sample text feature vector output by the text encoder obtained after each training and the sample visual features output by the image encoder obtained after each training, and then determining the loss value of the first loss function based on the sample text feature vector and the sample visual feature vector.

[0082] For example, see Figure 7 , because the electronic device can input multiple sample text information into the text encoder and multiple sample training images into the image encoder during the iterative training of the text encoder and the image encoder. The text encoder can determine the sample text feature vector of the input sample text information, and the image encoder can determine the sample visual feature vector of the input sample training image, and then the electronic device determines the loss value of the first loss function according to the sample text feature vector and the sample visual feature vector.

[0083] As an example, the first loss function may be an image-text contrast (ITC) loss function, and the first loss function may be as shown in the following first formula (1).

[0084]

[0085] It should be noted that in the above first formula (1), D w Represents two sample feature vectors (X 1 and X 2), P is the dimension of the feature, Y is the label of whether the two sample feature vectors match, where Y is 1 for similar or matching samples, Y is 0 for dissimilar or mismatching samples, m is the set threshold, L(W, (Y, X 1 , X 2 )) is the loss value.

[0086] As an example, the first loss function can be expressed as the first formula (1) above, and can certainly be expressed in other ways. For example, the first loss function can also be expressed as the second formula (2) below.

[0087]

[0088] It should be noted that in the above second formula (2), L itc is the loss value, s(I, T) and s(T, I) are the similarities between vectors, w and v are network parameters, g v (v cls ) is the text feature vector, g' w (w' cls ) is the visual feature vector, M is the number of scene categories, is the matching similarity between any text feature vector and the corresponding visual feature vector. is the matching similarity between any visual feature vector and the corresponding text feature vector.

[0089] As an example, the loss value converges when the loss value is less than or equal to a first preset value, and / or when the change in the loss value is less than or equal to a second preset value. Both the first preset value and the second preset value can be pre-set according to requirements.

[0090] It is worth noting that by training the text encoder and the image encoder, the target text encoder and the image encoder are obtained, so that the training accuracy is higher.

[0091] In some embodiments, the electronic device can not only determine the text encoder obtained when the loss value converges as the target text encoder, but also determine the image encoder obtained when the convergence is the target image encoder. The electronic device can also determine the number of training times of iterative training. When the number of training times is greater than or equal to the number threshold, the electronic device can determine the text encoder obtained when the number of training times is greater than or equal to the number threshold as the target text encoder, and determine the image encoder obtained when the number of training times is greater than or equal to the number threshold as the target image encoder.

[0092] It should be noted that the number threshold can also be pre-set according to needs, for example, the number threshold can be 100 times, 150 times, etc.

[0093] In some embodiments, the electronic device can not only determine the first text feature vector and the visual features of each sample image in the above manner, but also determine the first text feature vector of the first text information and the visual feature vector of each sample image in other ways. Exemplarily, the electronic device can determine the visual feature vector of each sample image through a visual basic network model (such as a Resent (Residual Neural Network) model, etc.), and determine the first text feature vector of the first text information through a multilingual text model. Alternatively, the electronic device vectorizes the first text information through a text vector model to obtain the first text feature vector of the first text information, and the text vector model can be determined based on a pre-trained language representation model (Bidirectional Encoder Representation from Transformers, BETR). The embodiments of the present application do not impose specific restrictions on this.

[0094] Step 602: Update the weight of the initial image classification model corresponding to the first scene category based on the first text feature vector.

[0095] As an example, the operation of an electronic device updating the weight of an initial image classification model corresponding to a first scene category based on a first text feature vector may include: reducing or increasing the dimension of the first text feature vector according to the dimension of the initial weight of the initial image classification model to obtain a first vector; and updating the weight of the initial image classification model to the first vector.

[0096] For example, see Figure 8 , a weight learning module may be provided in the electronic device, the electronic device may input the first text feature vector into the weight learning module, the weight learning module may reduce or increase the dimension of the first text feature vector according to the dimension of the initial weight of the initial image classification model, obtain a first vector, and send the first vector to the initial image classification model. The initial image classification model may update the weight to the first vector. The dimension of the first vector is the same as or similar to the dimension of the initial weight of the initial image classification model.

[0097] It should be noted that the weight learning model may be a model with a multi-layer perceptron (MLP) structure, and the embodiments of the present application do not impose any specific limitation on this.

[0098] In some embodiments, after obtaining the first vector, the electronic device may further process the first vector to obtain a second vector, and update the weight of the initial image classification model to the second vector. The electronic device may process the first vector by adding or multiplying the first vector to the initial weight of the initial image classification model to obtain the second vector.

[0099] As an example, when the dimension of the first text feature vector is the same as the dimension of the initial weights of the initial image classification model, the electronic device may also directly update the weights of the initial image classification model to the first text feature vector.

[0100] It is worth noting that by updating the weight of the initial classification model to the first vector, the first text information can be used as prior information of the initial classification model, thereby reducing the complexity of subsequent model training.

[0101] In some embodiments, the initial classification model may be a binary classifier.

[0102] Step 603: Based on the visual feature vector of each sample image, iteratively train the updated initial image classification model to obtain a first image classification model corresponding to the first scene category.

[0103] It should be noted that the first image classification model can identify images belonging to the first scene category.

[0104] For example, see Figure 8 Since the first text feature information is used as prior information in the updated initial image classification model, during the iterative training process, the electronic device can input the visual feature vector of each sample image into the updated initial image classification model for iterative training; during the iterative training process, the loss value of the second loss function between the classification result of the image classification model obtained after each training and the preset result is determined; when the loss value converges, the image classification model obtained at the time of convergence is used as the first image classification model.

[0105] It should be noted that the preset result is the classification label corresponding to each sample image.

[0106] As an example, the second loss function may be a binary cross entropy (BCE) loss function, or other loss functions, which are not specifically limited in the present embodiment.

[0107] It should be noted that the first image classification model can identify images belonging to the first scene category. In the case where there are multiple scene categories, the electronic device can obtain the image classification model corresponding to each of the multiple scene categories in the manner of steps 601 to 603. Figure 7 The electronic device can train the binary classifier corresponding to each scene category according to the operations of the above steps 601 to 603, and obtain an image classification model corresponding to each scene category in the multiple scene categories.

[0108] Exemplarily, when the electronic device needs to train to obtain a second image classification model corresponding to a second scene category, the electronic device may determine a second text feature vector of the second text information corresponding to the second scene category and a visual feature vector of each sample image in a sample image training set corresponding to the second scene category, and the second scene category is different from the first scene category. For example, the first scene category may be a scene category of flowers, and the second scene category may be a scene category of the ocean. Then, the electronic device may update the weight of the initial image classification model corresponding to the second scene category based on the second text feature vector; and iteratively train the updated initial image classification model (the model corresponding to the second scene category) based on the visual feature vector of each sample image corresponding to the second scene category to obtain a second image classification model corresponding to the second scene category.

[0109] Since the image classification model corresponding to each scene category can be trained separately, when the scene categories increase, an image classification model can be trained separately without affecting the image classification models corresponding to the existing scene categories.

[0110] In the embodiment of the present application, in the process of training the model, the weight of the image classification model to be trained can be fixed based on the first text feature vector, that is, the first text feature vector can be used as the prior information of the image classification model to be trained, and then the image classification model is trained based on the sample image training set, so there is no need to fuse the text information and the sample image, which reduces the operations in the training process, reduces the cumbersome training procedures, and improves the model training efficiency. In addition, in the process of training the image classification model, information of different modes of the same thing is used, so that the trained image classification model is more accurate.

[0111] It should be noted that, when the electronic device obtains multiple image classification models, the electronic device can identify different scene categories through different image classification models in the multiple image classification models. In order to understand the embodiments of the present application, the following explains how the electronic device identifies the scene category to which the image belongs through multiple image classification models.

[0112] Please refer to image 9, Fig. 9 The present invention is a flowchart of a method for image classification according to an exemplary embodiment. As an example but not a limitation, the method is described by taking application in an electronic device as an example, and the method may include some or all of the following contents:

[0113] Step 901: Obtain an image to be classified.

[0114] As an example, the operation of the electronic device acquiring the image to be classified includes: when the camera is turned on, determining that the preview image captured by the camera is the image to be classified; or, when an image selection operation is received, determining that the image selected by the image selection operation is the image to be classified. It can be seen that the image to be classified can be any image, for example, the image to be classified can be an image downloaded from the network, or a preview image captured by the camera of the electronic device, or any image stored in the electronic device.

[0115] It should be noted that the situation in which the camera is turned on includes the situation in which the camera is turned on to scan QR codes, bar codes and other information, or the situation in which the camera is turned on to perform text recognition, object recognition, or the situation in which the camera is turned on to take pictures.

[0116] As an example, the image selection operation may refer to a selection operation on any image in the gallery of the electronic device (composed of stored images), or it may be a download operation, a save operation, etc. on a network image. The embodiments of the present application do not impose any specific restrictions on this.

[0117] It is worth noting that since the image to be classified can be a preview image or an image selected by the user, the electronic device can realize scene recognition of any image, thereby increasing the application scenarios of the image classification model and improving the practicability of the image classification model.

[0118] In some embodiments, when the camera is turned on, the operation of the electronic device determining that the preview image captured by the camera is an image to be classified includes: when the camera is turned on, displaying a scene recognition control in the shooting interface, and performing image capture through the camera to obtain a preview image, the scene recognition control is used to control whether to perform scene recognition; in response to the operation of turning on the scene recognition control, determining that the preview image is an image to be classified. Exemplarily, the scene can refer to the above Figure 5 The application scenario shown.

[0119] Since not all scenes require scene recognition, in order to allow users to selectively perform scene recognition, the electronic device may also display a scene recognition control in the shooting interface.

[0120] As an example, when the scene recognition control is in a closed state, that is, when the user has not turned on the scene recognition control, the electronic device will not perform post-scene recognition (or image recognition) operations.

[0121] It is worth noting that by setting the scene recognition control, the user can independently choose whether to perform scene recognition, thereby increasing the interactivity with the user and saving the operating resources of the electronic device when there is no need to perform scene recognition.

[0122] Step 902: Input the images to be classified into multiple image classification models respectively.

[0123] It should be noted that the multiple image classification models may include a first image classification model and other image classification models determined by the above steps 601-603, such as a second image classification model. Of course, in some cases, the image classification model may also include an image classification model determined by other methods, and the present embodiment of the application does not impose specific restrictions on this. In order to understand the embodiment of the present application, the embodiment of the present application is described by taking each image classification model in the multiple image classification models as an example in which the image classification model is determined by the above steps 601-603.

[0124] Since different image classification models among the multiple image classification models can identify different scene categories, in order to accurately identify the scene category to which the image to be classified belongs, the electronic device can input the image to be classified into the multiple image classification models respectively.

[0125] Step 903: Determine the scene category to which the image to be classified belongs according to the classification results output by each image classification model in the multiple image classification models.

[0126] Since the scene category that can be recognized by each of the multiple image classification models is different from that of other image classification models, the electronic device can obtain multiple classification results.

[0127] It should be noted that the electronic device can represent the classification result through the classification label mentioned in the above step 601. Of course, it can also be represented in other ways, for example, by at least one of letters, numbers, patterns, identifiers and other information. The embodiment of the present application does not impose specific restrictions on this.

[0128] Exemplarily, the classification result output by the image classification model D can be represented by letters. When the letter output by the image classification model D is "yes", it means that the scene category to which the image to be classified belongs is the scene category corresponding to the image classification model D. When the letter output by the image classification model D is "no", it means that the scene category to which the image to be classified belongs is not the scene category corresponding to the image classification model D.

[0129] Since there may be objects corresponding to multiple scene categories in an image, the image to be classified may belong to at least one scene category of the multiple scene categories at the same time. In this case, the electronic device may determine that the scene category to which the image to be classified belongs is the at least one scene category.

[0130] Exemplarily, the image to be classified includes a dog, grass, trees and blue sky, and the multiple image classification models included in the electronic device include an image classification model corresponding to the sky scene category, an image classification model corresponding to the grass scene category, an image classification model corresponding to the night scene category, an image classification model corresponding to the cute pet scene category, an image classification model corresponding to the portrait scene category, an image classification model corresponding to the snow scene category, etc. The electronic device can input the image to be classified into multiple image classification models respectively. Each of the multiple image classification models can perform scene recognition on the image to be classified and output classification results respectively, and the classification results can be represented by the above classification labels. Among them, the classification label output by the image classification model corresponding to the sky scene category is [0, 1], the classification label output by the image classification model corresponding to the grass scene category is [0, 1], the classification label output by the image classification model corresponding to the night scene category is [1, 0], the classification label output by the image classification model corresponding to the cute pet scene category is [0, 1], the classification label output by the image classification model corresponding to the portrait scene category is [1, 0], and the classification label output by the image classification model corresponding to the snow scene category is [1, 0]. Based on the classification labels output by each image classification model, the electronic device can determine that the scene category to which the image to be classified belongs can be a cute pet scene category, a grass scene category, or a sky scene category.

[0131] As an example, when the image to be classified is a preview image, if the user is satisfied with the preview image, the user can trigger a shooting operation, and the electronic device can receive the shooting operation; in response to the shooting operation, the electronic device can store the preview image after exposure in an image folder corresponding to the scene category to which the preview image belongs. Figure 4 The scene shown.

[0132] From the above, it can be seen that the image to be classified may belong to multiple scene categories. Therefore, when it is necessary to store the image to be classified, the electronic device can store the image to be classified in the image folder corresponding to each scene category in the multiple scene categories to which the image to be classified belongs.

[0133] Of course, in order to save storage space, the electronic device may also store the image to be classified in any one of the multiple scene categories to which the image to be classified belongs. Alternatively, the electronic device stores an image to be classified and stores the image identifier of the image to be classified and the corresponding scene category. When it is necessary to display the image in a classified manner, the electronic device may obtain the image to be classified according to the image identifier of the image to be classified, and display the image to be classified in the classification display interface.

[0134] It is worth noting that by storing the exposed preview image in the image folder corresponding to the scene category to which the preview image belongs, it is convenient for users to search for images according to the scene category, thereby improving the interactivity with users and user stickiness.

[0135] As an example, when the image to be classified is a preview image, after determining the scene category to which the image to be classified belongs, the electronic device may also display a scene label in the preview image, and the scene label is used to describe the scene category to which the image to be classified belongs. For example, if the scene category to which the image to be classified belongs includes a cute pet scene category, a grass scene category, and a sky scene category, then the electronic device may display "cute pet", "grass", and "sky" in the preview image. The scene can refer to the above Figure 2 The application scenario shown.

[0136] As an example, when the image to be classified is a preview image, after determining the scene category to which the image to be classified belongs, the electronic device can also determine the imaging scheme of the image to be classified according to the scene category to which the image to be classified belongs, such as determining the exposure parameters, filter scheme, shooting mode, display resolution, etc. of the image to be classified. Figure 3 The scene shown.

[0137] For example, when the scene category to which the image to be classified belongs is a moon scene, the electronic device can reduce the exposure parameter of the image to be classified to obtain a clearer moon image. Alternatively, the electronic device can control the camera to enter the moon-viewing mode (also known as the moon-shooting mode) to change the imaging scheme of the image to be classified.

[0138] Exemplarily, when the scene category to which the image to be classified belongs is a text scene category or a scene category with rich texture, the electronic device may perform image super-resolution processing on the image to be classified, that is, enlarge the image display resolution of the image to be displayed.

[0139] In the embodiment of the present application, since there is no need to obtain the text feature vector again in the process of performing scene recognition on the image to be classified through multiple image classification models, the operation of the image classification model for scene recognition is reduced, and the efficiency of the image classification model for scene recognition is improved. In addition, since the scene recognition of the image to be classified can be performed separately through multiple image classification models, the scene category to which the image to be classified belongs can be obtained as much as possible, and then the electronic device can adopt different display schemes according to the scene category to which the image to be classified belongs, thereby improving the display quality and display effect of the image to be classified.

[0140] After explaining in detail the training method of the image classification model provided in the embodiment of the present application, the electronic device involved in the embodiment of the present application is explained.

[0141] As an example, the method can be applied to an electronic device capable of model training. As an example and not a limitation, the electronic device can be, but is not limited to, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, a vehicle-mounted device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a mobile phone, etc., and the embodiments of the present application are not limited to this.

[0142] In addition, the electronic device may also apply the trained image classification model, and the electronic device for training the image classification model and the electronic device for applying the image classification model may be the same electronic device or different electronic devices, and the embodiments of the present application do not impose any specific restrictions on this.

[0143] Fig.10 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.10The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0144] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0145] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0146] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0147] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0148] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0149] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0150] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0151] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is an integer greater than 1.

[0152] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0153] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0154] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is an integer greater than 1.

[0155] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0156] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0157] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0158] As an example, the NPU may include multiple image classification models provided by the embodiments of the present application, and the multiple image classification models include a first image classification model.

[0159] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0160] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the electronic device 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0161] The electronic device 100 can implement audio functions, such as music playback, recording, etc., through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D and the application processor.

[0162] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than a pressure threshold acts on a short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the pressure threshold acts on a short message application icon, an instruction to create a new short message is executed.

[0163] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0164] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, in a shooting scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0165] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0166] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0167] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0168] Next, the software system of the electronic device 100 will be described.

[0169] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software system of the electronic device 100.

[0170] Fig.11 1 is a block diagram of a software system of an electronic device 100 provided in an embodiment of the present application. Fig.11 The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system layer, and the kernel layer.

[0171] The application layer can include a series of application packages. Fig.11 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0172] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Fig.11As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make these data accessible to applications. These data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of the application, and the display interface can be composed of one or more views, for example, including a view for displaying a text message notification icon, including a view for displaying text, and including a view for displaying pictures. The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of an icon or scroll bar text, such as a notification of an application running in the background. The notification manager can also be a notification that appears on the screen in the form of a dialog window, such as a text message in the status bar, a reminder sound, an electronic device vibrating, an indicator light flashing, etc.

[0173] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0174] The system library can include multiple functional modules, such as: surface manager, media library, 3D graphics processing library (such as: OpenGL ES), 2D graphics engine (such as: SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0175] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0176] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0177] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the raw input event. For example, if the touch operation is a single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer, starts the camera application, and then calls the kernel layer to start the camera driver to capture static images or videos through the camera 193.

[0178] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0179] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. A training method for an image classification model, characterized in that: Applied in an electronic device, the method comprises: Determine a first text feature vector of first text information corresponding to a first scene category and a visual feature vector of each sample image in a sample image training set corresponding to the first scene category; updating a weight of an initial image classification model corresponding to the first scene category based on the first text feature vector; Iteratively training the updated initial image classification model based on the visual feature vector of each sample image to obtain a first image classification model corresponding to the first scene category, wherein the first image classification model can identify images belonging to the first scene category; Among them, the visual feature vector of each sample image is input into the updated initial image classification model for iterative training; during the iterative training process, the loss value of the second loss function between the classification result of the image classification model obtained after each training and the preset result is determined; when the loss value converges, the image classification model obtained at the time of convergence is determined as the first image classification model, and the preset result is the classification label corresponding to each sample image.

2. The method according to claim 1, characterized in that The determining of a first text feature vector of first text information corresponding to a first scene category and a visual feature vector of each sample image in a sample image training set corresponding to the first scene category includes: Acquire the first text information and the sample image training set corresponding to the first scene category; Processing the first text information through a pre-trained target text encoder to obtain the first text feature vector; Each sample image is processed respectively by a pre-trained target image encoder to obtain a visual feature vector of each sample image.

3. The method according to claim 1 or 2, characterized in that Before determining the first text feature vector of the first text information corresponding to the first scene category and the visual feature vector of each sample image in the sample image training set corresponding to the first scene category, the method further includes: Acquire a plurality of sample text information and a plurality of sample training images, wherein the plurality of sample text information corresponds to the plurality of sample training images one by one, and each piece of sample text information in the plurality of sample text information is used to describe a scene category to which a corresponding sample training image belongs; Iteratively training an initial text encoder based on the multiple sample text information, and iteratively training an initial image encoder based on the multiple sample training images; During the iterative training process, determining a loss value of a first loss function between a text encoder obtained after each training and an image encoder obtained after each training; When the loss value converges, the text encoder obtained at the time of convergence is determined as the target text encoder, and the image encoder obtained at the time of convergence is determined as the target image encoder. The target text encoder is used to determine the first text feature vector, and the target image encoder is used to determine the visual feature vector of each sample image.

4. The method according to claim 1, characterized in that The updating of the weight of the initial image classification model corresponding to the first scene category based on the first text feature vector includes: According to the dimension of the initial weight of the initial image classification model, the first text feature vector is reduced in dimension or increased in dimension to obtain a first vector; The weight of the initial image classification model is updated to the first vector.

5. The method according to any one of claims 1 to 4, characterized in that The electronic device includes a plurality of image classification models, the plurality of image classification models include the first image classification model, and different image classification models among the plurality of image classification models can identify different scene categories; After iteratively training the updated initial image classification model based on the visual feature vector of each sample image to obtain the first image classification model corresponding to the first scene category, the method further includes: Get the image to be classified; Inputting the images to be classified into the multiple image classification models respectively; The scene category to which the image to be classified belongs is determined according to the classification result output by each image classification model in the multiple image classification models.

6. The method according to claim 5, characterized in that The step of obtaining the image to be classified comprises: When the camera is turned on, determining that the preview image captured by the camera is the image to be classified; or, In the case where an image selection operation is received, it is determined that the image selected by the image selection operation is the image to be classified.

7. The method according to claim 6, characterized in that The step of determining, when the camera is turned on, that the preview image captured by the camera is the image to be classified includes: When the camera is turned on, a scene recognition control is displayed in the shooting interface, and image acquisition is performed through the camera to obtain the preview image, wherein the scene recognition control is used to control whether scene recognition is performed; In response to an opening operation of the scene recognition control, the preview image is determined to be the image to be classified.

8. The method according to claim 5, characterized in that The image to be classified is a preview image captured by a camera; After determining the scene category to which the image to be classified belongs according to the classification result output by each of the multiple image classification models, the method further includes: Receive shooting operation; In response to the shooting operation, the exposed preview image is stored in an image folder corresponding to the scene category to which the preview image belongs.

9. An electronic device, characterized in that: The structure of the electronic device includes a processor and a memory; The memory is used to store a program that supports the electronic device to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 8.

Images