Method for determining image classification model, image classification method, device and product

Abstract

The embodiment of the present specification provides a method for determining an image classification model, a device for determining an image classification model, an image classification method, an image classification device, a computer readable storage medium, an electronic device and a computer program product, wherein the method for determining an image classification model comprises: on the one hand, determining a pre-trained image classification model; on the other hand, obtaining first training samples corresponding to a classification task, that is, different samples are used for fine-tuning for different classification tasks, and specifically, the classification task is divided into a user portrait classification task and a scene portrait classification task. Further, the pre-trained image classification model is fine-tuned through the first training samples to obtain an image classification model used for executing the classification task.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of artificial intelligence, and in particular, to a method for determining an image classification model, a device for determining an image classification model, an image classification method, an image classification device, a computer readable storage medium, an electronic device, and a computer program product. BACKGROUND

[0002] At present, the main general process of a computer vision (CV) portrait based on a pre-training model is: model pre-training in the upstream, data labeling in the downstream task, and further fine-tuning of the pre-training model on the labeled data. The advantage of this scheme is that the pre-training model can fully utilize the image and text representation capabilities of the open world for data labeling, and the visual encoder can be used for image representation, which has strong fine-tuning performance on the downstream task. However, in the scheme provided by the related technology, the representation of the image is the representation of the whole image, which causes the problem of low portrait recognition accuracy.

[0003] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present specification, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0004] Embodiments of the present specification provide a method for determining an image classification model, a device for determining an image classification model, an image classification method, an image classification device, a computer readable storage medium, an electronic device, and a computer program product, which can improve the accuracy of portrait recognition.

[0005] Other characteristics and advantages of the embodiments of the present specification will become apparent from the following detailed description, or will be learned by practice of the present specification.

[0006] According to one aspect of the embodiments of the present specification, a method for determining an image classification model is provided, which includes: determining a pre-trained image classification model; obtaining a first training sample corresponding to a classification task, wherein the classification task is divided into a user portrait classification task and a scene portrait classification task; fine-tuning the pre-trained image classification model through the first training sample to obtain an image classification model for executing the classification task.

[0007] In an exemplary embodiment, based on the above scheme, in the case where the classification task is a user portrait classification task, the obtaining of the first training sample corresponding to the classification task includes: performing semantic segmentation on a first original image, and obtaining a user region image in the first original image; determining the first training sample according to the user region image.

[0008] In the example embodiment, based on the above scheme, in the case that the classification task is a scene portrait classification task, the first training sample corresponding to the classification task is obtained by performing semantic segmentation on the second original image to obtain a scene region image in the second original image, and determining the first training sample according to the scene region image.

[0009] In the example embodiment, based on the above scheme, the pre-trained image classification model includes a pre-trained image encoder and a classification layer, the pre-trained image classification model is fine-tuned by the first training sample by inputting the first training sample into the pre-trained image encoder for encoding processing to obtain an encoded sample feature, performing front connection processing on the encoded sample feature by the classification layer to obtain a predicted label of the first training sample, and determining a loss function by the preset label of the first training sample and the predicted label, and optimizing the parameters of the classification layer by the loss function.

[0010] In the example embodiment, based on the above scheme, the pre-trained image classification model is determined by obtaining a second training sample corresponding to the classification task, and pre-training an original classification model by the second training sample to obtain the pre-trained image classification model.

[0011] In the example embodiment, based on the above scheme, the second training sample corresponding to the classification task is obtained by, in the case that the classification task is a user portrait classification task, performing semantic segmentation on a third original image to obtain a user region image in the third original image, and determining the second training sample according to the user region image, in the case that the classification task is a scene portrait classification task, performing semantic segmentation on a fourth original image to obtain a user region image in the fourth original image, and determining the second training sample according to the scene region image, or performing semantic segmentation on a plurality of frames of fourth original images respectively to obtain scene regions corresponding to the plurality of frames of fourth original images respectively, performing reconstruction processing on the scene regions corresponding to the plurality of frames of fourth original images respectively, and determining the second training sample according to reconstructed images obtained after the reconstruction processing.

[0012] In the example embodiment, based on the above scheme, the pre-trained image classification model includes an image encoder and a classification layer in the contrastive image-text pre-training CLIP.

[0013] According to another aspect of the embodiments of the present disclosure, there is provided an image classification method, comprising: inputting a to-be-tested image into an image classification model to perform encoding processing and full connection processing by the image classification model;

[0014] determining a classification result of the to-be-tested image according to an output of the image classification model; wherein the image classification model is determined according to the determination method of the image classification model provided in the above aspect.

[0015] According to still another aspect of the embodiments of the present disclosure, there is provided a determination apparatus of an image classification model configured in a first gateway, wherein the apparatus comprises: a determination module, an acquisition module and a fine-tuning module.

[0016] The determination module is configured to determine a pre-trained image classification model; the acquisition module is configured to acquire first training samples corresponding to a classification task, wherein the classification task comprises a user portrait classification task and a scene portrait classification task; and the fine-tuning module is configured to fine-tune the pre-trained image classification model by using the first training samples to obtain an image classification model used for performing the classification task.

[0017] According to yet another aspect of the embodiments of the present disclosure, there is provided an image classification apparatus, wherein the apparatus comprises: an input module and a determination module.

[0018] The input module is configured to input a to-be-tested image into an image classification model to perform encoding processing and full connection processing by the image classification model; and the determination module is configured to determine a classification result of the to-be-tested image according to an output of the image classification model; wherein the image classification model is determined according to the determination method of the image classification model provided in the above aspect.

[0019] According to an aspect of the embodiments of the present disclosure, there is provided an electronic device, comprising: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the determination method of the image classification model or the image classification method provided in any one of the above aspects.

[0020] According to another aspect of the embodiments of the present disclosure, there is provided a computer readable storage medium, wherein the computer readable storage medium stores instructions, and the instructions, when executed on a computer or a processor, cause the computer or the processor to execute the determination method of the image classification model or the image classification method provided in any one of the above aspects.

[0021] According to another aspect of the embodiments of this specification, a computer program product including instructions is provided, which, when run on a computer or processor, causes the computer or processor to perform a method for determining an image classification model or an image classification method as provided in any of the foregoing aspects.

[0022] The image classification model determination method, image classification model determination device, image classification method, image classification device, computer-readable storage medium, electronic device, and computer program product provided in the embodiments of this specification have the following technical effects:

[0023] On one hand, a pre-trained image classification model is determined; on the other hand, first training samples corresponding to the classification task are obtained. That is, different classification tasks require different samples for fine-tuning. Specifically, the classification tasks are divided into user profile classification tasks and scene profile classification tasks. The pre-trained image classification model is further fine-tuned using the first training samples to obtain an image classification model for performing the aforementioned classification tasks. The samples used in the fine-tuning process provided in this embodiment are determined based on the model's execution task, which is beneficial for providing targeted representations of the profile, thereby improving the profile recognition accuracy of the fine-tuned model.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification. It is obvious that the drawings described below are merely some embodiments of this specification, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0026] Figure 1 This is a flowchart illustrating the method for determining the image classification model provided in the embodiments of this specification.

[0027] Figure 2 This is a schematic diagram of the process for determining the first training sample provided in the embodiments of this specification.

[0028] Figure 3 The embodiments provided in this specification are based on Figure 2 A flowchart illustrating the process of fine-tuning the model using the first selected training samples.

[0029] Figure 4 This is a schematic diagram of the process for determining the first training sample provided in one embodiment of this specification.

[0030] Figure 5 The method for determining the pre-training model provided by the embodiment of the present specification is as shown in the following. Figure 4 The flowchart of the model fine-tuning process of the determined first training sample provided by the embodiment of the present specification.

[0031] Figure 6 The flowchart of the method for determining the pre-training model provided by the embodiment of the present specification is as shown in the following.

[0032] Figure 7 The structural diagram of the determination device of the image classification model provided by the embodiment of the present specification is as shown in the following.

[0033] Figure 8 The structural diagram of the image classification device provided by the embodiment of the present specification is as shown in the following.

[0034] Figure 9 The structural diagram of the electronic device provided by the embodiment of the present specification is as shown in the following. DETAILED DESCRIPTION

[0035] For the purposes of the present specification, the following terms are defined below.

[0036] The following description relates to the drawings in which:

[0037] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art. The features, structures, or characteristics described in connection with the examples can be combined in any suitable manner in one or more implementations. In the following description, numerous specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware- specific details and

[0038] Furthermore, the accompanying drawings are included to provide a further understanding of principles of embodiments and no limitations as to the scope of the concepts are intended to be derived therefrom. Identical reference numerals have been used in the drawings and description of the preferred embodiments to refer to like components. Changes in the reference numerals can be made to designate like components in the various figures. It is intended that the application can be practiced without necessarily being limited to these choices. While specific implementations are discussed, various modifications are possible. It is intended that the appended claims and examples cover all modifications that fall within the scope of the application. Various features and advantages of the application are set forth in the following claims.

[0039] In the existing scheme, the understanding of the scene and the user is based on global image information, and the accuracy and coverage of the portrait of the scene and the user are reduced; in order to improve the accuracy of the scene and the portrait of the user, a multi-modal portrait perception system of the user and the scene based on portrait semantic segmentation is proposed.

[0040] In the embodiment, the method comprises the following steps. Figure 1 The flowchart of the information of the method for determining the image classification model provided by the embodiment of the present specification is shown in FIG. 1. Figure 1 The embodiment shown in the figure comprises S110-S130.

[0041] In S110, a pre-trained image classification model is determined.

[0042] The pre-trained image classification model comprises a pre-trained encoder part and a classification layer set for performing an image classification task, wherein the parameters of the classification layer are initialization parameters, and the parameters of the pre-trained encoder part are pre-trained parameters.

[0043] In S120, a first training sample corresponding to a classification task is obtained, wherein the classification task comprises a user portrait classification task and a scene portrait classification task.

[0044] The core work of the user portrait classification task is to label the user, and one of the important purposes of labeling is to enable people to understand and facilitate computer processing. For example, in the field of telecommunications, the following classification statistics can be made: how many users use iPhone mobile phones, and what is the proportion of men and women in the user group using iPhone mobile phones; data mining work can also be done, such as using association rules to calculate what package users of iPhone mobile phones usually use, and using clustering algorithms to analyze the user feature distribution of iPhone mobile phone users.

[0045] The classification task of the above scene portrait is classification for a scene industry category, and is specifically applicable to a semantic recognition scene and an instance counting scene, and can help a merchant to build a self-service business intelligence (BI) capability. The BI generally refers to a technology and a tool for business analysis, and can help a merchant to obtain, process, and transform raw data into valuable information to guide business actions.

[0046] In S130, the pre-trained image classification model is fine-tuned by using the first training sample to obtain an image classification model for performing the classification task.

[0047] The pre-training of the original model is common learning, and the fine-tuning of the pre-trained model is characteristic learning. For example, the original model is a contrastive language-image pre-training (CLIP) model based on a contrastive text-image pair, which includes a text encoder and an image encoder. In the fine-tuning stage provided in this embodiment, the image encoder and the classification layer in the pre-trained CLIP model are fine-tuned.

[0048] Specifically, the first training sample is input into the pre-trained image encoder for encoding processing to obtain an encoded sample feature. The classification layer is used to perform front connection processing on the encoded sample feature to obtain a predicted label of the first training sample. Further, the preset label of the first training sample and the predicted label thereof are used to determine a loss function (cross-entropy function), and the parameters of the classification layer are optimized by using the loss function.

[0049] Figure 1 In the scheme provided in the illustrated embodiment, a pre-trained image classification model is determined, and a first training sample corresponding to a classification task is obtained. The classification task includes a user portrait classification task and a scene portrait classification task. The pre-trained image classification model is fine-tuned by using the first training sample to obtain an image classification model for performing the classification task. The sample used in the fine-tuning process provided in this embodiment is determined according to the execution task of the model, which is beneficial to providing a portrait representation in a targeted manner, thereby improving the portrait recognition accuracy of the fine-tuned model.

[0050] In the case where the task type is the user portrait classification task, the obtained first training sample corresponds to the user portrait classification task. For example, the first training sample is a text-image pair, and the text is a user portrait description. Figure 2 The flowchart for obtaining the first training sample provided in this embodiment is shown in FIG. 2. As shown in FIG. 2, the first training sample is obtained by using a user portrait description and a corresponding image. Figure 2The first original image 20 is input into the semantic segmentation model 200 for semantic segmentation. The semantic segmentation model classifies the pixels of the first original image, and the output of the model is shown in image 22. That is, the pixels in the first original image 20 are divided into types corresponding to user profiles and types corresponding to scene profiles other than users. Further, based on image 22 and the first original image 20, the user region image 24 in the first original image 20 can be obtained. In this embodiment of the specification, the training samples (denoted as: first training samples) used for model fine-tuning will be determined based on the user region image 24.

[0051] In the case of a classification task where the task type is user profiling, the first training sample (e.g., ...) is determined for model fine-tuning. Figure 2 Following image 24, the pre-trained image classification model is further fine-tuned using this first training sample. For example, when image 24 is used as the first training sample, its preset labels may include, for example, "face," "single face," "trendy style," "young person," "female," "thin person," "long hair," and "dyed hair." (See reference...) Figure 3 The pre-trained image classification model 300 includes an image encoder 30, a classification layer 32, and a softmax layer 34. After receiving the first training sample (such as image 24), the pre-trained image classification model 300 can generate predicted labels, further determine the loss function based on the preset label and predicted label of the sample, and optimize the parameters of the upper classification layer through the loss function, thereby achieving the technical effect of fine-tuning the model through the first training sample.

[0052] exist Figure 2 and Figure 3 In the provided embodiments, during the model fine-tuning process for the user profile classification task, training samples containing only user information (i.e., without scene profiles) are used for fine-tuning. During the fine-tuning process, it is beneficial to provide targeted representations of user profiles, thereby improving the accuracy of the fine-tuned model in recognizing user profiles.

[0053] In the case of a scene portrait classification task, the first training sample obtained corresponds to the scene portrait classification task. For example, Figure 4 This is a schematic diagram illustrating the process of obtaining the first training sample as provided in an embodiment of this specification. (See reference...) Figure 4The second original image 40 is input into the semantic segmentation model for semantic segmentation. The semantic segmentation model classifies the pixels of the second original image, i.e., classifies the pixels in the second original image 40 into the type 42 corresponding to the user portrait and the type corresponding to the scene portrait other than the user. Further, based on the image 42 and the second original image 40, a scene region image in the second original image 40 can be obtained. In an embodiment of the present specification, the scene region image can be directly used as a training sample (also referred to as: first training sample) for model fine-tuning. In another embodiment of the present specification, multiple scene region images of the same scene are reconstructed to obtain image information of a larger perspective, and the image 430 after the reconstruction processing is determined as the first training sample.

[0054] In the case of a classification task of the scene portrait type, after the first training sample (such as the image 430) for model fine-tuning is determined, the pre-trained image classification model is further fine-tuned by the first training sample. For example, in the case of the image 430 as the first training sample, the preset labels thereof can include, for example, "indoor environment", "no face", and "supermarket". Figure 4 Figure 5 The pre-trained image classification model 500 includes an image encoder 50, a classification layer 52, and a softmax layer 54. After the pre-trained image classification model 500 receives the above-mentioned first training sample (such as the image 430), a predicted label can be generated. Further, a loss function is determined according to the preset label and the predicted label of the sample, and the parameters of the above-mentioned classification layer are optimized by the above-mentioned loss function, thereby achieving the technical effect of fine-tuning the above-mentioned model by the first training sample.

[0055] In the embodiments provided in Figure 4 and Figure 5 In the model fine-tuning process of the scene portrait classification task, only training samples containing scene information (i.e., not containing user portraits) are used for fine-tuning, which is beneficial to provide a representation of the scene portrait during the fine-tuning process, thereby improving the recognition accuracy of the fine-tuned model for the scene portrait.

[0056] In the above-mentioned embodiments, only samples after semantic segmentation are used in the fine-tuning stage. If the execution task of the model is the classification of user portraits, only samples containing user portrait information are used. If the execution task of the model is the classification of scene portraits, only samples containing scene portrait information are used, which is beneficial to provide a representation of the portrait, thereby improving the portrait recognition accuracy of the fine-tuned model. In order to further improve the model effect, samples after semantic segmentation can be used in the pre-training stage and the fine-tuning stage. For example, Figure 6 ​A flowchart of the determination method of the pre-trained model provided by the embodiments of the present specification can be used as a specific implementation of S110. Referring to Figure 6 The embodiment shown in the figure includes:

[0057] S1102, obtaining a second training sample corresponding to the classification task; S1104, pre-training the original classification model through the second training sample to obtain a pre-trained image classification model.

[0058] Specifically, in the case where the classification task is a user portrait classification task, the second training sample for training the original model is determined by the embodiment as shown in Figure 2 Specifically, the third original image is subjected to semantic segmentation to obtain a user region image in the third original image, the second training sample is determined according to the user region image, and model pre-training is performed.

[0059] In the case where the classification task is a scene portrait classification task, the second training sample for training the original model is determined by the embodiment as shown in Figure 4 Specifically, the fourth original image is subjected to semantic segmentation to obtain a user region image in the fourth original image; the second training sample is determined according to the scene region image; or, a plurality of frames of the fourth original image about the same scene are respectively subjected to semantic segmentation to obtain scene regions corresponding to the plurality of frames of the fourth original image respectively; the scene regions corresponding to the plurality of frames of the fourth original image respectively are subjected to reconstruction processing; the second training sample is determined according to the reconstructed image obtained after the reconstruction processing. Further, model pre-training is performed.

[0060] In the embodiment shown in Figure 6 The training sample used in the model pre-training stage is determined according to the execution task of the model, which is beneficial to providing a representation of the portrait in a targeted manner, that is, the parameters of the pre-trained image encoder are determined according to the user portrait information only / scene portrait information only. Further, the training sample used in the model fine-tuning stage is determined according to the execution task of the model, which is beneficial to providing a representation of the portrait in a targeted manner, that is, the parameters of the fine-tuned image encoder and the classification layer parameters are determined according to the user portrait information only / scene portrait information only, thereby improving the portrait recognition accuracy of the fine-tuned model and improving the overall model effect.

[0061] In the exemplary embodiments, the scheme provided by the embodiments of the present specification can be applied to CLIP. For example, in the model pre-training stage, according to the model classification task being user portrait classification or scene portrait classification, the sample for model pre-training is determined to contain only user portrait information or only scene portrait information. Further, the model pre-training stage is performed according to the determined sample.

[0062] Further, after the pre-training stage, the image encoder part can be used to determine the pre-trained image classification model as shown in Figure 3 Further, according to the model classification task being user portrait classification or scene portrait classification, the samples for model fine-tuning are determined to contain only user portrait information or only scene portrait information. Then, the model fine-tuning stage is performed according to the determined samples.

[0063] It can be seen that the scheme provided by the embodiments of the present specification can be applied to a multi-modal portrait perception system, where multi-modal means collaborative inference of multiple heterogeneous modal data. Multi-modal data analysis needs to be mutually promoted with advanced cognitive intelligence; in the field of artificial intelligence, it often means collaborative perception of information such as images, texts, and voices, so that the system can more accurately understand the external world.

[0064] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present specification, and are not for limiting purposes. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0065] The following is a device embodiment of the present specification, which can be used to execute the method embodiments of the present specification. For details not disclosed in the device embodiments of the present specification, please refer to the method embodiments of the present specification.

[0066] wherein, Figure 7 The structure diagram of the image classification model determination device provided by the embodiments of the present specification is shown in FIG. 7. Figure 7 The image classification model determination device shown in the figure can be realized by software, hardware or a combination of both to become all or part of an electronic device, and can also be integrated as an independent module in an electronic device.

[0067] In the embodiments of the present specification, the image classification model determination device 700 includes a determination module 710, an acquisition module 720, and a fine-tuning module 730.

[0068] The determination module 710 is configured to determine a pre-trained image classification model. The acquisition module 720 is configured to acquire first training samples corresponding to a classification task, where the classification task includes a user portrait classification task and a scene portrait classification task. The fine-tuning module 730 is configured to fine-tune the pre-trained image classification model through the first training samples to obtain an image classification model for performing the classification task.

[0069] In the example embodiment, based on the foregoing scheme, in the case that the classification task is a user portrait classification task, the obtaining module 720 is specifically configured to: perform semantic segmentation on the first original image, and obtain a user region image in the first original image; and determine the first training sample according to the user region image.

[0070] In the example embodiment, based on the foregoing scheme, in the case that the classification task is a scene portrait classification task, the obtaining module 720 is specifically configured to: perform semantic segmentation on the second original image, and obtain a scene region image in the second original image; determine the first training sample according to the scene region image; or perform semantic segmentation on a plurality of second original samples respectively about the same scene, to obtain scene regions corresponding to the plurality of second original images respectively; perform reconstruction processing on the scene regions corresponding to the plurality of second original images respectively; and determine the first training sample according to a reconstruction image obtained after the reconstruction processing.

[0071] In the example embodiment, based on the foregoing scheme, the pre-trained image classification model comprises a pre-trained image encoder and a classification layer; and the fine-tuning module 730 is specifically configured to: input the first training sample into the pre-trained image encoder for encoding processing, to obtain an encoded sample feature; perform pre-connection processing on the encoded sample feature through the classification layer, to obtain a predicted label about the first training sample; and determine a loss function according to a preset label of the first training sample and the predicted label, and optimize parameters of the classification layer through the loss function.

[0072] In the example embodiment, based on the foregoing scheme, the determining module 710 comprises an obtaining unit and a pre-training unit. The obtaining unit is configured to: obtain a second training sample corresponding to the classification task; and the pre-training unit is configured to: pre-train an original classification model through the second training sample, to obtain the pre-trained image classification model.

[0073] In an example embodiment, based on the foregoing scheme, the acquisition unit is specifically configured to: in a case where the classification task is a user portrait classification task, perform semantic segmentation on the third original image to obtain a user region image in the third original image; determine the second training sample according to the user region image; in a case where the classification task is a scene portrait classification task, perform semantic segmentation on a fourth original image to obtain a user region image in the fourth original image; determine the second training sample according to the scene region image; or perform semantic segmentation on multiple frames of fourth original images respectively to obtain scene regions corresponding to the multiple frames of fourth original images respectively; perform reconstruction processing on the scene regions corresponding to the multiple frames of fourth original images respectively; and determine the second training sample according to a reconstructed image obtained after the reconstruction processing.

[0074] In an example embodiment, based on the foregoing scheme, the pre-trained image classification model includes an image encoder and a classification layer in a contrastive image-text pre-training CLIP.

[0075] It should be noted that the determination apparatus of the image classification model provided in the above embodiments is only used as an example to illustrate the division of the functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above-described functions.

[0076] In addition, the determination apparatus of the image classification model and the determination method of the image classification model provided in the above embodiments belong to the same concept, so for details not disclosed in the apparatus embodiments of the present specification, please refer to the above-mentioned determination method of the image classification model of the present specification, which will not be described here.

[0077] wherein, Figure 8 The structure of the image classification apparatus provided in the embodiments of the present specification is shown in the following figure. Figure 8 The determination apparatus of the image classification model shown in the figure can be realized by software, hardware or a combination of both to become all or part of an electronic device, and can also be integrated as an independent module in an electronic device.

[0078] In the embodiments of the present specification, the image classification apparatus 800 includes an input module 810 and a determination module 820.

[0079] The input module 810 is configured to input a to-be-tested image into the image classification model to perform encoding processing and full connection processing on the to-be-tested image by using the image classification model; and the determination module 820 is configured to determine a classification result of the to-be-tested image according to an output of the image classification model. The image classification model is determined by using the method for determining an image classification model provided in the embodiments.

[0080] It should be noted that the determination apparatus for determining an image classification model provided in the embodiments is only used as an example to divide the functions of the apparatus, and in actual applications, the functions can be completed by different functional modules according to actual needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above.

[0081] In addition, the determination apparatus for determining an image classification model and the method for determining an image classification model provided in the embodiments belong to the same concept, and therefore, details not disclosed in the apparatus embodiments of the present specification are described with reference to the method embodiments of the determination apparatus for determining an image classification model provided in the present specification, which will not be described herein again.

[0082] The embodiments of the present specification also provide a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement the steps of the method of any of the preceding embodiments. The computer readable storage medium can include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD (Digital Video Disc), a CD-ROM (Compact Disc Read-Only Memory), a microdrive, and a magneto-optical disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable read only memory), a DRAM (Dynamic Random Access Memory), a VRAM (Video Random Access Memory), a flash memory device, a magnetic or optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0083] Figure 9An exemplary structural diagram of an electronic device is shown in the embodiments of the present specification. Please refer to Figure 9 As shown in the figure, the electronic device 900 comprises a processor 901 and a memory 902.

[0084] In the embodiments of the present specification, the processor 901 is the control center of the computer system, which can be the processor of a physical machine or the processor of a virtual machine. The processor 901 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 can be implemented in at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), Programmable Logic Array (PLA). The processor 901 can also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake-up state; the coprocessor is a low-power processor for processing data in the standby state.

[0085] In the embodiments of the present specification, when the electronic device executes the determination method of the image classification model, the processor 901 is specifically configured to: determine a pre-trained image classification model; obtain a first training sample corresponding to a classification task, wherein the classification task includes a user portrait classification task and a scene portrait classification task; and fine-tune the pre-trained image classification model through the first training sample to obtain an image classification model for executing the classification task.

[0086] Further, when the classification task is the user portrait classification task, the processor 901 is specifically configured to: perform semantic segmentation on a first original image to obtain a user region image in the first original image; and determine the first training sample according to the user region image.

[0087] Further, when the classification task is the scene portrait classification task, the processor 901 is specifically configured to: perform semantic segmentation on a second original image to obtain a scene region image in the second original image; determine the first training sample according to the scene region image; or perform semantic segmentation on multiple frames of second original samples respectively to obtain scene regions corresponding to the multiple frames of second original images respectively; perform reconstruction processing on the scene regions corresponding to the multiple frames of second original images respectively; and determine the first training sample according to the reconstructed image obtained after the reconstruction processing.

[0088] Further, the pre-trained image classification model comprises a pre-trained image encoder and a classification layer.

[0089] The pre-training image classification model is fine-tuned by the first training sample, comprising: inputting the first training sample into the pre-trained image encoder for encoding processing to obtain an encoded sample feature; performing pre-connection processing on the encoded sample feature by the classification layer to obtain a predicted label of the first training sample; determining a loss function according to the preset label of the first training sample and the predicted label, and optimizing the parameters of the classification layer by the loss function.

[0090] Further, the pre-trained image classification model comprises a pre-trained image encoder and a classification layer.

[0091] Further, the pre-training image classification model comprises a pre-trained image encoder and a classification layer.

[0092] Further, the pre-training image classification model comprises a pre-trained image encoder and a classification layer.

[0093] In the embodiment of the present specification, when the electronic device executes the image classification method, the processor 901 is configured to: input a to-be-tested image into an image classification model to perform encoding processing and full connection processing by the image classification model; and determine a classification result of the to-be-tested image according to an output of the image classification model; wherein the image classification model is determined according to the determination method of the image classification model provided in the embodiment.

[0094] The memory 902 can include one or more computer-readable storage media. The computer-readable storage media can be non-transitory. The memory 902 can also include high-speed random access memory and can include nonvolatile memory, such as one or more magnetic disk storage devices, optical storage devices, flash memory devices, or other nonvolatile solid-state storage devices. In some embodiments of the disclosure, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction for execution by the processor 901 to implement the method in the embodiments of the disclosure.

[0095] In some embodiments, the electronic device 900 further includes a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902, and the peripheral device interface 903 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 903 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a display screen 904, an image acquisition component (camera) 905, and an audio circuit 906.

[0096] The peripheral device interface 903 can be used to connect at least one peripheral device related to input / output (I / O) to the processor 901 and the memory 902. In some embodiments of the disclosure, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments of the disclosure, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on a separate chip or circuit board. The embodiments of the disclosure are not limited in this regard.

[0097] The display screen 904 is configured to display a user interface (UI). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 904 is a touch display screen, the display screen 904 is further configured to capture touch signals on or above the surface of the display screen 904. The touch signals can be input to the processor 901 as control signals for processing. In this case, the display screen 904 can also be configured to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments of the present specification, the display screen 904 can be one, arranged on the front panel of the electronic device 900; in some other embodiments of the present specification, the display screen 904 can be at least two, arranged on different surfaces of the electronic device 900 or in a folding design; in some other embodiments of the present specification, the display screen 904 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device 900. Even, the display screen 904 can also be arranged in an irregular shape other than a rectangle, i.e., a notched screen. The display screen 904 can be made of a material such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0098] The image capturing component (camera) 905 is configured to capture images or videos. Optionally, the image capturing component (camera) 905 includes a front camera and a rear camera. Generally, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a long-focus camera, to realize the background blurring function of the main camera and the depth-of-field camera, the panorama shooting and virtual reality (VR) shooting function of the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments of the present specification, the image capturing component (camera) 905 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0099] The audio circuit 906 can include a microphone and a speaker. The microphone is configured to capture sound waves of a user and the environment, and convert the sound waves into an electrical signal input to the processor 901 for processing. For the purpose of stereo sound capture or noise reduction, the microphone can be multiple, arranged at different parts of the electronic device 900. The microphone can also be an array microphone or an omnidirectional capture microphone.

[0100] The power supply 907 is configured to supply power to various components in the electronic device 900. The power supply 907 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 907 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery that is charged through a wired line, and the wireless charging battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0101] The structural block diagram of the electronic device shown in the embodiments of the present specification does not constitute a limitation on the electronic device 900, and the electronic device 900 can include more or fewer components than illustrated, or combine certain components, or adopt a different arrangement of components.

[0102] In the description of the present specification, it should be understood that the terms "first", "second" and the like are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance. For those skilled in the art, the specific meanings of the above terms in the present specification can be understood according to the specific circumstances. In addition, in the description of the present specification, "a plurality of" means two or more, unless otherwise specified. "And / or", which describes the association relationship of the associated objects, means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship.

[0103] The embodiments of the present specification also provide a computer readable storage medium, which stores instructions when running on a computer or a processor, so that the computer or the processor executes one or more steps in the above embodiments. The constituent modules of the available field determination device can be stored in the above computer readable storage medium if they are realized in the form of software functional units and sold or used as independent products. The constituent modules of the available field determination device can be stored in the above computer readable storage medium if they are realized in the form of software functional units and sold or used as independent products.

[0104] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The above computer program product includes one or more computer instructions. When loaded and executed by a computer, the above computer program instructions generate all or part of the processes or functions described in the embodiments of the present specification. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0105] It should be noted that the above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired result. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0106] The above is only a specific embodiment of the present specification, but the protection scope of the present specification is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present specification, which should be covered within the protection scope of the present specification. Therefore, equivalent changes made in accordance with the claims of the present specification are still within the scope covered by the present specification.

Claims

1. A method for determining an image classification model, wherein, The method includes: Determine the pre-trained image classification model; Obtain the first training sample corresponding to the classification task, wherein the classification task is divided into user profile classification task and scene profile classification task; The pre-trained image classification model is fine-tuned using the first training sample to obtain an image classification model for performing the classification task. Wherein, in the case that the classification task is a scene image classification task, obtaining the first training sample corresponding to the classification task includes: Semantic segmentation is performed on multiple frames of second original images of the same scene to obtain scene region images corresponding to the multiple frames of second original images respectively; reconstruction processing is performed on the scene region images corresponding to the multiple frames of second original images respectively; the first training sample is determined based on the reconstructed image obtained after the reconstruction processing. The step of performing semantic segmentation on multiple frames of second original images of the same scene to obtain scene region images corresponding to the multiple frames of second original images includes: The multiple frames of the second original images are input into the semantic segmentation model. The semantic segmentation model divides the pixels in each second original image into types corresponding to user profiles and types corresponding to scene profiles, and obtains the output images of each model. Based on the output images of each model and each second original image, scene region images corresponding to the multiple frames of second original images are obtained respectively.

2. The method according to claim 1, wherein, In the case where the classification task is a user profile classification task, obtaining the first training sample corresponding to the classification task includes: Perform semantic segmentation on the first original image and obtain the user region image in the first original image; The first training sample is determined based on the user region image.

3. The method according to any one of claims 1 to 2, wherein, The pre-trained image classification model includes a pre-trained image encoder and a classification layer; The step of fine-tuning the pre-trained image classification model using the first training samples includes: The first training sample is input into the pre-trained image encoder for encoding processing to obtain encoded sample features; The coded sample features are processed by the classification layer through a pre-connection process to obtain the predicted label for the first training sample. The preset label and the predicted label of the first training sample are used to determine the loss function, and the parameters of the classification layer are optimized by the loss function.

4. The method according to any one of claims 1 to 2, wherein, The determination of the pre-trained image classification model includes: Obtain a second training sample corresponding to the classification task; The pre-trained image classification model is obtained by pre-training the original classification model using the second training sample.

5. The method according to claim 4, wherein, The step of obtaining the second training sample corresponding to the classification task includes: In the case where the classification task is a user profile classification task, semantic segmentation is performed on the third original image to obtain the user region image in the third original image; the second training sample is determined based on the user region image. In the case where the classification task is a scene portrait classification task, semantic segmentation is performed on the fourth original image to obtain the user region image in the fourth original image; the second training sample is determined based on the scene region image; or, semantic segmentation is performed on multiple frames of the fourth original image about the same scene to obtain the scene regions corresponding to the multiple frames of the fourth original image respectively; the scene regions corresponding to the multiple frames of the fourth original image are reconstructed; the second training sample is determined based on the reconstructed image obtained after the reconstruction process.

6. The method according to any one of claims 1 to 2, wherein, The pre-trained image classification model includes an image encoder and a classification layer in the contrastive image-text pre-training CLIP.

7. An image classification method, wherein, The method includes: The image to be tested is input into the image classification model for encoding and fully connected processing. The classification result of the image to be tested is determined based on the output of the image classification model; The image classification model is determined by the method according to any one of claims 1 to 6.

8. A device for determining an image classification model, wherein, The device includes: The determination module is used to determine the pre-trained image classification model; The acquisition module is used to acquire the first training sample corresponding to the classification task, wherein the classification task is divided into user profile classification task and scene profile classification task. The fine-tuning module is used to fine-tune the pre-trained image classification model using the first training samples to obtain an image classification model for performing the classification task. In the case where the classification task is a scene portrait classification task, the acquisition module is specifically used to: perform semantic segmentation on multiple frames of second original images of the same scene to obtain scene region images corresponding to the multiple frames of second original images respectively; perform reconstruction processing on the scene region images corresponding to the multiple frames of second original images respectively; and determine the first training sample based on the reconstructed image obtained after the reconstruction processing. The step of performing semantic segmentation on multiple frames of second original images of the same scene to obtain scene region images corresponding to the multiple frames of second original images includes: inputting the multiple frames of second original images into a semantic segmentation model, dividing the pixels in each second original image into types corresponding to user profiles and types corresponding to scene profiles through the semantic segmentation model, and obtaining output images of each model; and obtaining scene region images corresponding to the multiple frames of second original images based on the output images of each model and the second original images.

9. An image classification device, wherein, The device includes: The input module is used to input the image to be tested into the image classification model, so that the image classification model can perform encoding processing and fully connected processing. The determination module is used to determine the classification result of the image to be tested based on the output of the image classification model; The image classification model is determined by the method according to any one of claims 1 to 6.

10. A computer-readable storage medium storing instructions, wherein, When the instructions are executed on a computer or processor, they cause the computer or processor to perform the method as described in any one of claims 1 to 7.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

12. A computer program product containing instructions, wherein, When the computer program product is run on a computer or processor, it causes the computer or processor to perform the method as described in any one of claims 1 to 7.

Images