Image classification method and system, medium and electronic equipment

Abstract

The invention provides an image classification method and system, a medium and electronic equipment. The method comprises the following steps: acquiring a to-be-detected image and a preset object category; based on the to-be-detected image and the preset object category, respectively obtaining the similarity between the to-be-detected image and the preset object category and an initial category result of the to-be-detected image; and classifying the to-be-detected image based on the similarity and the initial category result. According to the technical scheme, the accuracy of image classification can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image classification method and system, medium and electronic device. Background Technology

[0002] Image classification is a crucial part of visual language tasks. It typically relies on the expressive power of video and language information. Currently, most image classification is achieved through visual language models pre-trained on large-scale datasets. However, there may be mismatches between the video and language information of different images, resulting in unsatisfactory image classification accuracy.

[0003] Therefore, an image classification method is needed to improve the accuracy of image classification.

[0004] Application content

[0005] The purpose of this application is to provide an image classification method, system, medium, and electronic device to solve the technical problem that the accuracy of image classification is not ideal due to the mismatch between video and language information of different images.

[0006] Firstly, this application provides an image classification method. The image classification method includes: acquiring an image to be detected and a preset object category; acquiring a similarity between the image to be detected and the preset object category, and an initial category result of the image to be detected, based on the image to be detected and the preset object category, respectively; and classifying the image to be detected based on the similarity and the initial category result.

[0007] The preset object category includes at least one, each preset object category corresponds to a similarity with the image to be detected, and each preset object category corresponds to an initial category result of the image to be detected.

[0008] In one implementation of the first aspect, obtaining the image to be detected and the preset object category includes: obtaining the image to be detected; and obtaining the preset object category of the image to be detected through target detection.

[0009] In one implementation of the first aspect, obtaining the initial category result of the image to be detected based on the image to be detected and the preset object category includes: preprocessing the image to be detected; setting the preset object category according to a preset text format and obtaining the preset object category text; and inputting the preprocessed image to be detected and the preset object category text into a visual language model to obtain the initial category result of the image to be detected.

[0010] In one implementation of the first aspect, obtaining the similarity between the image to be detected and the preset object category based on the image to be detected and the preset object category includes: obtaining image features of the image to be detected and text features of the preset object category respectively; and calculating the similarity between the image features and the text features to obtain the similarity between the image to be detected and the preset object category.

[0011] In one implementation of the first aspect, the step of obtaining the image features of the image to be detected and the text features of the preset object category includes: preprocessing the image to be detected and the preset object category; processing the preprocessed image to be detected using an image encoder to obtain the image features of the image to be detected; and processing the preprocessed preset object category using a text encoder to obtain the text features of the preset object category.

[0012] In one implementation of the first aspect, classifying the image to be detected based on the similarity and the initial category result includes: determining whether the similarity is less than a preset threshold; if the similarity is less than the preset threshold, then the image to be detected does not include the preset object category corresponding to the similarity; if the similarity is not less than the preset threshold, then determining whether the initial category result corresponding to the similarity meets a preset requirement; if it does, then the image to be detected includes the preset object category corresponding to the similarity and the initial category result; if it does not, then the image to be detected does not include the preset object category.

[0013] Secondly, this application provides an image classification system. The image classification system includes: an acquisition module for acquiring an image to be detected and a preset object category; a processing module for acquiring, based on the image to be detected and the preset object category, a similarity score between the image to be detected and the preset object category, and an initial category result for the image to be detected; wherein the preset object category includes at least one, each preset object category corresponds to a similarity score with the image to be detected, and each preset object category corresponds to an initial category result for the image to be detected; and a classification module for classifying the image to be detected based on the similarity score and the initial category result.

[0014] In one implementation of the second aspect, the processing module includes: a first processing module and a second processing module; the first processing module is used to obtain the similarity between the image to be detected and the preset object category based on the image to be detected and the preset object category; the second processing module is used to obtain the initial category result of the image to be detected based on the image to be detected and the preset object category.

[0015] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the image classification method described in any of the first aspects of this application.

[0016] Fourthly, this application provides an electronic device, comprising: a processor and a memory; the memory for storing a computer program; the processor for executing the computer program stored in the memory to cause the electronic device to perform the image classification method according to any one of the first aspects of this application.

[0017] According to the image classification method, system, medium, and electronic device of this application, an initial category result is obtained for the image to be detected using a visual language model, the similarity between the image to be detected and the category to be classified is calculated using CLIP, and finally, the category of the image to be detected is further determined by combining the initial category result and the similarity. This can greatly improve the accuracy of the classification of the image to be detected. Attached Figure Description

[0018] Figure 1 The diagram shown is a scene illustration of the electronic device of this application in one embodiment.

[0019] Figure 2 The diagram shown is a flowchart of an embodiment of the image classification method described in this application.

[0020] Figure 3 The diagram shown is a flowchart of an embodiment of the method for obtaining the initial category result of an image to be detected as described in this application.

[0021] Figure 4 The diagram shown is a flowchart of an embodiment of the method for obtaining the similarity between an image to be detected and a preset object category as described in this application.

[0022] Figure 5 The diagram shown is a flowchart of an embodiment of the method for acquiring image features and text features as described in this application.

[0023] Figure 6 The diagram shown is a flowchart of an embodiment of the method for classifying images to be detected as described in this application.

[0024] Figure 7 The diagram shown is a flowchart of another embodiment of the method for classifying images to be detected as described in this application.

[0025] Figure 8 The diagram shown is a flowchart of another embodiment of the image classification method described in this application.

[0026] Figure 9The diagram shown is a structural schematic of an embodiment of the image classification system described in this application.

[0027] Figure 10 The diagram shown is a structural schematic of the electronic device described in an embodiment of this application.

[0028] Component designation explanation

[0029] 11 Mobile phone 12 Tablet computer 13 Laptop computer 900 Image classification system 910 Acquisition module 920 Processing module 921 First processing module 922 Second processing module 930 Classification module 101 Processing unit 102 Memory 1021 Random access memory 1022 Cache memory 1023 Storage system 1024 Program / utility 1025 Program module 103 Bus 104 Input / output interface 105 Network adapter S1-S3 Step S211-S213 Step S221-S222 Step S2211-S2213 Step S31-S32 Step Detailed Implementation

[0030] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0031] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0032] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0033] The following embodiments of this application provide an image classification method and system, medium, and electronic device. The method utilizes a visual language model to obtain initial category results for the image to be detected, uses CLIP to calculate the similarity between the image to be detected and the category to be classified, and finally combines the initial category results and the similarity to further determine the category of the image to be detected, which can greatly improve the accuracy of the classification of the image to be detected.

[0034] The image classification method of this application can be applied to, for example... Figure 1The electronic devices shown in this application may include mobile phones 11 with wireless charging capabilities, tablet computers 12, laptop computers 13, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This application does not impose any restrictions on the specific types of electronic devices.

[0035] For example, the electronic device may be a station (STAION, ST) in a WLAN with wireless charging capability, a cellular phone, cordless phone, Session Initiation Protocol (SIP) phone, Wireless Local Loop (WLL) station, Personal Digital Assistant (PDA) device, handheld device with wireless charging capability, computing device or other processing device, computer, laptop computer, handheld communication device, handheld computing device, and / or other devices for communication over a wireless system, as well as next-generation communication systems, such as mobile terminals in 5G networks, mobile terminals in future evolved Public Land Mobile Networks (PLMNs), or mobile terminals in future evolved Non-terrestrial Networks (NTNs).

[0036] For example, the electronic device can communicate with networks and other devices wirelessly. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0037] The principles and implementation methods of the image classification method and system, medium and electronic device described in the embodiments of this application will be explained in detail below with reference to the accompanying drawings, so that those skilled in the art can understand the image classification method and system, medium and electronic device of this embodiment without creative effort.

[0038] Please see Figure 2 The diagram shows a flowchart of the image classification method provided in this application embodiment. Figure 2 As shown, the image classification method includes the following steps S1 to S3.

[0039] Step S1: Obtain the image to be detected and the preset object category.

[0040] Specifically, an image to be detected is acquired, and a preset object category of the image to be detected is obtained through object detection.

[0041] Specifically, object detection methods include, but are not limited to, mainstream object detection models such as R-CNN and YOLO. By performing object detection on the image to be detected, object category information of the image to be detected under the object detection model is obtained. The object category information is the preset object category of the image to be detected.

[0042] It should be noted that the image to be detected, after target detection, includes at least one object category information, that is, at least one preset object category of the image to be detected can be obtained.

[0043] Step S2: Based on the image to be detected and the preset object category, obtain the similarity between the image to be detected and the preset object category, and the initial category result of the image to be detected.

[0044] Specifically, the preset object category includes at least one, each preset object category corresponds to a similarity score with the image to be detected, and each preset object category corresponds to an initial category result of the image to be detected. In some implementations, obtaining the initial category result of the image to be detected based on the image to be detected and the preset object category includes: preprocessing the image to be detected; setting the text format of the preset object category; and inputting the image to be detected and the preset object category into a visual language model to obtain the initial category result of the image to be detected.

[0045] like Figure 3 As shown, obtaining the initial category result of the image to be detected includes the following steps S211 to S213.

[0046] Step S211: Preprocess the image to be detected.

[0047] Specifically, before inputting the image to be detected into the visual language model, the image to be detected needs to be preprocessed, including: adjusting the size of the image to be detected, normalizing the image to be detected, and image enhancement.

[0048] Step S212: Set the preset object category according to the preset text format and obtain the preset object category text.

[0049] Specifically, the preset text format is: "Does the image contain any object named 'xx'?". Here, xx represents the preset object category obtained in step S1; the preset object category text is set and retrieved based on the actual number of detected objects of that preset object category.

[0050] It should be noted that setting and obtaining the preset object category text according to the above preset text format can help the visual language model better understand the image to be detected and improve the recognition results of the visual language model on the image to be detected.

[0051] In one embodiment, the preset object categories include: people, apples, brown tables, and windows.

[0052] Based on the text input requirements of the visual language model, the text format of the preset object category is set as follows: "Is there a 'person' object in the image? Is there an 'apple' object? Is there a 'brown table' object? Is there a 'window' object? If there is, answer yes; otherwise, answer no, no explanation is needed."

[0053] Step S213: Input the preprocessed image to be detected and the preset object category text into the visual language model to obtain the initial category result of the image to be detected.

[0054] Specifically, the preprocessed image to be detected and the preset object category text are input into the VLM model. The image to be detected is judged based on the input preset object category text, and the initial category result of the image to be detected is output.

[0055] In one embodiment, the initial category result includes "yes, no, no, yes", which means that the image to be detected contains a "person" object, no "apple" object, no "brown table" object, and a "window" object.

[0056] In some implementations, obtaining the similarity between the image to be detected and the preset object category based on the image to be detected and the preset object category includes: obtaining the image features of the image to be detected and the text features of the preset object category respectively; and calculating the similarity between the image features and the text features to obtain the similarity between the image to be detected and the preset object category.

[0057] like Figure 4 As shown, obtaining the similarity between the image to be detected and the preset object category includes the following steps S221 to S222.

[0058] Step S221: Obtain the image features of the image to be detected and the text features of the preset object category, respectively.

[0059] Specifically, in one embodiment, the image features of the image to be detected and the text features of the preset object category are obtained by CLIP image encoder and CLIP text encoder, respectively.

[0060] In some implementations, obtaining the image features of the image to be detected and the text features of the preset object category includes: preprocessing the image to be detected and the preset object category; processing the image to be detected using an image encoder to obtain the image features of the image to be detected; and processing the preset object category using a text encoder to obtain the text features of the preset object category.

[0061] like Figure 5 As shown, obtaining image features and text features includes the following steps S2211 to S2213.

[0062] Step S2211: Preprocess the image to be detected and the preset object category.

[0063] Specifically, for the image to be detected, before inputting it into the CLIP image encoder, the size of the image to be detected is adjusted. For example, if ResNet is used as the image encoder, the image size needs to be adjusted to... The pixel values ​​of the image to be detected are then converted from the range of [0, 255] to the range of [0, 1], and the average value of each color channel is subtracted and divided by the standard deviation.

[0064] Specifically, for the preset object category, the preset object category is cleaned by removing irrelevant punctuation marks, special characters and stop words to ensure that the input text is concise and closely related to the preset object category.

[0065] Step S2212: Process the preprocessed image to be detected using an image encoder to obtain the image features of the image to be detected.

[0066] Specifically, the preprocessed image to be detected is input into the CLIP image encoder. The CLIP image encoder processes the image to be detected through its convolutional neural network structure and outputs a high-dimensional vector, which is the image feature of the image to be detected. This vector contains the key visual features of the image to be detected.

[0067] Step S2213: Use a text encoder to process the preprocessed preset object category to obtain the text features of the preset object category.

[0068] Specifically, the CLIP text encoder, a Transformer-based model such as BERT or its variants, is used to process the pre-processed preset object categories of the input. The CLIP text encoder transforms the input preset object categories into a series of numerical representations, i.e., text embeddings.

[0069] Furthermore, the embedding vectors obtained through the CLIP text encoder are further processed to extract higher-level semantic features. The CLIP model optimizes these features through a contrastive learning task during its training process to capture the correlation between the text and the corresponding image.

[0070] It should be noted that when extracting text features from the preset object categories, each preset object category corresponds to an output vector, that is, each preset object category corresponds to a text feature.

[0071] Step S222: Calculate the similarity between the image features and the text features to obtain the similarity between the image to be detected and the preset object category.

[0072] Specifically, the image features and the text features are subjected to cosine similarity calculation.

[0073] In one embodiment, the preset object categories include: person, apple, brown table, and window, which corresponds to the acquisition of 4 text features, namely the first text feature, the second text feature, the third text feature, and the fourth text feature in the above order.

[0074] The image features are compared with the four text features to calculate cosine similarity, resulting in four similarity values: first similarity, second similarity, third similarity, and fourth similarity.

[0075] Step S3: Classify the image to be detected based on the similarity and the initial category result.

[0076] In some implementations, classifying the image to be detected based on the similarity and the initial category result includes: determining whether the similarity meets a preset threshold; if the similarity meets the preset threshold, determining whether the initial category result corresponding to the similarity meets a preset requirement; if it does, the image to be detected belongs to the preset object category; if it does not, the image to be detected does not belong to the preset object category; if the similarity does not meet the preset threshold, the image to be detected does not belong to the preset object category.

[0077] like Figure 6 As shown, classifying the image to be detected includes the following steps S31 to S32.

[0078] Step S31: Determine whether the similarity is not less than a preset threshold.

[0079] In one embodiment, the preset threshold is set to 0.6.

[0080] It should be noted that the preset threshold is not a unique fixed value and can be adjusted according to actual needs.

[0081] Step S32: If the similarity is less than the preset threshold, the image to be detected does not include the preset object category corresponding to the similarity; if the similarity is not less than the preset threshold, it is determined whether the initial category result corresponding to the similarity meets the preset requirements. If it does, the image to be detected includes the preset object category corresponding to the similarity and the initial category result. If it does not, the image to be detected does not include the preset object category.

[0082] Specifically, the preset requirement is that the output of the initial category result is yes.

[0083] Specifically, for the preset object category whose similarity is not less than the preset threshold, it is determined whether the corresponding initial category result is yes. If yes, the image to be detected includes the preset object category; if the corresponding initial category result is no, the image to be detected does not include the preset object category.

[0084] The preset object categories include: people, apples, brown tables, and windows; the initial category result is "yes, no, no, yes"; the similarity includes: first similarity of 0.6, second similarity of 0.8, third similarity of 0.3, and fourth similarity of 0.4; based on the above judgment results, the image to be detected includes objects of the category "people".

[0085] In some implementations, such as Figure 7 As shown, classifying the image to be detected based on the similarity and the initial category result can also be done by simultaneously determining whether the similarity is not less than a preset threshold and whether the initial category result meets a preset requirement; if the above determination conditions are met simultaneously, then the image to be detected includes the preset object category corresponding to the similarity and the initial category result.

[0086] Alternatively, classifying the image to be detected based on the similarity and the initial category result can also be as follows:

[0087] Determine whether the initial category result meets the preset requirements;

[0088] If the conditions are met, it is determined whether the similarity corresponding to the initial category result is not less than the preset threshold; if yes, the image to be detected includes the similarity and the preset object category corresponding to the initial category result; if no, the image to be detected does not include the similarity and the preset object category corresponding to the initial category result.

[0089] If the condition is not met, the image to be detected does not include the preset object category corresponding to the initial category result.

[0090] Please see Figure 8 The above is a flowchart of another embodiment of the image classification method described in this application.

[0091] like Figure 8 As shown, the image classification method includes:

[0092] Obtain the image to be detected and the preset object categories: Receive the image to be detected, perform target detection on the image to be detected, and obtain the preset object categories; the object categories include: object category 1, object category 2, and object category 3;

[0093] Obtaining initial category results: Input the preprocessed image to be detected and the preset object category into the VLM video language model, and input text format according to the preset object category to perform initial category judgment on the image to be detected, and output the initial category result of the image to be detected; the initial category result is: yes, yes, no;

[0094] Similarity acquisition: The preprocessed image to be detected is input into the CLIP image encoder to obtain the image features of the image to be detected, and the preprocessed preset object category is input into the CLIP text encoder to obtain the text features of the preset object category. The cosine similarity between the image features and the text features is calculated. The similarity includes: first similarity 0.6, second similarity 0.3, and third similarity 0.8.

[0095] The image to be detected is classified by: determining whether the similarity is not less than a preset threshold and whether the initial category result meets a preset requirement; if both conditions are met, the image to be detected includes the preset object category corresponding to the similarity and the initial category result; if neither condition is met, the image to be detected does not include the preset object category corresponding to the similarity and the initial category result; the preset threshold is 0.6, and the preset condition is: the initial category result is yes;

[0096] Therefore, in this embodiment, the classification result of the image to be detected is: the image to be detected includes object category 1.

[0097] It should be noted that the scope of protection of the image classification method described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0098] Please seeFigure 9 The diagram shown is a schematic representation of the image classification system described in an embodiment of this application.

[0099] like Figure 9 As shown, the image classification system 900 includes: an acquisition module 910, a processing module 920, and a classification module 930.

[0100] The acquisition module 910 is used to acquire the image to be detected and the preset object category.

[0101] The processing module 920 is used to obtain the similarity between the image to be detected and the preset object category and the initial category result of the image to be detected based on the image to be detected and the preset object category, respectively; wherein, the preset object category includes at least one, each preset object category corresponds to a similarity with the image to be detected, and each preset object category corresponds to an initial category result of the image to be detected.

[0102] Furthermore, the processing module 920 includes a first processing module 921 and a second processing module 922.

[0103] The first processing module 921 is used to obtain the similarity between the image to be detected and the preset object category based on the image to be detected and the preset object category.

[0104] The second processing module 922 is used to obtain the initial category result of the image to be detected based on the image to be detected and the preset object category.

[0105] The classification module 930 is used to classify the image to be detected based on the similarity and the initial category result.

[0106] It should be noted that the way each module of the image classification system described in this embodiment implements its corresponding function corresponds one-to-one with the image classification method described in the embodiments of this application, so it will not be repeated here.

[0107] It should be noted that the above division of modules is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, module x can be a separate processing element, or it can be integrated into a chip in the aforementioned device. Alternatively, it can be stored as program code in the memory of the aforementioned device, and its function can be called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0108] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to form a system-on-a-chip (SOC).

[0109] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0110] This application also provides an electronic device, including a processor and a memory.

[0111] Specifically, a memory is used to store computer programs; memory includes various media that can store program code, such as ROM, RAM, magnetic disks, USB flash drives, memory cards, or optical discs.

[0112] The processor is used to execute computer programs stored in memory to enable the electronic device to perform the image classification method described above.

[0113] like Figure 10 As shown, the electronic device of this application is embodied in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors or processing units 101, memory 102, and bus 103 connecting different system components (including memory 102 and processing unit 101).

[0114] Bus 103 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0115] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0116] Memory 102 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1021 and / or cache memory 1022. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 1023 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 10 Not shown; usually referred to as a "hard drive"). Although Figure 10 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 103 via one or more data media interfaces. Memory 102 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0117] A program / utility 1024 having a set (at least one) of program modules 1025 may be stored, for example, in memory 102. Such program modules 1025 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 1025 typically perform the functions and / or methods described in the embodiments of this application.

[0118] The electronic device can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), and with one or more devices that enable a user to interact with the electronic device, and / or with any device that enables the electronic device to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 104. Furthermore, the electronic device can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 105. Figure 10 As shown, network adapter 105 communicates with other modules of the electronic device via bus 103. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0120] In summary, the image classification method, system, medium, and electronic device of this application utilize a visual language model to obtain initial category results for the image to be detected, employs CLIP to calculate the similarity between the image to be detected and the category to be classified, and finally combines the initial category results and the similarity to further determine the category of the image to be detected. This significantly improves the accuracy of image classification. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial applicability.

[0121] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. An image classification method, characterized in that, The method includes: Obtain the image to be detected and the preset object category; Based on the image to be detected and the preset object category, the similarity between the image to be detected and the preset object category and the initial category result of the image to be detected are obtained respectively; wherein, the preset object category includes at least one, each preset object category corresponds to a similarity with the image to be detected, and each preset object category corresponds to an initial category result of the image to be detected; The image to be detected is classified based on the similarity and the initial category result.

2. The image classification method according to claim 1, characterized in that, The acquisition of the image to be detected and the preset object categories include: Acquire the image to be detected; The preset object category of the image to be detected is obtained through object detection.

3. The image classification method according to claim 1, characterized in that, The initial category result of the image to be detected based on the image to be detected and the preset object category includes: Preprocess the image to be detected; Set the preset object category according to the preset text format, and obtain the preset object category text; The preprocessed image to be detected and the preset object category text are input into the visual language model to obtain the initial category result of the image to be detected.

4. The image classification method according to claim 1, characterized in that, The step of obtaining the similarity between the image to be detected and the preset object category based on the image to be detected and the preset object category includes: The image features of the image to be detected and the text features of the preset object category are obtained respectively; The image features and text features are compared to calculate the similarity between them, so as to obtain the similarity between the image to be detected and the preset object category.

5. The image classification method according to claim 4, characterized in that, The steps of acquiring the image features of the image to be detected and the text features of the preset object category include: Preprocess the image to be detected and the preset object category; The preprocessed image to be detected is processed using an image encoder to obtain the image features of the image to be detected; The preprocessed preset object category is processed using a text encoder to obtain the text features of the preset object category.

6. The image classification method according to claim 1, characterized in that, The classification of the image to be detected based on the similarity and the initial category result includes: Determine whether the similarity is less than a preset threshold; If the similarity is less than the preset threshold, then the image to be detected does not include the preset object category corresponding to the similarity. If the similarity is not less than the preset threshold, then it is determined whether the initial category result corresponding to the similarity meets the preset requirements. If it does, the image to be detected includes the preset object category corresponding to the similarity and the initial category result. If it does not meet the requirements, the image to be detected does not include the preset object category.

7. An image classification system, characterized in that, The system includes: The acquisition module is used to acquire the image to be detected and the preset object category; The processing module is used to obtain the similarity between the image to be detected and the preset object category and the initial category result of the image to be detected based on the image to be detected and the preset object category, respectively; wherein, the preset object category includes at least one, each preset object category corresponds to a similarity with the image to be detected, and each preset object category corresponds to an initial category result of the image to be detected; A classification module is used to classify the image to be detected based on the similarity and the initial category result.

8. The image classification system according to claim 7, characterized in that, The processing module includes: a first processing module and a second processing module; The first processing module is used to obtain the similarity between the image to be detected and the preset object category based on the image to be detected and the preset object category; The second processing module is used to obtain the initial category result of the image to be detected based on the image to be detected and the preset object category.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program implements the image classification method as described in any one of claims 1 to 6.

10. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to cause the electronic device to perform the image classification method according to any one of claims 1 to 6.

Images