Image classification method and device, electronic equipment and computer readable medium

Abstract

The disclosure provides an image classification method, device and electronic equipment, and relates to the technical field of image processing, in particular to the technical field of artificial intelligence, computer vision and the like. The specific implementation scheme is as follows: inputting a to-be-classified image into a pre-trained target detection network to obtain at least one candidate region in the to-be-classified image; inputting a candidate region image corresponding to the candidate region into a pre-trained image feature extraction network to obtain an image feature corresponding to the candidate region; inputting the to-be-classified image into a pre-trained text recognition network to obtain a text feature corresponding to text in the to-be-classified image; splicing the image feature and the text feature to obtain a spliced feature, and inputting the spliced feature into a pre-trained classification network to obtain a classification result corresponding to the to-be-classified image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more particularly to the fields of artificial intelligence and computer vision. Specifically, this disclosure relates to an image classification method, apparatus, electronic device, and computer-readable medium. Background Technology

[0002] Some special forms of images (such as comics) are often harmful due to their exaggerated artistic features, and these images are often highly concealed and difficult to detect.

[0003] For example, the dissemination of harmful comics is not allowed on social media platforms and live streaming platforms. However, due to the strong concealment of comics, the risk control difficulty of social media platforms and live streaming platforms is increased. Summary of the Invention

[0004] This disclosure provides an image classification method, apparatus, electronic device, and computer-readable medium.

[0005] According to a first aspect of this disclosure, an image classification method is provided, the method comprising:

[0006] The image to be classified is input into a pre-trained object detection network to obtain at least one candidate region in the image to be classified.

[0007] The candidate region image corresponding to the candidate region is input into a pre-trained image feature extraction network to obtain the image features corresponding to the candidate region;

[0008] The image to be classified is input into a pre-trained text recognition network to obtain the text features corresponding to the text in the image to be classified.

[0009] The image features and text features are concatenated to obtain concatenated features, which are then input into a pre-trained classification network to obtain the classification result corresponding to the image to be classified.

[0010] According to a second aspect of this disclosure, an image classification apparatus is provided, comprising:

[0011] The region module is used to input the image to be classified into a pre-trained target detection network to obtain at least one candidate region in the image to be classified.

[0012] The image module is used to input the candidate region image corresponding to the candidate region into a pre-trained image feature extraction network to obtain the image features corresponding to the candidate region;

[0013] The text module is used to input the image to be classified into a pre-trained text recognition network to obtain the text features corresponding to the text in the image to be classified.

[0014] The classification module is used to concatenate the image features and the text features to obtain concatenated features, and input the concatenated features into a pre-trained classification network to obtain the classification result corresponding to the image to be classified.

[0015] According to a third aspect of this disclosure, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to at least one of the aforementioned processors; wherein,

[0018] The memory stores instructions that can be executed by at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the image classification method.

[0019] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the image classification method described above.

[0020] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described image classification method.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a schematic flowchart of an image classification method provided in an embodiment of this disclosure;

[0024] Figure 2 This is a flowchart illustrating some steps of an image classification method provided in an embodiment of this disclosure;

[0025] Figure 3 This is a flowchart illustrating some steps of an image classification method provided in an embodiment of this disclosure;

[0026] Figure 4 This is a schematic diagram of a specific embodiment of an image classification method provided in this disclosure;

[0027] Figure 5 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this disclosure;

[0028] Figure 6 This is a block diagram of an electronic device used to implement the image classification method of the embodiments of this disclosure. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] In some related technologies, object detection methods can be used to detect specific symbols in an image to determine whether the image is harmful. However, this method only considers the symbols involved in the image, lacks a global semantic understanding of the image, and does not fully utilize the joint information and relational information of the image context.

[0031] In some related technologies, OCR (Optical Character Recognition) can be used to extract text from images, and then a comprehensive judgment can be made on the text to determine whether the image is harmful. However, this method is not suitable for situations where the text information in the image is unclear.

[0032] In some related technologies, image features can be identified and combined with text in the image for joint judgment. However, the strategy for combining image and text information is manually formulated, and often cannot find the optimal solution for combination. Furthermore, simple combination cannot find the inherent connection between text and image.

[0033] The image classification methods, apparatuses, electronic devices, and computer-readable media provided in this disclosure are intended to solve at least one of the above-mentioned technical problems of the prior art.

[0034] The image classification method provided in this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0035] Figure 1A flowchart illustrating an image classification method provided in an embodiment of this disclosure is shown, such as... Figure 1 As shown, the method mainly includes steps S110, S120, S130, and S140.

[0036] In step S110, the image to be classified is input into a pre-trained target detection network to obtain at least one candidate region in the image to be classified.

[0037] In step S120, the candidate region image corresponding to the candidate region is input into a pre-trained image feature extraction network to obtain the image features corresponding to the candidate region;

[0038] In step S130, the image to be classified is input into a pre-trained text recognition network to obtain the text features corresponding to the text in the image to be classified.

[0039] In step S140, image features and text features are concatenated to obtain concatenated features, and the concatenated features are input into a pre-trained classification network to obtain the classification result corresponding to the image to be classified.

[0040] For example, in step S110, the image to be classified can be a cartoon image, and the image to be classified can contain text information.

[0041] In some possible implementations, the input to the object detection network is an image, and the output of the object detection network is a foreground region (i.e., an output candidate region) in the image that is not the background and may contain objects (such as symbols, people, vehicles, etc.), as well as the category of the object in the output candidate region and the corresponding confidence score.

[0042] The number of candidate regions output by an object detection network can be multiple.

[0043] Object detection networks can be one-stage networks, such as YOLO networks and SSD networks; pre-trained object detection networks can also be two-stage networks, such as Faster-RCNN networks.

[0044] In some possible implementations, candidate regions can be randomly determined from multiple output candidate regions.

[0045] In some possible implementations, candidate regions can be determined from the output candidate regions based on the confidence level corresponding to the output candidate regions.

[0046] In some possible implementations, the training data for the object detection network can include various types of images, such as images taken by a camera, comics, oil paintings, and other artificially generated images.

[0047] In some possible implementations, the training data for the object detection network can include an OCR dataset. That is, the object detection network can determine whether there is text in the image to be classified, and if there is text in the image to be classified, determine the region where the text is located.

[0048] In some possible implementations, in step S120, a matting operation is performed on the image to be classified, and the region images of the candidate regions in the image to be classified are used as candidate region images and input into the image feature extraction network to obtain the image features corresponding to each candidate region image.

[0049] In some possible implementations, the image feature extraction network can be the feature extraction part of a deep learning-based image classification network, such as the feature extraction part of a ResNet network.

[0050] In some possible implementations, the image feature extraction network can be the ImageEncoder of the CLIP (Contrastive Language-Image Pre-training) network.

[0051] The CLIP network consists of two models: a Text Encoder and an Image Encoder. The Text Encoder is used to extract features from text and can use the Text Transformer model commonly used in NLP (Natural Language Processing). The Image Encoder is used to extract features from images and can use the commonly used CNN (Convolutional Neural Network) model or the Vision Transformer.

[0052] The CLIP network was trained on a large dataset (400 million images). It has good image semantic information understanding ability, extracts image features accurately, and has good classification performance.

[0053] In some possible implementations, in step S130, the text recognition network can be any network capable of text recognition, such as the paddleOCR network.

[0054] In some possible implementations, the input to the character recognition network is an image, and the output of the character recognition network is the text in the image, the specific location of the text, and the relationship between the text and other text.

[0055] In some possible implementations, the image to be classified is input into an image recognition network to obtain the text input to the image recognition network. After the image is obtained through the image recognition network, the text is encoded to obtain the text features corresponding to the text.

[0056] In some possible implementations, in step S140, the image features and text features can be spliced ​​together by adding the image features and text features.

[0057] In some possible implementations, image features and text features can be concatenated in a certain order to obtain concatenated features.

[0058] In some possible implementations, image features and text features can be concatenated in the order of text features and image features. For example, image features can be fixed as the last few dimensions of the concatenated features, and text features can be used as the first few dimensions of the concatenated features.

[0059] In some possible implementations, the classification network can be a pre-trained model BERT (Bidirectional Encoder Representation from Transformers), which has certain relational analysis capabilities, can better understand spliced ​​features, and obtain better classification results.

[0060] In the image classification method provided in this embodiment, candidate regions in the image are obtained based on the object detection network. The classification result of the image is determined by splicing the image features of the candidate regions and the text features of the text in the image. This realizes the combination of image visual content and text content, helps the classification network to deepen its understanding of image semantics, and improves the accuracy of image classification.

[0061] The image classification method provided in the embodiments of this disclosure will be described in detail below.

[0062] As mentioned above, in some possible implementations, image features and text features are concatenated in a certain order to obtain concatenated features.

[0063] In some possible implementations, image features and text features can be concatenated in the order of image features and text features.

[0064] In other words, all image features are first stitched together, and then text features are stitched together after all image features have been stitched together.

[0065] In some specific implementations, each image feature and each text feature is a dimension of the spliced ​​feature, and the dimension number corresponding to each image feature is less than the dimension number corresponding to each text feature.

[0066] In some possible implementations, to more easily combine image features and text features, it is necessary to transform the image features and text features into features of the same dimension.

[0067] In some possible implementations, zero-padding can be used to transform image features and text features into features of the same dimension.

[0068] Since the image features of each candidate region are obtained through the same image feature extraction network, the dimensionality of the image features of each candidate region is consistent. Similarly, the dimensionality of each text feature is also consistent.

[0069] Therefore, if the dimensions of image features and text features are inconsistent, it is sufficient to perform the same zero-padding operation on all image features or on all text features.

[0070] In some possible implementations, if the dimension of the image feature is smaller than the dimension of the text feature, the image feature is padded with zeros to make the dimension of the image feature the same as the dimension of the text feature; if the dimension of the image feature is larger than the dimension of the text feature, the text feature is padded with zeros to make the dimension of the text feature the same as the dimension of the image feature.

[0071] Compared to other methods, zero padding is easy for computers to implement and requires less computing resources. Since zeros do not provide information, zero padding has little impact on image features, and consequently has little impact on the spliced ​​features obtained by splicing image features, as well as on the classification results of the classification network.

[0072] In some possible implementations, in order to make the acquired candidate regions contain more image information and thus help the image feature extraction network to obtain more and more accurate image features, a preset number of candidate regions can be determined based on the confidence level of the candidate regions.

[0073] Figure 2 A flowchart illustrating the steps of determining a preset number of candidate regions based on their corresponding confidence levels is shown. Figure 2 As shown, determining a preset number of candidate regions based on the confidence level corresponding to the candidate regions may include steps S210 and S220.

[0074] In step S210, the image to be classified is input into a pre-trained target detection network to obtain multiple output candidate regions and the confidence scores corresponding to the output candidate regions.

[0075] In step S220, a preset number of output candidate regions are selected from all output candidate regions based on the confidence level corresponding to the output candidate regions to determine them as candidate regions.

[0076] In some possible implementations, in step S210, the object detection network can be a Faster-RCNN network.

[0077] In some possible implementations, a partially cleaned OCR dataset is added to the training data of the object detection network to enable the network to detect text regions.

[0078] In some possible implementations, the confidence level corresponding to the output candidate region is the confidence level of the objects existing in the output candidate region as objects that affect the classification result (such as identifiers that can belong to a certain category, tools that belong to a certain category, etc.).

[0079] In some possible implementations, after obtaining the output candidate regions, the output candidate regions are filtered according to their size, and smaller output candidate regions are deleted to avoid the difficulty in extracting image features from candidate regions with too small a size, which would affect the classification results.

[0080] In some possible implementations, in step S220, the N output candidate regions with the highest confidence are determined from the output candidate regions in descending order of confidence as the final candidate regions.

[0081] Where N is the preset number of candidate regions.

[0082] Figure 3 This diagram illustrates the process of stitching image features and text features together in the order of image features and text features, given a preset number of candidate regions. Figure 3 As shown, when the number of candidate regions is preset, the process of splicing image features and text features in the order of image features and text features may include steps S310 and S320.

[0083] In step S310, when the number of candidate regions is a preset number, the first N dimensions of the spliced ​​features are determined as the image features corresponding to the candidate regions.

[0084] In step S320, the text features are concatenated according to the confidence level corresponding to the text features.

[0085] Where N is the preset number of candidate regions, and the confidence level corresponding to the text feature is the confidence level of the text corresponding to the text feature output by the text recognition network.

[0086] In some possible implementations, in step S310, the image features corresponding to the candidate regions can be stitched together according to the confidence level corresponding to the image features.

[0087] For example, the image feature with the highest confidence level is concatenated with the image feature with the second highest confidence level, and then the concatenated feature is concatenated with the image feature with the second highest confidence level, and so on, until all image features are concatenated.

[0088] In some possible implementations, in step S320, if the sum of the number of text features and the number of image features is less than the preset dimension of the splicing features, the splicing features are padded with zeros so that the dimension of the splicing features is the preset dimension.

[0089] In other words, if the number of text features obtained is insufficient to meet the splicing requirements, zero-padding is used to obtain splicing features of a preset dimension.

[0090] In some possible implementations, if the sum of the number of text features and the number of image features is greater than the preset dimension of the splicing features, the text features are discarded according to the confidence level of the text features, so that the dimension of the splicing features is the preset dimension.

[0091] In other words, if the number of acquired text features is insufficient to meet the splicing requirements, zero-padding is used to obtain splicing features of a preset dimension. If the number of acquired text features exceeds the splicing requirements, redundant or low-confidence text features are discarded according to their corresponding confidence levels, and splicing features of a preset dimension are obtained.

[0092] Determining the first N dimensions of the spliced ​​features as image features allows the classification network to better learn image-text consistency features. By padding with zeros or deleting text features, the dimensions of the spliced ​​features can be fixed, thereby enabling the spliced ​​features to adapt to various classification networks and facilitating the application of the image classification method provided in this embodiment.

[0093] The image classification method provided in this disclosure will be described in detail below with reference to a specific embodiment.

[0094] Figure 4 A specific embodiment of the image classification method provided in this disclosure is shown. For example... Figure 4 As shown in the embodiments of this disclosure, the image classification method can be used for the detection of harmful comics. Specifically, the image classification method provided in these embodiments may include:

[0095] Faster-RCNN was used as the object detection model, and the Visual Genome dataset and a partially cleaned OCR dataset were used as the training set to train the Faster-RCNN network.

[0096] In the actual inference process, the image to be classified is processed by the Faster-RCNN network to obtain a large number of output candidate regions. Only the top 64 output candidate regions with the highest quality and confidence are selected as candidate regions for subsequent processes.

[0097] Based on the 64 candidate regions obtained, the image to be classified is cut out to obtain the candidate region images corresponding to the candidate regions. The VIT branch (image encoder) of the CLIP model is used to extract features from the candidate region images to obtain image features. Each image feature has a dimension of 512. Therefore, 64 features with a dimension of 512 are obtained in the end. In order to facilitate the concatenation of image features and text features, each image feature is padded with zeros at the end to make it 768 dimensions.

[0098] Text extraction from images to be classified using PaddleOCR (e.g.) Figure 4 Four sets of words can be extracted, and the four sets of words are encoded to obtain four 768-dimensional text features.

[0099] Sixty-four image features are concatenated with four text features. To achieve better image-text consistency, the first 64 dimensions of the concatenated features are fixed as image features. Then, text information is concatenated according to text confidence. By padding or discarding zeros (i.e., if there are too many text features, the redundant low-confidence text is discarded, and if there are not enough text features, zeros are padded), the concatenated features are transformed into 256 dimensions, i.e., features of size 256*768.

[0100] The 256*768 feature map is fed into a classification network to determine whether the comic as a whole contains harmful content. The classification network is a BERT pre-trained model.

[0101] Based on and Figure 1 The method shown follows the same principle. Figure 5 A schematic diagram of the structure of an image classification apparatus provided in an embodiment of this disclosure is shown, such as... Figure 5 As shown, the image classification device 50 may include:

[0102] Region module 510 is used to input the image to be classified into a pre-trained object detection network to obtain at least one candidate region in the image to be classified.

[0103] Image module 520 is used to input the candidate region image corresponding to the candidate region into a pre-trained image feature extraction network to obtain the image features corresponding to the candidate region;

[0104] The text module 530 is used to input the image to be classified into a pre-trained text recognition network to obtain the text features corresponding to the text in the image to be classified.

[0105] The classification module 540 is used to concatenate image features and text features to obtain concatenated features, and input the concatenated features into a pre-trained classification network to obtain the classification result corresponding to the image to be classified.

[0106] In the image classification apparatus of this disclosure, candidate regions in an image are obtained based on an object detection network. The classification result of the image is determined by splicing the image features of the candidate regions and the text features of the text in the image. This achieves the combination of visual and textual content of the image, helping the classification network to deepen its understanding of the image semantics and improving the accuracy of image classification.

[0107] It is understood that the above-described modules of the image classification device in the embodiments of this disclosure have the ability to implement... Figure 1 The embodiments shown illustrate the functions of corresponding steps in the image classification method. These functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above. These modules can be software and / or hardware, and each module can be implemented individually or integrated from multiple modules. For a detailed description of the functions of each module in the image classification device, please refer to [link to relevant documentation]. Figure 1 The corresponding descriptions of the image classification methods in the embodiments shown are not repeated here.

[0108] In some possible implementations, the classification module 540 is used to: concatenate image features and text features in the order of image features and text features to obtain concatenated features.

[0109] In some possible implementations, the image classification device 50 also includes a zero-padding module, which pads the image features with zeros when the dimension of the image features is smaller than the dimension of the text features, so that the dimension of the image features is the same as the dimension of the text features; and pads the text features with zeros when the dimension of the image features is larger than the dimension of the text features, so that the dimension of the text features is the same as the dimension of the image features.

[0110] In some possible implementations, the image module 520 is used to: input the image to be classified into a pre-trained object detection network, obtain multiple output candidate regions and the confidence scores corresponding to the output candidate regions from the output candidate regions; and select a preset number of output candidate regions from all the output candidate regions based on the confidence scores corresponding to the output candidate regions to determine them as candidate regions.

[0111] In some possible implementations, the classification module 540 is used to: determine the first N dimensions of the concatenated features as the image features corresponding to the candidate regions; N is a preset number of candidate regions; concatenate the text features according to the confidence level corresponding to the text features; if the sum of the number of text features and the number of image features is less than the preset dimension of the concatenated features, pad the concatenated features with zeros so that the dimension of the concatenated features is the preset dimension; if the sum of the number of text features and the number of image features is greater than the preset dimension of the concatenated features, discard the text features according to the confidence level corresponding to the text features so that the dimension of the concatenated features is the preset dimension; wherein, the confidence level corresponding to the text features is the confidence level of the text corresponding to the text features output by the text recognition network.

[0112] In some possible implementations, the text module 530 is used to: input the image to be classified into a pre-trained character recognition network, obtain the text output by the character recognition network, encode the text, and obtain the text features corresponding to the text.

[0113] In some possible implementations, the classification network is the pre-trained model BERT.

[0114] In some possible implementations, the image feature extraction network is an image encoder based on CLIP, a pre-trained model of contrastive text-image pairs.

[0115] In some possible implementations, the object detection network is a Faster R-CNN (Fast Region Convolutional Neural Network); the training set for the object detection network includes a text recognition dataset.

[0116] In some possible implementations, the image to be classified is a cartoon image.

[0117] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0118] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0119] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform an image classification method as provided in the embodiments of this disclosure.

[0120] Compared with existing technologies, this electronic device obtains candidate regions in images based on object detection networks. It determines the image classification result by splicing the image features of the candidate regions and the text features of the text in the image. This realizes the combination of image visual content and text content, helps the classification network to deepen its understanding of image semantics, and improves the accuracy of image classification.

[0121] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform an image classification method as provided in the embodiments of this disclosure.

[0122] Compared with existing technologies, this readable storage medium obtains candidate regions in images based on object detection networks. It determines the image classification result by splicing the image features of the candidate regions and the text features of the text in the image. This realizes the combination of image visual content and text content, helps the classification network to deepen its understanding of image semantics, and improves the accuracy of image classification.

[0123] The computer program product includes a computer program that, when executed by a processor, implements the image classification method as provided in the embodiments of this disclosure.

[0124] Compared with existing technologies, this computer program product obtains candidate regions in images based on object detection networks. It determines the image classification result by splicing the image features of the candidate regions and the text features of the text in the image. This achieves the combination of image visual content and text content, helps the classification network to deepen its understanding of image semantics, and improves the accuracy of image classification.

[0125] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0126] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0127] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0128] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as image classification methods. For example, in some embodiments, the image classification method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the image classification method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the image classification method by any other suitable means (e.g., by means of firmware).

[0129] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0131] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0133] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0134] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0135] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0136] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image classification method, comprising: The image to be classified is input into a pre-trained object detection network to obtain at least one candidate region in the image to be classified. The image to be classified is a cartoon image; the target detection network is a Faster R-CNN (Fast Region Convolutional Neural Network). The candidate region image corresponding to the candidate region is input into a pre-trained image feature extraction network to obtain the image features corresponding to the candidate region; the image feature extraction network is an image encoder based on the CLIP pre-trained model of contrastive text-image pairs; The image to be classified is input into a pre-trained text recognition network to obtain the text features corresponding to the text in the image to be classified. The image features and text features are concatenated to obtain concatenated features, and the concatenated features are input into a pre-trained classification network to obtain the classification result corresponding to the image to be classified. The classification network is the pre-trained model BERT; The step of concatenating the image features and the text features to obtain the concatenated features includes: The first N dimensions of the spliced ​​features are determined as the image features corresponding to the candidate regions, where N is a preset number of candidate regions; the number of image features in the spliced ​​features is greater than the number of text features; The text features are concatenated according to the confidence level corresponding to the text features; If the sum of the number of text features and the number of image features is less than the preset dimension of the splicing feature, the splicing feature is padded with zeros so that the dimension of the splicing feature is the preset dimension. If the sum of the number of text features and the number of image features is greater than the preset dimension of the splicing feature, the text features are discarded according to the confidence level corresponding to the text features, so that the dimension of the splicing feature is the preset dimension. Wherein, the confidence level corresponding to the text feature is the confidence level of the text corresponding to the text feature output by the text recognition network.

2. The method according to claim 1, wherein, The step of concatenating the image features and the text features to obtain the concatenated features includes: The image features and text features are concatenated in the order of image features and text features to obtain the concatenated features.

3. The method according to claim 2, wherein, Before concatenating the image features and the text features to obtain the concatenated features, the method further includes: If the dimension of the image feature is smaller than the dimension of the text feature, the image feature is padded with zeros to make the dimension of the image feature the same as the dimension of the text feature. If the dimension of the image feature is greater than the dimension of the text feature, the text feature is padded with zeros so that the dimension of the text feature is the same as the dimension of the image feature.

4. The method according to claim 2, wherein, The step of inputting the image to be classified into a pre-trained object detection network to obtain at least one candidate region in the image to be classified includes: The image to be classified is input into a pre-trained target detection network to obtain multiple output candidate regions and the confidence scores corresponding to the output candidate regions. Based on the confidence level corresponding to the output candidate region, a preset number of output candidate regions are selected from all the output candidate regions to be determined as candidate regions.

5. The method according to claim 1, wherein, The step of inputting the image to be classified into a pre-trained text recognition network to obtain the text features corresponding to the text in the image to be classified includes: The image to be classified is input into a pre-trained character recognition network to obtain the text output by the character recognition network; The text is encoded to obtain the text features corresponding to the text.

6. The method according to claim 1, wherein, The training set for the object detection network includes a text recognition dataset.

7. An image classification apparatus, comprising: The region module is used to input the image to be classified into a pre-trained target detection network to obtain at least one candidate region in the image to be classified. The image to be classified is a cartoon image; the target detection network is a Faster R-CNN (Fast Region Convolutional Neural Network). The image module is used to input the candidate region image corresponding to the candidate region into a pre-trained image feature extraction network to obtain the image features corresponding to the candidate region; the image feature extraction network is an image encoder based on the CLIP pre-trained model of contrastive text-image pairs; The text module is used to input the image to be classified into a pre-trained text recognition network to obtain the text features corresponding to the text in the image to be classified. The classification module is used to concatenate the image features and the text features to obtain concatenated features, and input the concatenated features into a pre-trained classification network to obtain the classification result corresponding to the image to be classified; the classification network is a pre-trained model BERT; The classification module is specifically used to determine the first N dimensions of the spliced ​​features as the image features corresponding to the candidate regions, where N is a preset number of candidate regions; the number of image features in the spliced ​​features is greater than the number of text features; The text features are concatenated according to the confidence level corresponding to the text features; If the sum of the number of text features and the number of image features is less than the preset dimension of the splicing feature, the splicing feature is padded with zeros so that the dimension of the splicing feature is the preset dimension. If the sum of the number of text features and the number of image features is greater than the preset dimension of the splicing feature, the text features are discarded according to the confidence level corresponding to the text features, so that the dimension of the splicing feature is the preset dimension. Wherein, the confidence level corresponding to the text feature is the confidence level of the text corresponding to the text feature output by the text recognition network.

8. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images