A method, apparatus, device, and storage medium for image classification

Abstract

The embodiment of the present invention discloses a method, apparatus, device and storage medium for image classification. The method comprises: determining the global image category of the image to be classified according to the global information of the image to be classified; determining the image theme category of the image to be classified according to the local information of the image to be classified; and determining the image classification result of the image to be classified according to the global image category and the image theme category. In the embodiment of the present invention, the global image category and the image theme category of the image are obtained, and the classification result of the image is determined by comprehensively distinguishing the global image category and the image theme category, which solves the problem of inaccurate image classification caused by only determining the global image category through visual features in the prior art, takes into account the influence of local features on image classification, and improves the accuracy of image classification.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, and in particular to a method, apparatus, device and storage medium for image classification. Background Art

[0002] Image scene classification is a processing scenario in computer image classification. With the widespread application of deep learning technology, computational image classification is increasingly processed using deep learning methods.

[0003] At present, the image classification system based on deep learning generally feeds the visual features of the image into the deep neural network for classification mapping calculation, which mainly classifies the image content directly based on the visual features, using the global information of the image content, but lacks the full use of the visual features of the focus area. As a result, the classification results are not accurate enough for some images with complex scene contents. Summary of the invention

[0004] Embodiments of the present invention provide a method, apparatus, device and storage medium for image classification to improve the accuracy of image classification.

[0005] In a first aspect, an embodiment of the present invention provides a method for classifying an image, the method comprising:

[0006] Determining the global image category of the image to be classified according to the global information of the image to be classified;

[0007] Determining the picture subject category of the picture to be classified according to the local information of the picture to be classified;

[0008] According to the global picture category and the picture subject category, a picture classification result of the picture to be classified is determined.

[0009] In a second aspect, an embodiment of the present invention further provides a picture classification device, the device comprising:

[0010] A global category determination module, used to determine the global picture category of the picture to be classified according to the global information of the picture to be classified;

[0011] A theme category determination module, used to determine the picture theme category of the picture to be classified according to the local information of the picture to be classified;

[0012] The classification result determination module is used to determine the picture classification result of the picture to be classified according to the global picture category and the picture subject category.

[0013] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image classification method as described in any embodiment of the present invention is implemented.

[0014] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the image classification method as described in any embodiment of the present invention.

[0015] The embodiment of the present invention obtains the global image category and the image theme category of the image to be classified, and makes a comprehensive judgment on the global image category and the image theme category to determine the classification result of the image to be classified. This solves the problem of inaccurate image classification caused by only obtaining the global image category through visual features in the prior art, and considers the influence of the local features of the image on the image classification, thereby improving the accuracy of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of a method for classifying pictures in Embodiment 1 of the present invention;

[0017] Figure 2 is a schematic diagram of a picture to be classified in Embodiment 1 of the present invention;

[0018] Figure 3 is a flowchart of a method for classifying pictures in Embodiment 2 of the present invention;

[0019] Figure 4 is a structural block diagram of a picture classification device in Embodiment 3 of the present invention;

[0020] Figure 5 It is a structural diagram of a computer device in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0022] Embodiment 1

[0023] Figure 1 This is a flow chart of a method for classifying pictures provided in the first embodiment of the present invention. This embodiment is applicable to the case of classifying pictures. The method can be executed by a picture classification device. Figure 1 As shown, the method specifically comprises the following steps:

[0024] Step 110: Determine the global picture category of the picture to be classified according to the global information of the picture to be classified.

[0025] The image to be classified is obtained, and the image to be classified is input into a pre-trained image classification neural network model to output a global image category. The image classification neural network model can use multiple convolutional layers to extract visual features, and obtain the global image category of the image to be classified based on the overall visual features of the image to be classified.

[0026] The image classification neural network model is an image classification neural network model based on deep learning. For example, a neural network based on a residual structure or a Google Inception network with a multi-scale feature fusion mechanism can be used as an image classification neural network model. Global information refers to the overall visual features of the image to be classified, and the global image category refers to the image category obtained by referring to the global information of the image to be classified. Figure 2 Schematic diagram of pictures to be classified. Figure 2 A seagull is flying over the water. Figure 2 The global information and image classification neural network model can be obtained Figure 2 The global image category is water scene.

[0027] Step 120: Determine the picture subject category of the picture to be classified based on the local information of the picture to be classified.

[0028] The local information of the image to be classified refers to the information on different regions of the image to be classified, and the image theme category refers to the image theme displayed by the image to be classified obtained based on the local information, and the image theme category is obtained according to the image theme. The local features can be extracted through a pre-trained deep neural network model to output the image theme category.

[0029] In this embodiment, optionally, the picture theme category of the picture to be classified is determined based on the local information of the picture to be classified, including: determining the theme semantic text of the picture to be classified based on the local information of the picture to be classified; and obtaining the picture theme category of the picture to be classified based on the theme semantic text.

[0030] Specifically, the picture subject semantic model is a pre-trained deep neural network model, the picture to be classified is input into the pre-trained picture subject semantic model, the picture subject semantic model extracts the local features of the picture to be classified, and outputs the subject semantic text of the picture to be classified, the subject semantic text refers to the description of the content of the picture to be classified in the form of text information. After determining the subject semantic text, the subject semantic text is input into the pre-trained text classification model, and the text classification model outputs the picture subject category of the picture to be classified. A text classification model based on deep learning can be used, and the text classification model can classify text information. For example, the subject semantic text of the picture to be classified is "two kittens sleeping", then through the text classification model, it can be known that the subject semantic text belongs to the content of the animal class, therefore, the picture subject category of the picture to be classified is the animal class. The beneficial effect of such a setting is that the subject semantic text is first obtained, and then the subject semantic text is classified, which avoids the problem of information omission caused by directly classifying the picture to be classified, and improves the correctness of picture classification.

[0031] In this embodiment, optionally, the theme semantic text of the picture to be classified is determined based on the local information of the picture to be classified, including: dividing the picture to be classified into at least two local area pictures; inputting the theme semantic text of the current local area picture and the previous local area picture into a local feature extraction module in the picture theme semantic model to determine the embedding vector of the superimposed local area; wherein the superimposed local area includes the current local area picture and the local area pictures before the current local area picture; inputting the embedding vector into the semantic information description module in the picture theme semantic model to determine the theme semantic text of the superimposed local area.

[0032] Specifically, the image theme semantic model can be divided into a local feature extraction module and a semantic information description module. The local feature extraction module is used to extract features from the local information of the image to be classified and generate an embedding vector for the local area. The semantic information description module is used to predict the text to which the embedded features of the local area are most likely to be mapped based on the embedding vector, and obtain the theme semantic text of the local area.

[0033] The image to be classified is divided into at least two local area images. After obtaining the theme semantic text of the previous local area image, the theme semantic text of the current local area image and the previous local area image are input into the local feature extraction module to obtain the embedding vector of the superimposed local area. The superimposed local area may include the current local area image and each local area image before the current local area image, that is, the embedding vector of the superimposed local area refers to the embedding vector of the current local area image and each local area image before the current local area image. If the previous local area image is not the first local area image, the embedding vector of the previous local area image is the embedding vector of the previous local area image and each local area image before the previous local area image. The embedding vector of the superimposed local area in each cycle is input into the semantic information description module to obtain the theme semantic text of the superimposed local area. If all local areas are cycled, the theme semantic text of the superimposed local area obtained is the theme semantic text of the image to be classified.

[0034] For example, the image to be classified can be divided into three local areas, namely the first area, the second area and the third area. The first area is input into the local feature extraction module to obtain the first embedding vector of the first area. The first embedding vector of the first area is then input into the semantic information description module to obtain the first theme semantic text of the first area. The first theme semantic text of the first area and the second area are input into the local feature extraction module to obtain the second embedding vector superimposed with the first area and the second area. The second embedding vector is input into the semantic information description module to obtain the second theme semantic text of the first area and the second area. The second theme semantic text and the third area are input into the local feature extraction module to obtain the third embedding vector of the image to be classified. The third embedding vector is input into the semantic information description module to obtain the theme semantic text of the image to be classified. The beneficial effect of such a setting is that, since each input includes the output of the previous time, it is equivalent to using the attention mechanism of deep learning, so that the corresponding semantic text description generated by each area of ​​the image to be classified takes into account the influence of the semantic text input from the adjacent area, and quickly obtains a relatively smooth semantic information description oriented to the content theme of the image to be classified, which improves the efficiency and accuracy of the theme semantic text generation. For example, through the image theme semantic model, it can be obtained Figure 2 The theme semantic text is "A seagull flying over the water", which not only obtains the waterscape characteristics but also the animal characteristics of the seagull.

[0035] Step 130: Determine the image classification result of the image to be classified according to the global image category and the image subject category.

[0036] The image classification neural network model can output the global image category of the image to be classified, and the possible probability of the global image category being the image category to be classified. The image topic semantic model can output the image topic category of the image to be classified, and the possible probability of the image topic category being the image category to be classified. According to the possible probability of the global image category and the possible probability of the image topic category, the image classification result of the image to be classified is selected from the global image category and the image topic category.

[0037] In this embodiment, optionally, the image classification result of the image to be classified is determined based on the global image category and the image theme category, including: obtaining a first probability of the global image category based on a picture classification neural network model that determines the global image category; obtaining a second probability of the image theme category based on a picture theme semantic model; and comparing the first probability and the second probability to determine the image classification result of the image to be classified.

[0038] Specifically, the first probability is the probability of the global image category being the category of the image to be classified, and the second probability is the probability of the image theme category being the category of the image to be classified. The first probability and the second probability are compared, and the category with a larger probability is used as the image classification result of the image to be classified. For example, according to the image classification neural network model, the global image category of the image to be classified is obtained as "waterscape category" with a probability of 70%, and according to the image theme semantic model, the image theme category of the image to be classified is obtained as "animal category" with a probability of 90%, then the image classification result of the image to be classified is "animal category". The beneficial effect of such a setting is that by comparing the first probability of the global image category and the second probability of the image theme category to determine the final image classification result, the global information and local information of the image can be comprehensively considered, the use of the visual features of the focus area can be realized, and the image classification accuracy can be improved.

[0039] In this embodiment, optionally, after obtaining the second probability of the picture theme category according to the picture theme semantic model, it also includes: determining a first probability weight result and a second probability weight result of the picture to be classified according to the first probability, the second probability and a preset probability weight; comparing the first probability weight result and the second probability weight result to determine the picture classification result of the picture to be classified.

[0040] Specifically, a probability weight of the first probability and a probability weight of the second probability are preset. After obtaining the first probability and the second probability, the first probability weight result and the second probability weight result are calculated according to the preset probability weight. The first probability weight result and the second probability weight result are compared, and the image category corresponding to the larger value is used as the image classification result of the image to be classified. For example, the first probability is 10, the second probability is 12, the probability weight of the first probability is 20, and the probability weight of the second probability is 10. Although the first probability is less than the second probability, the first probability weight result is greater than the second probability weight result. Therefore, the image classification result of the image to be classified is the global image category. According to the importance of the global image category and the image subject category, the image classification result is determined to improve the accuracy of image classification.

[0041] In this embodiment, optionally, the first probability and the second probability are compared to determine the image classification result of the image to be classified, including: if the difference between the first probability and the second probability exceeds a preset difference, then taking the corresponding category with the larger probability value between the first probability and the second probability as the image classification result of the image to be classified; if the difference between the first probability and the second probability is within the preset difference, then taking the global image category and the image theme category as the image classification result of the image to be classified.

[0042] Specifically, the first probability and the second probability are compared. If the difference is greater than the preset difference, the image classification result corresponding to the large probability value can be directly used as the image classification result of the image to be classified; if the difference is less than or equal to the preset difference, it means that both the global image category and the image subject category can accurately represent the category of the image to be classified. The global image category and the image subject category are both used as the image classification results of the image to be classified, thereby improving the accuracy of image classification.

[0043] The technical solution of this embodiment obtains the global image category and the image theme category of the image to be classified, and makes a comprehensive judgment on the global image category and the image theme category to determine the classification result of the image to be classified. This solves the problem of inaccurate image classification caused by determining the classification result by only obtaining the global image category through visual features in the prior art, considers the influence of the local features of the image on the image classification, and improves the accuracy of image classification.

[0044] Embodiment 2

[0045] Figure 3 This is a flow chart of a method for image classification provided by the second embodiment of the present invention. This embodiment is further optimized based on the above embodiment. The method can be executed by an image classification device. Figure 3 As shown, the method specifically comprises the following steps:

[0046] Step 310: Determine the global picture category of the picture to be classified according to the global information of the picture to be classified.

[0047] Step 320: Determine the subject semantic text of the image to be classified based on the local information of the image to be classified.

[0048] Step 330: Obtain the picture theme category of the picture to be classified according to the theme semantic text.

[0049] After the subject semantic text is obtained, the picture subject category to which the subject semantic text belongs is determined according to the content of the subject semantic text.

[0050] In this embodiment, optionally, based on the theme semantic text, a picture theme category of the picture to be classified is obtained, including: inputting the theme semantic text into a pre-trained language pre-training model to obtain a standard theme semantic text; inputting the standard theme semantic text into a pre-trained text classification model to obtain a picture theme category of the picture to be classified.

[0051] Specifically, a language pre-training model is pre-trained, and the language pre-training model is used to modify incoherent or non-standard sentences into fluent sentences in a standard format to make the meaning of the sentences clearer. The topic semantic text output by the picture topic semantic model is input into the language pre-training model, and the language pre-training model outputs the standard topic semantic text. For example, the topic semantic text is "Seagulls fly on the water surface". After the topic semantic text is input into the language pre-training model, the standard topic semantic text can be obtained as "Seagulls fly on the water surface". The standard topic semantic text is input into the text classification model, and according to the natural language processing technology, the picture topic category to which the standard topic semantic text belongs is obtained, thereby obtaining the picture topic category of the picture to be classified. The beneficial effect of such a setting is that by converting the topic semantic text into the standard topic semantic text, the text classification model is avoided from incorrectly recognizing the topic semantic text, affecting the determination result of the picture topic category, and improving the confirmation accuracy of the picture topic category.

[0052] Step 340: Determine the image classification result of the image to be classified according to the global image category and the image subject category.

[0053] The embodiment of the present invention determines the image theme category by using a language pre-training model and a text classification model. After obtaining the global image category and the image theme category of the image to be classified, the global image category and the image theme category are comprehensively judged to determine the classification result of the image to be classified. The problem of inaccurate image classification caused by determining the classification result by obtaining the global image category only through visual features in the prior art is solved, and the influence of the local features of the image on the image classification is considered to improve the accuracy of image classification.

[0054] Embodiment 3

[0055] Figure 4This is a structural block diagram of a picture classification device provided in the third embodiment of the present invention, which can execute the picture classification method provided in any embodiment of the present invention and has the corresponding functional modules and beneficial effects of the execution method. Figure 4 As shown, the device specifically includes:

[0056] A global category determination module 401 is used to determine the global picture category of the picture to be classified according to the global information of the picture to be classified;

[0057] The theme category determination module 402 is used to determine the picture theme category of the picture to be classified according to the local information of the picture to be classified;

[0058] The classification result determination module 403 is used to determine the picture classification result of the picture to be classified according to the global picture category and the picture subject category.

[0059] Optionally, the subject category determination module 402 includes:

[0060] A topic semantic determination unit, used to determine the topic semantic text of the image to be classified according to the local information of the image to be classified;

[0061] The theme category generating unit is used to obtain the picture theme category of the picture to be classified according to the theme semantic text.

[0062] Optional, topic semantic determination unit, including:

[0063] A local area division unit, used to divide the image to be classified into at least two local area images;

[0064] An embedding vector determination unit, used to input the topic semantic text of the current local area picture and the previous local area picture into the local feature extraction module in the picture topic semantic model, and determine the embedding vector of the superimposed local area; wherein the superimposed local area includes the current local area picture and each local area picture before the current local area picture;

[0065] The semantic text determination unit is used to input the embedding vector into the semantic information description module in the picture theme semantic model to determine the theme semantic text of the superimposed local area.

[0066] Optional, subject category generation unit, including:

[0067] A standard text determination unit, used for inputting the subject semantic text into a pre-trained language pre-training model to obtain a standard subject semantic text;

[0068] The image theme category obtaining unit is used to input the standard theme semantic text into the pre-trained text classification model to obtain the image theme category of the image to be classified.

[0069] Optionally, the classification result determination module 403 includes:

[0070] A first probability determination unit, configured to obtain a first probability of a global picture category according to a picture classification neural network model for determining a global picture category;

[0071] A second probability determination unit, used to obtain a second probability of the picture theme category according to the picture theme semantic model;

[0072] The probability comparison unit is used to compare the first probability and the second probability to determine the image classification result of the image to be classified.

[0073] Optionally, the image classification neural network model is an image classification neural network model constructed based on deep learning.

[0074] Optionally, the device further comprises:

[0075] A weight determination module, used to determine a first probability weight result and a second probability weight result of the image to be classified according to the first probability, the second probability and a preset probability weight;

[0076] The weight comparison module is used to compare the first probability weight result and the second probability weight result to determine the image classification result of the image to be classified.

[0077] Optional, probability comparison unit, specifically used for:

[0078] If the difference between the first probability and the second probability exceeds a preset difference, taking the category corresponding to the larger probability value between the first probability and the second probability as the picture classification result of the picture to be classified;

[0079] If the difference between the first probability and the second probability is within a preset difference, the global image category and the image subject category are used as the image classification result of the image to be classified.

[0080] The embodiment of the present invention obtains the global image category and the image theme category of the image to be classified, and makes a comprehensive judgment on the global image category and the image theme category to determine the classification result of the image to be classified. This solves the problem of inaccurate image classification caused by only obtaining the global image category through visual features in the prior art, and considers the influence of the local features of the image on the image classification, thereby improving the accuracy of image classification.

[0081] Embodiment 4

[0082] Figure 5 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. Figure 5A block diagram of an exemplary computer device 500 suitable for use in implementing embodiments of the present invention is shown. Figure 5 The computer device 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0083] like Figure 5 As shown, the computer device 500 is in the form of a general computing device. The components of the computer device 500 may include, but are not limited to: one or more processors or processing units 501, a system memory 502, and a bus 503 connecting different system components (including the system memory 502 and the processing unit 501).

[0084] Bus 503 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0085] Computer device 500 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 500, including volatile and non-volatile media, removable and non-removable media.

[0086] System memory 502 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 504 and / or cache memory 505. Computer device 500 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 506 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 503 via one or more data medium interfaces. The memory 502 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0087] A program / utility 508 having a set (at least one) of program modules 507 may be stored, for example, in the memory 502, such program modules 507 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 507 generally perform the functions and / or methods of the embodiments described herein.

[0088] The computer device 500 may also communicate with one or more external devices 509 (e.g., keyboard, pointing device, display 510, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 500, and / or communicate with any device that enables the computer device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via input / output (I / O) interface 511. Furthermore, the computer device 500 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) via network adapter 512. As shown, the network adapter 512 communicates with other modules of the computer device 500 via bus 503. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0089] The processing unit 501 executes various functional applications and data processing by running the program stored in the system memory 502, for example, implementing a picture classification method provided by an embodiment of the present invention, including:

[0090] Determine the global image category of the image to be classified according to the global information of the image to be classified;

[0091] Determine the image subject category of the image to be classified according to the local information of the image to be classified;

[0092] According to the global image category and the image subject category, the image classification result of the image to be classified is determined.

[0093] Embodiment 5

[0094] Embodiment 5 of the present invention further provides a storage medium containing computer executable instructions, on which a computer program is stored. When the program is executed by a processor, a method for classifying an image provided in the embodiment of the present invention is implemented, including:

[0095] Determine the global image category of the image to be classified according to the global information of the image to be classified;

[0096] Determine the image subject category of the image to be classified according to the local information of the image to be classified;

[0097] According to the global image category and the image subject category, the image classification result of the image to be classified is determined.

[0098] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0099] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0100] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0101] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0102] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for image classification, characterized in that: include: Determining the global picture category of the picture to be classified according to the global information of the picture to be classified; wherein the global information refers to the overall visual features of the picture to be classified, and the global picture category is the picture category obtained based on the global information of the picture to be classified; Determining the picture subject category of the picture to be classified according to the local information of the picture to be classified; Determining a picture classification result of the picture to be classified according to the global picture category and the picture subject category; Wherein, determining the global picture category of the picture to be classified according to the global information of the picture to be classified includes: Obtain a picture to be classified, input the picture to be classified into a pre-trained picture classification neural network model, and output a global picture category of the picture to be classified; wherein the global picture category of the picture to be classified is obtained by extracting visual features of the picture to be classified by using a plurality of convolutional layers based on the overall visual features of the picture to be classified by the picture classification neural network model; Wherein, determining the picture subject category of the picture to be classified according to the local information of the picture to be classified includes: Determining the subject semantic text of the image to be classified according to the local information of the image to be classified; According to the subject semantic text, obtaining the picture subject category of the picture to be classified; The step of determining the subject semantic text of the image to be classified according to the local information of the image to be classified includes: Dividing the picture to be classified into at least two local area pictures; Inputting the topic semantic text of the current local area picture and the previous local area picture into the local feature extraction module in the picture topic semantic model to determine the embedding vector of the superimposed local area; wherein the superimposed local area includes the current local area picture and each local area picture before the current local area picture; Inputting the embedding vector into a semantic information description module in the picture theme semantic model to determine the theme semantic text of the superimposed local area; Wherein, obtaining the picture theme category of the picture to be classified according to the theme semantic text includes: Inputting the subject semantic text into a pre-trained language pre-training model to obtain a standard subject semantic text; Inputting the standard topic semantic text into a pre-trained text classification model to obtain the picture topic category of the picture to be classified; Wherein, determining the picture classification result of the picture to be classified according to the global picture category and the picture subject category includes: Obtaining a first probability of the global picture category according to a picture classification neural network model that determines the global picture category; wherein the picture classification neural network model is a picture classification neural network model constructed based on deep learning, and the first probability is a probability that the global picture category is a picture category to be classified; According to the picture theme semantic model, a second probability of the picture theme category is obtained; wherein the second probability is the probability that the picture theme category is the category of the picture to be classified; Determine a first probability weight result and a second probability weight result of the image to be classified according to the first probability, the second probability and a preset probability weight; Compare the first probability weight result and the second probability weight result to determine the image classification result of the image to be classified; wherein the classification result of the image to be classified is the image category corresponding to the larger probability weight value in the first probability weight result and the second probability weight result.

2. A picture classification device, characterized in that: include: A global category determination module, used to determine the global picture category of the picture to be classified according to the global information of the picture to be classified; wherein the global information refers to the overall visual features of the picture to be classified, and the global picture category is the picture category obtained based on the global information of the picture to be classified; A theme category determination module, used to determine the picture theme category of the picture to be classified according to the local information of the picture to be classified; A classification result determination module, used to determine the picture classification result of the picture to be classified according to the global picture category and the picture subject category; The global category determination module is specifically used to obtain a picture to be classified, input the picture to be classified into a pre-trained picture classification neural network model, and output the global picture category of the picture to be classified; wherein the global picture category of the picture to be classified is obtained by extracting visual features based on the overall visual features of the picture to be classified by the picture classification neural network model using multiple convolutional layers; Wherein, the subject category determination module includes: A topic semantics determination unit, used to determine the topic semantic text of the picture to be classified according to the local information of the picture to be classified; A theme category generating unit, used for obtaining the picture theme category of the picture to be classified according to the theme semantic text; Wherein, the subject semantic determination unit includes: A local area division unit, used to divide the picture to be classified into at least two local area pictures; An embedding vector determination unit, used to input the topic semantic text of the current local area picture and the previous local area picture into the local feature extraction module in the picture topic semantic model, and determine the embedding vector of the superimposed local area; wherein the superimposed local area includes the current local area picture and each local area picture before the current local area picture; A semantic text determination unit, used for inputting the embedding vector into a semantic information description module in the picture theme semantic model to determine the theme semantic text of the superimposed local area; Wherein, the subject category generating unit comprises: A standard text determination unit, used for inputting the subject semantic text into a pre-trained language pre-training model to obtain a standard subject semantic text; A picture theme category determination unit, used for inputting the standard theme semantic text into a pre-trained text classification model to obtain the picture theme category of the picture to be classified; Wherein, the classification result determination module includes: A first probability determination unit, configured to obtain a first probability of the global picture category according to a picture classification neural network model that determines the global picture category; wherein the picture classification neural network model is a picture classification neural network model constructed based on deep learning, and the first probability is a probability that the global picture category is a picture category to be classified; A second probability determination module, configured to obtain a second probability of the picture theme category according to the picture theme semantic model; wherein the second probability is a probability that the picture theme category is a picture category to be classified; The device also includes: A weight determination module, used to determine a first probability weight result and a second probability weight result of the image to be classified according to the first probability, the second probability and a preset probability weight; A weight comparison module is used to compare the first probability weight result and the second probability weight result to determine the image classification result of the image to be classified; wherein the classification result of the image to be classified is the image category corresponding to the larger probability weight value in the first probability weight result and the second probability weight result.

3. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image classification method as claimed in claim 1 is implemented.

4. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the image classification method as claimed in claim 1 when executed by a computer processor.

Images