Image-text content classification method and device, equipment and medium

Abstract

The embodiment of the invention provides an image-text content classification method and device, equipment and a medium. A specific embodiment of the method comprises the steps of obtaining a first text feature, determining first importance scores corresponding to a plurality of text feature sub-vectors in the first text feature, and changing the text feature sub-vectors with the first importance scores lower than a preset first threshold value into zero vectors to obtain a second text feature; obtaining a first image feature, determining second importance scores corresponding to a plurality of image feature sub-vectors in the first image feature, and changing the image feature sub-vectors with the second importance scores lower than a preset second threshold value into zero vectors to obtain a second image feature; and fusing the second text feature and the second image feature based on a cross attention mechanism to obtain a first image-text fusion feature, and classifying the first image and / or the first text according to the first image-text fusion feature. According to the method, the quality of the obtained image-text fusion feature can be improved, and the classification accuracy of the image-text content is further improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of machine learning technology, and more particularly to a method, device, equipment and medium for classifying graphic content. Background Art

[0002] The data of graphic content is a kind of multimodal data composed of image data and text data. At present, many application scenarios often have the task of classification based on graphic data. For example, the classification of text messages carrying pictures, or the classification of images with text titles or embedded text information. The existing schemes for classifying graphic data usually extract the modal features of image data and text data respectively, and then splice the modal features of image data and text data, and classify them according to the obtained graphic fusion features. However, this scheme also has the problem of poor classification effect in some scenarios due to the poor quality of the obtained modal features and graphic fusion features.

[0003] Therefore, a new classification method for graphic and text content is needed. Summary of the invention

[0004] The embodiments of the present disclosure describe a method, apparatus, device and medium for classifying graphic content.

[0005] According to a first aspect, a method for classifying graphic content is provided, comprising:

[0006] Acquire a first text feature of a first text, wherein the first text feature includes a plurality of text feature subvectors, determine first importance scores respectively corresponding to the plurality of text feature subvectors, and convert text feature subvectors in the first image feature whose first importance scores are lower than a preset first threshold into zero vectors to obtain a second text feature;

[0007] Acquire a first image feature of a first image, wherein the first image feature includes a plurality of image feature subvectors, determine second importance scores respectively corresponding to the plurality of image feature subvectors, and convert image feature subvectors of the first image feature whose second importance scores are lower than a preset second threshold into zero vectors to obtain a second image feature, wherein the first image is an image associated with the first text;

[0008] Based on the cross attention mechanism, the second text feature and the second image feature are fused to obtain a first image-text fusion feature;

[0009] Based on the first image-text fusion feature, the first image and / or the first text is classified.

[0010] According to a second aspect, a device for classifying graphic content is provided, comprising:

[0011] A first acquisition unit is configured to acquire a first text feature of a first text, wherein the first text feature includes a plurality of text feature sub-vectors, determine first importance scores respectively corresponding to the plurality of text feature sub-vectors, and convert text feature sub-vectors in the first image feature whose first importance scores are lower than a preset first threshold into zero vectors to obtain a second text feature;

[0012] A second acquisition unit is configured to acquire a first image feature of a first image, wherein the first image feature includes a plurality of image feature sub-vectors, determine second importance scores respectively corresponding to the plurality of image feature sub-vectors, and convert image feature sub-vectors of the first image feature whose second importance scores are lower than a preset second threshold into zero vectors to obtain a second image feature, wherein the first image is an image associated with the first text;

[0013] A cross-fusion unit is configured to fuse the second text feature and the second image feature based on a cross-attention mechanism to obtain a first image-text fusion feature;

[0014] The classification unit is configured to classify the first image and / or the first text based on the first image-text fusion feature.

[0015] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0016] According to a fourth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0017] According to an embodiment of the present disclosure, a method, device, equipment and medium for classifying graphic content are provided. A first text feature of a first text can be obtained, and first importance scores corresponding to multiple text feature subvectors in the first text feature are determined, and text feature subvectors with a first importance score lower than a preset first threshold are changed to zero vectors to obtain a second text feature. And, a first image feature of a first image is obtained, and second importance scores corresponding to multiple image feature subvectors in the first image feature are determined, and image feature subvectors with a second importance score lower than a preset second threshold are changed to zero vectors to obtain a second image feature. Then, based on a cross-attention mechanism, the second text feature and the second image feature are fused to obtain a first graphic fusion feature. Then, based on the first graphic fusion feature, the first image and / or the first text are classified. By using any one of the above methods, devices, storage media and electronic devices, the quality of the acquired graphic fusion feature can be improved, and the accuracy of classifying graphic content based on the graphic fusion feature can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram showing a classification scheme for graphic content;

[0019] Figure 2 A schematic diagram showing a method for classifying graphic content according to an embodiment of the present disclosure is shown;

[0020] Figure 3 A schematic diagram showing a flow chart of a method for classifying graphic content according to an embodiment of the present disclosure is shown;

[0021] Figure 4 A schematic diagram showing a text gate mechanism according to an embodiment of the present disclosure is shown;

[0022] Figure 5 A schematic diagram of obtaining a third text feature and a third image feature according to an embodiment of the present disclosure is shown;

[0023] Figure 6 A schematic block diagram of a device for classifying graphic content according to another embodiment of the present disclosure is shown;

[0024] Figure 7 A schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present disclosure is shown;

[0025] Figure 8 A schematic diagram of the structure of a storage medium suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0026] The technical solution provided in this specification is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant inventions, rather than to limit the inventions. It should also be noted that, for ease of description, only the parts related to the relevant inventions are shown in the accompanying drawings. It should be noted that, in the absence of conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0027] In the description of the implementations of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one / kind of implementation" or "the implementation" should be understood as "at least one / kind of implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below.

[0028] As mentioned above, graphic data is a kind of multimodal data composed of image data and text data. In many application scenarios, there are tasks to classify graphic data, such as the classification of text messages with pictures (pictures) or the classification of images with text titles or embedded text information. Figure 1 A schematic diagram showing a classification scheme for graphic content is shown, such as Figure 1 As shown, existing schemes for classifying image and text data usually extract the modal features of image data (or image modality) and text data (or text modality) separately, and then splice the modal features of image data (or image features, image modal features) and the modal features of text data (or text features, text modal features), and classify them according to the obtained image and text fusion features.

[0029] However, this solution also has the problem of poor classification effect in some scenarios. Specifically, in some scenarios, the acquired image data or text data often contains noise information, and noise information refers to redundant interference information existing in the image data or text data. For example, the noise information of the image data may be, for example, interference information introduced to the acquired image data due to the influence of the acquisition working environment, the material properties of the image sensor and the circuit structure during the image acquisition process. Or, in the process of image transmission and recording, interference information is introduced to the transmitted image data due to the imperfection of the transmission medium and the recording equipment. For another example, the noise information of the text data may be, for example, symbols, punctuation marks, numbers, letters, conversion characters, etc. that are irrelevant to the text semantics or have very weak correlation. The existence of noise information of the image data and text data affects the quality of the modal features extracted from the image data and text data images, thereby making the classification effect obtained by classification based on the modal features poor.

[0030] In other scenarios, there is a correlation between the image data and the text data in the graphic data. For example, a text message about a dog is accompanied by an image of a dog as an illustration. The text content or part of the text content in the text message is strongly correlated with the image information or part of the image information in the illustration. However, existing solutions for classifying graphic data usually simply concatenate image modal features and text modal features to obtain graphic fusion features. The graphic fusion features obtained in this way do not include the correlation information between the above-mentioned different modalities. As a result, the classification effect of classification based on the graphic fusion features that lack modal correlation information is poor.

[0031] In order to solve the above technical problems, the embodiments of the present disclosure provide a method for classifying graphic and text content. Figure 2 A schematic diagram of a method for classifying graphic content according to an embodiment of the present disclosure is shown. Figure 2 As shown, in some embodiments, for example, the modal features of the image and the text can be extracted respectively, and for the obtained first image feature (for example, I) and the first text feature (for example, T), the importance scores of each sub-vector included in the first image feature and the first text feature are calculated based on the text gate (Text-Gate) and the image gate (Image-Gate) mechanism, and the sub-vectors with importance scores lower than the predetermined threshold are converted into zero vectors to obtain the second image feature (for example, I0) and the second text feature (for example, T0). Then, I0 and T0 are fused based on the cross-attention mechanism, for example, to obtain the fused feature M, and the image and text content are classified according to M. In some embodiments, the zero vectors in I0 and T0 can also be deleted first, and the remaining sub-vectors in I0 and T0 are multiplied by the corresponding weights to obtain the third image feature and the third text feature, and then the third image feature and the third text feature are fused based on the cross-attention mechanism to obtain the fused feature M.

[0032] The advantages of this method are as follows: first, by scoring the importance of the subvectors included in the image features and text features, and converting the subvectors whose importance scores are lower than a predetermined threshold into zero vectors, the subvectors corresponding to the noise information in the image data and text data are identified, and the subvectors corresponding to the noise information are invalidated in the third image feature and the third text feature, thereby obtaining higher quality image features and text features. Second, by removing the invalid subvectors in the image features and text features, and fusing the image features and text features based on the cross attention mechanism. That is, the correlation information between different image and text modalities can be added to the fused features, which greatly improves the accuracy of classifying the image and text content according to the fused features. It can also reduce the dimensions of the image features and text features, and improve the computational efficiency of fusing the image features and text features, and classifying the image and text content according to the fused features. In addition, by applying weights to the remaining subvectors in the image features and text features, the proportion of the image features and text features in the fused features can be adjusted, further improving the classification effect.

[0033] The detailed process of this method is further described below.

[0034] Figure 3 FIG. 1 is a flow chart showing a method for classifying graphic content according to an embodiment of the present disclosure. Figure 3 As shown, the method comprises at least the following steps:

[0035] Step S301, obtaining a first text feature of a first text, wherein the first text feature includes a plurality of text feature sub-vectors, determining first importance scores respectively corresponding to the plurality of text feature sub-vectors, converting text feature sub-vectors whose first importance scores are lower than a preset first threshold into zero vectors, and obtaining a second text feature;

[0036] Step S303, obtaining a first image feature of a first image, wherein the first image feature includes a plurality of image feature sub-vectors, determining second importance scores respectively corresponding to the plurality of image feature sub-vectors, and converting image feature sub-vectors whose second importance scores are lower than a preset second threshold into zero vectors, to obtain a second image feature; the first image is an image associated with the first text;

[0037] Step S305: Based on the cross-attention mechanism, the second text feature and the second image feature are fused to obtain a first image-text fusion feature, and the first image and / or first text is classified based on the first image-text fusion feature.

[0038] First, in step S301, a first text feature of a first text is obtained. In this step, the first importance scores corresponding to the multiple text feature sub-vectors included in the first text feature can be determined, and the text feature sub-vectors whose first importance scores are lower than the preset first threshold are changed to zero vectors to obtain a second text feature. In different embodiments, the first text can be a text obtained by different specific methods, such as text obtained in different applications or businesses, and this specification does not limit this.

[0039] The first text may have an associated image, such as a first image. In different embodiments, depending on different specific applications, the specific manner in which the first text is associated with the first image may be different, and this specification does not limit this. For example, in one example, the first text may be a text message transmitted between different users, and the first image may be a drawing of the text message. In another example, the first image may be an image with a text title or embedded text information, and the first text may be a text title or embedded text information of the image. In yet another example, the first image may also be, for example, an image frame in a video, and the first text may be a subtitle or title on the image frame. It is understandable that the data involved in the disclosed scheme (such as the first text and the first image), including but not limited to the data itself, the acquisition or use of the data shall comply with the requirements of the corresponding laws, regulations and relevant provisions.

[0040] The first text feature may include multiple text feature sub-vectors. In one embodiment, the multiple text feature sub-vectors may be multiple word embedding vectors corresponding to multiple words in the first text. Therefore, in one embodiment, the multiple text feature sub-vectors may correspond to multiple words in the first text, respectively. In different specific embodiments, different specific methods of extracting word embedding vectors may be used, and this specification does not limit this.

[0041] Figure 4 FIG. 2 shows a schematic diagram of a text gate mechanism according to an embodiment of the present disclosure. Figure 4As shown, in order to determine the first importance scores corresponding to multiple text feature sub-vectors, in one embodiment, the first text feature (for example, T) can be input into the first fully connected layer (Fully Connected Layer, FCL) to obtain the first fully connected feature, and the first hidden feature can be input into the first Softmax (Soft Maximum) layer to obtain the first importance scores corresponding to the multiple text feature sub-vectors. After determining the first importance scores corresponding to the multiple text feature sub-vectors, the text feature sub-vectors whose first importance scores are lower than the preset first threshold can be changed to zero vectors to obtain second text features (for example, T0). In one embodiment, the value of the text feature sub-vector whose first importance score is lower than the preset first threshold can be multiplied by zero to make it a zero vector. In different specific embodiments, the value of the preset first threshold may be different.

[0042] Then, in step S303, the first image feature of the first image is obtained. In this step, the second importance scores corresponding to the multiple image feature sub-vectors included in the first image feature can be determined, and the image feature sub-vectors whose second importance scores are lower than the preset second threshold are changed to zero vectors to obtain the second image feature. As mentioned above, the first image is an image associated with the first text. In different embodiments, the first image can be obtained by different specific methods, and the specific methods of associating the first image with the first text can be different, which is not limited in this specification.

[0043] The first image feature may include multiple image feature sub-vectors. In one embodiment, the multiple image feature sub-vectors may correspond to multiple image channels (Channel) of the first image, respectively. In different specific embodiments, different specific methods for obtaining image feature sub-vectors corresponding to different image channels may be adopted, and this specification does not limit this. In a specific embodiment, for example, image feature sub-vectors corresponding to different image channels may be obtained by presetting a convolution kernel.

[0044] In order to determine the second importance scores corresponding to the multiple image feature sub-vectors, in one embodiment, the first image feature can be input into the second fully connected layer to obtain the second fully connected feature, and the second fully connected feature can be input into the second Softmax layer to obtain the second importance scores corresponding to the multiple text feature sub-vectors. After determining the second importance scores corresponding to the multiple image feature sub-vectors, the text feature sub-vectors whose second importance scores are lower than the preset second threshold can be changed to zero vectors to obtain second image features. In one embodiment, the value of the text feature sub-vector whose second importance score is lower than the preset second threshold can be multiplied by zero to make it a zero vector. In different specific embodiments, the value of the preset second threshold may be different.

[0045] Thereafter, in step S305, the second text feature and the second image feature may be fused based on a cross-attention mechanism to obtain a first image-text fusion feature, and the first image and / or first text may be classified based on the first image-text fusion feature.

[0046] In order to improve the computational efficiency of fusing the second image feature and the second text feature, and the computational efficiency of classifying the image and text content according to the fused features in the subsequent steps. In one embodiment, the zero vector included in the second text feature can be deleted, and the other text feature subvectors other than the zero vector included in the second text feature can be multiplied by the first weight to obtain the third text feature; the zero vector included in the second image feature can be deleted, and the other image feature subvectors other than the zero vector included in the second image feature can be multiplied by the second weight to obtain the third image feature; according to the cross attention mechanism, the third text feature and the third image feature can be fused to obtain the first image and text fusion feature. Figure 5 A schematic diagram of obtaining a third text feature and a third image feature according to an embodiment of the present disclosure is shown. Figure 5 As shown, in one example, the second text feature T0 includes, for example, a plurality of text sub-vectors (t1, t2, 0, t3, 0, ...), and the zero (0) vector therein may be deleted, and then the remaining text sub-vectors may be multiplied by the first weight α, thereby obtaining the third text feature T1 (αt1, αt2, αt3, ...). In one example, the second image feature I0 includes, for example, a plurality of image sub-vectors (i1, 0, i2, 0, i3, ...), and the zero (0) vector therein may be deleted, and then the remaining image sub-vectors may be multiplied by the second weight β, thereby obtaining the third image feature I1 (βi 1, βi2, βi3, ...). In different specific embodiments, the first weight and the second weight may be different specific weight values. In a specific embodiment, the first weight may be, for example, 0.5, and the second weight may be, for example, 0.5. Among them, by deleting the zero vector in the second text feature and the second image feature, the actually invalid sub-vectors in the second text feature and the second image feature can be deleted, the feature dimension can be reduced, and then the computational efficiency of feature fusion and the computational efficiency of classifying the text and image content based on the fused features can be improved. By applying the first weight and the second weight to the other text feature sub-vectors and image feature sub-vectors other than the zero vector, respectively, it is possible to adjust the proportion of the effective features in the second text feature and the second image feature in the fused feature according to the actual classification effect, so as to further improve the classification effect of the text and image content.

[0047] In a specific embodiment, the third text feature and the third image feature can be fused according to the cross-attention mechanism in the following manner: determine the first query matrix according to the third text feature, determine the first key matrix and the first value matrix according to the third image feature, and determine the first cross-attention feature matrix according to the first query matrix, the first key matrix and the first value matrix; determine the second query matrix according to the third image feature, determine the second key matrix and the second value matrix according to the third text feature, and determine the first cross-attention feature matrix according to the second query matrix, the second key matrix and the second value matrix; perform dimension normalization on the first cross-attention feature matrix and the second cross-attention feature matrix, and splice the dimensionally normalized first cross-attention feature matrix and the second cross-attention feature matrix to obtain the first image-text fusion feature. In the above manner, the correlation information between different image-text modalities can be added to the fusion feature, which greatly improves the accuracy of classifying image-text content according to the fusion feature.

[0048] In one embodiment, after obtaining the first image-text fusion feature, the first image-text fusion feature may be generalized to obtain a second image-text fusion feature. Furthermore, the first image and / or the first text may be classified based on the second image-text fusion feature. In a specific embodiment, the first image-text fusion feature may include multiple fusion sub-vectors. Some of the fusion sub-vectors in the multiple fusion sub-vectors may be randomly discarded by the Dropout function, and an intermediate fusion feature may be obtained according to the average value of the remaining fusion sub-vectors in the multiple fusion sub-vectors; the intermediate fusion feature may be normalized to obtain the second image-text fusion feature. In the above manner, the feature values ​​of different dimensions in the image-text fusion feature may be unified into the same numerical range, that is, the feature values ​​of different dimensions are comparable in numerical value, which may improve the accuracy of the classification of image-text content based on the image-text fusion feature.

[0049] Another embodiment of the present disclosure further provides a device for classifying graphic and text content. Figure 6 A schematic block diagram of a device for classifying graphic content according to another embodiment of the present disclosure is shown. The device is used to perform the following steps: Figure 3 The method shown, such as Figure 6 As shown, the device 600 includes:

[0050] The first acquisition unit 601 is configured to acquire a first text feature of a first text, wherein the first text feature includes a plurality of text feature sub-vectors, determine first importance scores respectively corresponding to the plurality of text feature sub-vectors, and convert text feature sub-vectors in the first image feature whose first importance scores are lower than a preset first threshold into zero vectors, to obtain a second text feature;

[0051] The second acquisition unit 602 is configured to acquire a first image feature of a first image, wherein the first image feature includes a plurality of image feature sub-vectors, determine second importance scores respectively corresponding to the plurality of image feature sub-vectors, and convert image feature sub-vectors of the first image feature whose second importance scores are lower than a preset second threshold into zero vectors to obtain a second image feature, wherein the first image is an image associated with the first text;

[0052] The fusion classification unit 603 is configured to fuse the second text feature and the second image feature based on a cross-attention mechanism to obtain a first image-text fusion feature, and classify the first image and / or first text based on the first image-text fusion feature.

[0053] The present disclosure also provides an electronic device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the following is achieved: Figure 3 The method shown.

[0054] You can also refer to the following Figure 7 , which shows a structural schematic diagram of an electronic device 800 suitable for implementing an embodiment of the present application. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0055] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701. The above-mentioned processing device 701 may be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), a microprocessor or a microcontroller, and may further include an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage device 708 to the random access memory (RAM) 703. In RAM703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, ROM702 and RAM703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0056] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0057] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the classification method of the graphic content provided in the embodiment of the present application are executed.

[0058] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the following steps provided in the present disclosure: Figure 1 The classification method of graphic content shown in . Figure 8 A schematic diagram of a storage medium for implementing an embodiment of the present application. Figure 8As shown, the storage medium 800 may be a non-transitory computer-readable storage medium for storing non-transitory computer-executable instructions 801. When the non-transitory computer-executable instructions 801 are executed by the processor, the classification method for graphic content provided in the embodiment of the present application may be implemented. For example, when the non-transitory computer-executable instructions 801 are executed by the processor, one or more steps in the classification method for graphic content provided in the embodiment of the present application may be executed. For example, the storage medium 800 may be applied to the above-mentioned electronic device. For example, the storage medium 800 may include a memory in the electronic device. For the description of the storage medium 800, reference may be made to the description of the memory in the embodiment of the electronic device, and the repeated parts will not be repeated. The specific functions and technical effects of the storage medium 800 may refer to the description of the classification method for graphic content provided in the embodiment of the present application, and will not be repeated here.

[0059] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a memory card of a smart phone, a storage component of a tablet computer, a portable computer disk, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0060] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the server, the electronic device implements the method for classifying graphic content provided in the embodiment of the present application.

[0061] Computer program code for performing the operations of the disclosed embodiments may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0062] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions. The units involved in the embodiments described in the present disclosure can be implemented by software or by hardware. Among them, the name of the unit does not constitute a limitation on the unit itself under certain circumstances. The functions described above in this article can be performed at least in part by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0063] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0064] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned public concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) to form a technical solution. In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a single embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0065] The specific implementation methods described above further describe in detail the purpose, technical solutions and beneficial effects of the embodiments of the present invention. Although the subject matter has been described in a language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. It should be understood that the above is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for classifying graphic content, comprising: Obtaining a first text feature of a first text, wherein the first text feature includes a plurality of text feature subvectors, determining first importance scores respectively corresponding to the plurality of text feature subvectors, and converting text feature subvectors whose first importance scores are lower than a preset first threshold into zero vectors, thereby obtaining a second text feature; Acquire a first image feature of a first image, wherein the first image feature includes a plurality of image feature subvectors, determine second importance scores respectively corresponding to the plurality of image feature subvectors, and convert image feature subvectors whose second importance scores are lower than a preset second threshold into zero vectors to obtain a second image feature; the first image is an image associated with the first text; Based on the cross attention mechanism, the second text feature and the second image feature are fused to obtain a first image-text fusion feature, and the first image and / or first text is classified based on the first image-text fusion feature.

2. The method according to claim 1, further comprising: After obtaining the first image-text fusion feature, generalizing the first image-text fusion feature to obtain a second image-text fusion feature; The classifying the first image and / or the first text based on the first image-text fusion feature includes: Based on the second image-text fusion feature, the first image and / or the first text is classified.

3. The method according to claim 1, wherein: Based on the cross attention mechanism, the second text feature and the second image feature are fused to obtain the first image-text fusion feature, including: The zero vector included in the second text feature is deleted, and other text feature subvectors other than the zero vector included in the second text feature are multiplied by the first weight to obtain a third text feature; Deleting the zero vector included in the second image feature, and multiplying other image feature subvectors other than the zero vector included in the second image feature by a second weight to obtain a third image feature; According to the cross attention mechanism, the third text feature and the third image feature are fused to obtain the first image-text fusion feature.

4. The method according to claim 3, wherein: The first weight is 0.5, and the second weight is 0.

5.

5. The method according to claim 3, wherein: Based on the cross attention mechanism, the third text feature and the third image feature are fused to obtain a first image-text fusion feature, including: Determine a first query matrix according to the third text feature, determine a first key matrix and a first value matrix according to the third image feature, and determine a first cross-attention feature matrix according to the first query matrix, the first key matrix, and the first value matrix; determine a second query matrix according to the third image feature, determine a second key matrix and a second value matrix according to the third text feature, and determine a first cross-attention feature matrix according to the second query matrix, the second key matrix, and the second value matrix; After dimension normalization processing is performed on the first cross-attention feature matrix and the second cross-attention feature matrix, the dimensionally normalized first cross-attention feature matrix and the second cross-attention feature matrix are spliced ​​to obtain a first image-text fusion feature.

6. The method according to claim 1, wherein: The multiple image feature sub-vectors respectively correspond to multiple image channels of the first image.

7. The method according to claim 1, wherein: The multiple text feature sub-vectors respectively correspond to multiple words in the first text.

8. The method according to claim 1, wherein: Determining first importance scores corresponding to the plurality of text feature sub-vectors respectively includes: Inputting the first text feature into a first fully connected layer to obtain a first fully connected feature, and inputting the first hidden feature into a first Softmax layer to obtain first importance scores corresponding to the multiple text feature sub-vectors respectively; Determining second importance scores corresponding to the plurality of image feature sub-vectors respectively includes: The first image feature is input into a second fully connected layer to obtain a second fully connected feature, and the second fully connected feature is input into a second Softmax layer to obtain second importance scores corresponding to the multiple text feature sub-vectors respectively.

9. The method according to claim 2, wherein: The first image-text fusion feature includes a plurality of fusion sub-vectors; The first image-text fusion feature is generalized to obtain a second image-text fusion feature, including: Randomly discarding some of the multiple fused sub-vectors by using a Dropout function, and obtaining an intermediate fusion feature according to an average value of the remaining fused sub-vectors in the multiple fused sub-vectors; The intermediate fusion feature is normalized to obtain a second image-text fusion feature.

10. A picture and text classification device, comprising: A first acquisition unit is configured to acquire a first text feature of a first text, wherein the first text feature includes a plurality of text feature sub-vectors, determine first importance scores respectively corresponding to the plurality of text feature sub-vectors, and convert text feature sub-vectors in the first image feature whose first importance scores are lower than a preset first threshold into zero vectors to obtain a second text feature; A second acquisition unit is configured to acquire a first image feature of a first image, wherein the first image feature includes a plurality of image feature sub-vectors, determine second importance scores respectively corresponding to the plurality of image feature sub-vectors, and convert image feature sub-vectors of the first image feature whose second importance scores are lower than a preset second threshold into zero vectors to obtain a second image feature, wherein the first image is an image associated with the first text; The fusion classification unit is configured to fuse the second text feature and the second image feature based on a cross-attention mechanism to obtain a first image-text fusion feature, and classify the first image and / or first text based on the first image-text fusion feature.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 9.

12. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 9 is implemented.

Images