Data processing method, device, computer equipment and storage medium

Abstract

The present application discloses a data processing method, apparatus, computer equipment and storage medium, which belongs to the field of computer technology. The present application extracts text features and image features from the text data and image data of the target article respectively, and uses the cross-modal interaction features between the two to predict the article category to which the target article belongs, while taking into account the contribution of the text modality and the image modality to the article category, rather than judging only from the text perspective. In addition, the extracted cross-modal interaction features are not a simple splicing of text features and image features, but can reflect richer and deeper inter-modal interaction information, greatly improving the recognition accuracy of article categories, and thus improving the accuracy of mining high-quality articles in the scenario of identifying high-quality articles.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development of computer technology, the task of discovering high-quality articles has gradually become a research hotspot. Currently, when mining high-quality articles, the content quality of the article is usually judged from the perspective of the text (without considering the contribution of images to content quality), or the embedding features of the text and images are concatenated to determine whether the article is high-quality based on the concatenated features.

[0003] In the above process, whether judging content quality from the perspective of text or using the features of text and image splicing to judge content quality, the accuracy of mining high-quality articles needs to be improved. Therefore, a method to improve the accuracy of mining high-quality articles is urgently needed. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method, apparatus, computer device, and storage medium that can improve the accuracy of identifying article categories and thereby improve the accuracy of mining high-quality articles. The technical solution is as follows:

[0005] In one aspect, a data processing method is provided, the method comprising:

[0006] Acquire text features and image features of a target article, wherein the text features are used to characterize text data in the target article, and the image features are used to characterize image data in the target article;

[0007] For the text feature, based on some features of the image feature that are associated with the text feature, a first interaction feature is obtained, where the first interaction feature is used to represent the text feature integrated with the image feature;

[0008] For the image feature, based on some features of the text feature associated with the image feature, obtaining a second interactive feature, wherein the second interactive feature is used to represent the image feature integrated with the text feature;

[0009] fusing the first interaction feature with the second interaction feature to obtain a cross-modal interaction feature;

[0010] Based on the cross-modal interaction features, the article category to which the target article belongs is determined.

[0011] In one aspect, a data processing device is provided, comprising:

[0012] A first acquisition module is used to acquire text features and image features of a target article, wherein the text features are used to characterize text data in the target article, and the image features are used to characterize image data in the target article;

[0013] A second acquisition module is configured to acquire, for the text feature, a first interaction feature based on a portion of features in the image feature that are associated with the text feature, wherein the first interaction feature is used to represent the text feature integrated with the image feature;

[0014] A third acquisition module is configured to acquire, for the image feature, a second interaction feature based on a portion of the text feature associated with the image feature, wherein the second interaction feature is used to represent the image feature integrated with the text feature;

[0015] a fusion module, configured to fuse the first interaction feature with the second interaction feature to obtain a cross-modal interaction feature;

[0016] A determination module is used to determine the article category to which the target article belongs based on the cross-modal interaction feature.

[0017] In one possible implementation, the first acquisition module includes:

[0018] A first extraction and fusion unit is used to extract text semantic features of the text data, and fuse the text semantic features with text position features to obtain the text features;

[0019] The second extraction and fusion unit is used to extract the image depth feature of the image data, and fuse the image depth feature with the image position feature to obtain the image feature.

[0020] In one possible implementation, the text data includes title data and text data; the text features include title features and text features;

[0021] The first extraction and fusion unit is used for:

[0022] extracting title semantic features of the title data and text semantic features of the text data;

[0023] Fusing the title semantic feature with the title position feature to obtain the title feature;

[0024] The text semantic feature and the text position feature are fused to obtain the text feature.

[0025] In one possible implementation, the first interaction feature includes a title interaction feature and a text interaction feature, and the second acquisition module includes:

[0026] A first acquisition unit is configured to acquire, from the title feature, the title interaction feature based on partial features of the text feature and the image feature that are respectively associated with the title feature, the title interaction feature being used to represent the title feature after the text feature and the image feature are integrated;

[0027] The second acquisition unit is used to acquire the text interaction feature based on the part of the title feature and the picture feature that are respectively associated with the text feature, wherein the text interaction feature is used to represent the text feature after the title feature and the picture feature are integrated.

[0028] In one possible implementation manner, the first acquiring unit is configured to:

[0029] Obtaining a first title interaction feature based on some features of the text features that are associated with the title features;

[0030] obtaining a second title interaction feature based on some features of the image feature that are associated with the title feature;

[0031] concatenating the first title interaction feature and the second title interaction feature to obtain a third title interaction feature;

[0032] The third title interaction feature is encoded and decoded to obtain the title interaction feature.

[0033] In one possible implementation manner, the second acquiring unit is configured to:

[0034] Obtaining a first text interaction feature based on some features of the title feature that are associated with the text feature;

[0035] Obtaining a second text interaction feature based on some features of the image feature that are associated with the text feature;

[0036] concatenating the first text interaction feature and the second text interaction feature to obtain a third text interaction feature;

[0037] The third text interaction feature is encoded and decoded to obtain the text interaction feature.

[0038] In one possible implementation, the third acquisition module includes:

[0039] The third acquisition unit is configured to acquire the second interaction feature based on the picture feature and some features of the title feature and the text feature that are respectively associated with the picture feature.

[0040] In one possible implementation, the third acquiring unit is configured to:

[0041] Obtaining a first picture interaction feature based on some features of the title feature that are associated with the picture feature;

[0042] Obtaining a second image interaction feature based on some features of the text feature that are associated with the image feature;

[0043] concatenating the first image interaction feature and the second image interaction feature to obtain a third image interaction feature;

[0044] The third image interaction feature is encoded and decoded to obtain the second interaction feature.

[0045] In a possible implementation, the text position feature and the image position feature are both relative position features between the text data and the image data, and the relative position features are used to characterize the sequence and distance between the text data and the image data.

[0046] In one possible implementation, the relative position feature is obtained by:

[0047] Determining position information of each of the plurality of texts and the plurality of images in the target article based on the plurality of texts in the text data and the plurality of images in the image data;

[0048] Based on the position information, a relative position coding matrix is ​​constructed, wherein any element in the relative position coding matrix is ​​used to represent the relative position information between the text corresponding to the column to which the element belongs and the picture corresponding to the row to which the element belongs;

[0049] Based on the relative position encoding matrix, a relative position feature between any text in the plurality of texts and any picture in the plurality of pictures is determined.

[0050] In one possible implementation, the determining module is configured to:

[0051] Performing full connection processing on the cross-modal interaction features to obtain fully connected features;

[0052] Performing exponential normalization on the fully connected features to obtain multiple predicted probabilities that the target article belongs to multiple categories;

[0053] The category corresponding to the predicted probability that meets the target condition is selected as the article category to which the target article belongs.

[0054] On the one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein the one or more memories store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors to implement a data processing method as described in any possible implementation manner.

[0055] On the one hand, a storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the data processing method as described in any possible implementation manner.

[0056] In one aspect, a computer program product or computer program is provided, comprising one or more program codes stored in a computer-readable storage medium. One or more processors of a computer device are capable of reading the one or more program codes from the computer-readable storage medium, and executing the one or more program codes by the one or more processors, so that the computer device can perform the data processing method of any possible embodiment described above.

[0057] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0058] By extracting text features and image features from the text data and image data of the target article respectively, and using the cross-modal interaction features between the two, the article category to which the target article belongs is predicted. At the same time, the contribution of the text modality and image modality to the article category is taken into account, rather than judging only from the text perspective. In addition, the extracted cross-modal interaction features are not a simple splicing of text features and image features, but can reflect richer and deeper inter-modal interaction information, greatly improving the accuracy of article category recognition, and thus improving the accuracy of mining high-quality articles in the scenario of identifying high-quality articles. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0060] Figure 1 This is a schematic diagram of an implementation environment of a data processing method provided in an embodiment of the present application;

[0061] Figure 2 This is a flow chart of a data processing method provided by an embodiment of the present application;

[0062] Figure 3 This is a flow chart of a data processing method provided by an embodiment of the present application;

[0063] Figure 4 This is a schematic diagram of the principle of extracting location information provided by an embodiment of the present application;

[0064] Figure 5 This is a schematic diagram of a cross-modal interaction model provided by an embodiment of the present application;

[0065] Figure 6 This is a schematic diagram of the principle of a multimodal fusion network combined with relative position coding provided in an embodiment of the present application;

[0066] Figure 7 This is a flow chart of a data processing method provided by an embodiment of the present application;

[0067] Figure 8 This is a schematic diagram of a multimodal fusion network provided in an embodiment of the present application;

[0068] Figure 9 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0069] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0070] Figure 11 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0072] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on the quantity and execution order.

[0073] In the present application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, a plurality of first positions means two or more first positions.

[0074] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0075] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0076] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0077] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0078] The solutions provided in the embodiments of this application involve artificial intelligence machine learning and other technologies, and in particular, multi-modal machine learning (MMML) technology. The following explains the terminology of multi-modal machine learning:

[0079] Modality: Each source or form of information can be called a modality. For example, people have touch, hearing, vision, and smell; the media of information include voice, video, text, etc.; a variety of sensors, such as radar, infrared, accelerometers, etc., each of the above can be called a modality. At the same time, modality can also have a very broad definition, such as considering two different languages ​​as two modalities, or even data sets collected under two different circumstances can be considered as two modalities. In an embodiment of the present application, for an article, it can be optionally divided into two modalities: text and picture, or divided into three modalities: title, text, and picture.

[0080] Multimodal machine learning: abbreviated as multimodal learning, aims to achieve the ability to process and understand multi-source modal information through machine learning methods. Among them, unimodality refers to a single modality, while multimodality refers to the combination of two or more modalities in various forms. Currently, the more popular research direction is multimodal learning between images, videos, audio, and semantics. Generally speaking, multimodal learning is divided into the following research directions: multimodal representation learning, modal conversion, alignment, multimodal fusion, collaborative learning, etc. Unimodal representation learning is responsible for representing information as a numerical vector that can be processed by a computer or further abstracted into a higher-level feature vector, while multimodal representation learning refers to learning better feature representations by utilizing the complementarity between multiple modalities and eliminating the redundancy between modalities.

[0081] Multimodal Fusion: It is a research direction of multimodal learning. Multimodal fusion is responsible for combining information from multiple modalities to perform target prediction (classification or regression). It is one of the earliest research directions of MMML and is also the most widely used direction. Multimodal fusion also has other common aliases, such as multi-source information fusion and multi-sensor fusion. In the embodiment of the present application, it involves the bimodal fusion of text modality and image modality in an article. Since the text modality can be divided into title modality and body modality, it also involves the trimodal fusion of title modality, body modality and image modality.

[0082] High-quality images and text: Starting from the perspective of the article itself, we construct high-quality articles that balance content quality and reading experience. This can help the recommendation side better understand and apply the articles (i.e., images and text) released by the content center. Optionally, when comprehensively evaluating the content quality of articles, we model them based on dimensions such as multimodal fusion of images and text, article layout experience, and account atomic features to ultimately identify high-quality articles.

[0083] Relative Position Embedding (RPE): refers to a position encoding method in the Transformer model. There are two ways of position encoding for the Transformer model: absolute position encoding and relative position encoding. Absolute position encoding is the currently commonly used position encoding method, that is, directly randomly initialize a position vector (Position Embedding) for the characters at different positions, add it to the input character vector (Word Embedding) sequence and input the model as a parameter for training. When using absolute position encoding, the position vectors corresponding to characters at different positions are different, but the relative meaning of characters at different positions cannot be explicitly obtained through absolute position encoding. For example, the distance between position 1 and position 2 is closer than the distance between position 3 and position 10, and the difference between position 1 and position 2 and position 3 and position 4 is only 1. When using absolute position encoding, only the relative relationship between positions can be learned implicitly. In the embodiment of the present application, by introducing relative position encoding, the feature representation of the relative relationship between positions can be enhanced.

[0084] Figure 1 This is a schematic diagram of an implementation environment of a data processing method provided in an embodiment of the present application. Figure 1 In this implementation environment, the following are included: a terminal 110 and a server 120, where the terminal 110 and the server 120 are both examples of computer devices.

[0085] Terminal 110 is used to browse various articles containing graphic content, for example, the articles include but are not limited to: web information, public account tweets, blogs, Weibo, etc., and the embodiment of the present application does not specifically limit the type of article. An application that supports browsing articles is installed and running on the terminal 110, for example, the application is a browser application, a social application, a graphic information application, a news viewing application, etc., and the embodiment of the present application does not specifically limit the type of the application. Schematically, the user starts the application on the terminal 110, and can browse high-quality articles pushed by the server 120 through the application. Optionally, the terminal 110 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this.

[0086] The terminal 110 and the server 120 can be connected directly or indirectly via wired or wireless communication, which is not limited in this application.

[0087] Server 120 is used to identify and push high-quality articles. That is, server 120 is used to provide background services to the application installed on terminal 110. Illustratively, server 120 collects articles published by creators on the platform, extracts the title, text, and images from the articles, and determines whether the corresponding articles are high-quality based on the corresponding title features, text features, and image features. During the recommendation phase, server 120 increases the recommendation weight of identified high-quality articles, making them more likely to be pushed to the user's terminal 110.

[0088] Optionally, the server 120 includes at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. For example, the server 120 performs primary computing tasks, and the terminal 110 performs secondary computing tasks; or, the server 120 performs secondary computing tasks, and the terminal 110 performs primary computing tasks; or, the terminal 110 and the server 120 perform collaborative computing using a distributed computing architecture.

[0089] In some embodiments, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0090] Those skilled in the art will appreciate that the number of terminals 110 may be greater or less. For example, there may be only one terminal 110, or there may be dozens, hundreds, or even more terminals 110. The embodiment of the present application does not limit the number or device type of terminals 110.

[0091] Figure 2 This is a flow chart of a data processing method provided by an embodiment of the present application. Figure 2 This embodiment is applied to a computer device and is described by taking the computer device as a server as an example. This embodiment includes the following steps:

[0092] 201. The server obtains text features and image features of a target article, where the text features are used to characterize text data in the target article, and the image features are used to characterize image data in the target article.

[0093] Among them, the target article refers to any article of the article category to be determined. The types of the target articles include but are not limited to: web information, public account tweets, blogs, Weibo, etc. The embodiment of this application does not specifically limit the type of the target article.

[0094] A server is an exemplary description of a computer device, including at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. In the embodiment of the present application, only the computer device is used as an example to illustrate, that is, the type identification of the target article is completed on the server side, for example, identifying whether the target article is a high-quality article. In some embodiments, the step of identifying the type of the target article can also be deployed on the terminal side, for example, the terminal independently identifies whether the target article is a high-quality article.

[0095] In some embodiments, the server obtains a target article. Optionally, the target article is an article stored in an article database, or the target article is the latest article uploaded by the terminal to the server, or the target article is an article downloaded from a distributed file system. The embodiments of the present application do not specifically limit the source of the target article.

[0096] In the scenario of identifying high-quality articles, if the text content of two articles is similar or the text quality is similar, but the image quality of the two articles is different, then the image quality of the two articles will affect the final judgment result of whether they are high-quality articles. Therefore, in addition to the text modality, the image modality, that is, the visual modality, will also affect the identification of high-quality articles. Therefore, in addition to the text modality, in the embodiment of the present application, the image modality is also introduced to comprehensively utilize multimodal data to accurately identify high-quality articles.

[0097] In some embodiments, after acquiring the target article, the server extracts the text data and image data from the target article separately, as text and images have different characteristics and require different feature extraction methods. The above process can be considered as a multimodal data extraction process for the target article. Optionally, dividing the data into only two modalities, text and image, can reduce the computational complexity of multimodal fusion.

[0098] In some embodiments, if the mode is divided into only two modes, text and picture, the server extracts the text semantic features of the text data and the picture depth features of the picture data respectively, fuses the text semantic features with the text position features to obtain text features, and fuses the picture depth features with the picture position features to obtain picture features.

[0099] In the above process, by extracting text features and image features separately, it is possible to use different methods to perform targeted feature extraction for the two different modalities of text and image data. Moreover, since the text features and image features each incorporate relevant information about the position, the same text or image will exhibit different features when appearing in different positions in the target article, thereby improving the expressive power of the text features and image features.

[0100] In some embodiments, since the title and body of a text usually have different characteristics, the server can extract the title data and body data separately when extracting text data to introduce more and richer feature information, that is, divide it into three modes: title, body and picture, so as to improve the recognition accuracy of the overall article category.

[0101] In some embodiments, if the three modes are divided into title, text and picture, the server extracts the title semantic features of the title data, the text semantic features of the text data and the picture depth features of the picture data respectively, and then fuses the title semantic features with the title position features to obtain the title features, fuses the text semantic features with the text position features to obtain the text features, and fuses the picture depth features with the picture position features to obtain the picture features.

[0102] In the above process, by extracting title features, text features and image features respectively, it is possible to perform targeted feature extraction for data of three different modalities, namely title, text and image, using different methods, and also integrate position-related information into each of the title features, text features and image features, so that when the same title, text or image appears in different positions in the target article, it will show different features, thereby improving the expressive power of the title features, text features and image features.

[0103] In some embodiments, the above-mentioned position features such as text position features, image position features, title position features, and body position features all refer to absolute position features obtained using absolute position encoding, which can simplify the training process of multimodal fusion.

[0104] In some embodiments, the aforementioned positional features, such as text positional features, image positional features, title positional features, and body positional features, all refer to relative positional features obtained using a relative position encoding method. These relative positional features are used to characterize the order and distance between corresponding text data, image data, title data, or body data. The relative position encoding method for the relative positional features will be described in detail in the next embodiment and will not be elaborated upon here.

[0105] 202. For the text feature, the server obtains a first interactive feature based on a portion of features in the image feature that are associated with the text feature. The first interactive feature is used to represent the text feature integrated with the image feature.

[0106] In some embodiments, for the fusion of text and image modalities, the server uses the text modality as the main modality and receives auxiliary information of the image modality, for example, obtaining some features of the image features that are associated with the text features, and on this basis obtains the first interactive feature.

[0107] In some embodiments, for the case of a three-modal fusion of title, text, and image, the first interaction feature at this time includes a title interaction feature and a text interaction feature. Optionally, the server uses the title modality as the main modality and receives auxiliary information of the text modality and the image modality respectively, for example, respectively obtaining partial features of the text features and the image features that are associated with the title features, and on this basis obtains the title interaction feature; in addition, uses the text modality as the main modality and receives auxiliary information of the title modality and the image modality respectively, for example, respectively obtaining partial features of the title features and the image features that are associated with the text features, and on this basis obtains the text interaction feature.

[0108] 203. For the image feature, the server obtains a second interactive feature based on a portion of features in the text feature that are associated with the image feature, where the second interactive feature is used to represent the image feature integrated with the text feature.

[0109] In some embodiments, for the fusion of text and image modalities, the server uses the image modality as the main modality and receives auxiliary information of the text modality, for example, obtaining some features of the text features that are associated with the image features, and on this basis obtains the second interactive feature.

[0110] In some embodiments, for the case of trimodal fusion of title, text and picture, the server takes the picture mode as the main mode and receives auxiliary information of the title mode and the text mode respectively. For example, it obtains some features of the title features and the text features that are associated with the picture features, and obtains the second interactive feature on this basis.

[0111] 204. The server fuses the first interaction feature with the second interaction feature to obtain a cross-modal interaction feature.

[0112] In some embodiments, the cross-modal interaction feature obtained in the above step 204 refers to the information obtained by integrating the features of each multimodal data with reference to the features of other modal data, so that the cross-modal interaction feature can strengthen the association between text data and image data. Among them, for each modality in the multimodality, it will receive the assistance of other modalities as the main modality to obtain a cross-modal feature, and the cross-modal features corresponding to each modality are fused to obtain the final cross-modal interaction feature.

[0113] For the case of text and image modal fusion, the server directly fuses the first interaction feature and the second interaction feature to obtain the final cross-modal interaction feature.

[0114] In the above process, for the target article, the text modality and the image modality are expressed differently, so there may be overlap (i.e., information redundancy) or complementarity (i.e., more information than single-modal features) between text features and image features. In general, the salient features of multimodal data are redundancy and complementarity. There may even be multiple different information interactions between modalities. Therefore, by extracting the first interaction feature with the text modality as the main modality and the second interaction feature with the image modality as the main modality, we can reasonably process the multimodal data in the target article and obtain richer interaction features.

[0115] For the case of trimodal fusion of title, text and picture, the server fuses the title interaction feature, text interaction feature and second interaction feature to obtain the final cross-modal interaction feature.

[0116] In the above process, for the target article, three modalities are divided into title, text and picture, and they are combined in pairs for directed cross-modal attention interaction. Each modality will serve as the main modality and receive auxiliary information from the other two modalities. Finally, the cross-modal interaction features are fused. Moreover, since more modal interaction information is introduced than the two-modal fusion, the recognition accuracy of high-quality articles can be greatly improved.

[0117] 205. The server determines the article category to which the target article belongs based on the cross-modal interaction feature.

[0118] In some embodiments, the server performs full connection processing on the cross-modal interaction feature to obtain a fully connected feature; performs exponential normalization on the fully connected feature to obtain multiple predicted probabilities that the target article belongs to multiple categories; and selects the category corresponding to the predicted probability that meets the target condition as the article category to which the target article belongs.

[0119] Optionally, the server inputs the cross-modal interaction feature into a fully connected layer or fully connected network, outputs the fully connected feature, and then maps the fully connected feature using an exponentially normalized Softmax function to obtain the predicted probability that the target article belongs to each category. Furthermore, from all the predicted probabilities, the predicted probability that meets the target condition is selected, and the category corresponding to the predicted probability that meets the target condition is determined as the article category to which the target article belongs.

[0120] In some embodiments, the target condition is to maximize the predicted probability. The server then determines the maximum predicted probability from the multiple predicted probabilities and determines the category corresponding to the maximum predicted probability as the article category to which the target article belongs. Alternatively, the server sorts the multiple predicted probabilities from largest to smallest and selects the category corresponding to the predicted probability ranked first as the article category to which the target article belongs.

[0121] In some embodiments, the target condition is that the predicted probability is greater than a probability threshold. The server then determines, from the multiple predicted probabilities, each predicted probability greater than the probability threshold, and randomly selects a category from each category corresponding to each predicted probability greater than the probability threshold as the article category to which the target article belongs. The predicted probability is any value greater than or equal to 0 and less than or equal to 1.

[0122] In some embodiments, the target condition is a random selection of the top K predicted probabilities (K ≥ 1). The server then sorts the multiple predicted probabilities in descending order, selects the top K predicted probabilities, and randomly selects a category from the K categories corresponding to the K predicted probabilities as the article category to which the target article belongs. K is an integer greater than or equal to 1.

[0123] In the above process, based on the multimodal fusion method combined with relative position coding, the article category to which the target article belongs can be identified. Optionally, the article category is divided according to whether the target article is a high-quality article, for example, divided into: high-quality articles, non-high-quality articles, etc.; Optionally, the article category is divided according to the field to which the main content of the target article belongs, for example, divided into: finance, entertainment, news, popular science, etc. The embodiment of the present application does not specifically limit the method of dividing the article category.

[0124] In an exemplary scenario, the article category is divided according to whether the target article is a high-quality article. Then it can be applied to the scenario of identifying high-quality articles, that is, identifying high-quality graphic content. For example, the article category is divided into: high-quality articles and non-high-quality articles, or the article category is divided into: high-quality articles, ordinary articles and low-quality articles, etc. The embodiment of the present application does not specifically limit the way of dividing the article category. In the high-quality article identification scenario, the cross-modal interaction information between adjacent graphics and texts (that is, text and pictures with similar positions) in the same target article is crucial. Since the text in the target article is usually represented in the form of a character sequence or a sentence sequence, and the pictures can also be arranged in order as a picture sequence, the text modality and the picture modality can achieve sequence-level interaction. Therefore, by constructing a sequence-level multimodal fusion network, when the features of the text and the picture are not aligned, the sequence-level interaction information between the modalities can be fully utilized to extract the interaction features between the multimodal features and put them into the prediction process of the article category to improve the recognition accuracy of the article category.

[0125] In an exemplary scenario, article categories are divided according to the field to which the main content of the target article belongs, so it can be applied to scenarios where precise push is performed according to object data. For example, article categories are divided into: finance, entertainment, news, science, etc. In this application scenario, optionally, the server determines whether to recommend the target article to the target user based on the similarity between the cross-modal interaction features of the target article and the target user's object data, thereby being able to recommend to the target user a target article that meets the user's long-term preferences. Alternatively, the server determines whether to recommend the target article to the target user based on the similarity between the cross-modal interaction features of the target article and the target user's average historical reading features, wherein the average historical reading features refer to the average features of the cross-modal interaction features of historical articles read by the target user in the last week (or within a specified time period such as one month or two months), thereby being able to recommend to the target user a target article that meets the user's recent preferences.

[0126] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0127] The method provided in the embodiment of the present application predicts the article category to which the target article belongs by extracting text features and image features from the text data and image data of the target article respectively, and utilizing the cross-modal interaction features between the two. At the same time, the method takes into account the contribution of the text modality and the image modality to the article category, rather than judging only from the text perspective. In addition, the extracted cross-modal interaction features are not a simple splicing of text features and image features, but can reflect richer and deeper inter-modal interaction information, greatly improving the accuracy of identifying article categories, and thus improving the accuracy of mining high-quality articles in the scenario of identifying high-quality articles.

[0128] Figure 3 This is a flow chart of a data processing method provided by an embodiment of the present application. Figure 3 This embodiment is applied to a computer device and is described by taking the computer device as a server as an example. Regarding the case where the target article is divided into only two modalities, text and image, this embodiment of the application will detail how to identify the article category of the target article based on a two-modal fusion method. This embodiment includes the following steps:

[0129] 301. The server obtains the text data and image data in the target article.

[0130] Among them, the target article refers to any article of the article category to be determined. The types of the target articles include but are not limited to: web information, public account tweets, blogs, Weibo, etc. The embodiment of this application does not specifically limit the type of the target article.

[0131] In some embodiments, the server obtains a target article. Optionally, the target article is an article stored in an article database, or the target article is the latest article uploaded by the terminal to the server, or the target article is an article downloaded from a distributed file system. The embodiments of the present application do not specifically limit the source of the target article.

[0132] In some embodiments, because text and images have different characteristics and require different feature extraction methods, the server extracts text data and image data from the target article separately. The above process can be considered as a multimodal data extraction process for the target article. Optionally, dividing the data into only two modalities, text and image, can reduce the computational complexity of multimodal fusion.

[0133] In some embodiments, since the title and body of a text usually have different characteristics, the server can extract the title data and body data separately when extracting text data to introduce more and richer feature information, that is, divide it into three modes: title, body and picture, so as to improve the recognition accuracy of the overall article category.

[0134] 302. The server extracts text semantic features of the text data, and fuses the text semantic features with text position features to obtain text features of the text data.

[0135] In some embodiments, the server extracts the text semantic features based on a text encoding model, and the text encoding model is used to extract the text semantic features of text data, that is, the server inputs the text data into the text encoding model, and encodes the text data by the text encoding model to extract the text semantic features. Optionally, the model structure of the text encoding model includes but is not limited to a combination of any one of the following or at least two: BERT (Bidirectional Encoder Representation From Transformers, a translation model using bidirectional encoding representation), Transformers (transformer, a classic translation model), ELMo (Embeddings From LanguageModels, a language model using embedded processing), NNLM (Neural Network Language Model, neural network language model), etc. The embodiment of the present application does not specifically limit the model structure of the text encoding model. For example, the text encoding model is a BERT model, to reduce the computational complexity of the feature extraction process, and for another example, the text encoding model is formed by cascading the encoder (Encoder) of the BERT model and the Transformers model.

[0136] Schematically, taking the text encoding model formed by the cascade of the encoders of the BERT model and the Transformers model as an example, it is assumed that the text data includes at least one sentence. The server performs word segmentation on each sentence to obtain at least one character contained in each sentence. The characters of each sentence are arranged in the order of appearance in the target article to form a character sequence, wherein [SEP] is added to the end of each sentence in the character sequence as a sentence separator, and [CLS] is added to the first position of the character sequence as a classifier, wherein the sentence separator is used to break sentences between adjacent sentences, and the classifier is used to represent the global semantic information of the entire character sequence.

[0137] The character sequence is input into the BERT model, which includes an embedding layer and at least one bidirectional encoding layer. Each bidirectional encoding layer is used to perform forward and reverse encoding on the input signal. The output of each bidirectional encoding layer serves as the input to the next bidirectional encoding layer, i.e., the bidirectional encoding layers are connected in series. Each bidirectional encoding layer includes two parts: an attention network and a forward fully connected layer. Each hidden layer in the attention network is obtained by weighted averaging the hidden layers of the previous layer, so that each hidden layer can be directly associated with all the hidden layers of the previous layer. The long sequence information of the input (i.e., the character sequence) can be used to obtain a hidden layer vector used to represent global information, and the forward fully connected layer is used to further process the global information obtained by the attention network to enhance the learning ability of the entire BERT model.

[0138] Optionally, the character sequence is first input into the embedding layer of the BERT model, and each character in the character sequence is embedded through the embedding layer. In other words, each character is mapped to the embedding space to obtain the embedding vector of each character, that is, a sequence of embedding vectors is obtained. Then, the embedding vector sequence is input into the at least one bidirectional encoding layer, and each embedding vector in the embedding vector sequence is bidirectionally encoded (including forward encoding and reverse encoding) through the at least one bidirectional encoding layer, and the semantic vector of each character is output, that is, a sequence of semantic vectors is obtained. Each character in the character sequence corresponds to an embedding vector in the embedding vector sequence, and each embedding vector in the embedding vector sequence corresponds to a semantic vector in the semantic vector sequence.

[0139] In the above process, the embedded vector sequence is forward encoded and reverse encoded respectively through the bidirectional encoding layer. Through forward encoding, the semantic vector corresponding to each character can integrate the relevant information of the historical characters that appeared before the character, while through reverse encoding, the semantic vector corresponding to each character can integrate the relevant information of future characters that appear after the character. The encoding operations in two directions can greatly improve the expressive ability of the semantic vectors of each character.

[0140] Take the first bidirectional coding layer as an example for explanation, which includes an attention network and a forward fully connected layer. The embedding vector sequence is input into the attention network of the first bidirectional coding layer, and the embedding vector sequence is weighted by the attention network to extract the attention feature sequence of the embedding vector sequence. The attention feature sequence is input into the forward fully connected layer of the first bidirectional coding layer, and the attention feature sequence is bidirectionally semantically encoded (including forward encoding and reverse encoding) by the forward fully connected layer to output a latent vector sequence, which is input into the second bidirectional coding layer. The processing logic of subsequent bidirectional coding layers is similar to that of the first bidirectional coding layer and will not be repeated here. Due to the introduction of the attention mechanism in the bidirectional coding layer, each character can focus on the character that is more closely associated with itself (more closely related) each time semantic encoding is performed, so that the semantic vector of each character finally obtained has higher accuracy.

[0141] Then, the semantic vector sequence output by the last bidirectional coding layer in the BERT model is input into the encoder of the Transformers model, wherein the Transformers model includes multiple cascaded encoders, for example, including N (N≥1) cascaded encoders, such as N=6 or other numbers, which are not specifically limited in the embodiments of the present application. Each encoder includes a multi-head attention layer and a feedforward neural network layer. The multi-head attention layer is used to comprehensively extract the association between each character in the character sequence from multiple expression subspaces, and the feedforward neural network layer is used to fully connect the feature vector output by the multi-head attention layer. A residual structure is provided after the multi-head attention layer and the feedforward neural network layer, that is, the input and output of the current layer are residually connected (i.e., spliced) and normalized before being input into the next layer. The input semantic vector sequence is encoded by multiple encoders of the Transformers model, and the text semantic features of the text data are output by the last encoder.

[0142] In the above process, the text encoding model formed by cascading the encoders of the BERT model and the Transformers model can extract text semantic features with stronger expressive power. Optionally, the text semantic features can be extracted using only the BERT model, or using only the encoder of the Transformers model to extract the text semantic features to reduce the computational complexity of extracting the text semantic features. The embodiments of the present application do not specifically limit this.

[0143] In some embodiments, after extracting the text semantic features using the text encoding model, the server may obtain text position features of the text data, which are used to represent the positional order of each character in the text data. Optionally, the position information of each character in the character sequence is encoded to obtain the text position features of the text data. Subsequently, the text semantic features and the text position features are concatenated to obtain the text features of the text data.

[0144] In some embodiments, when encoding the position information of each character, the server can use an absolute position encoding method or a relative position encoding method. The embodiment of the present application does not specifically limit the encoding method of the position information. Schematically, the relative position encoding method will be used as an example in the following step 303, and will not be described in detail here.

[0145] In some embodiments, if the dimensions of the text semantic features and the text position features are different, then the text semantic features and the text position features cannot be directly spliced ​​together. In this case, a 1-dimensional convolutional layer is used to transform the dimension of the text semantic features (i.e., increase or decrease the dimension) so that the dimension-transformed text semantic features and the text position features have the same dimension, thereby splicing the dimension-transformed text semantic features and the text position features to obtain the text features of the text data. The 1-dimensional convolutional layer refers to a convolutional layer with a convolution kernel size of 1×1.

[0146] In some embodiments, in addition to fusion by splicing, the text semantic features and text position features can also be fused by element-by-element addition, element-by-element multiplication, bilinear merging, etc. The embodiment of the present application does not specifically limit the fusion method.

[0147] 303. The server extracts the image depth feature of the image data, and fuses the image depth feature with the image position feature to obtain the image feature of the image data.

[0148] In some embodiments, the server extracts the image depth features based on an image depth model, where the image depth model is used to extract the image depth features of the image data. That is, the server inputs the image data into the image depth model and convolves the image data with the image depth model to extract the image depth features. Optionally, the image depth model includes, but is not limited to, convolutional neural networks (CNN), deep residual networks (ResNet), MobileNet (a lightweight neural network), etc. The embodiment of the present application does not specifically limit the model structure of the image depth model.

[0149] Schematically, the image depth model is a MobileNet model. The MobileNet model refers to replacing the standard convolution layer in the VGG (Visual Geometry Group) model with a depthwise separable convolution layer. Depthwise separable convolution is a decomposable convolution operation that can be decomposed into depthwise convolution and pointwise convolution. Depthwise convolution is different from standard convolution. The convolution kernel of standard convolution is used on all channels of the input feature map, while depthwise convolution uses a different convolution kernel for each input channel, that is, one convolution kernel corresponds to one input channel, and pointwise convolution is a one-dimensional convolution, that is, a standard convolution with a convolution kernel size of 1x1. By using depthwise separable convolution, the amount of computation and the number of parameters of the image depth model can be greatly reduced.

[0150] In some embodiments, the MobileNet model includes a 3x3 standard convolutional layer, multiple stacked depthwise separable convolutional layers, a mean pooling layer, and a fully connected layer. Downsampling can be performed after the standard convolutional layer before inputting into the next layer. Similarly, downsampling can be performed after some depthwise separable convolutional layers before inputting into the next layer. The mean pooling layer is used to mean pool the feature map output by the last depthwise separable convolutional layer, and the fully connected layer is used to fully connect the feature map output by the mean pooling layer. Schematically, the MobileNet model contains a total of 28 layers, of which 13 are depthwise separable convolutional layers.

[0151] In some embodiments, since the target article usually contains multiple pictures, the multiple pictures can constitute a picture sequence. The server inputs the picture sequence into the MobileNet model, performs a standard convolution operation on the picture sequence through a standard convolution layer to obtain a first feature map, and inputs the first feature map into multiple cascaded depthwise separable convolution layers. Each depthwise separable convolution layer performs a depthwise separable convolution operation on the feature map output by the previous layer. The last depthwise separable convolution layer outputs a second feature map, and inputs the second feature map into the mean pooling layer. The second feature map is mean pooled through the mean pooling layer to obtain a third feature map. The third feature map is input into the fully connected layer, and the third feature map is fully connected through the fully connected layer to obtain the deep features of the picture.

[0152] In some embodiments, after extracting the image depth features using the image depth model, the server may obtain the image position features of the image data, which are used to characterize the order of the positions of each image in the image data. Optionally, the position information of each image in the image sequence is encoded to obtain the image position features of the image data. Subsequently, the image depth features and the image position features are concatenated to obtain the image features of the image data.

[0153] In some embodiments, when encoding the location information of each image, the server can use an absolute position encoding method or a relative position encoding method. The embodiment of the present application does not specifically limit the encoding method of the location information.

[0154] Schematically, the relative position encoding method is used as an example to illustrate. In this case, the text position feature and the image position feature are both relative position features between the text data and the image data. The relative position feature is used to characterize the order and distance between the text data and the image data.

[0155] In some embodiments, the method for obtaining the relative position feature includes: determining the position information of the multiple texts and the multiple pictures in the target article based on the multiple texts in the text data and the multiple pictures in the picture data; constructing a relative position coding matrix based on the position information, and any element in the relative position coding matrix is ​​used to represent the relative position information between the text corresponding to the column to which the element belongs and the picture corresponding to the row to which the element belongs; based on the relative position coding matrix, determining the relative position feature between any text in the multiple texts and any picture in the multiple pictures.

[0156] Figure 4This is a schematic diagram of the principle of extracting position information provided by an embodiment of the present application, as shown in 400. Assume that the target article includes 5 paragraphs of text and 4 pictures, where the position number of the text sequence composed of the 5 paragraphs of text is {1, 3, 5, 7, 8}, and the position number of the picture sequence composed of the 4 pictures is {0, 2, 4, 6}. Then the absolute position relationship extracted from the target article can be expressed as: pos-0 (img), pos-1 (text), pos-2 (img), pos-3 (text), pos-4 (img), pos-5 (text), pos-6 (img), pos-7 (text), pos-8 (text). Schematically, based on the above absolute position relationship, the text sequence represents the matrix column and the picture sequence represents the matrix row, and the relative position coding matrix shown in Table 1 below is constructed:

[0157] Table 1

[0158] 1 3 5 7 8 0 1 3 5 7 8 2 -1 1 3 5 6 4 -3 -1 1 3 4 6 -5 -3 -1 1 2

[0159] Each element in the relative position encoding matrix represents the relative position between the text in the column to which the element belongs and the image in the row to which the element belongs. For example, the element "3" in the second row and third column of the relative position encoding matrix represents the relative position between the text "5" in the third column and the image "2" in the second row: 3 = 5 - 2.

[0160] After constructing the relative position coding matrix, the relative position information between each text and each image can be determined through the relative position coding matrix, and the corresponding relative position features can be obtained by encoding the relative position information.

[0161] In the above process, by using relative position features, explicit relative position information can be introduced into both text features and image features, thereby improving the expressive power of each of the text features and image features.

[0162] In some embodiments, the text position features and image position features obtained using the traditional absolute position encoding method are both absolute position features, which can implicitly learn the positional correlation between different text paragraphs and different article illustrations, and fully consider the modal internal position relationship of text sequences and image sequences.

[0163] In some embodiments, if the image depth feature and the image position feature have different dimensions, then the image depth feature and the image position feature cannot be directly spliced ​​together. In this case, a 1-dimensional convolutional layer is used to transform the dimension of the image depth feature (i.e., increase or decrease the dimension) so that the dimension-transformed image depth feature and the image position feature have the same dimension, thereby splicing the dimension-transformed image depth feature and the image position feature to obtain the image feature of the image data. The 1-dimensional convolutional layer refers to a convolutional layer with a convolution kernel size of 1×1.

[0164] In some embodiments, in addition to fusion by splicing, the image depth features and image position features can also be fused by element-by-element addition, element-by-element multiplication, bilinear merging, etc. The embodiment of the present application does not specifically limit the fusion method.

[0165] In the above steps 302-303, a possible implementation method is provided for the server to obtain the text features and image features of the target article in the case of bimodal fusion of text and image, wherein the text features are used to characterize the text data in the target article, and the image features are used to characterize the image data in the target article. In the next embodiment, how the server obtains the title features of the title data, the text features of the body data, and the image features of the image data in the case of trimodal fusion of title, text, and image will be described, and will not be repeated here.

[0166] 304. For the text feature, the server obtains a first interactive feature based on some features of the image feature that are associated with the text feature. The first interactive feature is used to represent the text feature integrated with the image feature.

[0167] In some embodiments, the text modality is used as the primary modality, and auxiliary information from the image modality is received. Specifically, the server obtains a portion of the image features that is associated with the text features, and processes the text features and the portion using a cross-modal interaction model to obtain the first interaction feature. Optionally, the cross-modal interaction model includes, but is not limited to, a Transformers model or a variant of the Transformers model.

[0168] Figure 5 This is a schematic diagram of the principle of a cross-modal interaction model provided in an embodiment of the present application, as shown in 500. Taking the cross-modal interaction model as an example, the cross-modal Transformers model is described. The cross-modal Transformers model includes D+1 (D≥0) cross-modal interaction layers. Assuming that the α modality is the main modality (e.g., text modality) and the β modality is the auxiliary modality (e.g., image modality), the process of extracting interaction features from the β modality to the α modality (β→α) is as follows:

[0169] Input the features of the α modality at time t=0 to the 0th layer of the cross-modal Transformers model and the characteristics of the β mode at t = 0 Since each layer in the cross-modal Transformers model processes the features of the two input modalities in a similar way, the processing logic of the i-th layer is used as an example to illustrate, where i is any integer greater than or equal to 0 and less than or equal to D+1.

[0170] The input signal of the i-th layer includes: the interaction features from the β mode to the α mode at time t = i-1 and the characteristics of the β mode at t = 0 The features Input to the layer normalization (LayerNorm, LN) layer for normalization to obtain the feature Q α Similarly, the feature Input into another LN layer for normalization to obtain feature K β and V β . The above feature Q α , K β and V β Input to the Multi-Head Attention Layer, through which the input feature Q α , K β and V β Weighted attention mechanism is used to extract cross-modal features between input signals. Cross-modal features and the characteristic Q in the input signal α Perform element-wise addition to obtain a fused feature. Input the fused feature into another LN layer for normalization, and input the normalized fused feature into a position-wise fully connected feed-forward network layer for full connection. Add the output feature of the position-wise fully connected feed-forward network layer and the fused feature input into the position-wise fully connected feed-forward network layer element-wise to obtain the output feature of the i-th layer, that is, the interaction feature from the β mode to the α mode at time t = i. And so on, the final interactive features are output by the Dth layer. (i.e., the first interaction feature.) The two element-wise addition operations are equivalent to performing residual connections on the input and output of the multi-head attention layer and the position-based fully connected feedforward network layer, respectively.

[0171] In the above process, the cross-modal Transformers model can receive the sequence features of the main modality and the auxiliary modality as input signals, and after two-level processing of the multi-head attention layer and the position fully connected feedforward network layer, it finally outputs the main modality representation (i.e., the first interaction feature) that integrates the auxiliary modality information. The multi-head attention layer can be regarded as a transformation of the self-attention layer, which changes the input features K and V to the feature sequence K of the auxiliary modality β. β and V β , and feature Q is the feature sequence Q of the main mode α α , using the main modality to select the auxiliary modal information that has an interactive relationship with itself in the auxiliary modality, so that the extracted first interaction feature has stronger feature expression ability. Furthermore, the structure of the cross-modal Transformers model is universal and flexible. It can be customized according to the importance of the modality during model design. In addition, the cross-modal attention mechanism in the multi-head attention layer is directed, that is, for the same pair of input modalities {text, image}, the interaction features extracted when the text is the main modality and when the image is the main modality are different. For example, when the text is the main modality in this step 304, the first interaction feature is extracted, and when the image is the main modality in the following step 305, the second interaction feature is extracted. This helps the model to make more full use of the interaction information between modalities. In addition, the stacking of multiple cross-modal interaction layers in the cross-modal Transformers model can integrate more high-order interaction information compared to traditional single-layer interaction solutions.

[0172] In the above process, directly outputting the first interaction feature from the cross-modal Transformers model can reduce the computational complexity of obtaining the first interaction feature. In some embodiments, the feature output by the cross-modal Transformers model is used as an intermediate interaction feature, which is input into a basic Transformers model for encoding and decoding, and the basic Transformers model ultimately outputs the first interaction feature.

[0173] Optionally, the server inputs the intermediate interaction feature into a Transformers model, which includes N cascaded encoders and N cascaded decoders. The N cascaded encoders are called to encode the intermediate interaction feature, and the encoded feature is input into the N cascaded decoders for decoding to obtain the first interaction feature. Wherein, N is an integer greater than or equal to 1, for example, N = 6 or other values.

[0174] In some embodiments, each of the N cascaded encoders includes a multi-head attention layer and a feedforward neural network layer. The multi-head attention layer is used to comprehensively extract the correlation between the feature vectors at each moment from multiple expression subspaces, and the feedforward neural network layer is used to fully connect the feature vectors output by the multi-head attention layer. A residual structure is provided after the multi-head attention layer and the feedforward neural network layer, that is, the input and output of the current layer are residually connected (i.e., spliced) and normalized before being input into the next layer. The input vector is encoded by N cascaded encoders, and the features output by the last encoder are input into N cascaded decoders.

[0175] Each of the N cascaded decoders includes a masked multi-head attention layer, a fused multi-head attention layer and a feedforward neural network layer. The masked multi-head attention layer is similar to the multi-head attention layer, but only focuses on the translation results before the current moment. Therefore, the translation results after the current moment need to be masked (occluded). The fused multi-head attention layer is also similar to the multi-head attention layer, but in addition to the output of the masked multi-head attention layer of the decoder as input, the fused multi-head attention layer also takes the output of the feedforward neural network layer of the encoder with the corresponding sequence number (referring to the output of the feedforward neural network layer of the encoder after the current moment). The decoder uses the residual connection and normalization result) as input. This design is used to focus on the encoded information of the encoder. In other words, the decoder predicts the interaction features of the next moment by looking at the output of the encoder and the self-attention to its own output. The feedforward neural network layer of the decoder is similar to the feedforward neural network layer of the encoder, which will not be described here. Similarly, the masked multi-head attention layer, fused multi-head attention layer, and feedforward neural network layer of the decoder are also provided with a residual structure, that is, the input and output of the current layer are residually connected (i.e. spliced) and normalized before being input into the next layer. Among them, the number of cascaded encoders and the number of cascaded decoders need to be consistent. The encoded features can be decoded by N cascaded decoders, and the last decoder outputs the first interaction feature.

[0176] 305. For the image feature, the server obtains a second interactive feature based on a portion of features in the text feature that are associated with the image feature. The second interactive feature is used to represent the image feature integrated with the text feature.

[0177] In some embodiments, the image modality is used as the primary modality, and auxiliary information from the text modality is received. Specifically, the server obtains a portion of the text features that is associated with the image features, and processes the image features and the portion using a cross-modal interaction model to obtain the second interaction feature. Optionally, the cross-modal interaction model includes, but is not limited to, a Transformers model or a variant of the Transformers model.

[0178] The above step 305 is similar to the above step 304, except that the main mode α is changed to the image mode and the auxiliary mode β is changed to the text mode, which will not be described in detail here.

[0179] In the above process, the second interaction feature is directly output by the cross-modal Transformers model, which can reduce the computational complexity when obtaining the second interaction feature. In some embodiments, the feature output by the cross-modal Transformers model is used as an intermediate interaction feature, and the intermediate interaction feature is input into a basic Transformers model for encoding and then decoding, and finally the second interaction feature is output by the basic Transformers model. Optionally, the server inputs the intermediate interaction feature into the Transformers model, and the Transformers model includes N cascaded encoders and N cascaded decoders, calls the N cascaded encoders to encode the intermediate interaction feature, and inputs the encoded feature into the N cascaded decoders for decoding to obtain the first interaction feature. Wherein, N is an integer greater than or equal to 1, for example, N = 6 or other values. The internal processing logic of each encoder and decoder in the basic Transformers model has been introduced in the above step 304 and will not be repeated here.

[0180] 306. The server fuses the first interaction feature with the second interaction feature to obtain a cross-modal interaction feature.

[0181] In some embodiments, the server concatenates the first interaction feature and the second interaction feature to obtain a final cross-modal interaction feature, thereby reducing the amount of computation during feature fusion.

[0182] In other embodiments, the server performs element-wise addition, element-wise multiplication, or bilinear merging of the first interaction feature and the second interaction feature, which can make the feature fusion more complete. The embodiment of the present application does not specifically limit the feature fusion method.

[0183] In the above steps 304-306, a possible implementation method of the server obtaining cross-modal interaction features between the two modalities in the case of the fusion of text and picture is shown. In the next embodiment, for the case of the fusion of title, text and picture, how the server obtains cross-modal interaction features between the three modalities will be introduced, which will not be repeated here.

[0184] 307. The server determines the article category to which the target article belongs based on the cross-modal interaction feature.

[0185] The above step 307 is similar to the above step 205 and will not be described in detail here.

[0186] Figure 6 This is a schematic diagram of a multimodal fusion network combined with relative position coding provided by an embodiment of the present application, such as Figure 6 As shown, the multimodal fusion network includes a text encoding model 601, an image encoding model 602 and a cross-modal interaction part 603.

[0187] Schematically, the text encoding model 601 is formed by cascading the BERT model 6011 obtained by fine-tuning the basic BERT model (Finetune) and the encoder 6012 of the Transformers model. The character sequence of the text data (referred to as the text sentence sequence) is input into the BERT model 6011, and a semantic vector sequence is output. The semantic vector sequence is then input into the encoder 6012 of the Transformers model, and the text semantic features of the text data are output. The text semantic features are input into a 1-dimensional convolutional layer (Conv1D) layer for dimensionality transformation, and then spliced ​​with the text position features to obtain the text features of the text data.

[0188] Schematically, the image coding model 602 is a pre-trained MobileNet model. The image sequence of the image data is input into the image coding model 602, and the image depth features of the image data are output. The image depth features are input into the Conv1D layer for dimensionality transformation and then spliced ​​with the image position features to obtain the image features of the image data.

[0189] Schematically, the cross-modal interaction part 603 includes two cross-modal Transformers models and two basic Transformers models. With the text modality as the main modality, the cross-modal Transformers model is used to extract the intermediate interaction features from the image modality to the text modality. This intermediate interaction feature is input into the basic Transformers model for encoding and decoding, and the first interaction feature is output. With the image modality as the main modality, the cross-modal Transformers model is used to extract the intermediate interaction features from the text modality to the image modality. This intermediate interaction feature is input into the basic Transformers model for encoding and decoding, and the second interaction feature is output.

[0190] Furthermore, the first interaction feature and the second interaction feature are concatenated to obtain a final cross-modal interaction feature between the two modalities, and the cross-modal interaction feature is then used to predict the article category (Classification) to which the target article ultimately belongs.

[0191] In some embodiments, if the relative position encoding method introduced in the above step 303 is adopted, then the absolute position features of each Transformers model in the cross-modal interaction part 603 need to be modified into relative position features. For example, the original character embedding (embedding vector) and position embedding (position vector) are separated, and after the column form is expanded, the position vector of the absolute position encoding method is converted into the position vector of the relative position encoding method, that is, the relative position relationship is integrated into the self-attention layer when any two modalities perform interaction calculations.

[0192] In the Transformers model, the self-attention layer is usually expressed as:

[0193]

[0194] Among them, Attention(Q,K,V) refers to the attention coefficient calculated based on the Q (Query) matrix, K (Key) matrix and V (Value) matrix, softmax() refers to the exponential normalization function, Q refers to the Q matrix of the current character, K refers to the K matrix of the current character, V refers to the V matrix of the current character, K T is the transposed matrix of the K matrix, is the scaling factor.

[0195] Using the multiplication distributive law expansion, in the case of absolute position encoding, the attention coefficient between the i-th element in modal 1 and the j-th element in modal 2 is The expansion of is as follows:

[0196]

[0197] Among them, E represents the text vector, U refers to the position vector, and W refers to the parameter matrix.

[0198] That is to say, The transposed matrix of the text vector representing the i-th element in modality 1, represents the transposed matrix of the parameter matrix of the Q matrix, W k represents the parameter matrix of the K matrix, The text vector representing the jth element in modality 2, U j represents the position vector of the jth element in mode 2, Represents the transposed matrix of the position vector of the i-th element in mode 1.

[0199] Position vector R using relative position encoding i-j To replace the position vector U of the absolute position encoding method j and U i , the above expansion can be transformed into:

[0200]

[0201] Among them, E represents the text vector, U refers to the position vector, and W refers to the parameter matrix.

[0202] That is to say, The transposed matrix of the text vector representing the i-th element in modality 1, represents the transposed matrix of the parameter matrix of the Q matrix, W k，E Represents the parameter matrix related to the K matrix and text vector under relative position encoding, The text vector representing the jth element in modality 2, R i-j represents the relative position encoding vector between the i-th element in modality 1 and the j-th element in modality 2, W k，R Represents the parameter matrix related to the K matrix and the relative position encoding vector under relative position encoding, u T and v T They represent the parameter vectors to be learned that are independent of the position of the i-th element in mode 1,

[0203] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0204] The method provided in the embodiment of the present application predicts the article category to which the target article belongs by extracting text features and image features from the text data and image data of the target article respectively, and utilizing the cross-modal interaction features between the two. At the same time, the method takes into account the contribution of the text modality and the image modality to the article category, rather than judging only from the text perspective. In addition, the extracted cross-modal interaction features are not a simple splicing of text features and image features, but can reflect richer and deeper inter-modal interaction information, greatly improving the accuracy of identifying article categories, and thus improving the accuracy of mining high-quality articles in the scenario of identifying high-quality articles.

[0205] Figure 7 This is a flow chart of a data processing method provided by an embodiment of the present application. Figure 7 This embodiment is applied to a computer device and is described by taking the computer device as a server as an example. Regarding the case where a target article is divided into three modalities: title, text, and image, this embodiment of the application will describe in detail how to identify the article category of the target article based on a three-modal fusion method. This embodiment includes the following steps:

[0206] 701. The server obtains the title data, text data, and image data in the target article.

[0207] The title data and body data may be collectively referred to as text data.

[0208] The above step 701 is similar to the above step 301 and will not be described in detail here. Optionally, after the server obtains the text data and the image data, it extracts the title data and the body data from the text data.

[0209] 702. The server extracts the title semantic features of the title data, and fuses the title semantic features with the title position features to obtain the title features of the title data.

[0210] In some embodiments, the server extracts the title semantic features based on a title encoding model. The title encoding model is used to extract the title semantic features of the title data. That is, the server inputs the title data into the title encoding model, and encodes the title data using the title encoding model to extract the title semantic features. Optionally, the model structure of the title encoding model includes, but is not limited to, a BERT model, a Transformers model, an ELMo model, an NNLM model, etc. This embodiment of the present application does not specifically limit the model structure of the title encoding model.

[0211] Schematically, taking the title encoding model as the BERT model as an example, assuming that the title data includes at least one title, the server performs word segmentation on each title to obtain at least one character contained in each title, and arranges the characters of each title in the order of appearance in the target article to form a character sequence, wherein [SEP] is added to the end of each title in the character sequence as a sentence separator, and [CLS] is added to the first position of the character sequence as a classifier, wherein the sentence separator is used to break sentences between adjacent titles, and the classifier is used to represent the global semantic information of the entire character sequence.

[0212] The character sequence is input into the BERT model, which includes an embedding layer and at least one bidirectional encoding layer. Each bidirectional encoding layer is used to encode the input signal in both forward and reverse directions. The output of each bidirectional encoding layer serves as the input to the next bidirectional encoding layer, meaning that the bidirectional encoding layers are connected in series. Each bidirectional encoding layer consists of two parts: an attention network and a forward fully connected layer. Each hidden layer in the attention network is a weighted average of the hidden layers in the previous layer, allowing each hidden layer to be directly associated with all the hidden layers in the previous layer. The input long sequence information (i.e., the character sequence) can be used to obtain a hidden layer vector that represents global information. The forward fully connected layer is used to further process the global information obtained by the attention network to enhance the learning ability of the entire BERT model.

[0213] Optionally, the character sequence is first input into the embedding layer of the BERT model, and each character in the character sequence is embedded through the embedding layer. In other words, each character is mapped to the embedding space to obtain the embedding vector of each character, that is, a sequence of embedding vectors is obtained. Then, the embedding vector sequence is input into the at least one bidirectional encoding layer, and each embedding vector in the embedding vector sequence is bidirectionally encoded (including forward encoding and reverse encoding) through the at least one bidirectional encoding layer, and the semantic vector of each character is output, that is, a sequence of semantic vectors is obtained. Finally, the last bidirectional encoding layer outputs the title semantic features of the title data. Each character in the character sequence corresponds to an embedding vector in the embedding vector sequence, and each embedding vector in the embedding vector sequence corresponds to a semantic vector in the semantic vector sequence.

[0214] In the above process, the embedded vector sequence is forward encoded and reverse encoded respectively through the bidirectional encoding layer. Through forward encoding, the semantic vector corresponding to each character can integrate the relevant information of historical characters that appeared before the character, while through reverse encoding, the semantic vector corresponding to each character can integrate the relevant information of future characters that appear after the character. The encoding operations in two directions can greatly improve the expressive ability of the semantic vectors of each character.

[0215] Taking the first bidirectional encoding layer as an example, the bidirectional encoding layer includes an attention network and a forward fully connected layer. The embedded vector sequence is input into the attention network of the first bidirectional encoding layer, and the embedded vector sequence is weighted by the attention network to extract the attention feature sequence of the embedded vector sequence. The attention feature sequence is input into the forward fully connected layer of the first bidirectional encoding layer. The attention feature sequence is bidirectionally semantically encoded (including forward encoding and reverse encoding) by the forward fully connected layer, and a latent vector sequence is output. The latent vector sequence is input into the second bidirectional encoding layer, and so on. The processing logic of subsequent bidirectional encoding layers is similar to that of the first bidirectional encoding layer and will not be repeated here. Finally, the title semantic features of the title data are output by the last bidirectional encoding layer. Due to the introduction of the attention mechanism in the bidirectional encoding layer, each character can focus on the character that is more closely associated with itself during semantic encoding, so that the semantic vector of each character finally obtained has higher accuracy.

[0216] In some embodiments, after extracting the title semantic features using the title encoding model, the server may obtain a title position feature of the title data. The title position feature is used to represent the sequence of positions of each character in the title data. Optionally, the position information of each character in the character sequence is encoded to obtain the title position feature of the title data. The title semantic feature and the title position feature are then concatenated to obtain the title feature of the title data.

[0217] In some embodiments, the server can use an absolute position encoding method or a relative position encoding method when encoding the position information of each character. The embodiment of this application does not specifically limit the encoding method of the position information. Both position encoding methods have been introduced in the previous embodiment and will not be repeated here.

[0218] In some embodiments, if the dimensions of the title semantic feature and the title position feature are different, then the title semantic feature and the title position feature cannot be directly concatenated. In this case, a 1-dimensional convolutional layer is used to transform the dimension of the title semantic feature (i.e., increase or decrease the dimension) so that the dimension-transformed title semantic feature and the title position feature have the same dimension. The transformed title semantic feature and the title position feature are then concatenated to obtain the title feature of the title data. The 1-dimensional convolutional layer refers to a convolutional layer with a convolution kernel size of 1×1.

[0219] In some embodiments, in addition to fusion by splicing, the title semantic features and title position features can also be fused by element-by-element addition, element-by-element multiplication, bilinear merging, etc. The embodiment of the present application does not specifically limit the fusion method.

[0220] 703. The server extracts the text semantic features of the text data, and fuses the text semantic features with the text position features to obtain the text features of the text data.

[0221] Among them, the title feature and the text feature can be collectively referred to as text features.

[0222] In some embodiments, the server extracts the text semantic features based on a text encoding model, and the text encoding model is used to extract the text semantic features of the text data, that is, the server inputs the text data into the text encoding model, and encodes the text data through the text encoding model to extract the text semantic features. Optionally, the model structure of the text encoding model includes but is not limited to any one of the following or a combination of at least two: a BERT model, a Transformers model, an ELMo model, an NNLM model, etc. The embodiment of the present application does not specifically limit the model structure of the text encoding model. Schematically, the text encoding model is formed by cascading the encoders of the BERT model and the Transformers model. The text encoding model of this structure processes the text data similarly to the text encoding model in step 302 above, and will not be elaborated here.

[0223] In some embodiments, after extracting the text semantic features using the text encoding model, the server may obtain text position features of the text data. The text position features are used to represent the sequence of positions of each character in the text data. Optionally, the position information of each character in the character sequence is encoded to obtain the text position features of the text data. The text semantic features and the text position features are then concatenated to obtain the text features of the text data.

[0224] In some embodiments, the server can use an absolute position encoding method or a relative position encoding method when encoding the position information of each character. The embodiment of this application does not specifically limit the encoding method of the position information. Both position encoding methods have been introduced in the previous embodiment and will not be repeated here.

[0225] In some embodiments, if the dimensions of the text semantic features and the text position features are different, then the text semantic features and the text position features cannot be directly spliced ​​together. In this case, a 1-dimensional convolutional layer is used to transform the dimension of the text semantic features (i.e., increase or decrease the dimension) so that the dimension-transformed text semantic features and the text position features have the same dimension, thereby splicing the dimension-transformed text semantic features and the text position features to obtain the text features of the text data. The 1-dimensional convolutional layer refers to a convolutional layer with a convolution kernel size of 1×1.

[0226] In some embodiments, in addition to fusion by splicing, the text semantic features and text position features can also be fused by element-by-element addition, element-by-element multiplication, bilinear merging, etc. The embodiment of this application does not specifically limit the fusion method.

[0227] In the above steps 702-703, a possible implementation method is provided in which, when the text data is divided into title data and body data, the server extracts the text semantic features of the text data, fuses the text semantic features with the text position features, and obtains the text features of the text data. By dividing the text data into title data and body data, more and richer feature information can be extracted.

[0228] 704. The server extracts the image depth feature of the image data, and fuses the image depth feature with the image position feature to obtain the image feature of the image data.

[0229] The above step 704 is similar to the above step 303 and will not be described in detail here.

[0230] 705. For the title feature, the server obtains a title interaction feature based on the text feature and the partial features of the image feature that are respectively associated with the title feature. The title interaction feature is used to represent the title feature after the text feature and the image feature are integrated.

[0231] In some embodiments, the server obtains the first title interaction feature based on the partial features associated with the title feature in the main text feature, that is, with the title mode as the main mode, the server receives the auxiliary information of the main text mode. Optionally, the server obtains the partial features associated with the title feature in the main text feature, and uses the cross-modal interaction model to process the title feature and the partial features to obtain the first title interaction feature. Optionally, the cross-modal interaction model includes but is not limited to: a Transformers model or a variant of the Transformers model. For example, the cross-modal interaction model is a cross-modal Transformers model, but the main mode α is changed to the title mode, and the auxiliary mode β is changed to the main text mode. The model structure and processing logic are similar to those of the above step 304 and are not repeated here.

[0232] In some embodiments, the server obtains a second title interaction feature based on a portion of the features in the picture feature that is associated with the title feature, that is, with the title modality as the main modality, the server receives auxiliary information of the picture modality. Optionally, the server obtains a portion of the features in the picture feature that is associated with the title feature, and uses a cross-modal interaction model to process the title feature and the portion of the features to obtain the second title interaction feature. Optionally, the cross-modal interaction model includes but is not limited to: a Transformers model or a variant of the Transformers model. For example, the cross-modal interaction model is a cross-modal Transformers model, but the main modality α is changed to the title modality, and the auxiliary modality β is changed to the picture modality. The model structure and processing logic are similar to those of the above step 304 and are not repeated here.

[0233] In some embodiments, the server concatenates the first title interaction feature and the second title interaction feature to obtain a third title interaction feature, which can reduce the computational complexity when fusing the first title interaction feature and the second title interaction feature. Optionally, fusion methods such as element-by-element addition, element-by-element multiplication, and bilinear merging can also be adopted, and this embodiment of the present application does not specifically limit this.

[0234] In some embodiments, the server encodes and decodes the third title interaction feature to obtain the title interaction feature. Optionally, the server inputs the third title interaction feature into a Transformers model, which includes N cascaded encoders and N cascaded decoders. The N cascaded encoders are called to encode the third title interaction feature to obtain an intermediate title interaction feature, and the intermediate title interaction feature is input into the N cascaded decoders for decoding to obtain the title interaction feature. Wherein, N is an integer greater than or equal to 1, for example, N = 6 or other values.

[0235] In some embodiments, each of the N cascaded encoders includes a multi-head attention layer and a feedforward neural network layer. The multi-head attention layer is used to comprehensively extract the association between the characters in the title data from multiple expression subspaces, and the feedforward neural network layer is used to fully connect the feature vector output by the multi-head attention layer. A residual structure is provided after the multi-head attention layer and the feedforward neural network layer, that is, the input and output of the current layer are residually connected (i.e., spliced) and normalized before being input into the next layer. The input vector is encoded by N cascaded encoders, and the last encoder outputs the intermediate title interaction feature.

[0236] Then, the intermediate title interaction feature is input into N cascaded decoders. Each of the N cascaded decoders includes a masked multi-head attention layer, a fused multi-head attention layer and a feedforward neural network layer. The masked multi-head attention layer is similar to the multi-head attention layer, but only focuses on the translation results before the current moment. Therefore, the translation results after the current moment need to be masked (occluded). The fused multi-head attention layer is also similar to the multi-head attention layer, but in addition to the output of the masked multi-head attention layer of the decoder as input, the fused multi-head attention layer also takes the output of the feedforward neural network layer of the encoder with the corresponding sequence number (referring to the output of the feedforward neural network layer of the encoder after the current moment). The decoder uses the residual connection and normalization result) as input. This design is used to focus on the encoded information of the encoder. In other words, the decoder predicts the interaction features of the next moment by looking at the output of the encoder and the self-attention to its own output. The feedforward neural network layer of the decoder is similar to the feedforward neural network layer of the encoder, which will not be elaborated here. Similarly, the masked multi-head attention layer, fused multi-head attention layer, and feedforward neural network layer of the decoder are also provided with a residual structure, that is, the input and output of the current layer are residually connected (i.e. spliced) and normalized before being input into the next layer. Among them, the number of cascaded encoders and the number of cascaded decoders need to be consistent. The intermediate title interaction features can be decoded by N cascaded decoders, and the final title interaction features are output by the last decoder.

[0237] 706. For the text feature, the server obtains a text interaction feature based on the title feature and the partial features of the image feature that are respectively associated with the text feature. The text interaction feature is used to represent the text feature after the title feature and the image feature are integrated.

[0238] Since in the case of trimodal fusion of title, text, and picture, the first interaction feature includes the title interaction feature and the text interaction feature, the above steps 705-706 illustrate a possible implementation method of how to obtain the first interaction feature.

[0239] In some embodiments, the server obtains the first main text interaction feature based on the partial features of the title feature that are associated with the main text feature, that is, with the main text mode as the main mode, the server receives the auxiliary information of the title mode. Optionally, the server obtains the partial features of the title feature that are associated with the main text feature, and uses a cross-modal interaction model to process the main text feature and the partial features to obtain the first main text interaction feature. Optionally, the cross-modal interaction model includes but is not limited to: a Transformers model or a variant of the Transformers model. For example, the cross-modal interaction model is a cross-modal Transformers model, but the main mode α is changed to the main text mode, and the auxiliary mode β is changed to the title mode. The model structure and processing logic are similar to those of the above step 304 and are not repeated here.

[0240] In some embodiments, the server obtains the second text interaction feature based on the partial features of the image feature that are associated with the text feature, that is, with the text modality as the main modality, the server receives the auxiliary information of the image modality. Optionally, the server obtains the partial features of the image feature that are associated with the text feature, and uses a cross-modal interaction model to process the text feature and the partial features to obtain the second text interaction feature. Optionally, the cross-modal interaction model includes but is not limited to: a Transformers model or a variant of the Transformers model. For example, the cross-modal interaction model is a cross-modal Transformers model, but the main modality α is changed to the text modality, and the auxiliary modality β is changed to the image modality. The model structure and processing logic are similar to those of the above step 304 and are not repeated here.

[0241] In some embodiments, the server concatenates the first text interaction feature and the second text interaction feature to obtain a third text interaction feature, which can reduce the computational complexity when fusing the first text interaction feature and the second text interaction feature. Optionally, fusion methods such as element-by-element addition, element-by-element multiplication, and bilinear merging can also be adopted, and the embodiments of the present application do not specifically limit this.

[0242] In some embodiments, the server encodes and decodes the third text interaction feature to obtain the text interaction feature. Optionally, the server inputs the third text interaction feature into the Transformers model, encodes the third text interaction feature through N cascaded encoders in the Transformers model to obtain an intermediate text interaction feature, and inputs the intermediate text interaction feature into N cascaded decoders for decoding to obtain the text interaction feature. Wherein, N is an integer greater than or equal to 1, for example, N = 6 or other values. The internal processing logic of the encoder and decoder of the Transformers model has been described in detail in the above step 705 and will not be repeated here.

[0243] 707. For the image feature, the server obtains a second interactive feature based on the title feature and the partial features of the text feature that are respectively associated with the image feature. The second interactive feature is used to represent the image feature after the title feature and the text feature are integrated.

[0244] In some embodiments, the server obtains the first picture interaction feature based on the partial features associated with the picture feature in the title feature, that is, with the picture mode as the main mode, the server receives the auxiliary information of the title mode. Optionally, the server obtains the partial features associated with the picture feature in the title feature, and uses the cross-modal interaction model to process the picture feature and the partial features to obtain the first picture interaction feature. Optionally, the cross-modal interaction model includes but is not limited to: a Transformers model or a variant of the Transformers model. For example, the cross-modal interaction model is a cross-modal Transformers model, but the main mode α is changed to the picture mode, and the auxiliary mode β is changed to the title mode. The model structure and processing logic are similar to the above step 304 and are not repeated here.

[0245] In some embodiments, the server obtains the second image interaction feature based on the partial features associated with the image feature in the text feature, that is, with the image modality as the main modality, the server receives the auxiliary information of the text modality. Optionally, the server obtains the partial features associated with the image feature in the text feature, and uses the cross-modal interaction model to process the image feature and the partial features to obtain the second image interaction feature. Optionally, the cross-modal interaction model includes but is not limited to: a Transformers model or a variant of the Transformers model. For example, the cross-modal interaction model is a cross-modal Transformers model, but the main modality α is changed to the image modality, and the auxiliary modality β is changed to the text modality. The model structure and processing logic are similar to the above step 304 and are not repeated here.

[0246] In some embodiments, the server concatenates the first image interaction feature and the second image interaction feature to obtain a third image interaction feature, which can reduce the computational complexity when fusing the first image interaction feature and the second image interaction feature. Optionally, fusion methods such as element-by-element addition, element-by-element multiplication, and bilinear merging can also be adopted. The embodiments of the present application do not specifically limit this.

[0247] In some embodiments, the server encodes and decodes the third image interaction feature to obtain the second interaction feature. Optionally, the server inputs the third image interaction feature into the Transformers model, encodes the third image interaction feature through N cascaded encoders in the Transformers model to obtain an intermediate image interaction feature, and inputs the intermediate image interaction feature into N cascaded decoders for decoding to obtain the second interaction feature. Wherein, N is an integer greater than or equal to 1, for example, N = 6 or other values. The internal processing logic of the encoder and decoder of the Transformers model has been described in detail in the above step 705 and will not be repeated here.

[0248] 708. The server fuses the title interaction feature, the text interaction feature, and the second interaction feature to obtain a cross-modal interaction feature.

[0249] In some embodiments, the server concatenates the title interaction feature, the text interaction feature, and the second interaction feature to obtain a final cross-modal interaction feature among the three modalities, thereby reducing the amount of computation during feature fusion.

[0250] In other embodiments, the server performs element-by-element addition, element-by-element multiplication, or bilinear merging of the title interaction feature, the text interaction feature, and the second interaction feature, which can make the feature fusion more complete. The embodiments of the present application do not specifically limit the feature fusion method.

[0251] In the above steps 705-708, a possible implementation method for the server to obtain cross-modal interaction features is provided, namely, by dividing the text data into title data and body data, the original two-modal fusion is expanded to a three-modal fusion, which can make full use of the sequence-level interaction information between the modalities, and perform directed cross-modal attention weighting on the three modalities of title, body and picture in pairs (a total of 6 combinations). Each modality will receive auxiliary information of the other two modalities as the main modality, which greatly improves the expressive ability of the cross-modal interaction features finally obtained, and greatly improves the accuracy of the final prediction based on the cross-modal interaction features.

[0252] 709. The server determines the article category to which the target article belongs based on the cross-modal interaction feature.

[0253] The above step 709 is similar to the above step 205 and will not be described in detail here.

[0254] Figure 8 This is a schematic diagram of a multimodal fusion network provided in an embodiment of the present application. Figure 8 As shown, the multimodal fusion network includes a title encoding model 801, a text encoding model 802, a picture encoding model 803 and a cross-modal interaction part 804.

[0255] Schematically, the title encoding model 801 is a BERT model obtained by fine-tuning the basic BERT model (Finetune). The character sequence of the title data (referred to as the title sequence for short) is input into the title encoding model 801, and the title semantic features of the title data are output. The title semantic features are input into a 1-dimensional convolutional layer (Conv1D) for dimensionality transformation and then spliced ​​with the title position features to obtain the title features of the title data.

[0256] Schematically, the text encoding model 802 is formed by cascading the fine-tuned BERT model 8021 and the encoder 8022 of the Transformers model. The character sequence of the text data (referred to as the text sentence sequence) is input into the BERT model 8021, which outputs a semantic vector sequence. The semantic vector sequence is then input into the encoder 8022 of the Transformers model, which outputs the text semantic features of the text data. The text semantic features are input into the Conv1D layer for dimensionality transformation and then spliced ​​with the text position features to obtain the text features of the text data.

[0257] Schematically, the image coding model 803 is a pre-trained MobileNet model. The image sequence of the image data is input into the image coding model 803, and the image depth features of the image data are output. The image depth features are input into the Conv1D layer for dimensionality transformation and then spliced ​​with the image position features to obtain the image features of the image data.

[0258] Schematically, the cross-modal interaction portion 804 includes six cross-modal Transformers models and three basic Transformers models. With the title modality as the primary modality, the cross-modal Transformers model is used to extract the first title interaction feature from the text modality to the title modality, and the second title interaction feature from the image modality to the title modality. The first title interaction feature and the second title interaction feature are concatenated to obtain a third title interaction feature. The third title interaction feature is input into the Transformers model for encoding and decoding, and the title interaction feature is output. Furthermore, with the text modality as the primary modality, the cross-modal Transformers model is used to extract the first text interaction feature from the title modality to the text modality, and the second text interaction feature from the image modality to the text modality. The first text interaction feature and the second text interaction feature are concatenated to obtain a third text interaction feature. The third text interaction feature is input into the Transformers model for encoding and decoding, and the text interaction feature is output. In addition, with the picture modality as the main modality, the cross-modal Transformers model is used to extract the first picture interaction feature from the title modality → picture modality, and the second picture interaction feature from the text modality → picture modality. The first picture interaction feature and the second picture interaction feature are spliced ​​together to obtain the third picture interaction feature. The third picture interaction feature is input into the Transformers model for encoding and then decoding, and the second interaction feature is output.

[0259] Furthermore, the title interaction feature, the text interaction feature, and the second interaction feature are concatenated to obtain a final cross-modal interaction feature among the three modalities. This cross-modal interaction feature is then used to predict the article category (Classification) to which the target article ultimately belongs. It should be noted that in the case of trimodal fusion, a relative position encoding method can also be introduced based on a method similar to that in the above embodiment, which will not be elaborated here.

[0260] Compared with traditional multimodal fusion methods, due to the different sampling rates of each modality, the data of each modality is essentially non-aligned, and there are long-term dependencies between the elements of different modalities. Traditional multimodal fusion methods cannot improve these two problems, and therefore the recognition accuracy of article categories is low. The above-mentioned multimodal fusion network constructs a cross-modal interaction method in the non-aligned case of high-quality image and text recognition. The cross-modal interaction part of the three modalities can fully utilize the sequence-level interaction information between the modalities. After the cross-modal interaction model of each of the six combination methods fuses the information of two modalities, the self-attention-based Transformers model is used to continue modeling in combination with the context. Finally, the three sets of features (the title interaction feature, the text interaction feature, and the second interaction feature) are spliced ​​for prediction. Through longitudinal comparative experiments, it is found that the model effect is best in the three-way combination scenario of title, text, and image, that is, the interaction information between any two modalities has a significant enhancing effect on the model effect.

[0261] The multimodal fusion network combined with relative position encoding can be applied to high-quality image and text recognition scenarios. In this scenario, the modal interactivity between adjacent images and text is crucial. At the same time, by introducing relative position encoding, it can enhance the learning of the relative position relationship between text and image sequences, thereby improving the recognition accuracy of the overall model. In addition, in the era of self-media, many factors affect the evaluation of article quality. In addition to text quality, the overall matching effect between images and text is also crucial. The multimodal fusion network combined with relative position encoding completes the construction of a multimodal module for high-quality image and text recognition scenarios.

[0262] In the test task of judging the quality of graphic content in the content center, the model evaluation accuracy reached 95%. However, traditional supervised methods of identifying high-quality graphics and texts, such as judging content quality only from the text perspective, or simply splicing text embeddings and image embeddings to judge content quality, consider very single dimensions and are unable to learn the modal interaction information between adjacent texts and images. As a result, the overall accuracy is less than 95%. Therefore, the method provided in the embodiment of the present application can greatly improve the recognition accuracy for article categories.

[0263] In addition, in the above-mentioned test tasks, the coverage rate of high-quality text and image content reached 17%. By conducting a recommendation weighting experiment on the identified high-quality text and image content on the browser side, it was achieved that high-quality content with good text and image matching effects and excellent user experience was recommended to users first, and good business results were achieved on the business side compared to historical application versions. Schematically, in the content processing link of the content center, all text and image content is scored for content quality, then released and distributed to the end side. The end side performs hierarchical recommendation weighting based on the content quality score, for example, recommending weighted recommendations for identified high-quality content and downgrading recommendations for low-quality content. This recommendation method can effectively improve the user's reading experience and is an innovation in recommendation algorithms based on specific business scenarios.

[0264] In addition, after using the image and text prior quality recognition algorithm provided in the embodiment of the present application to conduct a weighted recommendation experiment for high-quality content, the overall click PV (Page View) on the browser side increased by 0.38%, the exposure efficiency increased by 0.43%, the CTR (Click-Through-Rate) increased by 0.394%, and the user's stay time increased by 0.17%; at the same time, the next-day retention of DAU (Daily Active User) increased by 0.165%, and the average sharing per person in the interactive indicator data increased by 1.705%, the average like per person increased by 4.215%, and the average comment per person increased by 0.188%.

[0265] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0266] The method provided in the embodiment of the present application predicts the article category to which the target article belongs by extracting text features and image features from the text data and image data of the target article respectively, and utilizing the cross-modal interaction features between the two. At the same time, the method takes into account the contribution of the text modality and the image modality to the article category, rather than judging only from the text perspective. In addition, the extracted cross-modal interaction features are not a simple splicing of text features and image features, but can reflect richer and deeper inter-modal interaction information, greatly improving the accuracy of identifying article categories, and thus improving the accuracy of mining high-quality articles in the scenario of identifying high-quality articles.

[0267] Figure 9 This is a structural diagram of a data processing device provided in an embodiment of the present application. Please refer to Figure 9 , the device comprises:

[0268] A first acquisition module 901 is used to acquire text features and image features of a target article, wherein the text features are used to characterize text data in the target article, and the image features are used to characterize image data in the target article;

[0269] A second acquisition module 902 is configured to acquire a first interaction feature for the text feature based on a portion of the image feature associated with the text feature, wherein the first interaction feature is used to represent the text feature integrated with the image feature;

[0270] A third acquisition module 903 is configured to acquire a second interactive feature for the image feature based on a portion of the text feature associated with the image feature, wherein the second interactive feature is used to represent the image feature integrated with the text feature;

[0271] A fusion module 904 is configured to fuse the first interaction feature with the second interaction feature to obtain a cross-modal interaction feature;

[0272] The determination module 905 is configured to determine the article category to which the target article belongs based on the cross-modal interaction feature.

[0273] The device provided in the embodiment of the present application extracts text features and image features from the text data and image data of the target article respectively, and uses the cross-modal interaction features between the two to predict the article category to which the target article belongs. At the same time, it takes into account the contribution of the text modality and the image modality to the article category, rather than judging only from the text perspective. In addition, the extracted cross-modal interaction features are not a simple splicing of text features and image features, but can reflect richer and deeper inter-modal interaction information, greatly improving the accuracy of identifying article categories, and thus improving the accuracy of mining high-quality articles in the scenario of identifying high-quality articles.

[0274] In one possible implementation, based on Figure 9 The first acquisition module 901 includes:

[0275] A first extraction and fusion unit is used to extract text semantic features of the text data, and fuse the text semantic features with text position features to obtain the text features;

[0276] The second extraction and fusion unit is used to extract the image depth feature of the image data, and fuse the image depth feature with the image position feature to obtain the image feature.

[0277] In one possible implementation, the text data includes title data and text data; the text features include title features and text features;

[0278] The first extraction and fusion unit is used for:

[0279] Extracting title semantic features of the title data and text semantic features of the text data;

[0280] The title semantic feature is fused with the title position feature to obtain the title feature;

[0281] The text semantic feature is fused with the text position feature to obtain the text feature.

[0282] In a possible implementation, the first interaction feature includes a title interaction feature and a text interaction feature. Figure 9 The second acquisition module 902 includes:

[0283] A first acquisition unit is configured to acquire, from the title feature, a title interaction feature based on the text feature and the partial features of the image feature that are respectively associated with the title feature, the title interaction feature being used to represent the title feature after the text feature and the image feature are integrated;

[0284] The second acquisition unit is used to obtain the text interaction feature based on the part of the title feature and the picture feature that are respectively associated with the text feature, wherein the text interaction feature is used to represent the text feature after the title feature and the picture feature are integrated.

[0285] In one possible implementation, the first acquiring unit is configured to:

[0286] Obtaining a first title interaction feature based on some features in the text feature that are associated with the title feature;

[0287] Obtaining a second title interaction feature based on some features of the image feature that are associated with the title feature;

[0288] splicing the first title interaction feature and the second title interaction feature to obtain a third title interaction feature;

[0289] The third title interaction feature is encoded and decoded to obtain the title interaction feature.

[0290] In one possible implementation, the second acquiring unit is configured to:

[0291] Obtaining a first text interaction feature based on some features of the title feature that are associated with the text feature;

[0292] Obtaining a second text interaction feature based on some features of the image feature that are associated with the text feature;

[0293] splicing the first text interaction feature and the second text interaction feature to obtain a third text interaction feature;

[0294] The third text interaction feature is encoded and decoded to obtain the text interaction feature.

[0295] In one possible implementation, based on Figure 9 The third acquisition module 903 includes:

[0296] The third acquisition unit is configured to acquire the second interaction feature based on the title feature and some features of the text feature that are respectively associated with the picture feature.

[0297] In one possible implementation, the third acquiring unit is configured to:

[0298] Obtaining a first picture interaction feature based on some features in the title feature that are associated with the picture feature;

[0299] Obtaining a second image interaction feature based on some features of the text feature that are associated with the image feature;

[0300] splicing the first image interaction feature and the second image interaction feature to obtain a third image interaction feature;

[0301] The third image interaction feature is encoded and decoded to obtain the second interaction feature.

[0302] In a possible implementation, the text position feature and the image position feature are both relative position features between the text data and the image data, and the relative position features are used to represent the sequence and distance between the text data and the image data.

[0303] In one possible implementation, the relative position feature is obtained by:

[0304] Determining position information of each of the plurality of texts and the plurality of images in the target article based on the plurality of texts in the text data and the plurality of images in the image data;

[0305] Based on the position information, a relative position coding matrix is ​​constructed, where any element in the relative position coding matrix is ​​used to represent the relative position information between the text corresponding to the column to which the element belongs and the image corresponding to the row to which the element belongs;

[0306] Based on the relative position encoding matrix, a relative position feature between any text in the plurality of texts and any picture in the plurality of pictures is determined.

[0307] In one possible implementation, the determining module 905 is configured to:

[0308] Perform full connection processing on the cross-modal interaction feature to obtain a fully connected feature;

[0309] Exponential normalization is performed on the fully connected features to obtain multiple predicted probabilities that the target article belongs to multiple categories;

[0310] The category corresponding to the predicted probability that meets the target conditions is selected as the article category to which the target article belongs.

[0311] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0312] It should be noted that the data processing device provided in the above embodiments is merely illustrated by the division of the above functional modules when processing data. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing device provided in the above embodiments and the data processing method embodiment are based on the same concept. The specific implementation process is detailed in the data processing method embodiment and will not be repeated here.

[0313] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Please refer to Figure 10 , taking a computer device as terminal 1000 as an example for explanation, in this case, terminal 1000 can independently complete the process of identifying the article category of the target article. Optionally, the device type of terminal 1000 includes: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), an MP4 player (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4), a laptop computer, or a desktop computer. Terminal 1000 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0314] Typically, the terminal 1000 includes a processor 1001 and a memory 1002 .

[0315] Optionally, the processor 1001 includes one or more processing cores, such as a 4-core processor, an 8-core processor, etc. Optionally, the processor 1001 is implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). In some embodiments, the processor 1001 includes a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1001 is integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1001 also includes an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0316] In some embodiments, the memory 1002 includes one or more computer-readable storage media, optionally, the computer-readable storage medium is non-transitory. Optionally, the memory 1002 also includes a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one program code, and the at least one program code is used to be executed by the processor 1001 to implement the data processing method provided in each embodiment of the present application.

[0317] In some embodiments, terminal 1000 may optionally include a peripheral device interface 1003 and at least one peripheral device. Processor 1001, memory 1002, and peripheral device interface 1003 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1003 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1009.

[0318] The peripheral device interface 1003 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 are implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0319] The RF circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1004 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. Optionally, the RF circuit 1004 communicates with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1004 also includes circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0320] The display screen 1005 is used to display a UI (User Interface). Optionally, the UI includes graphics, text, icons, videos, and any combination thereof. When the display screen 1005 is a touch screen display, the display screen 1005 also has the ability to collect touch signals on the surface or above the surface of the display screen 1005. The touch signal can be input as a control signal to the processor 1001 for processing. Optionally, the display screen 1005 is also used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there is one display screen 1005, which is set on the front panel of the terminal 1000; in other embodiments, there are at least two display screens 1005, which are respectively set on different surfaces of the terminal 1000 or in a folding design; in still other embodiments, the display screen 1005 is a flexible display screen, which is set on the curved surface or folding surface of the terminal 1000. Even, optionally, the display screen 1005 is set into a non-rectangular irregular shape, that is, a special-shaped screen. Optionally, the display screen 1005 is made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0321] The camera assembly 1006 is used to capture images or videos. Optionally, the camera assembly 1006 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1006 also includes a flash. Optionally, the flash is a monochrome temperature flash, or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which is used for light compensation at different color temperatures.

[0322] In some embodiments, the audio circuit 1007 includes a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 1001 for processing, or input into the radio frequency circuit 1004 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there are multiple microphones, which are respectively arranged in different parts of the terminal 1000. Optionally, the microphone is an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. Optionally, the speaker is a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1007 also includes a headphone jack.

[0323] Power supply 1009 is used to power various components in terminal 1000. Optionally, power supply 1009 is AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 1009 includes a rechargeable battery, the rechargeable battery supports wired charging or wireless charging. The rechargeable battery is also configured to support fast charging technology.

[0324] In some embodiments, the terminal 1000 further includes one or more sensors 1010 , including but not limited to: an acceleration sensor 1011 , a gyroscope sensor 1012 , a pressure sensor 1013 , an optical sensor 1015 , and a proximity sensor 1016 .

[0325] In some embodiments, the accelerometer 1011 detects the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 1000. For example, the accelerometer 1011 is used to detect the components of gravity acceleration along the three coordinate axes. Optionally, the processor 1001 controls the display screen 1005 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 1011. The accelerometer 1011 is also used to collect game or user motion data.

[0326] In some embodiments, the gyroscope sensor 1012 detects the orientation and rotation angle of the terminal 1000. The gyroscope sensor 1012 and the acceleration sensor 1011 work together to collect the user's 3D movements on the terminal 1000. The processor 1001 implements the following functions based on the data collected by the gyroscope sensor 1012: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0327] Optionally, the pressure sensor 1013 is provided on the side frame of the terminal 1000 and / or the lower layer of the display screen 1005. When the pressure sensor 1013 is provided on the side frame of the terminal 1000, it can detect the user's grip signal of the terminal 1000, and the processor 1001 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 1013. When the pressure sensor 1013 is provided on the lower layer of the display screen 1005, the processor 1001 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 1005. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0328] Optical sensor 1015 is used to detect ambient light intensity. In one embodiment, processor 1001 controls the display brightness of display screen 1005 based on the ambient light intensity detected by optical sensor 1015. Specifically, when the ambient light intensity is high, the display brightness of display screen 1005 is increased; when the ambient light intensity is low, the display brightness of display screen 1005 is decreased. In another embodiment, processor 1001 also dynamically adjusts the capture parameters of camera assembly 1006 based on the ambient light intensity detected by optical sensor 1015.

[0329] Proximity sensor 1016, also known as a distance sensor, is typically located on the front panel of terminal 1000. Proximity sensor 1016 is used to detect the distance between the user and the front of terminal 1000. In one embodiment, when proximity sensor 1016 detects that the distance between the user and the front of terminal 1000 is gradually decreasing, processor 1001 controls display screen 1005 to switch from the screen-on state to the screen-off state. When proximity sensor 1016 detects that the distance between the user and the front of terminal 1000 is gradually increasing, processor 1001 controls display screen 1005 to switch from the screen-off state to the screen-on state.

[0330] Those skilled in the art will understand that Figure 10 The structure shown in the figure does not constitute a limitation on the terminal 1000, and the terminal 1000 can include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0331] Figure 111 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 1100 may have relatively large differences due to different configurations or performance. The computer device 1100 includes one or more processors (Central Processing Units, CPU) 1101 and one or more memories 1102, wherein the memory 1102 stores at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1101 to implement the data processing methods provided in the above embodiments. Optionally, the computer device 1100 also has components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The computer device 1100 also includes other components for implementing device functions, which are not described here.

[0332] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program. The at least one computer program can be executed by a processor in a terminal to implement the data processing methods described in each of the above embodiments. For example, the computer-readable storage medium includes ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device.

[0333] In an exemplary embodiment, a computer program product or computer program is also provided, comprising one or more program codes stored in a computer-readable storage medium. One or more processors of a computer device can read the one or more program codes from the computer-readable storage medium, and the one or more processors can execute the one or more program codes, so that the computer device can perform the data processing method of the above-described embodiment.

[0334] Those skilled in the art will understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. Optionally, the program is stored in a computer-readable storage medium. Optionally, the above-mentioned storage medium is a read-only memory, a disk or an optical disk, etc.

[0335] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Determining position information of each of the plurality of texts and the plurality of images in the target article based on the plurality of texts in the text data of the target article and the plurality of images in the image data of the target article; Based on the position information, a relative position coding matrix is ​​constructed, wherein any element in the relative position coding matrix is ​​used to represent the relative position information between the text corresponding to the column to which the element belongs and the picture corresponding to the row to which the element belongs; Determine, based on the relative position coding matrix, a relative position feature between the text data and the image data, wherein the relative position feature is used to characterize the order and distance between the text data and the image data, and the relative position feature includes a text position feature and an image position feature; Extracting text semantic features of the text data, fusing the text semantic features with the text position features to obtain text features, wherein the text features are used to characterize the text data; Extracting a picture depth feature of the picture data, fusing the picture depth feature with the picture position feature to obtain a picture feature, wherein the picture feature is used to characterize the picture data; For the text feature, based on some features of the image feature that are associated with the text feature, a first interaction feature is obtained, where the first interaction feature is used to represent the text feature integrated with the image feature; For the image feature, based on some features of the text feature associated with the image feature, obtaining a second interactive feature, wherein the second interactive feature is used to represent the image feature integrated with the text feature; fusing the first interaction feature with the second interaction feature to obtain a cross-modal interaction feature; Based on the cross-modal interaction features, the article category to which the target article belongs is determined.

2. The method according to claim 1, characterized in that The text data includes title data and text data; the text features include title features and text features; The extracting of text semantic features of the text data comprises: extracting title semantic features of the title data and text semantic features of the text data; The text features are obtained by fusing the text semantic features with the text position features, including: Fusing the title semantic feature with the title position feature to obtain the title feature; The text semantic feature and the text position feature are fused to obtain the text feature.

3. The method according to claim 2, characterized in that The first interaction feature includes a title interaction feature and a text interaction feature. The obtaining of the first interaction feature based on a portion of the image features associated with the text features includes: For the title feature, based on partial features of the text feature and the image feature that are respectively associated with the title feature, obtaining the title interaction feature, wherein the title interaction feature is used to represent the title feature after the text feature and the image feature are integrated; For the text features, the text interaction features are obtained based on the partial features of the title features and the picture features respectively associated with the text features. The text interaction features are used to represent the text features after the title features and the picture features are integrated.

4. The method according to claim 3, characterized in that The acquiring of the title interaction feature based on partial features of the text feature and the image feature that are respectively associated with the title feature includes: Obtaining a first title interaction feature based on some features of the text features that are associated with the title features; obtaining a second title interaction feature based on some features of the image feature that are associated with the title feature; concatenating the first title interaction feature and the second title interaction feature to obtain a third title interaction feature; The third title interaction feature is encoded and decoded to obtain the title interaction feature.

5. The method according to claim 3, characterized in that The acquiring of the text interaction feature based on partial features of the title feature and the image feature that are respectively associated with the text feature includes: Obtaining a first text interaction feature based on some features of the title feature that are associated with the text feature; Obtaining a second text interaction feature based on some features of the image feature that are associated with the text feature; concatenating the first text interaction feature and the second text interaction feature to obtain a third text interaction feature; The third text interaction feature is encoded and decoded to obtain the text interaction feature.

6. The method according to claim 2, characterized in that The acquiring, for the picture feature, a second interaction feature based on some features of the text feature associated with the picture feature comprises: For the picture feature, the second interaction feature is obtained based on partial features of the title feature and the text feature that are respectively associated with the picture feature.

7. The method according to claim 6, characterized in that The acquiring of the second interaction feature based on the partial features of the title feature and the text feature respectively associated with the picture feature includes: Obtaining a first picture interaction feature based on some features of the title feature that are associated with the picture feature; Obtaining a second image interaction feature based on some features of the text feature that are associated with the image feature; concatenating the first image interaction feature and the second image interaction feature to obtain a third image interaction feature; The third image interaction feature is encoded and decoded to obtain the second interaction feature.

8. The method according to claim 1, characterized in that Determining the article category to which the target article belongs based on the cross-modal interaction feature includes: Performing full connection processing on the cross-modal interaction features to obtain fully connected features; Performing exponential normalization on the fully connected features to obtain multiple predicted probabilities that the target article belongs to multiple categories; The category corresponding to the predicted probability that meets the target condition is selected as the article category to which the target article belongs.

9. A data processing device, characterized in that: The device comprises: A first acquisition module is configured to determine position information of each of the plurality of texts and the plurality of images in the target article based on the plurality of texts in the text data of the target article and the plurality of images in the image data of the target article; construct a relative position coding matrix based on the position information, wherein any element in the relative position coding matrix is ​​used to represent relative position information between the text corresponding to the column to which the element belongs and the image corresponding to the row to which the element belongs; and determine relative position features between the text data and the image data based on the relative position coding matrix, wherein the relative position features are used to represent the order and distance between the text data and the image data, and the relative position features include text position features and image position features; The first acquisition module is further configured to extract text semantic features of the text data, fuse the text semantic features with the text position features to obtain text features, and the text features are used to characterize the text data; The first acquisition module is further configured to extract a picture depth feature of the picture data, and fuse the picture depth feature with the picture position feature to obtain a picture feature, wherein the picture feature is used to represent the picture data; A second acquisition module is configured to acquire, for the text feature, a first interaction feature based on a portion of features in the image feature that are associated with the text feature, wherein the first interaction feature is used to represent the text feature integrated with the image feature; A third acquisition module is configured to acquire, for the image feature, a second interaction feature based on a portion of the text feature associated with the image feature, wherein the second interaction feature is used to represent the image feature integrated with the text feature; a fusion module, configured to fuse the first interaction feature with the second interaction feature to obtain a cross-modal interaction feature; A determination module is used to determine the article category to which the target article belongs based on the cross-modal interaction feature.

10. The device according to claim 9, characterized in that The text data includes title data and text data; the text features include title features and text features; the first acquisition module is further used to: extracting title semantic features of the title data and text semantic features of the text data; The text features are obtained by fusing the text semantic features with the text position features, including: Fusing the title semantic feature with the title position feature to obtain the title feature; The text semantic feature and the text position feature are fused to obtain the text feature.

11. The device according to claim 10, characterized in that The first interaction feature includes a title interaction feature and a text interaction feature, and the second acquisition module includes: A first acquisition unit is configured to acquire, from the title feature, the title interaction feature based on partial features of the text feature and the image feature that are respectively associated with the title feature, the title interaction feature being used to represent the title feature after the text feature and the image feature are integrated; The second acquisition unit is used to acquire the text interaction feature based on the part of the title feature and the picture feature that are respectively associated with the text feature, wherein the text interaction feature is used to represent the text feature after the title feature and the picture feature are integrated.

12. The device according to claim 11, characterized in that The first acquiring unit is configured to: Obtaining a first title interaction feature based on some features of the text features that are associated with the title features; obtaining a second title interaction feature based on some features of the image feature that are associated with the title feature; concatenating the first title interaction feature and the second title interaction feature to obtain a third title interaction feature; The third title interaction feature is encoded and decoded to obtain the title interaction feature.

13. The device according to claim 11, characterized in that The second acquiring unit is configured to: Obtaining a first text interaction feature based on some features of the title feature that are associated with the text feature; Obtaining a second text interaction feature based on some features of the image feature that are associated with the text feature; concatenating the first text interaction feature and the second text interaction feature to obtain a third text interaction feature; The third text interaction feature is encoded and decoded to obtain the text interaction feature.

14. The device according to claim 10, characterized in that The third acquisition module includes: The third acquisition unit is configured to acquire the second interaction feature based on the picture feature and some features of the title feature and the text feature that are respectively associated with the picture feature.

15. The device according to claim 14, characterized in that The third acquiring unit is used for: Obtaining a first picture interaction feature based on some features of the title feature that are associated with the picture feature; Obtaining a second image interaction feature based on some features of the text feature that are associated with the image feature; concatenating the first image interaction feature and the second image interaction feature to obtain a third image interaction feature; The third image interaction feature is encoded and decoded to obtain the second interaction feature.

16. The device according to claim 9, characterized in that The determining module is used for: Performing full connection processing on the cross-modal interaction features to obtain fully connected features; Performing exponential normalization on the fully connected features to obtain multiple predicted probabilities that the target article belongs to multiple categories; The category corresponding to the predicted probability that meets the target condition is selected as the article category to which the target article belongs.

17. A computer device, characterized in that: The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the data processing method according to any one of claims 1 to 8.

18. A storage medium, characterized in that The storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the data processing method according to any one of claims 1 to 8.

19. A computer program product, characterized in that The computer program product includes at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the data processing method according to any one of claims 1 to 8.

Images