Text entity recognition method, device, equipment, storage medium and program product

Abstract

The present application provides a method, apparatus, device, storage medium, and program product for entity recognition of text. The method comprises: obtaining a text to be recognized and an image to be recognized associated with the text to be recognized; encoding the image to be recognized to obtain an image vector, and encoding the text to be recognized to obtain a text vector; modally fusing the image vector and the text vector to obtain a target multimodal vector; performing entity recognition on the text to be recognized based on the target multimodal vector to obtain an initial entity in the text to be recognized; performing entity recognition on the text to be recognized based on the text vector to obtain a revised entity in the text to be recognized; and revising the initial entity based on the revised entity to obtain a target entity in the text to be recognized. The present application can effectively improve the accuracy of entity recognition.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text entity recognition method, apparatus, device, storage medium, and program product. Background Art

[0002] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0003] In related technologies, pre-established rules are usually used to perform entity recognition on the text to be recognized. For example, to recognize the prices of items in the text, if the prices of all commodities in the text to be recognized are in the form of "number + yuan", entity recognition can be performed using regular expressions. As the amount of corpus increases, conflicts may occur between the preset rules, making the accuracy of entity recognition extremely low. Summary of the Invention

[0004] The embodiments of the present application provide a text entity recognition method, device, electronic device, computer-readable storage medium and computer program product, which can effectively improve the accuracy of entity recognition.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] This embodiment of the present application provides a method for identifying text entities, including:

[0007] Acquire a text to be recognized and an image to be recognized associated with the text to be recognized;

[0008] Encoding the image to be recognized to obtain an image vector, and encoding the text to be recognized to obtain a text vector;

[0009] Performing modal fusion on the image vector and the text vector to obtain a target multimodal vector;

[0010] Performing entity recognition on the text to be recognized based on the target multimodal vector to obtain an initial entity in the text to be recognized; performing entity recognition on the text to be recognized based on the text vector to obtain a revised entity in the text to be recognized;

[0011] Based on the modified entity, the initial entity is modified to obtain a target entity in the text to be recognized.

[0012] The present invention provides a text entity recognition device, comprising:

[0013] An acquisition module, configured to acquire text to be recognized and an image to be recognized associated with the text to be recognized;

[0014] an encoding module, configured to encode the image to be recognized to obtain an image vector, and to encode the text to be recognized to obtain a text vector;

[0015] A modality fusion module, configured to perform modality fusion on the image vector and the text vector to obtain a target multimodal vector;

[0016] An entity recognition module is configured to perform entity recognition on the text to be recognized based on the target multimodal vector to obtain an initial entity in the text to be recognized; and perform entity recognition on the text to be recognized based on the text vector to obtain a revised entity in the text to be recognized;

[0017] The correction module is used to correct the initial entity based on the correction entity to obtain the target entity in the text to be recognized.

[0018] In some embodiments, the modal fusion is implemented through a modal fusion network; the modal fusion network includes a first modal fusion network, a second modal fusion network, and a third modal fusion network; the above-mentioned modal fusion module is also used to call the first modal fusion network to perform modal fusion on the image vector and the text vector to obtain a first multimodal vector; call the second modal fusion network to perform modal fusion on the image vector and the text vector to obtain a second multimodal vector; call the third modal fusion network to perform modal fusion on the text vector and the first multimodal vector to obtain a third multimodal vector; perform weighted fusion on the third multimodal vector and the second multimodal vector to obtain a weighted fusion vector; and perform vector splicing on the weighted fusion vector and the third multimodal vector to obtain the target multimodal vector.

[0019] In some embodiments, the first modal fusion network includes a multi-head attention network, a residual connection network and a feedforward neural network; the above-mentioned modal fusion module is also used to call the multi-head attention network to perform modal fusion on the image vector and the text vector to obtain a fourth multimodal vector; call the residual connection network to perform residual connection on the fourth multimodal vector and the image vector to obtain a residual connection result; call the feedforward neural network to perform activation processing on the residual connection result to obtain an activation result; call the residual connection network to perform residual connection on the activation result and the residual connection result to obtain the first multimodal vector.

[0020] In some embodiments, the above-mentioned modal fusion module is also used to determine the image vector as the query vector of the first modal fusion network, and to determine the text vector as the key vector and value vector of the first modal fusion network respectively; call the multi-head attention network to perform modal fusion on the query vector, the key vector and the value vector to obtain the fourth multimodal vector; the above-mentioned modal fusion module is also used to call the residual connection network to perform residual connection on the fourth multimodal vector and the query vector to obtain the residual connection result.

[0021] In some embodiments, the network structures of the second modal fusion network, the third modal fusion network and the first modal fusion network are the same; the above-mentioned modal fusion module is further used to determine the text vector as the query vector of the second modal fusion network, and determine the image vector as the key vector and value vector of the second modal fusion network respectively; call the second modal fusion network to perform modal fusion on the query vector, the key vector and the value vector to obtain a second multimodal vector; the above-mentioned modal fusion module is further used to determine the text vector as the query vector of the third modal fusion network, and determine the first multimodal vector as the key vector and value vector of the third modal fusion network respectively; call the third modal fusion network to perform modal fusion on the query vector, the key vector and the value vector to obtain a third multimodal vector.

[0022] In some embodiments, the above-mentioned modal fusion module is also used to obtain a first weight matrix of the third multimodal vector and a second weight matrix of the second multimodal vector; multiply the third multimodal vector and the first weight matrix to obtain a first product result; multiply the second multimodal vector and the second weight matrix to obtain a second product result; add the first product result and the second product result to obtain a sum result; normalize the sum result to obtain a normalized vector; multiply the normalized vector and the second multimodal vector to obtain the weighted fusion vector.

[0023] In some embodiments, the target multimodal vector includes word vectors corresponding to each word in the text to be recognized, and the above-mentioned entity recognition module is also used to perform the following processing on the word vectors corresponding to each word in the text to be recognized: based on the word vectors, the entity category of the word is predicted to obtain the initial category probability values ​​of the word corresponding to each category; the category corresponding to the largest initial category probability value is determined as the initial category corresponding to the word; when the initial category represents that the word is a named entity, the entity category of the named entity is obtained, and the word is determined as the initial entity corresponding to the entity category in the text to be recognized.

[0024] In some embodiments, the text vector includes a word text vector corresponding to each word in the text to be recognized, and the above-mentioned entity recognition module is also used to perform the following processing on the word text vector corresponding to each word in the text to be recognized: based on the word text vector, the entity category of the word is predicted to obtain the corrected category probability value of the word corresponding to each category; the category corresponding to the largest corrected category probability value is determined as the corrected category corresponding to the word; when the corrected category represents that the word is a named entity, the entity category of the named entity is obtained, and the word is determined as the corrected entity corresponding to the entity category in the text to be recognized.

[0025] In some embodiments, the number of the revised entity and the number of the initial entity are both at least one, and the revision module is further used to perform the following processing for each revised entity: comparing the revised entity with each initial entity to obtain a comparison result, wherein the comparison result is used to characterize whether there is an initial entity identical to the revised entity in the at least one initial entity; in response to the comparison result characterizing the existence of the initial entity identical to the revised entity, the initial entity identical to the revised entity is determined as a candidate entity in the text to be recognized; in response to the comparison result characterizing the absence of the initial entity identical to the revised entity, the word text vector and the word vector are spliced ​​to obtain a spliced ​​vector; performing entity recognition on the text to be recognized based on the spliced ​​vector to obtain a candidate entity in the text to be recognized; calling the entity knowledge graph to perform entity disambiguation on the candidate entity to obtain a target entity in the text to be recognized.

[0026] In some embodiments, the splicing vector includes word splicing vectors corresponding to each word in the text to be recognized, and the correction module is further used to perform the following processing on the word splicing vectors corresponding to each word in the text to be recognized: based on the word splicing vector, the entity category of the word is predicted to obtain the target category probability value of each category corresponding to the word; the category corresponding to the largest target category probability value is determined as the target category corresponding to the word; when the target category represents that the word is a named entity, the entity category of the named entity is obtained, and the word is determined as a candidate entity corresponding to the entity category in the text to be recognized.

[0027] In some embodiments, the above-mentioned encoding module is also used to determine the context vector, word vector and position vector corresponding to each word in the text to be recognized; sum the context vector, the word vector and the position vector to obtain a fusion vector; encode the fusion vector to obtain the text vector; the above-mentioned encoding module is also used to obtain the dimension of the text vector, and based on the dimension of the text vector, resize the image to be recognized to obtain a standard image; encode the standard image to obtain an image vector with the same dimension as the text vector.

[0028] An embodiment of the present application provides an electronic device, including:

[0029] a memory for storing executable instructions;

[0030] The processor is used to implement the text entity recognition method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0031] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute instructions to implement the text entity recognition method provided in the embodiment of the present application.

[0032] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the text entity recognition method described in the present application.

[0033] The embodiments of the present application have the following beneficial effects:

[0034] By combining the text to be recognized and the image to be recognized associated with the text to be recognized, a target multimodal vector is determined. Entity recognition is performed on the text to be recognized based on the target multimodal vector, thereby accurately determining the initial entity in the text to be recognized. Since the determined initial entity comprehensively considers the image to be recognized associated with the text to be recognized, the semantic information of the image to be recognized assists in entity recognition of the text to be recognized, thereby effectively improving the accuracy of the determined initial entity. At the same time, by performing entity recognition on the text to be recognized based on the text vector, a corrected entity is obtained. The initial entity combined with the semantic information of the image to be recognized is corrected by the corrected entity to obtain the target entity in the text to be recognized. On the one hand, by performing entity recognition in combination with the associated image to be recognized, the accuracy of entity recognition is effectively improved; on the other hand, by correcting the entity, any recognition errors that may exist in the initial entity are corrected to obtain the target entity, thereby further improving the accuracy of entity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the structure of the text entity recognition system architecture provided by the embodiment of the present application;

[0036] Figure 2 Schematic diagram of the structure of the text entity recognition device provided in the embodiment of the present application;

[0037] Figures 3A to 3E Schematic diagram of the flow of the text entity recognition method provided by the embodiment of the present application;

[0038] Figures 4A to 4D Schematic diagram of the principle of the text entity recognition method provided by the embodiment of the present application;

[0039] Figure 4E It is a schematic diagram of the effect of the text entity recognition method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0041] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0044] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0045] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0046] 1) Artificial Intelligence (AI): The theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics.

[0047] 2) Convolutional Neural Networks (CNN): These are a type of feedforward neural network (FNN) that incorporates convolutional computations and possesses a deep structure. They are a representative algorithm for deep learning. CNNs possess representation learning capabilities and can perform shift-invariant classification on input images based on their hierarchical structure.

[0048] 3) Conditional Random Field (CRF): A discriminative probability model and a type of random field, commonly used to annotate or analyze sequence data, such as natural language text or biological sequences. A CRF is a conditional probability distribution model P(Y|X), representing a Markov random field of output random variables Y given a set of input random variables X. In other words, a CRF assumes that the output random variables form a Markov random field. CRFs can be viewed as a generalization of the maximum entropy Markov model for annotation problems.

[0049] 4) Named Entity Recognition (NER): It is an important basic tool in application fields such as information extraction, question-answering systems, syntactic analysis, and machine translation, and plays a vital role in the practical application of natural language processing technology.

[0050] 5) Self-Attention Network: It is an attention mechanism that focuses on the correlation between different parts of the entire input.

[0051] 6) Natural Language Processing (NLP): A key area of ​​research in computer science and artificial intelligence, NLP studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0052] 7) Feedforward Neural Network (FFN): This is an artificial neural network in which neurons are arranged in layers, with each neuron connected only to the neurons in the previous layer. Each layer receives the output of the previous layer and outputs it to the next layer, with no feedback between layers. Feedforward neural networks include perceptron networks, backpropagation networks, and radial basis function (RBF) networks. The perceptron network is the simplest feedforward network and is primarily used for pattern classification, but can also be used in learning control and multimodal control based on pattern classification. Perceptron networks can be divided into single-layer and multi-layer perceptron networks. A backpropagation network is a feedforward network that uses the backpropagation learning algorithm to adjust connection weights. Unlike a perceptron, the BP network uses a sigmoid function as its neuron transformation function, resulting in a continuous output between 0 and 1, enabling arbitrary nonlinear mapping from input to output. An RBF network is a feedforward network whose hidden layer neurons consist of RBF neurons. RBF neurons are neurons whose transformation function is the RBF (Radial Basis Function). A typical RBF network consists of three layers: an input layer, one or more RBF layers (hidden layers) composed of RBF neurons, and an output layer composed of linear neurons.

[0053] 8) Residual Network (Res Net): Also known as a residual network, this network is easy to optimize and can improve accuracy by increasing its depth. Its internal residual blocks use skip connections, which alleviates the vanishing gradient problem associated with increasing the depth of deep neural networks.

[0054] 9) Long Short-Term Memory (LSTM): This is a type of recurrent neural network specifically designed to address the long-term dependency issues common in recurrent neural networks. All recurrent neural networks have a chain-like structure of repeating neural network modules. In standard recurrent neural networks, this repeating module has a very simple structure.

[0055] During the implementation of the embodiments of this application, the applicant discovered that the related technology has the following problems:

[0056] Named Entity Recognition (NER) is a subtask of information extraction that can locate and classify named entities in text into pre-defined categories, such as people, organizations, locations, time expressions, quantities, percentages, etc.

[0057] In the related technology of named entity recognition, entity recognition is usually performed directly on the text to be recognized, thereby determining the entities in the text to be recognized. For entities with special context in the text to be recognized, or text in which the entity itself has multiple features, pre-established rules are used to perform entity recognition on the text to be recognized. For example, to recognize the price of an item in a text, if the price of all goods in the text to be recognized is in the form of "number + yuan", regular expressions can be used to extract them. As the amount of corpus increases, the situation will become more complicated, and conflicts may occur between the preset rules. The entire recognition system may also become unmaintainable. Therefore, this entity recognition method has poor universality and extremely low recognition accuracy.

[0058] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium, and computer program product for text entity recognition, which can effectively improve the accuracy of entity recognition. The following describes an exemplary application of the text entity recognition device provided by the embodiments of the present application. The device provided by the embodiments of the present application can be implemented as various types of user terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as a server.

[0059] See also Figure 1 , Figure 1 This is an architectural diagram of a text entity recognition system 100 provided in an embodiment of the present application. To implement an application scenario of entity recognition, a terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0060] The terminal 400 is used for the user to use the client 410, which is displayed on a graphical interface 410-1 (graphic interface 410-1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.

[0061] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart voice interaction device, smart home appliance, car terminal, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0062] In some embodiments, the server 200 obtains the text to be recognized from the terminal 400 , determines a target entity in the text to be recognized, and sends the determined target entity to the terminal 400 .

[0063] In other embodiments, the terminal 400 obtains the text to be recognized, determines the target entity in the text to be recognized, and sends the determined target entity to the server 200 .

[0064] In other embodiments, the embodiments of the present application can be implemented with the help of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or local area network to realize data calculation, storage, processing, and sharing.

[0065] Cloud technology is a general term for network, information, integration, management platform, and application technologies used in the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a key support. The backend services of technical network systems require a large amount of computing and storage resources.

[0066] See also Figure 2 , Figure 2 2 is a structural diagram of a server 200 of the text entity recognition method provided in an embodiment of the present application. Figure 2 The server 200 shown includes: at least one processor 210, a memory 250, and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0067] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0068] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0069] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0070] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0071] The operating system 251 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks.

[0072] The network communication module 252 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 220. Exemplary network interfaces 220 include Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB).

[0073] In some embodiments, the text entity recognition device provided by the embodiments of the present application can be implemented in a software manner. Figure 2 Entity recognition device 255 for text stored in memory 250 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: acquisition module 2551, encoding module 2552, modality fusion module 2553, entity recognition module 2554, and correction module 2555. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0074] In other embodiments, the text entity recognition device provided in the embodiments of the present application can be implemented in hardware. As an example, the text entity recognition device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the text entity recognition method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0075] The text entity recognition method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server or terminal provided in the embodiment of the present application.

[0076] See also Figure 3A , Figure 3A This is a flow chart of the text entity recognition method provided by the embodiment of the present application, which will be combined with Figure 3A Steps 101 to 105 are shown for illustration. The execution subject of the following steps 101 to 105 can be a server or a terminal. The following description will be made by taking the execution subject as a server as an example.

[0077] In step 101, a text to be recognized and an image to be recognized associated with the text to be recognized are obtained.

[0078] In some embodiments, in audio and video application scenarios, the text to be recognized can be the complete subtitle text of the video to be played, and the image to be recognized associated with the text to be recognized can be each image frame of the video to be played. Then, an image to be recognized can correspond to at least one sentence in the text to be recognized, that is, there are subtitles in the video. An image to be recognized can also not correspond to the text to be recognized, that is, there are no subtitles in the video. Each sentence in the text to be recognized corresponds to at least one image to be recognized, that is, for each subtitle in the video, there is an image to be recognized associated with the subtitle.

[0079] As an example, the text to be recognized is "Xiao Ming: Have you eaten? Xiao Zhang: I just finished my meal", and the images to be recognized associated with the text to be recognized may be Image 1 and Image 2. Then, Image 1 may be the image to be recognized associated with the sentence "Xiao Ming: Have you eaten?" in the text to be recognized, and Image 2 may be the image to be recognized associated with the sentence "Xiao Zhang: I just finished my meal" in the text to be recognized. The picture content of Image 1 includes Xiao Ming, and the picture content of Image 2 includes Xiao Zhang.

[0080] In some embodiments, in mobile app (APP) applications, for example, news apps, the text to be recognized can be any piece of news content within the app, and the image to be recognized associated with the text to be recognized can be a news image corresponding to the news text. Thus, one image to be recognized can correspond to at least one sentence in the text to be recognized, meaning that the news image and at least one sentence in the news text correspond, effectively improving the readability of the news.

[0081] As an example, the text to be recognized obtained is "Today the weather is sunny, and tourists are going out; tomorrow it will be cloudy and turn to heavy rain, so it is recommended to take an umbrella when going out." The images to be recognized associated with the text to be recognized may be image 3 and image 4. Then, image 3 may be the image to be recognized associated with the sentence in the text to be recognized "Today the weather is sunny, and tourists are going out" and image 4 may be the image to be recognized associated with the sentence in the text to be recognized "Tomorrow it will be cloudy and turn to heavy rain, so it is recommended to take an umbrella when going out." The picture content of image 3 represents sunny weather, and the picture content of image 4 represents rain.

[0082] In step 102, the image to be recognized is encoded to obtain an image vector, and the text to be recognized is encoded to obtain a text vector.

[0083] In some embodiments, the image vector is a vector representation of the image to be recognized, and the text vector is a vector representation of the text to be recognized.

[0084] In some embodiments, encoding the text to be recognized in the above step 102 to obtain a text vector can be achieved in the following manner: determining the context vector, word vector and position vector corresponding to each word in the text to be recognized; summing the context vector, word vector and position vector to obtain a fusion vector; encoding the fusion vector to obtain a text vector.

[0085] In some embodiments, the context vector corresponding to a word represents the semantic information of the context of the word. Through the context vector, the semantic information of the context of each word in the text to be recognized can be accurately represented.

[0086] As an example, in a text, a word may have multiple different semantic meanings, and its semantic meaning can be accurately recognized only in combination with the context. For example, the word "day" has different semantic meanings in different contexts. In the context of "heaven, earth, and man", and in the context of "oh my god", the semantic meaning of the word "day" is completely different.

[0087] In some embodiments, the word vector corresponding to a word represents the semantic information of the word itself, and the position vector corresponding to the word represents the relative position of the word in the text to be recognized. For example, when the text to be recognized is "Xiaozhang is a chubby young man", the position of the word "small" in the text to be recognized is the 9th.

[0088] In some embodiments, the above-mentioned encoding of the fusion vector to obtain the text vector can be achieved by the following method: call a bidirectional encoding network to encode the fusion vector to obtain the text vector.

[0089] As an example, see Figure 4A , Figure 4A is a schematic diagram of the principle of the entity recognition method for the text provided in the embodiments of the present application. Figure 4A The text to be recognized shown is "The article is so... beautiful", and the context vectors corresponding to each word in the text to be recognized are: E0, E1, E2, E3... E A , A , A , [SEP] , A , A , A , Jordan ,E n+1 ;The word vectors corresponding to each word in the text to be recognized are: E [cls] ,E Kevin ,E Durant ,E enters ……E Jordan ,E [SEP] ;The position vectors corresponding to each word in the text to be recognized are: E A ,E A ,E A ,E A ……E A ,E A 。

[0090] As an example, see Figure 4A ,The context vectors: E0, E1, E2, E3... E n ,E n+1 、The word vectors: E [cls] ,E Kevin ,E Durant ,E enters ……E Jordan ,E [SEP] and the position vectors: E [[ID=6%]] A ,E A ,E A ,E A ……E A ,E ASum up and get the fusion vector; call the bidirectional encoding network to encode the fusion vector and get the text vector: c0, c1, c2, c3...c n , c n+1 .

[0091] In some embodiments, a bidirectional encoder representation from a transformer (BERT) network can be a pre-trained language representation model. Instead of using a traditional unidirectional language model or a shallow concatenation of two unidirectional language models for pre-training, BERT uses a new masked language model (MLM) to generate deep bidirectional language representations. The goal of a bidirectional encoder network is to train on large-scale unlabeled corpora and obtain results that contain rich semantic information, namely, semantic representations of text.

[0092] In some embodiments, encoding the image to be recognized in the above step 102 to obtain an image vector can be achieved in the following manner: obtaining the dimension of the text vector, and based on the dimension of the text vector, resizing the image to be recognized to obtain a standard image; encoding the standard image to obtain an image vector with the same dimension as the text vector.

[0093] In some embodiments, the dimension of a vector refers to the number of components included in the vector, and the dimension of a text vector represents the number of components included in the text vector. Since the dimension of an image vector is affected by the image size, to obtain an image vector with the same dimension as a text vector, the image to be recognized can be resized to obtain a standard image, and the standard image can be encoded to obtain an image vector with the same dimension as the text vector.

[0094] In some embodiments, the above encoding of the standard image to obtain an image vector with the same dimension as the text vector can be achieved by calling a convolutional neural network to encode the standard image to obtain an image vector with the same dimension as the text vector.

[0095] For example, see Figure 4B , Figure 4B This is a schematic diagram of the principle of the text entity recognition method provided by the embodiment of the present application. The convolutional neural network is called to encode the standard image to obtain the image vector with the same dimension as the text vector: V1, V2...V n+1 .

[0096] In some embodiments, convolutional neural networks (CNNs) are a type of feed-forward neural network with a deep structure that incorporates convolutional computations. They are a representative algorithm for deep learning. CNNs possess representation learning capabilities and can perform shift-invariant classification on input information based on their hierarchical structure.

[0097] In some embodiments, the complete text to be recognized and at least one matching image to be recognized can be directly encoded to obtain an image vector and a text vector. This makes entity recognition more efficient and allows direct entity recognition results. However, since it is impossible to accurately match an image with at least one sentence, the accuracy may be insufficient.

[0098] In some embodiments, a single sentence in the complete text to be recognized and the image to be recognized that matches the sentence can be encoded to obtain an image vector and a text vector for each sentence. In this way, since each sentence must be processed separately, the efficiency of entity recognition will be relatively low, and different sentences may correspond to the same image, and the same image may be executed repeatedly, so the efficiency will naturally be low. However, since each sentence accurately corresponds to an image to be recognized, the recognition accuracy will be relatively high. The number of operations of the algorithm is equal to the number of sentences in the text to be recognized. If the number of images to be recognized is greater than the number of sentences in the text to be recognized, this processing method can effectively ensure high recognition accuracy while taking into account the execution efficiency of the algorithm.

[0099] In some embodiments, each image to be recognized can be matched to at least one sentence in the text to be recognized, and the at least one matching sentence and the image to be recognized can be encoded to obtain an image vector and a text vector corresponding to the at least one sentence. In this way, the number of encoding operations is equal to the number of images to be recognized. When the number of images to be recognized is smaller than the number of sentences, this processing method can effectively ensure high recognition accuracy while also balancing algorithm execution efficiency.

[0100] In this way, for different scenarios, the number of sentences in the text to be recognized and the number of images to be recognized have different size relationships. Therefore, the embodiments of the present application adopt different processing methods for the text to be recognized and the images to be recognized in different scenarios. When the number of images to be recognized is less than the number of sentences, at least one matching sentence and the image to be recognized are encoded; when the number of images to be recognized is greater than the number of sentences in the text to be recognized, a single sentence in the complete text to be recognized and the image to be recognized that matches the sentence are encoded. Therefore, in different usage scenarios, it is possible to effectively ensure high recognition accuracy while taking into account the execution efficiency of the algorithm.

[0101] In step 103, the image vector and the text vector are modally fused to obtain a target multimodal vector.

[0102] In some embodiments, the above step 103 can be implemented as follows: calling the self-attention network to convert the text vector to obtain the hidden vector of the text vector; performing modal fusion on the image vector and the text vector to obtain the target multimodal vector.

[0103] In some embodiments, the multimodal vector has at least two modalities, and the multimodal vector can integrate different modalities, such as visual modality, auditory modality, tactile modality, etc. Therefore, the multimodal vector can not only convey the language text information of the text vector, but also can mine the multimodal information hidden in the language text information of the text vector.

[0104] For example, see Figure 4C , Figure 4C This is a schematic diagram of the principle of the text entity recognition method provided by the embodiment of the present application, calling the self-attention network to analyze the text vector (c0, c1, c2, c3...c n , c n+1 ) is converted to obtain the hidden vector of the text vector (r0, r1, r2, r3...r n , r n+1 ); the image vector (V1, V2...V n+1 ) and hidden vectors (r0, r1, r2, r3…r n , r n+1 ) performs modal fusion to obtain the target multimodal vector.

[0105] In some embodiments, modality fusion is achieved by Figure 4D The modal fusion network shown is used to fuse the image vector and the text vector to obtain a multimodal vector (h0, h1, h2, h3...h n , h n+1 ); Figure 4DThe modal fusion network shown includes a first modal fusion network 41, a second modal fusion network 42, and a third modal fusion network 43, wherein the first modal fusion network 41, the second modal fusion network 42, and the third modal fusion network 43 have the same network structure but different network inputs.

[0106] In some embodiments, see Figure 3B , Figure 3B is a flow chart of a text entity recognition method provided in an embodiment of the present application, Figure 3B Step 103 shown may be implemented by executing the following steps 1031 to 1035 .

[0107] In step 1031, a first modal fusion network is called to perform modal fusion on the image vector and the text vector to obtain a first multimodal vector.

[0108] In some embodiments, the above-mentioned first modal fusion network is used to perform modal fusion on image vectors and text vectors, wherein modal fusion refers to the process of fusion between different modalities. For example, the text vector is a text modality, and the image vector is an image modality. The first multimodal vector after modal fusion has information of both image and text modalities.

[0109] For example, see Figure 4D , calling the first modality fusion network 41, and performing the image vector (V1, V2...V n+1 ) and the hidden vectors of the text vector (r0, r1, r2, r3...r n , r n+1 ) to perform modal fusion and obtain the first multimodal vector (p1, p2...p n+1 ).

[0110] In some embodiments, the first modality fusion network includes a multi-head attention network, a residual connection network, and a feedforward neural network.

[0111] In some embodiments, see Figure 3C , Figure 3C is a flow chart of a text entity recognition method provided in an embodiment of the present application, Figure 3C The illustrated step 1031 may be implemented by executing the following steps 10311 to 10314 .

[0112] In step 10311, a multi-head attention network is called to perform modal fusion on the image vector and the text vector to obtain a fourth multimodal vector.

[0113] In some embodiments, a multi-head attention network is used to fuse vectors of different modalities. The expression of the multi-head attention network can be:

[0114] MH-CA(V,R)=W'[CA1(V,R),...,CA m (V,R)] T (1)

[0115] Among them, MH-CA(V, R) represents the multi-head attention network of the first modality fusion network, CA1 represents the self-attention of the first cross-modality, and CA m represents the second cross-modal self-attention, and W' represents the weight matrix.

[0116] In some embodiments, the i-th cross-modal self-attention can be expressed as:

[0117]

[0118] Among them, CA i Represents the self-attention of the i-th cross-modality, represents the weight matrix, R represents the hidden vector, m represents the number of cross-modal self-attention, and d represents the number of sub-vectors in the hidden vector.

[0119] In some embodiments, the above step 10311 can be implemented as follows: determine the image vector as the query vector of the first modal fusion network, and determine the text vector as the key vector and value vector of the first modal fusion network respectively; call the multi-head attention network to perform modal fusion on the query vector, key vector and value vector to obtain a fourth multimodal vector.

[0120] For example, see Figure 4D , the image vector (V1, V2...V n+1 ) is determined as the query vector (Query, Q) of the first modality fusion network 41, and the hidden vectors (r0, r1, r2, r3...r n , r n+1 ) are respectively determined as the key vector (Key, K) and value vector (Value, V) of the first modal fusion network 41; the multi-head attention network is called to perform modal fusion on the query vector (Q), the key vector (K) and the value vector (V) to obtain a fourth multimodal vector.

[0121] In step 10312, the residual connection network is called to perform residual connection on the fourth multimodal vector and the image vector to obtain a residual connection result.

[0122] In some embodiments, the residual connection network is used to perform residual connection on the input of the residual connection network to obtain a residual connection result, wherein the residual connection includes residual summation and horizontal normalization (Layer Normalization), the residual summation is used to make the residual connection network focus on the current difference, and the horizontal normalization is used to accelerate the convergence of the residual connection network.

[0123] In some embodiments, the horizontal normalization expression may be:

[0124] P=LN(MH-CA(V,R)) (3)

[0125] Among them, P represents the horizontal normalization result, MH-CA(V, R) represents the multi-head attention network representing the first modality fusion network, and LN represents the horizontal normalization.

[0126] In some embodiments, the above step 10312 can be implemented by calling a residual connection network, performing a residual connection on the fourth multimodal vector and the query vector, and obtaining a residual connection result.

[0127] For example, see Figure 4D , calling the residual connection network of the first modal fusion network 41, performing residual connection on the fourth multimodal vector and the query vector (Q) of the first modal fusion network 41, and obtaining a residual connection result.

[0128] In step 10313, the feedforward neural network is called to activate the residual connection result to obtain an activation result.

[0129] In some embodiments, a feedforward neural network is used to enhance the expressive power of the first modality fusion network. The feedforward neural network can be implemented by a nonlinear activation function, and the nonlinear activation function is used to activate the residual connection result.

[0130] In some embodiments, the neurons in the feedforward neural network are arranged in layers, with each neuron connected only to the neurons in the previous layer. The neurons receive the output of the previous layer and output it to the next layer, with no feedback between layers.

[0131] In step 10314, the residual connection network is called to perform residual connection on the activation result and the residual connection result to obtain a first multimodal vector.

[0132] For example, see Figure 4D , call the residual connection network of the first modality fusion network 41, perform residual connection on the activation result 411 and the residual connection result 412, and obtain the first multimodal vector (p0, p1, p2, p3...p n , p n+1 ).

[0133] In step 1032, the second modal fusion network is called to perform modal fusion on the image vector and the text vector to obtain a second multimodal vector.

[0134] In some embodiments, the second modality fusion network and the first modality fusion network have the same network structure. Step 1032 can be implemented as follows: determining the text vector as the query vector of the second modality fusion network, and determining the image vector as the key vector and value vector of the second modality fusion network, respectively; and calling the second modality fusion network to perform modal fusion on the query vector, key vector, and value vector to obtain a second multimodal vector.

[0135] For example, see Figure 4D , the hidden vector of the text vector (r0, r1, r2, r3...r n , r n+1 ) is determined as the query vector (Q) of the second modality fusion network 42, and the image vector (V1, V2...V n+1 ) are respectively determined as the key vector (K) and value vector (V) of the second modal fusion network 42; the second modal fusion network 42 is called to perform modal fusion on the query vector (Q), the key vector (K) and the value vector (V) to obtain a second multimodal vector.

[0136] In some embodiments, the second modal fusion network includes a multi-head attention network, a residual connection network and a feedforward neural network; the above-mentioned calling of the second modal fusion network to perform modal fusion on the query vector, the key vector and the value vector to obtain the second multimodal vector can be achieved as follows: calling the multi-head attention network to perform modal fusion on the query vector, the key vector and the value vector to obtain a fifth multimodal vector; calling the residual connection network to perform residual connection on the fifth multimodal vector and the query vector to obtain a residual connection result; calling the feedforward neural network to perform activation processing on the residual connection result to obtain an activation result; calling the residual connection network to perform residual connection on the activation result and the residual connection result to obtain a second multimodal vector.

[0137] For example, see Figure 4D , the second modal fusion network 42 includes a multi-head attention network, a residual connection network and a feedforward neural network; the above-mentioned second modal fusion network 42 is called to perform modal fusion on the query vector (Q), the key vector (K) and the value vector (V) to obtain the second multimodal vector (q0, q1, q2, q3...q n ,q n+1), which can be achieved as follows: calling a multi-head attention network to perform modal fusion on the query vector (Q), the key vector (K) and the value vector (V) to obtain a fifth multimodal vector 423; calling a residual connection network to perform residual connection on the fifth multimodal vector 423 and the query vector (Q) to obtain a residual connection result 422; calling a feedforward neural network to perform activation processing on the residual connection result 422 to obtain an activation result 421; calling a residual connection network to perform residual connection on the activation result 421 and the residual connection result 422 to obtain a second multimodal vector (q0, q1, q2, q3...q n ,q n+1 ).

[0138] In step 1033, the third modal fusion network is called to perform modal fusion on the text vector and the first multimodal vector to obtain a third multimodal vector.

[0139] For example, see Figure 4D , calling the third modality fusion network 43, the hidden vectors of the text vector (r0, r1, r2, r3...r n , r n+1 ) and the first multimodal vector (q0, q1, q2, q3...q n ,q n+1 ) to perform modal fusion and obtain the third multimodal vector (a0, a1, a2, a3...a n , a n+1 ).

[0140] In some embodiments, the third modal fusion network and the first modal fusion network have the same network structure. Step 1033 can be implemented as follows: determining the text vector as the query vector of the third modal fusion network, and determining the first multimodal vector as the key vector and value vector of the third modal fusion network, respectively; and invoking the third modal fusion network to perform modal fusion on the query vector, key vector, and value vector to obtain a third multimodal vector.

[0141] For example, see Figure 4D , the hidden vector of the text vector (r0, r1, r2, r3...r n , r n+1 ) is determined as the query vector (Q) of the third modality fusion network 43, and the first multimodal vector (q0, q1, q2, q3...q n ,q n+1 ) are respectively determined as the key vector (K) and value vector (V) of the third modal fusion network 43; the third modal fusion network 43 is called to perform modal fusion on the query vector (Q), the key vector (K) and the value vector (V) to obtain the third multimodal vector (a0, a1, a2, a3...a n , an+1 ).

[0142] In some embodiments, the third modal fusion network includes a multi-head attention network, a residual connection network and a feedforward neural network; the above-mentioned calling of the third modal fusion network to perform modal fusion on the query vector, the key vector and the value vector to obtain the third multimodal vector can be achieved as follows: calling the multi-head attention network to perform modal fusion on the query vector, the key vector and the value vector to obtain a sixth multimodal vector; calling the residual connection network to perform residual connection on the sixth multimodal vector and the query vector to obtain a residual connection result; calling the feedforward neural network to perform activation processing on the residual connection result to obtain an activation result; calling the residual connection network to perform residual connection on the activation result and the residual connection result to obtain the third multimodal vector.

[0143] For example, see Figure 4D , the third modal fusion network 43 includes a multi-head attention network, a residual connection network and a feedforward neural network; the third modal fusion network 43 is called to perform modal fusion on the query vector, the key vector and the value vector to obtain the third multimodal vector (a0, a1, a2, a3...a n , a n+1 ), which can be achieved by: calling a multi-head attention network to perform modal fusion on the query vector, key vector and value vector to obtain a sixth multimodal vector 433; calling a residual connection network to perform residual connection on the sixth multimodal vector 433 and the query vector (Q) to obtain a residual connection result 432; calling a feedforward neural network to perform activation processing on the residual connection result 432 to obtain an activation result 431; calling a residual connection network to perform residual connection on the activation result 431 and the residual connection result 432 to obtain a third multimodal vector (a0, a1, a2, a3...a n , a n+1 ).

[0144] In step 1034, weighted fusion is performed on the third multimodal vector and the second multimodal vector to obtain a weighted fusion vector.

[0145] In some embodiments, weighted fusion includes multiplication, addition, and normalization processes.

[0146] In some embodiments, see Figure 3D , Figure 3D is a flow chart of a text entity recognition method provided in an embodiment of the present application, Figure 3D The illustrated step 1034 may be implemented by executing the following steps 10341 to 10345 .

[0147] In step 10341, a first weight matrix of the third multimodal vector and a second weight matrix of the second multimodal vector are obtained.

[0148] As an example, the first weight matrix of the third multimodal vector can be expressed as: The second weight matrix of the second multimodal vector:

[0149] In step 10342, the third multimodal vector is multiplied by the first weight matrix to obtain a first product result; the second multimodal vector is multiplied by the second weight matrix to obtain a second product result.

[0150] As an example, the expression for the first product result can be:

[0151]

[0152] Among them, G1 represents the first product result, represents the first weight matrix, and A represents the third multimodal vector.

[0153] As an example, the expression for the second product result can be:

[0154]

[0155] Among them, G2 represents the second product result, represents the second weight matrix, and Q represents the second multimodal vector.

[0156] In step 10343, the first multiplication result and the second multiplication result are added to obtain a sum result.

[0157] As an example, the expression for the sum result can be:

[0158]

[0159] In step 10344, the summation result is normalized to obtain a normalized vector.

[0160] In some embodiments, the normalization process is used to dynamically control the correspondence between the features of the text to be recognized and the features of the image to be recognized through a visual detection (Visual Gate) mechanism, and the visual detection mechanism is implemented through a visual detection function.

[0161] As an example, the expression for a normalized vector can be:

[0162]

[0163] Among them, g represents the normalized vector, G3 and Characterize the summation result, represents the second weight matrix, Q represents the second multimodal vector, represents the first weight matrix, A represents the third multimodal vector, and σ represents the visual detection function.

[0164] In step 10345, the normalized vector and the second multimodal vector are multiplied to obtain a weighted fusion vector.

[0165] As an example, the expression of the weighted fusion vector can be:

[0166] B=gQ=(b0, b1, ..., b n+1 ) (8)

[0167] Among them, B and (b0, b1, ..., b n+1 ) represents the weighted fusion vector, Q represents the second multimodal vector, g represents the normalized vector, b0 to b n+1 Represents each component in the weighted fusion vector.

[0168] In step 1035 , the weighted fusion vector and the third multimodal vector are concatenated to obtain a target multimodal vector.

[0169] For example, see Figure 4D , the weighted fusion vector (b0, b1, ..., b n+1 ) and the third multimodal vector (a0, a1, a2, a3...a n , a n+1 ) to perform vector splicing and obtain the target multimodal vector (h0, h1, h2, h3...h n , h n+1 ).

[0170] In this way, through the first modal fusion network, the second modal fusion network and the third modal fusion network provided in the embodiment of the present application, the text vector and the image vector are fused from multiple different angles, so that the obtained multimodal vector fully integrates the features of the image to be identified and the text to be identified, providing strong data support for subsequent accurate entity recognition.

[0171] In step 104, entity recognition is performed on the text to be recognized based on the target multimodal vector to obtain initial entities in the text to be recognized; and entity recognition is performed on the text to be recognized based on the text vector to obtain corrected entities in the text to be recognized.

[0172] In some embodiments, named entity recognition (NER) is an information extraction technology used to obtain entities including names of people, places, roles, actors, etc. from the text to be recognized.

[0173] In some embodiments, the target multimodal vector includes word vectors corresponding to each word in the text to be recognized. In the step 104, entity recognition is performed on the text to be recognized based on the target multimodal vector to obtain the initial entity in the text to be recognized. This can be achieved by performing processing on the word vectors corresponding to each word in the text to be recognized: based on the word vectors, entity category prediction is performed on the words to obtain the initial category probability values ​​corresponding to each category; the category corresponding to the largest initial category probability value is determined as the initial category corresponding to the word; when the initial category representation word is a named entity, the entity category of the named entity is obtained, and the word is determined as the initial entity of the corresponding entity category in the text to be recognized.

[0174] In some embodiments, the entity category prediction described above can be implemented using a discriminant probability model, which is a conditional random field (CRF) used to annotate or analyze sequence data. A CRF is a vectorless graph model in which vertices represent random variables, and lines between vertices represent dependencies between random variables. In a CRF, the distribution of random variables is a conditional probability, and a given observation is a random variable. In principle, the CRF graph model layout can be arbitrarily given, and a commonly used layout is a linked architecture. By predicting words using the CRF, the initial category probability values ​​corresponding to each category can be determined.

[0175] In some embodiments, the categories may include entity categories and non-entity categories. When the category of a word is determined to be a non-entity category by the conditional random field, the word is characterized as not a named entity. When the category of a word is determined to be an entity category by the conditional random field, the word is characterized as a named entity.

[0176] In some embodiments, the entity categories may include a person name entity category, an organization entity category, a music name entity category, and the like.

[0177] For example, see Figure 4C Based on the word vector E1, the entity category of the word is predicted to obtain the initial category probability value of the word corresponding to each category (for example, the probability of the word corresponding to the non-entity category O is 0.1, and the probability of the word corresponding to the entity category is 0.9); the category (entity category) corresponding to the largest initial category probability value 0.9 is determined as the initial category corresponding to the word; when the initial category representation word is a named entity (the probability of the word corresponding to the person name entity category is 0.5, the probability of the word corresponding to the organization entity category is 0.1, and the probability of the word corresponding to the music name entity category is 0.2) (the person name entity category is a named entity), the entity category of the named entity (person name entity category) is obtained, and the word is determined as the initial entity of the corresponding entity category (person name entity category) in the text to be recognized.

[0178] For example, see Figure 4C Based on the word vector E3, the entity category of the word is predicted to obtain the initial category probability value of the word corresponding to each category (for example, the probability of the word corresponding to the non-entity category O is 0.9, and the probability of the word corresponding to the entity category is 0.1); the category corresponding to the largest initial category probability value of 0.9 (non-entity category) is determined as the initial category corresponding to the word; when the initial category representation word is not a named entity, the word is not determined as the initial entity in the text to be recognized.

[0179] In some embodiments, the above-mentioned text vector includes a word text vector corresponding to each word in the text to be recognized. In the above-mentioned step 104, entity recognition is performed on the text to be recognized based on the text vector to obtain the corrected entity in the text to be recognized. The following processing can be performed on the word text vector corresponding to each word in the text to be recognized: based on the word text vector, the entity category of the word is predicted to obtain the corrected category probability value corresponding to each category of the word; the category corresponding to the largest corrected category probability value is determined as the corrected category corresponding to the word; when the corrected category representation word is a named entity, the entity category of the named entity is obtained, and the word is determined as the corrected entity of the corresponding entity category in the text to be recognized.

[0180] For example, see Figure 4C Based on the word text vector F1, the entity category of the word is predicted, and the modified category probability values ​​of the word corresponding to each category are obtained (for example, the modified category probability of the word corresponding to the non-entity category O is 0.1, and the modified category probability of the word corresponding to the entity category is 0.9); the category (entity category) corresponding to the largest modified category probability value 0.9 is determined as the modified category corresponding to the word; when the modified category representation word is a named entity (the probability of the word corresponding to the person name entity category is 0.5, the probability of the word corresponding to the organization entity category is 0.1, and the probability of the word corresponding to the music name entity category is 0.2) (the person name entity category is a named entity), the entity category of the named entity (person name entity category) is obtained, and the word is determined as the modified entity of the corresponding entity category (person name entity category) in the text to be recognized.

[0181] For example, see Figure 4C Based on the word vector F3, the entity category of the word is predicted to obtain the modified category probability value corresponding to each category (for example, the probability of the word corresponding to the non-entity category O is 0.9, and the probability of the word corresponding to the entity category is 0.1); the category corresponding to the largest modified category probability value of 0.9 (non-entity category) is determined as the modified category corresponding to the word; when the modified category representation word is not a named entity, the word is not determined as the modified entity in the text to be recognized.

[0182] In this way, entity categories of words are predicted based on word text vectors and word vectors respectively, and the corrected entities and initial entities of the corresponding entity categories in the text to be recognized are obtained. Since the word vector integrates the features of the image to be recognized, the initial entity obtained based on the word vector effectively combines the features of the image to be recognized, thereby making the accuracy of the initial entity higher.

[0183] In step 105, the initial entity is corrected based on the corrected entity to obtain the target entity in the text to be recognized.

[0184] In some embodiments, since the initial entity obtained based on the word vector, the visual features of the image to be recognized introduced will cause an offset in the determination of the result of the initial entity, therefore, by determining the corrected entity based on the word text vector, the offset caused by the introduction of visual features can be reduced, thereby effectively improving the accuracy of the target entity in the determined text to be recognized.

[0185] In some embodiments, the number of the modified entity and the initial entity is at least one, see Figure 3E , Figure 3E is a flow chart of a text entity recognition method provided in an embodiment of the present application, Figure 3E The illustrated step 105 may be implemented by executing the following steps 1051 to 1055 for each correction entity.

[0186] In step 1051, the revised entity is compared with each initial entity to obtain a comparison result.

[0187] In some embodiments, the comparison result is used to indicate whether there is an initial entity identical to the revised entity in at least one initial entity.

[0188] As an example, the revised entity 11 is compared with the initial entity 21 , the initial entity 22 , the initial entity 23 , the initial entity 24 , and the initial entity 25 , respectively, to obtain comparison results. For example, the comparison results indicate that the revised entity 11 is the same as the initial entity 24 .

[0189] In step 1052 , in response to the comparison result indicating that there is an initial entity identical to the revised entity, the initial entity identical to the revised entity is determined as a candidate entity in the text to be recognized.

[0190] As an example, in response to the comparison result indicating that there is an initial entity identical to the revised entity 11 , the initial entity 24 identical to the revised entity 11 is determined as a candidate entity in the text to be recognized.

[0191] In step 1053 , in response to the comparison result indicating that there is no initial entity identical to the revised entity, the word text vector and the word vector are concatenated to obtain a concatenated vector.

[0192] In some embodiments, splicing refers to the process of splicing two vectors of the same dimension.

[0193] As an example, in response to the comparison result indicating that there is no initial entity identical to the revised entity, the word text vector and the word text vector are concatenated to obtain a concatenated vector.

[0194] In step 1054 , entity recognition is performed on the text to be recognized based on the concatenated vector to obtain candidate entities in the text to be recognized.

[0195] In some embodiments, the splicing vector includes word splicing vectors corresponding to each word in the text to be recognized. The above step 1054 can perform the following processing on the word splicing vectors corresponding to each word in the text to be recognized: based on the word splicing vector, the entity category of the word is predicted to obtain the target category probability value of each category corresponding to the word; the category corresponding to the largest target category probability value is determined as the target category corresponding to the word; when the target category representation word is a named entity, the entity category of the named entity is obtained, and the word is determined as a candidate entity of the corresponding entity category in the text to be recognized.

[0196] In step 1055, the entity knowledge graph is called to perform entity disambiguation on the candidate entities to obtain the target entity in the text to be identified.

[0197] In some embodiments, the entity knowledge graph is a series of various graphs that show the development process and structural relationships of entity knowledge. The entity knowledge graph includes three types of nodes: entity nodes, concept nodes, and attribute nodes. Entity nodes refer to things that are distinguishable and exist independently, such as a person, a city, a plant, and a commodity. Entity nodes are the most basic elements in the knowledge graph, and different entity nodes have different relationships. Concept nodes refer to a collection of entity nodes with the same characteristics, including books, names, etc. Attribute nodes are features used to distinguish concept nodes. Different concept nodes have different attributes, and different attribute value types correspond to edges of different types of attributes. When an attribute value corresponds to a concept node or an entity node, the attribute node is used to describe the relationship between two entity nodes, which is called an object attribute. When an attribute value corresponds to a specific numerical value, it is called a data attribute. The entity knowledge graph is used to describe the relationship between entities.

[0198] As an example, the entity knowledge graph includes a personal name entity knowledge graph, which includes a surname entity knowledge graph, a given name entity knowledge graph, a prefix entity knowledge graph (for example, prefix entities include Lao Wang and Xiao Li), and a suffix entity knowledge graph (for example, suffix entities include Mr. Wang and Mr. Zhang).

[0199] In some embodiments, step 1055 can be implemented as follows: comparing the candidate entity with each entity in the entity knowledge graph to determine a matching entity in the entity knowledge graph that is identical to the candidate entity. Obtaining a matching entity type of the matching entity in the entity knowledge graph, and comparing the matching entity type with the entity type of the candidate entity; in response to the comparison result indicating that the matching entity type is identical to the entity type of the candidate entity, determining the entity type of the candidate entity as the entity type of the target entity in the text to be recognized; in response to the comparison result indicating that the matching entity type is different from the entity type of the candidate entity, determining the entity type of the matching entity as the entity type of the target entity in the text to be recognized.

[0200] In this way, entity categories of words are predicted based on word text vectors and word vectors respectively, and the corrected entities and initial entities of the corresponding entity categories in the text to be recognized are obtained. Since the word vector integrates the features of the image to be recognized, the initial entity obtained based on the word vector effectively combines the features of the image to be recognized, so that the accuracy of the initial entity is higher. At the same time, in order to further improve the accuracy of the initial entity, the initial entity is further corrected by the corrected entity determined based on the word text vector. Through the entity knowledge graph, ambiguous entity types can be effectively eliminated, thereby effectively improving the accuracy of entity recognition.

[0201] In this way, by comparing the revised entity with each initial entity, if the comparison result indicates that the same initial entity exists as the revised entity, the initial entity that is the same as the revised entity is directly determined as the candidate entity in the text to be recognized. If the comparison result indicates that the same initial entity does not exist as the revised entity, the word text vector and the word text vector are concatenated to obtain a concatenated vector. Based on the concatenated vector, entity recognition is performed on the text to be recognized to obtain a candidate entity. Therefore, based on different comparison results, different methods are used to determine the target entity, which effectively improves the accuracy of the determined target entity.

[0202] In this way, by combining the text to be recognized and the image to be recognized associated with the text to be recognized, a multimodal vector is determined, and entity recognition is performed on the text to be recognized based on the multimodal vector, thereby accurately determining the initial entity in the text to be recognized. Since the determined initial entity comprehensively considers the image to be recognized associated with the text to be recognized, the semantic information of the image to be recognized assists in the entity recognition of the text to be recognized, thereby effectively improving the accuracy of the determined initial entity. At the same time, by performing entity recognition on the text to be recognized based on the text vector, a corrected entity is obtained, and the initial entity combined with the semantic information of the image to be recognized is corrected by the corrected entity to obtain the target entity in the text to be recognized. On the one hand, by performing entity recognition in combination with the associated image to be recognized, the accuracy of entity recognition is effectively improved; on the other hand, by correcting the entity, any recognition errors that may exist in the initial entity are corrected to obtain the target entity, thereby further improving the accuracy of entity recognition.

[0203] Below, an exemplary application of the embodiment of the present application in an actual entity recognition application scenario will be described.

[0204] See also Figure 4E , Figure 4E It is a schematic diagram of the effect of the text entity recognition method provided in an embodiment of the present application. Figure 4E The text to be recognized is: This Xiao Ming has a cuter face. The image to be recognized associated with the text to be recognized is Figure 4E The text entity recognition method provided by the embodiment of the present application is used to identify the kitten. Figure 4E When performing entity recognition on the text to be recognized: "This Xiao Ming has a cuter face", you can combine Figure 4E When the image to be identified is used to identify the text to be identified, the entity "cat" in the text to be identified is obtained. Figure 4E When the text to be identified is shown as: "This Xiao Ming has a cuter face", the entity "Xiao Ming" in the text to be identified is obtained. Since "Xiao Ming" in the text to be identified actually refers to Figure 4E Therefore, the entities identified by the entity recognition method of the related art are not accurate. The text recognition method provided in the embodiment of the present application can accurately identify entities that conform to the semantic environment of the text to be identified, and the accuracy of entity recognition is higher.

[0205] In audio and video applications, the text to be recognized can be the complete subtitles of the video to be played, and the images to be recognized associated with the text to be recognized can be each frame of the video to be played. Therefore, an image to be recognized can correspond to at least one sentence in the text to be recognized, meaning that subtitles are present in the video. An image to be recognized can also not correspond to text to be recognized, meaning that subtitles are not present in the video. Each sentence in the text to be recognized corresponds to at least one image to be recognized, meaning that each subtitle in the video has an associated image to be recognized.

[0206] In mobile app applications, for example, news apps, the text to be recognized can be any piece of news content within the app, and the image to be recognized associated with the text to be recognized can be a news image corresponding to the news content. Therefore, each image to be recognized can correspond to at least one sentence in the text to be recognized, meaning that the news image and the news content can correspond to at least one sentence, effectively improving the readability of the news.

[0207] In some embodiments, see Figure 4C , Figure 4C The entity recognition model of the text shown includes: an encoding network 52 for the image to be recognized, an encoding network 51 for the text to be recognized, an entity recognition network 53 for plain text, a multimodal entity recognition network 54 and a knowledge graph network 55.

[0208] In some embodiments, since the multimodal entity recognition network introduces image content, which may cause a visual bias to the target entity determined by the text entity recognition model, a text-only entity recognition network (Auxiliary Entity Span Detection Module) is added to reduce the bias.

[0209] In some embodiments, Figure 4C In the illustrated text entity recognition model, the inputs to the text entity recognition model are the text to be recognized and the image to be recognized associated with the text to be recognized. The image to be recognized is encoded by the encoding network 52 for the image to be recognized. First, the image to be recognized is subjected to feature transformation and size transformation using a convolutional neural network. After passing through the residual layer and the fully connected layer, the image to be recognized is converted into an image vector with the same dimensions as the text to be recognized.

[0210] In some embodiments, the text to be recognized is encoded by the encoding network 51 of the text to be recognized. Each input sentence is added with two special vectors, namely: vector (CLS) at the beginning of the sentence and vector (SEP) at the end of the sentence. Let S′ = (s0, s1, ..., s n+1) is the input sentence, s0, s n+1 are two special vectors, let X=(x0,x1,…,x n+1 ) is the word representation of S′, x i Indicates that each s i Semantic and position embedding information. Take X as the input vector of the bidirectional encoding network, and the output vector C=(c0,c1,…,c n+1 ) represents the context representation of X after passing through the bidirectional encoding network.

[0211] In some embodiments, the text modality and the image modality are fused, and the multimodal entity recognition network 54 is as follows: Figure 4C As shown in the figure, the left and right sides of the picture correspond to the image-aware word representation and the text image representation respectively. There are two modal information inputs.

[0212] In some embodiments, the hidden representation R after the text modality is activated is R=(r0, r1, ..., r n+1 ) Among them, r i Represents each x i Hidden representation of .

[0213] In some embodiments, the self-attention network can fuse information from different modalities together. The embodiment of the present application uses a multi-head self-attention network, which helps the model obtain more information.

[0214] In some embodiments, a multi-head attention network is used to fuse vectors of different modalities. The expression of the multi-head attention network can be:

[0215] MH-CA(V,R)=W'[CA1(V,R),...,CA m (V,R)] T (9)

[0216] Among them, MH-CA(V, R) represents the multi-head attention network of the first modality fusion network, CA1 represents the self-attention of the first cross-modality, and CA m represents the second cross-modal self-attention, and W' represents the weight matrix.

[0217] In some embodiments, the i-th cross-modal self-attention can be expressed as:

[0218]

[0219] Among them, CA i Represents the self-attention of the i-th cross-modality, represents the weight matrix, R represents the hidden vector, m represents the number of cross-modal self-attention, and d represents the number of sub-vectors in the hidden vector.

[0220] In some embodiments, the horizontal normalization expression may be:

[0221] P=LN(MH-CA(V,R)) (11)

[0222] Among them, P represents the horizontal normalization result, MH-CA(V, R) represents the multi-head attention network of the first modality fusion network, and LN represents the horizontal normalization.

[0223] In some embodiments, the input R of the CMT layer is queries, and V is keys and values. Output Q = (q0, q1, ..., q n+1 ), not every word requires visual information to correspond to it, such as: of, good, is, so we introduce a mechanism to dynamically control the correspondence between visual features and words.

[0224] In some embodiments, the expression of the normalized vector may be:

[0225]

[0226] Among them, g represents the normalized vector, G3 and Characterize the summation result, represents the second weight matrix, Q represents the second multimodal vector, represents the first weight matrix, A represents the third multimodal vector, and σ represents the visual detection function.

[0227] In some embodiments, the visual representation of each word after adding the gating mechanism is: B = gQ = (b0, b1, ..., b n+1 ), adding this gating mechanism is equivalent to controlling which word needs the corresponding information. The features A and B of the same dimension are concatenated and input into the hidden layer H = (h0,h1,…,h n+1 ).

[0228] In some embodiments, H is used as the input of a random conditional field CRF, and the output is the probability of the entity type in the sequence. For the input sentence S and image V, define:

[0229]

[0230] in, Represents category y i Convert to y i+1 The score, is the emission score, i represents the corresponding i-th word. y i Weight matrix. The output is an n-dimensional probability vector: E=(E1,E2,…,E n ).

[0231] In some embodiments, the entity naming is finally determined by combining the input F output by the auxiliary module and E output by the multimodal module.

[0232] In some embodiments, see Figure 4C ,Knowledge Graph Network 55,Processing Stage: After the previous two stages of processing, the text entity recognition model outputs entity names with types. In order to eliminate the ambiguity of entity representation as much as possible, the knowledge graph module is introduced. This module can be used to more accurately extract entity types. In the input stage of the text entity recognition model, the residual network is used to convert image information into feature information input into the model. Then, the visual feature information and text feature information are combined through the multimodal entity recognition network. After the output of the multimodal entity recognition network, the knowledge graph network is added to eliminate ambiguous entity types.

[0233] For a specific entity recognition application scenario, we extracted the play title, character name, and actor name from the text data. Without using the text entity recognition model provided by the embodiment of this application, the accuracy of the verification on the test set was 86.7%. After using the text entity recognition model provided by the embodiment of this application, the accuracy reached 95.1%, basically reaching the level of manual annotation.

[0234] In the embodiment of the present application, unlike fusing information of other modalities in the data preprocessing stage, the embodiment of the present application simultaneously inputs image and text features when inputting the model, and fully utilizes the information of the two modalities when fusing them into the same model, which can reduce error propagation and increase model accuracy.

[0235] In this embodiment of the present application, by adding a text-only entity recognition network, the offset caused by the introduction of visual content into the multimodal entity recognition network to the text entity recognition model is further reduced. In addition, by introducing knowledge graph post-processing, entity ambiguity can be eliminated and accuracy can be improved.

[0236] It is understandable that in the embodiments of the present application, the relevant data such as the text to be recognized and the image to be recognized, when the embodiments of the present application are applied to specific products or technologies, need to obtain user permission or consent, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0237] The following continues to describe the exemplary structure of the text entity recognition device 255 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules in the text entity recognition device 255 stored in the memory 240 may include: an acquisition module 2551, used to acquire the text to be recognized and the image to be recognized associated with the text to be recognized; an encoding module 2552, used to encode the image to be recognized to obtain an image vector, and encode the text to be recognized to obtain a text vector; a modal fusion module 2553, used to modally fuse the image vector and the text vector to obtain a target multimodal vector; an entity recognition module 2554, used to perform entity recognition on the text to be recognized based on the target multimodal vector to obtain an initial entity in the text to be recognized; perform entity recognition on the text to be recognized based on the text vector to obtain a revised entity in the text to be recognized; and a correction module 2555, used to correct the initial entity based on the revised entity to obtain a target entity in the text to be recognized.

[0238] In some embodiments, the modal fusion is implemented through a modal fusion network; the modal fusion network includes a first modal fusion network, a second modal fusion network, and a third modal fusion network; the above-mentioned modal fusion module 2553 is also used to call the first modal fusion network to perform modal fusion on the image vector and the text vector to obtain a first multimodal vector; call the second modal fusion network to perform modal fusion on the image vector and the text vector to obtain a second multimodal vector; call the third modal fusion network to perform modal fusion on the text vector and the first multimodal vector to obtain a third multimodal vector; perform weighted fusion on the third multimodal vector and the second multimodal vector to obtain a weighted fusion vector; and perform vector splicing on the weighted fusion vector and the third multimodal vector to obtain the target multimodal vector.

[0239] In some embodiments, the first modal fusion network includes a multi-head attention network, a residual connection network and a feedforward neural network; the above-mentioned modal fusion module 2553 is also used to call the multi-head attention network to perform modal fusion on the image vector and the text vector to obtain a fourth multimodal vector; call the residual connection network to perform residual connection on the fourth multimodal vector and the image vector to obtain a residual connection result; call the feedforward neural network to perform activation processing on the residual connection result to obtain an activation result; call the residual connection network to perform residual connection on the activation result and the residual connection result to obtain the first multimodal vector.

[0240] In some embodiments, the above-mentioned modal fusion module 2553 is also used to determine the image vector as the query vector of the first modal fusion network, and determine the text vector as the key vector and value vector of the first modal fusion network respectively; call the multi-head attention network to perform modal fusion on the query vector, the key vector and the value vector to obtain the fourth multimodal vector; the above-mentioned modal fusion module is also used to call the residual connection network to perform residual connection on the fourth multimodal vector and the query vector to obtain the residual connection result.

[0241] In some embodiments, the network structures of the second modal fusion network, the third modal fusion network and the first modal fusion network are the same; the above-mentioned modal fusion module 2553 is also used to determine the text vector as the query vector of the second modal fusion network, and determine the image vector as the key vector and value vector of the second modal fusion network respectively; call the second modal fusion network to perform modal fusion on the query vector, the key vector and the value vector to obtain a second multimodal vector; the above-mentioned modal fusion module is also used to determine the text vector as the query vector of the third modal fusion network, and determine the second multimodal vector as the key vector and value vector of the third modal fusion network respectively; call the third modal fusion network to perform modal fusion on the query vector, the key vector and the value vector to obtain a third multimodal vector.

[0242] In some embodiments, the above-mentioned modal fusion module 2553 is also used to obtain a first weight matrix of the third multimodal vector and a second weight matrix of the second multimodal vector; multiply the third multimodal vector and the first weight matrix to obtain a first product result; multiply the second multimodal vector and the second weight matrix to obtain a second product result; add the first product result and the second product result to obtain a sum result; normalize the sum result to obtain a normalized vector; multiply the normalized vector and the second multimodal vector to obtain the weighted fusion vector.

[0243] In some embodiments, the target multimodal vector includes word vectors corresponding to each word in the text to be recognized, and the above-mentioned entity recognition module 2554 is also used to perform the following processing on the word vectors corresponding to each word in the text to be recognized: based on the word vector, the entity category of the word is predicted to obtain the corrected category probability value of the word corresponding to each category; the category corresponding to the largest corrected category probability value is determined as the corrected category corresponding to the word; when the corrected category represents that the word is a named entity, the entity category of the named entity is obtained, and the word is determined as the initial entity corresponding to the entity category in the text to be recognized.

[0244] In some embodiments, the text vector includes a word text vector corresponding to each word in the text to be recognized, and the above-mentioned entity recognition module 2554 is also used to perform the following processing on the word text vector corresponding to each word in the text to be recognized: based on the word text vector, the entity category of the word is predicted to obtain the corrected category probability value corresponding to each category of the word; the category corresponding to the largest corrected category probability value is determined as the corrected category corresponding to the word; when the corrected category represents that the word is a named entity, the entity category of the named entity is obtained, and the word is determined as the corrected entity corresponding to the entity category in the text to be recognized.

[0245] In some embodiments, the number of the correction entity and the number of the initial entity are both at least one, and the correction module 2555 is further used to perform the following processing for each of the correction entities: compare the correction entity with each of the initial entities to obtain a comparison result, wherein the comparison result is used to characterize whether there is an initial entity identical to the correction entity in the at least one initial entity; in response to the comparison result characterizing the existence of the initial entity identical to the correction entity, the initial entity identical to the correction entity is determined as a candidate entity in the text to be recognized; in response to the comparison result characterizing the absence of the initial entity identical to the correction entity, the word text vector and the word vector are spliced ​​to obtain a spliced ​​vector; entity recognition is performed on the text to be recognized based on the spliced ​​vector to obtain a candidate entity in the text to be recognized; and the entity knowledge graph is called to perform entity disambiguation on the candidate entity to obtain a target entity in the text to be recognized.

[0246] In some embodiments, the splicing vector includes word splicing vectors corresponding to each word in the text to be recognized, and the correction module 2555 is further used to perform the following processing on the word splicing vectors corresponding to each word in the text to be recognized: based on the word splicing vector, the entity category of the word is predicted to obtain the target category probability value of each category corresponding to the word; the category corresponding to the largest target category probability value is determined as the target category corresponding to the word; when the target category represents that the word is a named entity, the entity category of the named entity is obtained, and the word is determined as the target entity corresponding to the entity category in the text to be recognized.

[0247] In some embodiments, the above-mentioned encoding module 2552 is also used to determine the context vector, word vector and position vector corresponding to each word in the text to be recognized; sum the context vector, the word vector and the position vector to obtain a fusion vector; encode the fusion vector to obtain the text vector; the above-mentioned encoding module 2552 is also used to obtain the dimension of the text vector, and based on the dimension of the text vector, resize the image to be recognized to obtain a standard image; encode the standard image to obtain an image vector with the same dimension as the text vector.

[0248] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the text entity recognition method described in the present application.

[0249] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the text entity recognition method provided by the embodiment of the present application, for example, Figure 3A Shown is a method for entity recognition of text.

[0250] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0251] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0252] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0253] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0254] In summary, the embodiments of the present application have the following beneficial effects:

[0255] (1) By combining the text to be recognized and the image to be recognized associated with the text to be recognized, a multimodal vector is determined, and entity recognition is performed on the text to be recognized based on the multimodal vector, thereby accurately determining the initial entity in the text to be recognized. Since the determined initial entity comprehensively considers the image to be recognized associated with the text to be recognized, the semantic information of the image to be recognized assists in the entity recognition of the text to be recognized, thereby effectively improving the accuracy of the determined initial entity. At the same time, by performing entity recognition on the text to be recognized based on the text vector, a corrected entity is obtained. The initial entity combined with the semantic information of the image to be recognized is corrected by the corrected entity to obtain the target entity in the text to be recognized. On the one hand, by performing entity recognition in combination with the associated image to be recognized, the accuracy of entity recognition is effectively improved; on the other hand, by correcting the entity, the recognition errors that may exist in the initial entity are corrected to obtain the target entity, thereby further improving the accuracy of entity recognition.

[0256] (2) In different scenarios, the number of sentences in the text to be recognized and the number of images to be recognized have different size relationships. Therefore, the embodiments of the present application adopt different processing methods for the text to be recognized and the images to be recognized in different scenarios. When the number of images to be recognized is less than the number of sentences, at least one matching sentence and the image to be recognized are encoded; when the number of images to be recognized is greater than the number of sentences in the text to be recognized, a single sentence in the complete text to be recognized and the image to be recognized that matches the sentence are encoded. In this way, in different usage scenarios, high recognition accuracy can be effectively guaranteed while taking into account the execution efficiency of the algorithm.

[0257] (3) Through the first modality fusion network, the second modality fusion network and the third modality fusion network provided in the embodiment of the present application, the text vector and the image vector are fused from multiple different angles, so that the obtained text vector fully integrates the features of the image to be recognized and the text to be recognized, providing strong data support for subsequent accurate entity recognition.

[0258] (4) Based on the word text vector and word vector, the entity category of the word is predicted, and the corrected entity and the initial entity of the corresponding entity category in the text to be recognized are obtained. Since the word vector integrates the features of the image to be recognized, the initial entity obtained based on the word vector effectively combines the features of the image to be recognized, so that the accuracy of the initial entity is higher.

[0259] (5) Based on the word text vector and word vector respectively, the entity category of the word is predicted, and the corrected entity and the initial entity of the corresponding entity category in the text to be recognized are obtained. Since the word vector integrates the features of the image to be recognized, the initial entity obtained based on the word vector effectively combines the features of the image to be recognized, so that the accuracy of the initial entity is higher. At the same time, in order to further improve the accuracy of the initial entity, the initial entity is further corrected by the corrected entity determined based on the word text vector, thereby effectively improving the accuracy of entity recognition.

[0260] (6) Since the initial entity obtained based on the word vector and the visual features of the image to be identified will cause an offset in the determination of the result of the initial entity, the offset caused by the introduction of visual features is reduced by determining the corrected entity based on the word text vector, thereby effectively improving the accuracy of the target entity in the text to be identified.

[0261] (7) By comparing the revised entity with each initial entity, if the comparison result indicates that there is an initial entity identical to the revised entity, the initial entity identical to the revised entity is directly determined as the target entity in the text to be recognized; if the comparison result indicates that there is no initial entity identical to the revised entity, the word text vector and the word text vector are concatenated to obtain a concatenated vector, and entity recognition is performed on the text to be recognized based on the concatenated vector to obtain the target entity. Thus, based on different comparison results, different methods are used to determine the target entity, thereby effectively improving the accuracy of the determined target entity.

[0262] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A text entity recognition method, characterized in that: The method comprises: Acquire a text to be recognized and an image to be recognized associated with the text to be recognized; Encoding the image to be recognized to obtain an image vector, and encoding the text to be recognized to obtain a text vector; Calling a first modal fusion network of the modal fusion network to perform modal fusion on the image vector and the text vector to obtain a first multimodal vector; calling a second modal fusion network of the modal fusion network to perform modal fusion on the image vector and the text vector to obtain a second multimodal vector; calling a third modal fusion network of the modal fusion network to perform modal fusion on the text vector and the first multimodal vector to obtain a third multimodal vector; Performing weighted fusion on the third multimodal vector and the second multimodal vector to obtain a weighted fusion vector; Performing vector concatenation on the weighted fusion vector and the third multimodal vector to obtain a target multimodal vector; Performing entity recognition on the text to be recognized based on the target multimodal vector to obtain an initial entity in the text to be recognized; performing entity recognition on the text to be recognized based on the text vector to obtain a revised entity in the text to be recognized; Based on the modified entity, the initial entity is modified to obtain a target entity in the text to be recognized.

2. The method according to claim 1, characterized in that The first modality fusion network includes a multi-head attention network, a residual connection network and a feedforward neural network; The calling of the first modal fusion network of the modal fusion network to perform modal fusion on the image vector and the text vector to obtain a first multimodal vector includes: Calling the multi-head attention network to perform modal fusion on the image vector and the text vector to obtain a fourth multimodal vector; Calling the residual connection network to perform a residual connection on the fourth multimodal vector and the image vector to obtain a residual connection result; Calling the feedforward neural network to perform activation processing on the residual connection result to obtain an activation result; The residual connection network is called to perform a residual connection on the activation result and the residual connection result to obtain the first multimodal vector.

3. The method according to claim 2, characterized in that The calling of the multi-head attention network to perform modal fusion on the image vector and the text vector to obtain a fourth multimodal vector includes: Determining the image vector as a query vector of the first modal fusion network, and determining the text vector as a key vector and a value vector of the first modal fusion network respectively; Calling the multi-head attention network to perform modal fusion on the query vector, the key vector, and the value vector to obtain the fourth multimodal vector; The calling of the residual connection network to perform a residual connection on the fourth multimodal vector and the image vector to obtain a residual connection result includes: The residual connection network is called to perform a residual connection on the fourth multimodal vector and the query vector to obtain the residual connection result.

4. The method according to claim 1, wherein The second modality fusion network, the third modality fusion network and the first modality fusion network have the same network structure; The calling of the second modal fusion network of the modal fusion network to perform modal fusion on the image vector and the text vector to obtain a second multimodal vector includes: Determining the text vector as a query vector of the second modality fusion network, and determining the image vector as a key vector and a value vector of the second modality fusion network respectively; calling the second modal fusion network to perform modal fusion on the query vector, the key vector, and the value vector to obtain the second multimodal vector; The calling of the third modal fusion network to perform modal fusion on the text vector and the first multimodal vector to obtain a third multimodal vector includes: Determining the text vector as a query vector of the third modal fusion network, and determining the first multimodal vector as a key vector and a value vector of the third modal fusion network respectively; The third modal fusion network is called to perform modal fusion on the query vector, the key vector, and the value vector to obtain the third multimodal vector.

5. The method according to claim 1, wherein The performing weighted fusion on the third multimodal vector and the second multimodal vector to obtain a weighted fusion vector includes: Obtaining a first weight matrix of the third multimodal vector and a second weight matrix of the second multimodal vector; Multiplying the third multimodal vector and the first weight matrix to obtain a first product result; multiplying the second multimodal vector and the second weight matrix to obtain a second product result; Adding the first multiplication result and the second multiplication result to obtain a summed result; Normalizing the summed result to obtain a normalized vector; The normalized vector and the second multimodal vector are multiplied to obtain the weighted fusion vector.

6. The method according to claim 1, characterized in that The target multimodal vector includes a word vector corresponding to each word in the text to be recognized, and the entity recognition of the text to be recognized based on the target multimodal vector to obtain an initial entity in the text to be recognized includes: The following processing is performed on the word vector corresponding to each word in the text to be recognized: Based on the word vector, the entity category of the word is predicted to obtain the initial category probability value of the word corresponding to each category; Determining the category corresponding to the largest initial category probability value as the initial category corresponding to the word; When the initial category indicates that the word is a named entity, the entity category of the named entity is obtained, and the word is determined as an initial entity corresponding to the entity category in the text to be recognized.

7. The method according to claim 6, characterized in that The text vector includes a word text vector corresponding to each word in the text to be recognized, and performing entity recognition on the text to be recognized based on the text vector to obtain a corrected entity in the text to be recognized includes: The following processing is performed on the word text vector corresponding to each word in the text to be recognized: Based on the word text vector, the entity category of the word is predicted to obtain the modified category probability value of the word corresponding to each category; Determining the category corresponding to the largest modified category probability value as the modified category corresponding to the word; When the correction category indicates that the word is a named entity, the entity category of the named entity is obtained, and the word is determined as a correction entity corresponding to the entity category in the text to be recognized.

8. The method according to claim 7, characterized in that The number of the modified entity and the number of the initial entity are both at least one, and the modifying the initial entity based on the modified entity to obtain the target entity in the text to be recognized includes: The following processing is performed for each of the correction entities: Comparing the revised entity with each of the initial entities to obtain a comparison result, wherein the comparison result is used to indicate whether there is an initial entity identical to the revised entity in the at least one initial entity; In response to the comparison result indicating that there is an initial entity identical to the revised entity, determining the initial entity identical to the revised entity as a candidate entity in the to-be-recognized text; In response to the comparison result indicating that there is no initial entity identical to the revised entity, concatenating the word text vector and the word vector to obtain a concatenated vector; Performing entity recognition on the text to be recognized based on the splicing vector to obtain candidate entities in the text to be recognized; The entity knowledge graph is called to perform entity disambiguation on the candidate entities to obtain the target entity in the text to be identified.

9. The method according to claim 8, characterized in that The splicing vector includes a word splicing vector corresponding to each word in the text to be recognized, and performing entity recognition on the text to be recognized based on the splicing vector to obtain candidate entities in the text to be recognized includes: The following processing is performed on the word concatenation vector corresponding to each word in the text to be recognized: Based on the word concatenation vector, the entity category of the word is predicted to obtain the target category probability value of each category corresponding to the word; Determining the category corresponding to the largest target category probability value as the target category corresponding to the word; When the target category represents that the word is a named entity, the entity category of the named entity is obtained, and the word is determined as a candidate entity corresponding to the entity category in the text to be recognized.

10. The method according to claim 1, characterized in that The encoding of the to-be-recognized text to obtain a text vector includes: Determine the context vector, word vector, and position vector corresponding to each word in the text to be recognized; Summing the context vector, the word vector, and the position vector to obtain a fusion vector; Encoding the fusion vector to obtain the text vector; The encoding of the image to be identified to obtain an image vector includes: Obtaining the dimension of the text vector, and based on the dimension of the text vector, performing size transformation on the image to be recognized to obtain a standard image; The standard image is encoded to obtain an image vector having the same dimension as the text vector.

11. A text entity recognition device, characterized in that: The device comprises: An acquisition module, configured to acquire text to be recognized and an image to be recognized associated with the text to be recognized; an encoding module, configured to encode the image to be recognized to obtain an image vector, and to encode the text to be recognized to obtain a text vector; A modal fusion module is configured to call a first modal fusion network of a modal fusion network to perform modal fusion on the image vector and the text vector to obtain a first multimodal vector; call a second modal fusion network of the modal fusion network to perform modal fusion on the image vector and the text vector to obtain a second multimodal vector; call a third modal fusion network of the modal fusion network to perform modal fusion on the text vector and the first multimodal vector to obtain a third multimodal vector; perform weighted fusion on the third multimodal vector and the second multimodal vector to obtain a weighted fusion vector; and perform vector concatenation on the weighted fusion vector and the third multimodal vector to obtain a target multimodal vector. An entity recognition module is configured to perform entity recognition on the text to be recognized based on the target multimodal vector to obtain an initial entity in the text to be recognized; and perform entity recognition on the text to be recognized based on the text vector to obtain a revised entity in the text to be recognized; The correction module is used to correct the initial entity based on the correction entity to obtain the target entity in the text to be recognized.

12. The device according to claim 11, characterized in that The first modality fusion network includes a multi-head attention network, a residual connection network and a feedforward neural network; The modal fusion module is further configured to call the multi-head attention network to perform modal fusion on the image vector and the text vector to obtain a fourth multimodal vector; and call the residual connection network to perform a residual connection on the fourth multimodal vector and the image vector to obtain a residual connection result. Calling the feedforward neural network to perform activation processing on the residual connection result to obtain an activation result; The residual connection network is called to perform a residual connection on the activation result and the residual connection result to obtain the first multimodal vector.

13. The device according to claim 12, characterized in that The modality fusion module is further configured to determine the image vector as a query vector of the first modality fusion network, and to determine the text vector as a key vector and a value vector of the first modality fusion network, respectively; Calling the multi-head attention network to perform modal fusion on the query vector, the key vector, and the value vector to obtain the fourth multimodal vector; The modal fusion module is further used to call the residual connection network to perform a residual connection on the fourth multimodal vector and the query vector to obtain the residual connection result.

14. The device according to claim 11, characterized in that The second modality fusion network, the third modality fusion network and the first modality fusion network have the same network structure; The modality fusion module is further configured to determine the text vector as a query vector of the second modality fusion network, and to determine the image vector as a key vector and a value vector of the second modality fusion network, respectively; calling the second modal fusion network to perform modal fusion on the query vector, the key vector, and the value vector to obtain the second multimodal vector; The modal fusion module is further configured to determine the text vector as a query vector of the third modal fusion network, and to determine the first multimodal vector as a key vector and a value vector of the third modal fusion network respectively; The third modal fusion network is called to perform modal fusion on the query vector, the key vector, and the value vector to obtain the third multimodal vector.

15. The device according to claim 11, characterized in that The modal fusion module is further configured to obtain a first weight matrix of the third multimodal vector and a second weight matrix of the second multimodal vector; and multiply the third multimodal vector and the first weight matrix to obtain a first product result; Multiplying the second multimodal vector and the second weight matrix to obtain a second product result; The first product result and the second product result are added to obtain a sum result; the sum result is normalized to obtain a normalized vector; and the normalized vector and the second multimodal vector are multiplied to obtain the weighted fusion vector.

16. The device according to claim 11, characterized in that The target multimodal vector includes word vectors corresponding to each word in the text to be recognized; The entity recognition module is further configured to perform the following processing on the word vector corresponding to each word in the to-be-recognized text: based on the word vector, predict the entity category of the word to obtain initial category probability values ​​corresponding to each category of the word; and determine the category corresponding to the largest initial category probability value as the initial category corresponding to the word; When the initial category indicates that the word is a named entity, the entity category of the named entity is obtained, and the word is determined as an initial entity corresponding to the entity category in the text to be recognized.

17. The device according to claim 16, characterized in that The text vector includes a word text vector corresponding to each word in the text to be recognized; The entity recognition module is further configured to perform the following processing on the word text vector corresponding to each word in the to-be-recognized text: based on the word text vector, predict the entity category of the word to obtain a modified category probability value corresponding to each category of the word; and determine the category corresponding to the largest modified category probability value as the modified category corresponding to the word; When the correction category indicates that the word is a named entity, the entity category of the named entity is obtained, and the word is determined as a correction entity corresponding to the entity category in the text to be recognized.

18. An electronic device, characterized in that: The electronic device comprises: a memory for storing executable instructions; A processor, configured to implement the text entity recognition method according to any one of claims 1 to 10 when executing the executable instructions or computer program stored in the memory.

19. A computer-readable storage medium storing executable instructions or a computer program, characterized in that: When the executable instructions are executed by a processor, the text entity recognition method according to any one of claims 1 to 10 is implemented.

20. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the text entity recognition method according to any one of claims 1 to 10 is implemented.

Images