Double-tower entity identification method and device fusing multi-modal features, equipment and medium

Abstract

The invention discloses a double-tower entity recognition method and device fusing multi-modal features, equipment and a medium, and the method comprises the steps: obtaining to-be-recognized text data which comprises a tag; performing cascade representation on the to-be-identified text data and the tag through a preset prompt strategy to obtain an output sequence; on the basis of the output sequence, obtaining a chessman image feature, a tactile perception feature and a visual perception feature, and carrying out feature fusion to obtain an image dense feature vector; based on the output sequence, predicting to obtain a text dense feature vector; performing feature fusion on the image dense feature vector and the text dense feature vector to obtain a fusion feature; and based on the fusion feature, determining a starting boundary and an ending boundary of the to-be-extracted entity to extract an entity segment of the to-be-extracted entity. Isolated texts and labels are fused, flat entity recognition and classification tasks are carried out from the perspective of reading understanding, and text feature mining ability is enhanced by fusing three image features of chessman, touch and vision.

Description

Technical Field

[0001] The present application relates to the field of entity recognition technology, and in particular to a dual-tower entity recognition method, device, computer equipment and storage medium integrating multimodal features. Background Art

[0002] The development of the Internet has led to an exponential growth of data in various vertical fields, such as communications, medical fields, and news fields. The huge amount of information is a challenge to the efficient knowledge of data, and the information extraction task has emerged. According to the data organization form of information, data can be divided into structured data, semi-structured data, and unstructured data. Among them, unstructured data accounts for the highest proportion, and it is also used as a carrier for the dissemination of information such as audio and video, and is often used as a data source for information extraction. The information extraction task refers to extracting entities, relationships, events and other information specified in the established ontology design from unstructured natural language texts, and outputting structured data. According to the division of the International Conference on Automatic Content Extraction (ACE), named entity recognition and classification is one of the key tasks of information extraction. This is because entities are the basis of a series of other complex tasks, such as knowledge graph construction, machine translation, and question-answering systems. In these tasks, only when the smallest granularity of knowledge nodes is accurately identified can subsequent high-level information be correctly extracted, such as triples in knowledge graphs, domain terms in machine translation, and answers to question-answering systems.

[0003] Taking the flat named entity (FNE) as an example, the recognition and classification tasks of FNE are the basis for the subsequent complex task learning. Good extraction performance can effectively reduce the error propagation problems faced by downstream complex tasks. Therefore, the performance of flat entity recognition and classification tasks can improve the subsequent information extraction effect and improve the level of information mastery in various vertical fields. However, there are currently two technical shortcomings in flat entity recognition and classification: the first is that because Chinese is a bamboo-like structure with the characteristics of structural openness that controls the whole with the zero, the viewpoint of natural text can be constantly switched and transformed, resulting in poor mining of the inherent meaning characteristics of natural language text. The second is that natural text and labels are usually isolated and divided, that is, the former is used as input and the latter is used as a golden label to guide the learning of the model, resulting in a lack of knowledge prompts and low entity recognition accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a dual-tower entity recognition method, device, equipment and medium that integrates multimodal features to address the above-mentioned technical problems, so as to solve at least one problem existing in the above-mentioned prior art.

[0005] In a first aspect, the embodiment of the present application is implemented by providing a dual-tower entity recognition method integrating multimodal features, including: Acquire text data to be recognized, wherein the text data to be recognized includes a label; By using a preset prompt strategy, the text data to be recognized and the label are cascaded to obtain an output sequence; Based on the output sequence, obtaining the inscription image features, tactile perception features and visual perception features, and performing feature fusion on the inscription image features, tactile perception features and visual perception features to obtain a dense feature vector of the image; Based on the output sequence, predicting a dense feature vector of the text; Performing feature fusion on the image dense feature vector and the text dense feature vector to obtain a fused feature; Based on the fusion features, a starting boundary and an ending boundary of the entity to be extracted are determined, so as to extract entity fragments of the entity to be extracted based on the starting boundary and the ending boundary.

[0006] In a second aspect, a dual-tower entity recognition device integrating multimodal features is provided, comprising: A to-be-recognized text data acquisition unit, used to acquire the to-be-recognized text data, wherein the to-be-recognized text data includes a label; A text and label combining unit, used to concatenate the text data to be identified and the label through a preset prompt strategy to obtain an output sequence; An image dense feature vector generating unit is used to obtain the inscription image features, tactile perception features and visual perception features based on the output sequence, and perform feature fusion on the inscription image features, tactile perception features and visual perception features to obtain an image dense feature vector; A text dense feature vector generating unit, used for predicting and obtaining a text dense feature vector based on the output sequence; A multimodal feature fusion unit, used for fusing the image dense feature vector and the text dense feature vector to obtain a fused feature; The entity recognition unit is used to determine the starting boundary and the ending boundary of the entity to be extracted based on the fusion feature, so as to extract the entity fragment of the entity to be extracted based on the starting boundary and the ending boundary.

[0007] In a third aspect, a computer device is provided, comprising a memory, a processor, and computer-readable instructions stored in the memory and running on the processor, wherein when the processor executes the computer-readable instructions, the dual-tower entity recognition method integrating multimodal features as described above is implemented.

[0008] In a fourth aspect, a readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the dual-tower entity recognition method integrating multimodal features as described above is implemented.

[0009] The above-mentioned dual-tower entity recognition method, device, computer equipment and storage medium that integrate multimodal features, and its method implementation include: obtaining text data to be recognized, the text data to be recognized includes labels; through a preset prompt strategy, cascading the text data to be recognized and the labels to obtain an output sequence; based on the output sequence, obtaining the inscription image features, tactile perception features and visual perception features, and performing feature fusion on the inscription image features, tactile perception features and visual perception features to obtain an image dense feature vector; based on the output sequence, predicting a text dense feature vector; performing feature fusion on the image dense feature vector and the text dense feature vector to obtain a fusion feature; based on the fusion feature, determining the starting boundary and the ending boundary of the entity to be extracted, so as to extract the entity fragment of the entity to be extracted based on the starting boundary and the ending boundary. In the embodiment of the present application, isolated text and labels are fused, and the task of flat entity recognition and classification is carried out from the perspective of reading comprehension. In addition, the three image features of inscription, vision, and touch are fused at the granularity of natural language characters to enhance the text feature mining capability, improve the semantic grasp level of the smallest granularity, and thus improve the accuracy of entity classification and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0011] Figure 1 This is a schematic diagram of a network architecture of an MPL model in an embodiment of the present application; Figure 2 is a schematic diagram of flattening named entities in one embodiment of the present application; Figure 3 It is a flow chart of a double-tower entity recognition method integrating multimodal features in one embodiment of the present application; Figure 4 is a schematic diagram of Braille in one embodiment of the present application; Figure 5 is a schematic diagram in italics of an embodiment of the present application; Figure 6 This is a schematic diagram of a network architecture of a visual-tactile perception fusion layer in one embodiment of the present application; Figure 7 This is a schematic diagram of marking text using the 0, 1 marking strategy in an embodiment of the present application; Figure 8 It is a structural schematic diagram of a dual-tower entity recognition device integrating multimodal features in one embodiment of the present application; Fig. 9 It is a schematic diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] The twin tower entity recognition method integrating multimodal features provided in this embodiment can be applied in the following aspects: Figure 1 In the network architecture of the model MPL shown, the model MPL may specifically include an image tower, a text tower, a multimodal feature fusion layer, and a head and tail identifier. It should be noted that the image tower may include a knowledge prompt generator and a visual tactile perception fusion layer, the text tower may include a knowledge prompt generator and a text embedding layer, and the head and tail identifier may include a head identifier and a tail identifier. Specifically, the knowledge prompt generator is used to jointly represent the input natural language text, such as a sentence, with the label to obtain an output sequence, and input it into the visual tactile perception fusion layer and the text embedding layer respectively. The visual tactile fusion layer extracts braiile image features, handwriting image features, and inscription image features based on the input output sequence, and performs multi-feature fusion; the text tower can capture the bidirectional contextual semantic information of the output sequence, for example, it can be captured by a Transformer series encoder. Then the multimodal feature fusion layer fuses the outputs of the image tower and the text tower as input vectors, and inputs them into the head and tail identifier to identify the starting position and the ending position of the entity respectively to extract the entity fragment.

[0014] Among them, entity, or named entity, is the basic unit of knowledge graph and also an important language unit that carries information in this article, such as name of person, place, organization, and product. Taking the flat named entity (FNE) as an example, its specific Figure 2 As shown, a schematic diagram of a common flat entity is provided, which may include information such as a company name and a name.

[0015] It should be noted that the Qiwen can be used to study the semantic correction effect of the origin of Chinese characters on modern Chinese characters; handwriting can be used to study the enhancement effect of handwriting on the visual perception semantic space of modern Chinese characters; Braille can be used to study the semantic expansion effect of tactile perception semantic space on modern Chinese characters. Therefore, the ability to mine text features can be enhanced by integrating the three image features of Qiwen, handwriting, and Braille.

[0016] In one embodiment, if Figure 3 As shown, a dual-tower entity recognition method integrating multimodal features is provided, comprising the following steps: In step S110, the text data to be recognized is obtained, and the text data to be recognized includes a label; In the embodiment of the present application, the text data to be identified can be standard short text data, and according to actual needs, it can include various types, such as video, music, characters, place names, etc. It should be noted that the label refers to the label used to characterize the entity in the text data to be identified. For example, the name, organization, etc. can be obtained by manual annotation.

[0017] In step S120, the text data to be recognized and the label are concatenated to obtain an output sequence through a preset prompt strategy; In an embodiment of the present application, by inputting the text data to be recognized into the knowledge prompt generator, the label can be first converted into a standard language of natural language. For example, "PER" needs to be converted into "Person", and in the Chinese flat named entity recognition, it needs to be further converted into a natural text expression "name". Then, according to the converted label, prompt information is generated according to the preset prompt strategy as question information. All question information can be constructed as a prompt information sequence of the label, and then the question information and the text to be recognized are generated through a large language model to generate answer information, and all answer information can constitute an output sequence. Through natural language, the label and the input text can be regarded as completely equal, which can effectively alleviate the isolation problem of the label and the natural text sequence.

[0018] The preset prompt strategy can be used to generate the prompt information of the tag. Specifically, it can include at least one of keyword prompt, rule prompt, encyclopedia definition prompt, synonym prompt, implication and sample prompt. Taking the tag "organization" as an example, its prompt strategy can be specifically shown in Table 1 below: Table 1: Prompt strategy table

[0019] As shown in Table 1 above, the prompts generated by labels can guide the learning of the model in the role of prior information. Among them, zero prompts are the current mainstream label2id and id2label mapping prompts; keywords are natural text expressions of prompt labels; rule prompts are organized into natural language questions by using rule generation based on keyword prompts; encyclopedia definition prompts are based on keyword prompts and use the encyclopedia definition of keywords as prompt information from the perspective of interpretation; synonym prompts are based on keyword prompts and use the synonyms of keywords in Chinese Cilin as an expansion of prompt information; implication and sample prompts are based on synonym prompts and add implication information and sample information.

[0020] In step S130, based on the output sequence, the inscription image features, tactile perception features and visual perception features are obtained, and the inscription image features, tactile perception features and visual perception features are fused to obtain a dense feature vector of the image; In an embodiment of the present application, an image tower can be constructed, and the output sequence output by the knowledge prompt generator can be used as the input sequence of the image tower. Specifically, an image data set is first constructed, and the image data set can include inscription images, handwritten images, and Braille images. After the output sequence is obtained, each character in the output sequence can be split, and then each character can be matched with the image in the image data set to obtain the inscription image, handwritten image, and Braille image corresponding to each character. Then, feature extraction is performed on the inscription image, handwritten image, and Braille image respectively, and the inscription image features, tactile perception features, and visual perception features can be obtained. The obtained inscription image features, tactile perception features, and visual perception features are losslessly spliced ​​and affine processed to obtain the image dense feature vector.

[0021] Among them, the oracle bone inscription image features help the neural network to mine the original meaning in unstructured natural text and enrich the representation of underlying features. Oracle bone inscriptions, originating from engraving on bones, are the oldest known mature writing system. Oracle bone inscriptions of the Shang Dynasty are widely regarded as the origin of Chinese characters. In terms of writing style, oracle bone inscriptions use a knife as a pen. For those used for engraving, the writing styles are mostly vertical or horizontal. Even curves are composed of short horizontal lines. In terms of image learning, the distinction between clear horizontal and vertical features is more obvious. In terms of character structure, some characters contain vivid pictorial drawings, accurately restoring the meaning inherent in the characters themselves, which can reduce semantic noise to a certain extent. In terms of layout, the font size is flexible, determined by the meaning, and can provide some semantic features in terms of font size. For example: "人 (rén)" and "入 (rù)": It is difficult to distinguish them in most modern commonly used Chinese characters, but the difference between them is obvious in oracle bone inscriptions. The former shows the head, arms, body, and legs from top to bottom, containing the pictorial meaning information of a person standing upright with a side view and hanging arms; the latter shows the lid of a box from top to bottom, containing the pictorial meaning information of collection and sealing. In terms of writing style, their features are obvious and in sharp contrast; in terms of character structure, their pictorial drawings are clearly distinguishable, containing the original meaning of the natural text itself; in terms of layout, the width-to-height ratios determined by the meaning have a semantic hinting effect on the natural text, and there are obvious differences in the width and height ratios of the characters "人 (rén)" and "入 (rù)".

[0022] Among them, the tactile perception feature is a mapped writing system through tactile perception, consisting of 63 characters. Each character is composed of 1 to 6 raised dots, and the raised dots are distributed in a matrix unit of three rows and two columns. The writing and reading processes are in opposite directions: when writing, the dot contacts are made from right to left; when reading, the dot contacts are obtained from left to right. Braille is the first systematic binary-coded writing system, which can represent 64 combinations in the matrix unit. This system can map 133 languages such as English, Chinese, and Arabic. The Braille codes of the Chinese characters "日 (rì)" and "月 (yuè)" are as Figure 4 shown.

[0023] Among them, the visual perception feature refers to the visual perception image feature of the handwriting. For example, taking Kaiti as an example, Kaiti, also known as Zhengshu and Zhenshu, is a font of modern Chinese characters, which evolved from Lishu. It gradually evolved from the concrete pictograms of the Shang Dynasty inscriptions to the abstract symbols of Kaiti in the late Han Dynasty. In modern times, Kaiti has gradually become simplified, and the text expression has become more abstract, becoming the modern popular handwritten regular characters. Exemplarily, Yan Zhenqing's Kaiti can be used as the visual feature of the writing image. Yan Zhenqing's Kaiti is light in horizontal strokes and heavy in vertical strokes, thin in left strokes and thick in right strokes, and has obvious expression; in terms of word formation, the long strokes are gathered, the heart is solid and the edges are empty, the inside is loose and the outside is tight, so the overall font is square, dense, plump and thick; in terms of composition, the edges of the single words are neatly extended, so the whole article is arranged vertically and horizontally, open and comfortable, neat and magnificent. This is an important reason for the development of the font from the late Han Dynasty to the Tang Dynasty and then to the modern popular font. It is understandable that other fonts can also be used as visual perception image features, which can be set according to actual conditions, and this application does not limit it here. Figure 5 As shown, a schematic diagram of Yan Zhenqing's regular script handwritten characters is provided.

[0024] In step S140, based on the output sequence, a text dense feature vector is predicted; In an embodiment of the present application, a text tower can be constructed, and the output sequence of the knowledge prompt generator can be used as the input sequence of the text tower. The text tower can include a pre-trained prediction model, such as the Enire model, to improve the semantic embedding quality of Chinese characters and obtain more accurate data dense vectors. Specifically, through the Enire model, each character vector can be predicted as a text dense feature vector.

[0025] The training process of the Enire model can be specifically as follows: construct a training sample data set, which may include a large amount of training sample data. The training sample data set can be masked at the entity granularity, and then input into the pre-constructed Enire model for iterative training. After each iteration, the loss value can be calculated by the loss function, such as cross entropy, regularization, etc. Based on the loss value, the model parameters are adjusted until the loss value is less than the preset threshold, or the number of iterations reaches the maximum number of iterations, then the trained Enire model can be obtained.

[0026] Optionally, the generation process of the text dense feature vector is specifically as follows: Treat the output of the knowledge hint generator as an input sequence , SP represents the output sequence of the knowledge prompt generator, n represents the total length of the sequence, and m represents the total length of the prompt information. The vectorization process of the text input sequence is as follows: ; in, The character vector of the i-th token of the sequence SP representing the knowledge hint is obtained through the pre-trained Enire model. The character vector is used as the dense vector of the text feature.

[0027] As an implementation method, the text data to be recognized can be processed through the embedding layer to obtain a text vector. Alternatively, the digital identifier corresponding to each word in the text can be queried through a dictionary, and the digital identifier corresponding to each word can be converted into a vector form to obtain a character vector. Among them, the dictionary stores the mapping relationship between text and digital ID, so based on the dictionary, each Chinese character in the text can be converted into a corresponding digital ID to realize the generation of a character vector. The character vector is used as the dense vector of the text feature.

[0028] In step S150, the image dense feature vector and the text dense feature vector are subjected to feature fusion to obtain a fusion feature; In an embodiment of the present application, after obtaining a text dense feature vector through a text tower and obtaining an image dense feature vector through an image tower, feature fusion can be performed on the text dense feature vector and image dense feature vector output by the top of the image tower and the text tower. For example, different fusion algorithms can be used to perform feature fusion at different granularities, such as token granularity (word granularity) and sentence granularity (sentence granularity).

[0029] In step S160 , a starting boundary and an ending boundary of the entity to be extracted are determined based on the fusion features, so as to extract entity segments of the entity to be extracted based on the starting boundary and the ending boundary.

[0030] In the embodiment of the present application, a head and tail identifier can be constructed, which includes a head identifier and a tail identifier, and two linear layers are used to identify the head and tail boundaries of the entity respectively. Based on the identified head boundary, i.e., the start boundary, and the tail boundary, i.e., the end boundary, the entity fragment is extracted. For example, if the entity is a name, for the text to be identified "Xiao Ming has booked a restaurant at 10 o'clock", the identified start boundary and end boundary can be marked with symbols, characters, numbers, etc., for example, " Booked a restaurant at 10 o'clock", the extracted entity can be .

[0031] In an embodiment of the present application, a dual-tower entity recognition method integrating multimodal features is provided, including: obtaining text data to be recognized, the text data to be recognized includes a label; through a preset prompt strategy, the text data to be recognized and the label are cascaded to obtain an output sequence; based on the output sequence, the image features, tactile perception features and visual perception features of the inscription are obtained, and the image features, tactile perception features and visual perception features of the inscription are feature fused to obtain an image dense feature vector; based on the output sequence, a text dense feature vector is predicted; the image dense feature vector and the text dense feature vector are feature fused to obtain a fusion feature; based on the fusion feature, the starting boundary and the ending boundary of the entity to be extracted are determined, so as to extract the entity fragment of the entity to be extracted based on the starting boundary and the ending boundary. In an embodiment of the present application, isolated text and labels are fused, and the task of flattening entity recognition and classification is carried out from the perspective of reading comprehension, and the three image features of inscription, handwriting and Braille are fused at the granularity of natural language characters to enhance the text feature mining capability, improve the semantic grasping level of the minimum granularity, and improve the accuracy of entity classification and recognition.

[0032] In an embodiment of the present application, the text data to be recognized and the label are concatenated to obtain an output sequence by using a preset prompt strategy, including: Convert the label into the standard language of the text data to be identified to obtain a natural language text; Based on the preset prompt strategy, generating prompt information of the natural language text as question information; Based on the question information and the text data to be recognized, answer information is generated.

[0033] Optionally, the label is converted into a standard language of natural language, for example, "PER" needs to be converted into "Person", and in Chinese flat named entity recognition, it needs to be further converted into a natural text expression "name". Then, according to the converted label, according to the preset prompt strategy, prompt information is generated as question information. All question information can be constructed as a prompt information sequence of the label, and then the prompt information sequence can be combined with the text sequence of the input text to be recognized. For example, the label and the natural text can be combined in a left-right parallel manner. The cascade operation of the label and the input sequence transforms the original separation state of the two into a joint state. The cascade process can be shown as follows.

[0034] ; Among them, sp represents the output sequence of the knowledge prompt generator, CAT represents the parallel operation from left to right, p represents the token of the prompt information sequence in Table 1 above, m represents the length of the prompt information, x represents the natural language text input sequence, and n represents the length of the input sequence.

[0035] In an embodiment of the present application, the inscription image features, tactile perception features and visual perception features are obtained based on the output sequence, including: Constructing an image data set, wherein the image data set includes a deed image, a preset font image, and a Braille image; Matching each character in the output sequence with the inscription image, the preset font image and the Braille image to obtain the inscription image, the preset font image and the Braille image corresponding to each character; Encoding the inscription image, the preset font image and the Braille image corresponding to each character into single-channel image features respectively, so as to serve as the inscription image features, the tactile perception features and the visual perception features; Among them, the described Qiwen image features are used to study the semantic correction effect of the source of Chinese characters on modern Chinese characters, the described visual perception features are used to study the enhancement effect of handwriting on the visual perception semantic space of modern Chinese characters, and the described tactile perception features are used to study the semantic expansion effect of tactile perception semantic space on modern Chinese characters.

[0036] The preset fonts may be handwritten, printed, or other font types.

[0037] It should be noted that the Chinese character system has a long history of development as a carrier of information transmission. The trend of changes in the structural design of Chinese characters is from easy to draw to easy to write; the trend of changes in the expression of meaning is from pictographic to symbolic. In this change, the pictographic meaning of the Chinese character itself is inevitably discarded, resulting in the blurring of the recognition and distinction between Chinese characters. According to the human cognitive system, humans often understand images faster than text, and their understanding accuracy is often better than text. Therefore, by using the image of the inscription as the source feature of the Chinese character, the image of the Braille as the tactile perception feature, and the image of the font as the visual perception feature, it is possible to deeply mine the text features and enrich the text feature representation to improve the accuracy of entity recognition.

[0038] Optionally, an image data set can be constructed, which includes a Qiwen image, a preset font image, for example, a Kaiti image, a Songti image, etc., and a Braille image. After obtaining the output sequence, the text of the output sequence can be compared with the image in the image data set to obtain the Qiwen image, the preset font image and the Braille image corresponding to the text, and after encoding processing, the Qiwen image features, tactile perception features and visual perception features can be obtained.

[0039] In one embodiment of the present application, the feature fusion of the inscribed text image features, tactile perception features, and visual perception features to obtain an image dense feature vector includes: Stitch the inscribed text image features, tactile perception features, and visual perception features to obtain a stitched feature; Perform an affine process on the stitched feature to obtain the image dense feature vector.

[0040] Optionally, a single-channel image with a token as the smallest granularity is used as the input of the above three image features, and each image is reconstructed into an image with a specification of 28*28 pixels for subsequent fusion. After obtaining the inscribed text image features, tactile perception features, and visual perception features, the following formula can be used for feature fusion: ; where fusion represents the mixed multi-modal features, Affine represents the affine transformation of the feature vector, Stack represents the lossless stacking operation, represents the inscribed text image features, tactile perception features, and visual perception features. Taking the Chinese character "mountain" as an example, the network architecture of this visual-tactile perception fusion layer is specifically as Figure 6 shown.

[0041] In one embodiment of the present application, the determination of the start boundary and end boundary of the entity to be extracted based on the fusion features includes: Input the fusion features into the head recognizer and the tail recognizer respectively, and perform activation processing on the fusion features through a preset activation function to obtain activation values, where the activation values represent the start boundary probability or end boundary probability of the entity to be extracted; Determine the start boundary or end boundary of the entity to be extracted based on the start boundary probability or end boundary probability.

[0042] Optionally, a head and tail recognizer can be constructed, which includes a head recognizer and a tail recognizer. Input the fusion features into the head recognizer and the tail recognizer respectively, and separate the head recognizer and the tail recognizer to perform the recognition process. Two linear layers can be used respectively to be responsible for the recognition of the start boundary and end boundary of the entity. During the recognition process, an activation function can be used. For example, in the head recognizer and the tail recognizer, the fusion features input into the head recognizer and the tail recognizer can be activated by Sigmoid respectively to obtain an activation value, which can be used to represent the probability of becoming the start boundary and end boundary of the entity, and the threshold can be set to 0.5. That is, when the recognized probability is greater than 0.5, it can be considered as the start boundary or end boundary, otherwise, the recognition is performed again. The specific recognition process can be represented by the following formula: ; Among them, SE represents the output sequence of the head and tail identifier, corresponding to the probability of each token in the sp sequence as the starting and ending boundary; v represents the output vector of the multimodal feature fusion layer; fusion represents the names of three fusion strategies, and its values ​​represent token granularity, sentence granularity, and weighted addition respectively.

[0043] In an embodiment of the present application, the starting boundary and the ending boundary include a plurality of each, and extracting the entity fragment of the entity to be extracted based on the starting boundary and the ending boundary includes: Pairing the plurality of start boundaries and end boundaries to obtain a plurality of entity segments, wherein the entity segments include a start boundary and an end boundary matching the start boundary; Extracting the entity fragments; Optionally, a nearest handshake strategy may be used for pairing. It can be understood that when pairing, each start boundary may be paired with an adjacent end boundary, or each start boundary may be paired with the end boundary closest to it, or the content between each start boundary and the end boundary may be semantically analyzed to determine the pairing result, or the determined start boundary and end boundary may be paired first, and the remaining start boundaries may be automatically paired with the end boundaries, etc.

[0044] Exemplarily, in a text, entities: name 1, name 2 and name 3 may be included, so three starting boundaries and three ending boundaries may be included, for example, their positional relationship is: starting boundary A, ending boundary A, ending boundary B, starting boundary B, ending boundary C, starting boundary C. Since starting boundary A is adjacent to ending boundary A, starting boundary A and ending boundary A may be paired, that is, the text between the two may be extracted as an entity segment. There are ending boundary B and ending boundary C on the left and right sides of starting boundary B, then the distance between starting boundary B and ending boundary B and ending boundary C may be determined, and starting boundary B may be paired with the nearest ending boundary, or semantic analysis may be performed on the text between starting boundary B and ending boundary B, and the text between starting boundary B and ending boundary C respectively to determine whether an entity can be formed, and if so, the corresponding entity segment may be extracted, or, since there is starting boundary C on the right side of ending boundary C, ending boundary C is the nearest ending boundary for starting boundary C, so starting boundary C may be paired with ending boundary C directly, and in this case, starting boundary B may be automatically paired with ending boundary B.

[0045] In an embodiment of the present application, the image dense feature vector and the text dense feature vector are subjected to feature fusion to obtain fused features, including: Performing a horizontal or vertical augmentation operation on the image dense feature vector and the text dense feature vector; Affine processing is performed on the image dense feature vector and the text dense feature vector after the augmentation operation by using a preset affine function to obtain the fusion feature.

[0046] Optionally, after obtaining the text dense feature vector through the text tower and the image dense feature vector through the image tower, the text dense feature vector and the image dense feature vector output by the top of the image tower and the text tower can be feature fused. For example, different fusion algorithms can be used to perform feature fusion at different granularities, such as token granularity and sentence granularity.

[0047] The token granularity can be as follows: ; in, It represents the fused feature vector at the token granularity, Affine represents the affine function, and Augmentation represents the horizontal or vertical augmentation operation. Accordingly, different augmentation matrices have different affine weight matrices.

[0048] The sentence granularity can be as follows: ; in, It represents the fused feature vector at the sentence granularity, Affine represents the affine function, and Augmentation represents the horizontal or vertical augmentation operation. Accordingly, different augmentation matrices have different affine weight matrices.

[0049] In the embodiment of the present application, in order to further verify the effectiveness of the MPL model provided by the present application, experiments are conducted with a public data set to verify the effectiveness of the present application. Among them, the data set can be derived from CLUE NER 2020, which is a subset of THUCNEWS. The training set and validation set of the data set can be 10748 and 1343 respectively, containing ten entity categories, and the corresponding categories and labels are shown in Table 2.

[0050] Table 2: Entity categories and labels correspondence and proportion table

[0051] In order to use the head and tail identifier to perform entity recognition, the 0, 1 labeling strategy can be used to transform the data. The specific labeling examples are as follows: Figure 7 shown.

[0052] The statistical information of the experimental data can be shown in Table 3 below.

[0053] Table 3: Statistics of the CLUENER2020 dataset

[0054] The evaluation process can be expressed using the following formula: ; ; ; The number of correctly identified entities refers to the named entities extracted by the encoder and decoder.

[0055] It can be run on an Ubuntu server. The specific training environment configuration is shown in Table 4 below.

[0056] Table 4: Training environment configuration table

[0057] As shown in Table 4 above, after comprehensive data analysis of the input data of the data set and the corresponding prompt information, the maximum sequence length is determined to be 200. After multiple experimental adjustments, the batch size (Batch_Size) is determined to be 32 and the training round (Epoch) is determined to be 100. This experiment uses the early stopping strategy to control and optimize the experimental training process, with the tolerance (Patience) set to 5 and the fluctuation threshold (Delta) set to 0.00002. The relevant parameter configurations of the MPL model are shown in Table 5.

[0058] Table 5: MPL model parameter configuration table

[0059] In order to verify the effectiveness of the MPL model proposed in this proposal, the current mainstream models can be used as comparative experiments for comparative analysis. The first model is the LSTM-CRF model proposed by Zhiheng Huang et al. of Baidu Research Institute in 2015, which is the mainstream method in many vertical fields; the second model is the BERT entity recognition model released by the huggingface team in 2020, which is also a more mainstream method at present; the third model is the RoBERTa model entity recognition model proposed by Cui Yiming et al. of Harbin Institute of Technology at the IEEE conference in 2021; the fourth model is to add an LSTM layer on the basis of Experiment 2; the fifth model is the BERT+LSTM+CRF model that is currently widely used in various vertical fields; the sixth model is the performance of humans under the CLUENER dataset; the seventh is the MPL model proposed in this application. The detailed setting information of the comparative experiment is shown in Table 6.

[0060] Table 6: Comparative test information table

[0061] Among them, the 0th LSTM-CRF method uses a long short-term memory neural network to capture semantic information of sequence information. The captured information can be divided into two data flow lines: the first is the information flow through both ends of the sequence, which is used to alleviate the problem of gradient disappearance; the second is the unit information flow controlled by the gating mechanism, and the gating mechanism determines whether the information passes or is forgotten. The semantic information of natural text is captured mainly through the two information flows, and then a CRF linear chain is connected to constrain unreasonable labels. Due to the use of the Viterbi algorithm for decoding by the CRF linear chain, the 0th method faces the problem of breaking keys for long entities. In addition, polysemy is also a severe challenge faced by this model. The 1st BERT method uses the BERT pre-trained model as a word vector encoder, and then uses a linear layer to decode the information vector for single-label classification at the Token granularity. Although this method alleviates the problem of breaking keys for long entities in the 0th LSTM-CRF, there is a problem of insufficient learning focus when a linear layer completes the two tasks of head and tail recognition. From the perspective of the regularity of Chinese natural language text expression, the mask mechanism of the 1st BERT has an inherent defect: the fixed co-occurrence phenomenon of transliterated words, disyllabic bound morphemes, reduplicated words, and onomatopoeic words will cause information leakage problems, such as "踌躇" (disyllabic bound morpheme with alliteration), "灿烂" (disyllabic bound morpheme with assonance), "辗转" (disyllabic bound morpheme with both alliteration and assonance), "颠沛" (disyllabic bound morpheme without alliteration or assonance), "纷纷" (reduplicated word), and "滴答" (onomatopoeic word). To alleviate the defect of the mask mechanism of the BERT model, the 2nd RoBERTa method replaces BERT with the RoBERTa pre-trained model with word as the smallest mask granularity on the basis of the 1st method. The 3rd method, on the basis of the 2nd method, uses BERT as a word vector encoder and then connects an LSTM network to capture semantic information bidirectionally, and then uses a linear layer for decoding to obtain labels. The 3rd and 1st methods can study the enhancement of the LSTM's ability to encode BERT information. The 4th method is the current mainstream flat entity recognition method, which combines the BERT pre-trained model of the 1st method and the LSTM+CRF of the 0th method. On the one hand, it can utilize the pre-training results of BERT to capture semantics bidirectionally, alleviating the problem of polysemy to a certain extent; on the other hand, it uses LSTM to filter the semantics of long texts with weights and uses the CRF decoder to constrain unreasonable labels. The 5th method is the performance of humans on the CLUENER dataset. The 6th method is the MPL model used in the proposed two-tower entity recognition method that fuses multi-modal features in this proposal. It fuses tactile features and visual features to form visual space feature vectors and tactile space feature vectors to enrich the expression of the underlying semantic space vectors of the text, and uses prompts to effectively cascade the originally fragmented text space and label controls to enrich the source of information capture.

[0062] In order to study the impact of different feature fusion methods on the final performance of the model, three groups of comparative experiments were set up.

[0063] Table 7: Comparative test information table

[0064] Among them, Experiment 0 adopts the mainstream fusion method, that is, linear accumulation at the element level. Experiments 1 and 2 are weighted ratios of Token granularity and Sentence granularity, respectively, to explore the performance of feature fusion at word granularity and sentence granularity in multimodal entity recognition.

[0065] In order to verify the effect of the components in the MPL method proposed in this proposal, the experiment sets multimodal fusion and prompt information as ablation items, and analyzes the contribution of each component to the experimental results through ablation research. The first group does not start the prompt information component and does not mount the multimodal feature fusion module; the second group starts the prompt information component but does not mount the multimodal fusion module; the third group does not start the prompt information component and mounts the multimodal fusion module; the fourth group starts the prompt information component and mounts the multimodal fusion module. The detailed settings of the ablation experiment are shown in Table 8.

[0066] Table 8: Ablation test information table

[0067] The MPL model uses LSTM+CRF as the baseline model and selects various methods widely used in various vertical fields as comparison items. After multiple experiments, the final comparative experimental performance is shown in Table 9.

[0068] Table 9: Comparative test results

[0069] From the analysis of Table 9, we can find that the F1 value of LSTM+CRF is 6.59% higher than that of human performance in the comparison between the benchmark experiment No. 0 and the experiment No. 5. There may be three reasons for this: First, the prior knowledge of the classification of entities such as banks and research institutes into company categories, agency categories, or organizational groups in the experiment No. 0 is better than that of humans. For example, "XX Research Institute" belongs to companies, "XX Economic Research Institute" belongs to government agencies, and "XX International Financial Research Institute" belongs to organizational groups; second, the names of movies, games, books, and newspapers are all regarded as the application objects of book marks. The method of multi-neural network collaboration of these prior knowledge will learn better than human performance; third, the participants of the experiment No. 5 did not have a more comprehensive definition of the entity. The F1 value of the experiment No. 1 increased by 8.82% compared with the benchmark experiment No. 0. The reason may be that BERT is a self-encoding style denoising model, and its ability to capture semantics in both directions through the self-attention mechanism and the mask mechanism is better than that of the benchmark experiment. Experiment 2 has 10.48% and 1.6% performance improvement compared to Experiment 0 and Experiment 1, respectively. The reason may be that the Whole Word Masking (WWM) strategy of the RoBERTa model alleviates the information leakage of the Mask of the BERT model, and the mining performance of text features based on the BERT model is better. Experiment 3 has 5.03% performance improvement compared to Experiment 0, but has 3.79% and 5.39% significant performance decline compared to Experiment 1 and Experiment 2, respectively: the reason why Experiment 3 has a significant improvement over the baseline experiment 0 may be that the semantic capture ability of the BERT pre-trained model combined with the LSTM network under the constraint of the wireless CRF chain has been enhanced; the reason why the performance of Experiment 3 has declined compared to Experiments 1 and 2 may be that the pre-trained model of the Transformer series is sufficient to learn the semantic relationship of the Token sequence by fitting natural text, but after connecting to the LSTM network, it will learn noise and cause performance decline. Experiment 4 achieved a 1.53% performance improvement compared to Experiment 3. The reason may be that the linear CRF chain constrained unreasonable labels, which provided positive guidance for label classification; however, compared to Experiment 2, the linear CRF chain did not make up for the 3.86% semantic mining improvement brought by the WWM strategy, which shows that enriching the underlying semantic expression plays an important role in improving the semantic mining level of natural language text. Experiment 5 was evaluated manually, and there were large individual differences in the mastery of prior knowledge. Therefore, Experiment 0 was selected as the baseline in this proposal.Experiment 6 is a dual-tower entity recognition method that integrates multimodal features proposed in this proposal. Compared with Experiment 0, Experiment 1, Experiment 2, Experiment 3, Experiment 4, and Experiment 5, it has a significant performance improvement of 18.22%, 9.4%, 7.8%, 13.19%, 11.66%, and 24.81%, respectively. There may be three reasons for this: First, the prompt method proposed in this proposal cascades the labels and sentences that were originally separated, which makes the semantic space of the natural language interpretation of the label play an important guiding role in the semantic capture of the sentence sequence; second, the dual-tower model proposed in this proposal fully and effectively learns text and pixel-level image information, and the features of the multi-perceptual space expand the natural text features. The third point is that the model architecture proposed in this proposal converts the task of entity recognition into a reading comprehension perspective. From the bottom of the tower to the top of the tower, the cascaded text is encoded into multimodal information, and then the head and tail identifiers are used to identify the start and end positions respectively. Compared with the current mainstream method of performing sequence labeling horizontally and fragment classification vertically, this model architecture is more suitable for entity recognition tasks.

[0070] After multiple experiments, the effects of three feature fusion methods on the experimental results are compared. The comparative experimental results are shown in Table 10.

[0071] Table 10: Comparative test results

[0072] From the analysis of Table 10, we can find that the accumulation effect based on the semantic space in Experiment 0 is the best, which is 0.39% and 1.27% higher than the F values ​​of Experiments 1 and 2, respectively. The reason for the former may be that the weight ratio of the Token granularity fits some of the wrong information, and the multimodal features are fused with some noise; the reason for the latter may be that the weight coefficient granularity faced by the Sentence granularity is too large, and it is impossible to enrich and express the inaccurate feature expressions in the sentence under this order of magnitude of data set. This comparative experiment shows that the accumulation effect based on the semantic space in the feature fusion stage is the best, and the fusion granularity is moderate.

[0073] In order to verify the impact of the core components of the MPL method proposed in this proposal on the experimental performance, the research work was analyzed according to four groups of ablation experiments. The ablation experiment results are shown in Table 11.

[0074] Table 11: Ablation experiment results

[0075] The effectiveness of the method proposed in this proposal can be verified by analyzing the above Table 11 in the vertical and horizontal directions. In the vertical dimension: Experiment 0 has a 1.82% performance decline compared to Experiment 3, indicating that the prompt component proposed in this proposal has a significant positive contribution to the experimental results by cascading the sentence and label and the synergy of the multi-perceptual spatial features constructed by the image tower. Experiment 1 has a 0.81% performance decline compared to Experiment 3, indicating that the visual feature space and tactile feature space fused by the image tower proposed in this proposal have a rich effect on the underlying text features. Experiment 2 has a 0.65% performance decline compared to Experiment 3, indicating that the prompt proposed in this proposal cascades the sentence semantic space and the label natural text semantic space, effectively alleviating the information loss problem caused by the isolation of labels and text sequences, enriching knowledge prompts, and improving the level of information utilization. In the horizontal dimension: An analysis of Experiments 0, 1, 2, and 3 revealed that the performance gains produced by the two components proposed in this proposal are not linearly additive. The reason may be that the two components introduce some noise during the collaborative process.

[0076] In the embodiment of the present application, from the perspective of mining the underlying features of natural language text, tactile perception features and visual perception features are used as entry points to enrich the semantic features of Chinese characters in coordination with handwriting; on the other hand, the labels are naturalized and prompt information is constructed to be cascaded with the input sequence, and the head and tail recognition tasks are carried out from the perspective of reading comprehension, which greatly reduces the learning difficulty of the model, can improve the extraction effect of entity recognition in vertical fields, and provide a solid foundation for improving the performance of downstream tasks. Compared with the existing technology, this proposal has the following advantages: the level of mining the underlying features of natural language text is significantly improved: a multimodal innovation point of integrating multi-perceptual space is proposed, and the pictographic features of Chinese characters are deeply studied, which greatly alleviates the problem of fuzzy distinction of Chinese characters. Full use of gold label data: isolated text sequences and label sequences are integrated, and a variety of prompt methods are proposed to transform labels and texts from isolated states to joint states, and the tasks of flattening entity recognition and classification are carried out from the perspective of reading comprehension. It can bring significant commercial value to downstream business tasks in various vertical fields. For example, in the medical field, it can improve the information extraction effect to promote the informatization, digitization and intelligence of medical work; in the financial industry, it can improve personalized services by accurately identifying customer information; in the healthcare field, it can use entity extraction to optimize clinical analysis; in e-commerce, it can enhance the recommendation system with the help of commodity entity recognition; in the legal industry, it can accelerate document review through case entity extraction; in the media industry, it can use entity recognition to improve content classification and recommendation quality; in some key industries, it can strengthen intelligence analysis through entity monitoring; in supply chain management, entity recognition can help demand forecasting. These applications not only improve business efficiency, but also enhance decision support and user experience.

[0077] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0078] In one embodiment, a dual-tower entity recognition device integrating multimodal features is provided, and the dual-tower entity recognition device integrating multimodal features corresponds one-to-one to the dual-tower entity recognition method integrating multimodal features in the above embodiment. Figure 8 As shown, the dual-tower entity recognition device integrating multimodal features includes a text data acquisition unit 10 to be recognized, a text and label combination unit 20, an image dense feature vector generation unit 30, a text dense feature vector generation unit 40, a multimodal feature fusion unit 50 and an entity recognition unit 60. Each functional module is described in detail as follows: The to-be-recognized text data acquisition unit 10 is used to acquire the to-be-recognized text data, wherein the to-be-recognized text data includes a tag; The text and label combining unit 20 is used to concatenate the text data to be identified and the label through a preset prompt strategy to obtain an output sequence; An image dense feature vector generating unit 30 is used to obtain the inscription image features, tactile perception features and visual perception features based on the output sequence, and to perform feature fusion on the inscription image features, tactile perception features and visual perception features to obtain an image dense feature vector; A text dense feature vector generating unit 40, configured to predict and obtain a text dense feature vector based on the output sequence; A multimodal feature fusion unit 50, used for fusing the image dense feature vector and the text dense feature vector to obtain a fused feature; The entity recognition unit 60 is used to determine the starting boundary and the ending boundary of the entity to be extracted based on the fusion feature, so as to extract the entity fragment of the entity to be extracted based on the starting boundary and the ending boundary.

[0079] In one embodiment of the present application, the text and label combining unit 20 is further used for: Convert the label into the standard language of the text data to be identified to obtain a natural language text; Based on the preset prompt strategy, generating prompt information of the natural language text as question information; Based on the question information and the text data to be recognized, answer information is generated.

[0080] In one embodiment of the present application, the image dense feature vector generating unit 30 is further used for: Constructing an image data set, wherein the image data set includes a deed image, a preset font image, and a Braille image; Matching each character in the output sequence with the inscription image, the preset font image and the Braille image to obtain the inscription image, the preset font image and the Braille image corresponding to each character; Encoding the inscription image, the preset font image and the Braille image corresponding to each character into single-channel image features respectively, so as to serve as the inscription image features, the tactile perception features and the visual perception features; Among them, the described Qiwen image features are used to study the semantic correction effect of the source of Chinese characters on modern Chinese characters, the described visual perception features are used to study the enhancement effect of handwriting on the visual perception semantic space of modern Chinese characters, and the described tactile perception features are used to study the semantic expansion effect of tactile perception semantic space on modern Chinese characters.

[0081] In one embodiment of the present application, the image dense feature vector generating unit 30 is further used for: splicing the inscription image features, tactile perception features, and visual perception features to obtain splicing features; Affine processing is performed on the splicing features to obtain the dense feature vector of the image.

[0082] In one embodiment of the present application, the entity identification unit 60 is further configured to: The fused features are respectively input into the head identifier and the tail identifier, and the fused features are activated by a preset activation function to obtain an activation value, wherein the activation value represents the starting boundary probability or the ending boundary probability of the entity to be extracted; Based on the starting boundary probability or the ending boundary probability, the starting boundary or the ending boundary of the entity to be extracted is determined.

[0083] In an embodiment of the present application, the start boundary and the end boundary respectively include a plurality of entity identification units 60, which are further used for: Based on the nearest handshake strategy, a plurality of the start boundaries and the end boundaries are paired to obtain a plurality of entity fragments, wherein the entity fragments include a start boundary and an end boundary matching the start boundary; The entity fragments are extracted.

[0084] In one embodiment of the present application, the multimodal feature fusion unit 50 is further used for: Performing a horizontal or vertical augmentation operation on the image dense feature vector and the text dense feature vector; Affine processing is performed on the image dense feature vector and the text dense feature vector after the augmentation operation by using a preset affine function to obtain the fusion feature.

[0085] In the embodiment of the present application, from the perspective of mining the underlying features of natural language text, tactile perception features and visual perception features are used as entry points to enrich the semantic features of Chinese characters in coordination with handwriting; on the other hand, the labels are naturalized and prompt information is constructed to be cascaded with the input sequence, and the head and tail recognition tasks are carried out from the perspective of reading comprehension, which greatly reduces the learning difficulty of the model, can improve the extraction effect of entity recognition in vertical fields, and provide a solid foundation for improving the performance of downstream tasks. Compared with the existing technology, this proposal has the following advantages: the level of mining the underlying features of natural language text is significantly improved: a multimodal innovation point of integrating multi-perceptual space is proposed, and the pictographic features of Chinese characters are deeply studied, which greatly alleviates the problem of fuzzy distinction of Chinese characters. Full use of gold label data: isolated text sequences and label sequences are integrated, and a variety of prompt methods are proposed to transform labels and texts from isolated states to joint states, and the tasks of flattening entity recognition and classification are carried out from the perspective of reading comprehension. It can bring significant commercial value to downstream business tasks in various vertical fields. For example, in the medical field, it can improve the information extraction effect to promote the informatization, digitization and intelligence of medical work; in the financial industry, it can improve personalized services by accurately identifying customer information; in the healthcare field, it can use entity extraction to optimize clinical analysis; in e-commerce, it can enhance the recommendation system with the help of commodity entity recognition; in the legal industry, it can accelerate document review through case entity extraction; in the media industry, it can use entity recognition to improve content classification and recommendation quality; in the government, it can strengthen intelligence analysis through entity monitoring; and in supply chain management, entity recognition can help demand forecasting. These applications not only improve business efficiency, but also enhance decision support and user experience.

[0086] For the specific definition of the dual-tower entity recognition device that integrates multimodal features, please refer to the definition of the dual-tower entity recognition method that integrates multimodal features above, which will not be repeated here. Each module in the above-mentioned dual-tower entity recognition device that integrates multimodal features can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0087] In one embodiment, a computer device is provided. The computer device may be a terminal device, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer-readable instructions. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, a dual-tower entity recognition method that integrates multimodal features is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0088] In an embodiment of the present application, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the steps of the double-tower entity recognition method integrating multimodal features as described above are implemented.

[0089] In an embodiment of the application, a readable storage medium is provided, wherein the readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the double-tower entity recognition method integrating multimodal features as described above are implemented.

[0090] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0091] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0092] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A twin-tower entity recognition method integrating multimodal features, characterized in that: The method comprises: Acquire text data to be recognized, wherein the text data to be recognized includes a label; By using a preset prompt strategy, the text data to be recognized and the label are cascaded to obtain an output sequence; Based on the output sequence, obtaining the inscription image features, tactile perception features and visual perception features, and performing feature fusion on the inscription image features, tactile perception features and visual perception features to obtain a dense feature vector of the image; Based on the output sequence, predicting a dense feature vector of the text; Performing feature fusion on the image dense feature vector and the text dense feature vector to obtain a fused feature; Based on the fusion features, a starting boundary and an ending boundary of the entity to be extracted are determined, so as to extract entity fragments of the entity to be extracted based on the starting boundary and the ending boundary.

2. The method for identifying a twin-tower entity by integrating multimodal features as claimed in claim 1, characterized in that: The method of concatenating the text data to be identified and the label to obtain an output sequence by using a preset prompt strategy includes: Convert the label into the standard language of the text data to be identified to obtain a natural language text; Based on the preset prompt strategy, generating prompt information of the natural language text as question information; Based on the question information and the text data to be recognized, answer information is generated.

3. The method for identifying a twin-tower entity by integrating multimodal features as claimed in claim 1, characterized in that: The method of obtaining the inscription image features, tactile perception features and visual perception features based on the output sequence includes: Constructing an image data set, wherein the image data set includes a deed image, a preset font image, and a Braille image; Matching each character in the output sequence with the inscription image, the preset font image and the Braille image to obtain the inscription image, the preset font image and the Braille image corresponding to each character; Encoding the inscription image, the preset font image and the Braille image corresponding to each character into single-channel image features respectively, so as to serve as the inscription image features, the tactile perception features and the visual perception features; Among them, the described Qiwen image features are used to study the semantic correction effect of the source of Chinese characters on modern Chinese characters, the described visual perception features are used to study the enhancement effect of handwriting on the visual perception semantic space of modern Chinese characters, and the described tactile perception features are used to study the semantic expansion effect of tactile perception semantic space on modern Chinese characters.

4. The method for identifying a twin-tower entity by integrating multimodal features as claimed in claim 1 or 3, characterized in that: The step of fusing the inscription image features, tactile perception features, and visual perception features to obtain a dense image feature vector includes: splicing the inscription image features, tactile perception features, and visual perception features to obtain splicing features; Affine processing is performed on the splicing features to obtain the dense feature vector of the image.

5. The method for identifying a twin-tower entity by integrating multimodal features as claimed in claim 1, characterized in that: The step of determining the starting boundary and the ending boundary of the entity to be extracted based on the fusion feature includes: The fused features are respectively input into the head identifier and the tail identifier, and the fused features are activated by a preset activation function to obtain an activation value, wherein the activation value represents the starting boundary probability or the ending boundary probability of the entity to be extracted; Based on the starting boundary probability or the ending boundary probability, the starting boundary or the ending boundary of the entity to be extracted is determined.

6. The method for identifying a twin-tower entity by integrating multimodal features as claimed in claim 1 or 5, characterized in that: The starting boundary and the ending boundary include a plurality of boundaries respectively, and extracting the entity fragment of the entity to be extracted based on the starting boundary and the ending boundary includes: Pairing the plurality of start boundaries and end boundaries to obtain a plurality of entity segments, wherein the entity segments include a start boundary and an end boundary matching the start boundary; The entity fragments are extracted.

7. The method for identifying a twin-tower entity by integrating multimodal features as claimed in claim 1, characterized in that: The step of fusing the image dense feature vector and the text dense feature vector to obtain a fused feature includes: Performing a horizontal or vertical augmentation operation on the image dense feature vector and the text dense feature vector; Affine processing is performed on the image dense feature vector and the text dense feature vector after the augmentation operation by using a preset affine function to obtain the fusion feature.

8. A dual-tower entity recognition device integrating multimodal features, characterized in that: The device comprises: A to-be-recognized text data acquisition unit, used to acquire the to-be-recognized text data, wherein the to-be-recognized text data includes a label; A text and label combining unit, used to concatenate the text data to be identified and the label through a preset prompt strategy to obtain an output sequence; An image dense feature vector generating unit is used to obtain the inscription image features, tactile perception features and visual perception features based on the output sequence, and to perform feature fusion on the inscription image features, tactile perception features and visual perception features to obtain an image dense feature vector; A text dense feature vector generating unit, used for predicting and obtaining a text dense feature vector based on the output sequence; A multimodal feature fusion unit, used for fusing the image dense feature vector and the text dense feature vector to obtain a fused feature; The entity recognition unit is used to determine the starting boundary and the ending boundary of the entity to be extracted based on the fusion feature, so as to extract the entity fragment of the entity to be extracted based on the starting boundary and the ending boundary.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executed on the processor, characterized in that: When the processor executes the computer-readable instructions, the double-tower entity recognition method integrating multimodal features as described in any one of claims 1 to 7 is implemented.

10. A readable storage medium having computer readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the double-tower entity recognition method integrating multimodal features as described in any one of claims 1 to 7 is implemented.

Images