Model training and named entity recognition method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a model training and named entity recognition method and device, electronic equipment and a storage medium. The model training method comprises: obtaining a target text and a picture description text of an associated picture, the associated picture matching the target text; fusing the target text and the picture description text to obtain a fused text; and training a named entity recognition model based on the fused text and a named entity label of the target text. The embodiments of the present application improve the training effect and recognition effect of the named entity recognition model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model training and named entity recognition method, apparatus, electronic device and storage medium. Background Technology

[0002] Named Entity Recognition (NER) refers to the identification of entities with specific meanings in text, mainly including names of people, places, organizations, and proper nouns. NER is an important foundational tool for natural language processing tasks such as information extraction, question answering systems, syntactic analysis, and machine translation.

[0003] The accuracy of named entity recognition determines the effectiveness of downstream natural language processing tasks. Current named entity recognition does not fully consider the contextual semantic factors in the text to be recognized, resulting in limited recognition accuracy. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a model training and named entity recognition method, apparatus, electronic device, and storage medium to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of the present invention, a model training method is provided, comprising: acquiring target text and image description text of an associated image, wherein the associated image matches the target text; fusing the target text and the image description text to obtain fused text; and training a named entity recognition model based on the named entity tags of the fused text and the target text.

[0006] In another implementation of the present invention, obtaining the image description text of the associated image includes: inputting the associated image into a pre-trained image description model to obtain the image description text.

[0007] In another implementation of the present invention, the step of fusing the target text and the image description text to obtain fused text includes: concatenating the dimensional representation of the target text and the dimensional representation of the image description text to obtain fused text.

[0008] In another implementation of the present invention, the named entity recognition model includes a context fusion layer and a conditional random field (CRF) processing layer, wherein the input of the context fusion layer is connected to the input of the CRF processing layer. Training the named entity recognition model based on the fused text and the named entity tags of the target text includes: training the named entity recognition model using the fused text as input to the context fusion layer and the named entity tags of the target text as output to the CRF processing layer.

[0009] In another implementation of the present invention, the named entity recognition model includes a dimension alignment layer, the input of the context fusion layer is connected to the input of the conditional random field processing layer via the dimension alignment layer, and the dimension alignment layer is used to extract the features of context fusion from the dimensions of the fused text to the dimensions of the target text.

[0010] In another implementation of the present invention, the context fusion layer is a transformer encoder.

[0011] According to a second aspect of the present invention, a named entity recognition method is provided, comprising: acquiring a text to be recognized and an associated image matching the text to be recognized; extracting image description text of the associated image; fusing the text to be recognized and the image description text to obtain a fused text; and inputting the fused text into a named entity recognition model to obtain named entity information of the text to be recognized, wherein the named entity recognition model is trained by the method described in the first aspect.

[0012] According to a third aspect of the present invention, a named entity recognition method is provided, comprising: acquiring product description text and a product image corresponding to the product description text; extracting image description text from the product image; fusing the product description text and the image description text to obtain fused text; and inputting the fused text into a named entity recognition model to obtain named entity information of the product description text, wherein the named entity recognition model is trained by the method described in the first aspect.

[0013] According to a fourth aspect of the present invention, a model training apparatus is provided, comprising: an acquisition module for acquiring target text and image description text of an associated image, wherein the associated image matches the target text; a fusion module for fusing the target text and the image description text to obtain fused text; and a training module for training a named entity recognition model based on the named entity tags of the fused text and the target text.

[0014] According to a fifth aspect of the present invention, a named entity recognition device is provided, comprising: an acquisition module for acquiring text to be recognized and associated images matching the text to be recognized; an extraction module for extracting image description text of the associated images; a fusion module for fusing the text to be recognized and the image description text to obtain fused text; and a recognition module for inputting the fused text into a named entity recognition model to obtain named entity information of the text to be recognized, wherein the named entity recognition model is trained by the method described in the first aspect.

[0015] According to a sixth aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first aspect.

[0016] According to a seventh aspect of the present invention, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0017] In the embodiment of the present invention, the fused text incorporates information from the target text and its associated images. Training is performed based on the fused text, which, compared to the target text, adds contextual semantic factors during training, improving the recognition effect of the named entity recognition model. Furthermore, the fused text includes image description text of associated images, which belongs to the text space and is easy to fuse with the target text, thereby improving the training effect. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 A schematic block diagram of an example named entity recognition method.

[0020] Figure 2A This is a flowchart of the steps of a model training method according to an embodiment of the present invention.

[0021] Figure 2B for Figure 2A The flowchart of the corresponding named entity recognition method.

[0022] Figure 3A This is a schematic diagram of an image description generation method according to another embodiment of the present invention.

[0023] Figure 3B for Figure 2A and Figure 2B A schematic block diagram of an example text processing flow of an embodiment.

[0024] Figure 4 This is a structural block diagram of an apparatus according to another embodiment of the present invention.

[0025] Figure 5This is a structural block diagram of an apparatus according to another embodiment of the present invention.

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device according to another embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0028] The specific implementation of the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0029] Figure 1 A schematic block diagram of an example named entity recognition method. Figure 1 The named entity recognition process uses a pre-trained named entity recognition model 120. Specifically, the target text 110 is input into the named entity recognition model 120 to obtain named entity information 130.

[0030] NER is a sequence labeling problem, and the data labeling methods follow the same principles as sequence labeling problems, primarily BIO and BIOES. As an example, the explanations of the various labeling symbols in BIOES are as follows:

[0031] B stands for Begin, indicating the start; I stands for Intermediate, indicating the middle; E stands for End, indicating the end; S stands for Single, indicating a single character; O stands for Other, indicating other characters and used to mark irrelevant characters.

[0032] Specifically, when the target text is the sentence "[Bob and Alice posing for a picture]", the named entity information obtained through the above-mentioned named entity annotation process is "[B-PER,E-PER,O,B-PER,E-PER O,O,O,O]", and subsequent natural language processing related to the target text can be performed based on the named entity information.

[0033] Figure 2AThis is a flowchart illustrating the steps of a model training method according to an embodiment of the present invention. The solution of this embodiment can be applied to any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, tablets, etc.), and PCs. For example, in the model training phase, a computing device (e.g., a data center) configured with a CPU (processor unit) + GPU (acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. The computing device, such as a data center, can be deployed in cloud servers such as dedicated clouds, private clouds, or hybrid clouds. Correspondingly, in the inference phase, a computing device configured with a CPU (processor unit) + GPU (acceleration unit) architecture can also be used for inference operations.

[0034] The model training method in this embodiment includes:

[0035] S210: Obtain the target text and the image description text of the associated image, and match the associated image with the target text.

[0036] It should be understood that the text in the embodiments of the present invention only includes text in the form of characters (including words and Chinese characters), sentences, paragraphs, and chapters. The text in the training samples may not have undergone word embedding processing (characters as units), and word embedding processing may be performed before participating in training; the text in the training samples may be text that has undergone word embedding processing, and the text after word embedding processing can directly participate in model training.

[0037] It should also be understood that target text and associated images can have a matching relationship. In one example, the descriptive object of the target text matches or is consistent with the descriptive object of the associated image. The descriptive object can be an abstract event or an object. For example, the target text can indicate the descriptive information of the target object or product included in the associated image, the positional relationship between target objects, event relationships, etc.

[0038] It should also be understood that image description text can be obtained by annotating associated images, by performing object detection on associated images to obtain text descriptions of target objects, or by using a pre-trained image description model to identify associated images and obtain image description text. Image description text can be at least one sentence, at least one paragraph, at least one character, or multiple character texts or multiple sentence texts combined in a predetermined manner. The predetermined manner can indicate a random sorting method or a sorting method based on contextual semantics. The following will combine... Figure 3A A detailed description of the image description model is provided.

[0039] S220: Merge the target text and the image description text to obtain the merged text.

[0040] It should be understood that the fusion process can be achieved by adding the dimensional representations of the individual texts. That is, the target text and the image description text are aligned, and then each element in the dimensional representation of the target text is added to each element in the dimensional representation of the image description text to obtain the dimensional representation of the fused text. Alternatively, a concatenation process can be used. For example, each element in the dimensional representation of the target text is concatenated with each element in the dimensional representation of the image description text, resulting in a fused text whose number of dimensions is the sum of the number of dimensions of the target text and the number of dimensions of the image description text.

[0041] It should also be understood that when each text is represented by at least one word embedding, the dimension of the text refers to the dimension corresponding to the word embedding, not the dimension of the word embedding itself. In other words, the number of word embeddings in the text corresponds to the number of dimensions of the text.

[0042] S230: Train a named entity recognition model based on the named entity tags of the fused text and the target text.

[0043] It should be understood that the training process in this embodiment can be supervised training, and the model used for training can be any neural network model, such as a feedforward neural network for classification, a transformer-based neural network, a neural network based on RNN, CNN, or LSTM, etc.

[0044] In the embodiment of the present invention, the fused text incorporates information from the target text and its associated images. Training is performed based on the fused text, which, compared to the target text, adds contextual semantic factors during training, improving the recognition effect of the named entity recognition model. Furthermore, the fused text includes image description text of associated images, which belongs to the text space and is easy to fuse with the target text, thereby improving the training effect.

[0045] Figure 2B for Figure 2A The corresponding named entity recognition method's flowchart is shown. The named entity recognition method in this embodiment includes:

[0046] S260: Obtain the text to be recognized and the associated image that matches the text to be recognized.

[0047] S270: Extract the image description text of the associated image.

[0048] S280: Merge the text to be recognized with the image description text to obtain the merged text.

[0049] S290: Input the fused text into the named entity recognition model to obtain the named entity information of the text to be recognized. The named entity recognition model is trained by the model training method.

[0050] In other words, the text to be recognized in the named entity recognition stage corresponds to the target text in the model training stage, and the associated image is related to the text to be recognized.

[0051] In one scenario, the text to be identified can be product description text, and the associated image can be a product image. Users can provide the target product image and its description text to the server. The server can deploy a named entity recognition model to perform named entity recognition processing on the product description text based on the product image and the product description text. Then, knowledge information, such as multimedia information, can be constructed based on the product description text annotated with named entities. Accordingly, when a search request for a product is received, the server can obtain accurate user search intent based on this knowledge information, thereby providing recommendations or accurate search results to the user.

[0052] In other examples, obtaining the image description text of associated images includes: inputting the associated images into a pre-trained image description model to obtain the image description text. This provides accurate information about the associated images and enables information fusion processing between the target text and associated images in the text space.

[0053] In other examples, the target text and image description text are merged to obtain the merged text, including: concatenating the dimensional representations of the target text and the image description text to obtain the merged text.

[0054] Therefore, the concatenation operation improves the data processing efficiency of merging target text and image description text, and increases the flexibility of data processing compared to the method of performing addition processing based on the dimensional representation of both after alignment processing.

[0055] In other examples, the named entity recognition model includes a context fusion layer and a conditional random field (CRF) processing layer, with the input of the context fusion layer connected to the input of the CRF processing layer. Furthermore, the named entity recognition model can be trained based on the fused text as input to the context fusion layer and on the named entity labels of the target text as output to the CRF processing layer.

[0056] In other examples, the named entity recognition model includes a dimension alignment layer. The input to the context fusion layer is connected to the input to the Conditional Random Field (CRF) processing layer via the dimension alignment layer. The dimension alignment layer is used to extract the features of context fusion from the dimensions of the fused text to the dimensions of the target text. Thus, the dimensions of the text input to the CRF processing layer match the dimensions of the output named entity labels, enabling the context fusion of the target text and the image description text to be performed relatively independently of the CRF processing layer. In other words, the dimension alignment layer decouples the context fusion layer from the CRF processing layer.

[0057] More specifically, the context fusion layer is a transformer encoder. Transformer encoders are highly capable of processing characters in text sequences, and their attention mechanism effectively fuses various dimensions of features for context fusion. Therefore, compared to the method of aligning the target text and image description text before adding them together, the training process is significantly easier. Furthermore, since the image description text resides in text space rather than image space, the context fusion effect is improved, which benefits the generalization ability of the trained named entity recognition model.

[0058] It can be adopted Figure 3A The image description generation method shown processes associated images to obtain image description text. (Reference) Figure 3A The target text 301 is “[Bob and Alice posing for a picture]”, and the image that matches the target text is the associated image 302. The associated image 302 is input into the pre-trained image description model 3000 to obtain the image description text 303 “[Alice hair Bob tie]” or its corresponding “[Bob wearing a tie next to Alice with hair]”.

[0059] It should be understood that the image description model 3000 can be a model such as VinVL. The number of dimensions of the target text 301 can be the same as or different from the number of dimensions of the image description text 303. In this example, the target text 301 has 9 dimensions, that is, the target text 301 includes 9 words (for example, characters). As an example, the target text 301 can be a text vector obtained after word embedding of these 9 words. Furthermore, the image description text 303 has 4 dimensions, that is, the image description text 303 includes 4 words. In fact, the words included in the image description text 303 have contextual semantic relationships. These words do not necessarily form a coherent sentence, and their order can be varied. Preferably, an order that better reflects the contextual semantic relationships can be chosen. In this example, "[Alicehair Bob tie]" is one possible order, but other ordering methods can also be used, such as "[tieBobAlicehair]". Since the association between "tie" and "Bob" is greater than that between "tie" and "Alice", "[tie BobAlice hair]" reflects the more accurate contextual semantic relationships than "[tie AliceBob hair]". Furthermore, "Alice" and "Bob" are more important features than "tie" and "hair", therefore, "[Alice hair Bob tie]" reflects the more accurate contextual semantic relationships than "[tie BobAlice hair]".

[0060] Furthermore, Figure 3B An exemplary text processing framework is shown, and the following sections will focus on the training and inference phases respectively. Figure 3B The framework explains and describes the text processing flow of this example.

[0061] The text processing framework in this example includes a transformer encoder 310, a dimension alignment layer 320, and a CRF layer 330 connected in sequence.

[0062] During the training phase, the target text "[Bob and Alice posing for a picture]" and the picture description text "[Alice hair Bob tie]" are concatenated to obtain the merged text "[Bob and Alice posing for a picture]". <x>Alice hair Bob tie].

[0063] Accordingly, during the model inference phase, the text to be identified can be used as the target text in the text processing framework of this example, and the associated image can be matched with the text to be identified.

[0064] It should be understood that in the merged text, the position of the target text and the position of the image description text are distinguished by specific symbols. For example, each character in the image description text may be marked with a symbol before or after it. <x>Alternatively, it can split the target text string and the image description text string based on specific symbols.

[0065] It should also be understood that the characters in the image description text can be continuous strings or scattered strings. For example, the image description text consists of a first part and a second part, and in the merged text, the target text is located between the first part and the second part.

[0066] In this example, the target text has 9 dimensions, the image description text has 4 dimensions, and the fused text has 13 dimensions. As a concrete example, the fused text is "[Bob and Alice posing for a picture]". <x>Alice hair Bob tie].

[0067] Then, the fused text is input into the named entity recognition model, passing sequentially through a transformer encoder 310, a dimension alignment layer 320, and a CRF layer 330 to obtain the named entity information of the target text: "[B-PER,E-PER,O,B-PER,E-PERO,O,O,O]". Specifically, the 13-dimensional fused text is input into the transformer encoder 310 to perform contextual semantic fusion between characters in each dimension, and outputs 13-dimensional text features accordingly.

[0068] Then, the 13-dimensional text features are input to the dimension alignment layer 320 for dimension alignment processing to align with the number of dimensions of the target text, in other words, the number of input dimensions of the CRF layer 330. In this example, 9-dimensional text features are determined from the 13-dimensional text features and used as the input to the CRF layer 330. At this point, the 9-dimensional text features incorporate the features of the image description text, that is, they incorporate information related to the image.

[0069] Then, the CRF layer 330 processes the input 9-dimensional text features and outputs 9-dimensional named entity information "[B-PER,E-PER,O,B-PER,E-PER O,O,O,O]".

[0070] Specifically, word features are fed into a CRF layer to obtain conditional probabilities:

[0071]

[0072] Where ψ is the potential function, θ represents the model parameters, and Y represents the set of all possible label sequences for a given sentence. y_0 is defined as a special start symbol.

[0073] The dimension alignment layer is used to extract the features of context fusion from the dimensions of the fused text to the dimensions of the target text. That is, it processes the 13-dimensional text features to obtain 9-dimensional text features. For example, based on the position indicated by the specific symbol mentioned above, the dimensions of the target text can be truncated from the dimensions of the fused text. In this example, the first 9 characters are truncated from the 13-dimensional text features as 9-dimensional text features.

[0074] Figure 4 This is a structural block diagram of an apparatus according to another embodiment of the present invention. The solution of this embodiment can be applied to any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, tablets, etc.), and PCs. For example, in the model training phase, a computing device (e.g., a data center) configured with a CPU (processor unit) + GPU (acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. The computing device, such as a data center, can be deployed in cloud servers such as dedicated clouds, private clouds, or hybrid clouds. Correspondingly, in the inference phase, a computing device configured with a CPU (processor unit) + GPU (acceleration unit) architecture can also be used for inference operations.

[0075] The model training device in this embodiment includes:

[0076] The acquisition module 410 acquires the target text and the image description text of the associated image, wherein the associated image matches the target text;

[0077] The fusion module 420 fuses the target text and the image description text to obtain fused text;

[0078] Training module 430 trains a named entity recognition model based on the named entity tags of the fused text and the target text.

[0079] In the embodiment of the present invention, the fused text incorporates information from the target text and its associated images. Training is performed based on the fused text, which, compared to the target text, adds contextual semantic factors during training, improving the recognition effect of the named entity recognition model. Furthermore, the fused text includes image description text of associated images, which belongs to the text space and is easy to fuse with the target text, thereby improving the training effect.

[0080] In other examples, the acquisition module is specifically used to: input associated images into a pre-trained image description model to obtain image description text.

[0081] In other examples, the fusion module is specifically used to: concatenate the dimensional representation of the target text and the dimensional representation of the image description text to obtain fused text.

[0082] In other examples, the named entity recognition model includes a context fusion layer and a conditional random field (CRF) processing layer, with the input of the context fusion layer connected to the input of the CRF processing layer. The training module is specifically used to: train the named entity recognition model based on the fused text as input to the context fusion layer and based on the named entity tags of the target text as output to the CRF processing layer.

[0083] In other examples, the named entity recognition model includes a dimension alignment layer, through which the input of the context fusion layer is connected to the input of the conditional random field processing layer, the dimension alignment layer being used to extract the features of context fusion from the dimensions of the fused text to the dimensions of the target text.

[0084] The apparatus of this embodiment is used to implement the corresponding methods in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each module in the apparatus of this embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0085] Figure 5 This is a structural block diagram of a device according to another embodiment of the present invention. The named entity recognition device of this embodiment includes:

[0086] The acquisition module 510 acquires the text to be recognized and the associated image matching the text to be recognized;

[0087] Extraction module 520 extracts the image description text of the associated image;

[0088] The fusion module 530 fuses the text to be identified with the image description text to obtain fused text;

[0089] The recognition module 540 inputs the fused text into the named entity recognition model to obtain the named entity information of the text to be recognized. The named entity recognition model is trained by the method according to any one of claims 1-6.

[0090] In the embodiment of the present invention, the fused text incorporates information from the target text and its associated images. Training is performed based on the fused text, which, compared to the target text, adds contextual semantic factors during training, thus improving the recognition performance of the named entity recognition model. Furthermore, the fused text includes image description text of the associated images, which belongs to the text space and is easy to fuse with the target text, thereby improving the training effect and ultimately enhancing the recognition performance of the trained model.

[0091] Reference Figure 6 The diagram shows a schematic of an electronic device according to another embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the electronic device.

[0092] like Figure 6 As shown, the electronic device may include: a processor 602, a communications interface 604, a memory 606, and a communications bus 608.

[0093] in:

[0094] The processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608.

[0095] Communication interface 604 is used for communication with other electronic devices or servers.

[0096] The processor 602 is used to execute program 610, specifically the relevant steps in the above method embodiments.

[0097] Specifically, program 610 may include program code that includes computer operation instructions.

[0098] Processor 602 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0099] Memory 606 is used to store program 610. Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0100] Specifically, program 610 can be used to cause processor 602 to perform the following operations: obtain target text and image description text of associated images, wherein the associated images match the target text; fuse the target text and the image description text to obtain fused text; and train a named entity recognition model based on the named entity tags of the fused text and the target text.

[0101] Alternatively, program 610 can specifically be used to cause processor 602 to perform the following operations: acquire the text to be identified and the associated image matching the text to be identified; extract the image description text of the associated image; fuse the text to be identified and the image description text to obtain fused text; input the fused text into a named entity recognition model to obtain the named entity information of the text to be identified, wherein the named entity recognition model is trained by a model training method.

[0102] Furthermore, the specific implementation of each step in procedure 610 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0103] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of the present invention can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0104] The methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0105] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of the present invention.

[0106] The above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims.< / x> < / x> < / x>

Claims

1. A model training method, comprising: Obtain the target text and the image description text of the associated image, wherein the associated image matches the target text; The target text and the image description text are merged to obtain merged text, wherein the merged text includes the target text, the image description text, and a preset symbol, and the preset symbol distinguishes the position of the target text from the position of the image description text; Based on the named entity tags of the fused text and the target text, a named entity recognition model is trained. The named entity recognition model is used to fuse the contextual semantics of the fused text to obtain context-fused text features. The named entity recognition model is also used to extract the text features from the dimension of the fused text to the dimension of the target text using the preset symbols. The step of obtaining the image description text of the associated image includes: The associated image is input into a pre-trained image description model to obtain the image description text, wherein each text in the image description text is ordered according to the context semantics.

2. The method according to claim 1, wherein, The fusion of the target text and the image description text to obtain the fused text includes: By concatenating the dimensional representations of the target text and the image description text, a fused text is obtained.

3. The method according to claim 1, wherein, The named entity recognition model includes a context fusion layer and a conditional random field (CRF) processing layer, wherein the input of the context fusion layer is connected to the input of the CRF processing layer. The process of training a named entity recognition model based on the named entity tags of the fused text and the target text includes: A named entity recognition model is trained based on the fused text as input to the context fusion layer and the named entity tags of the target text as output to the conditional random field processing layer.

4. The method according to claim 3, wherein, The named entity recognition model includes a dimension alignment layer. The input of the context fusion layer is connected to the input of the conditional random field processing layer via the dimension alignment layer. The dimension alignment layer is used to extract the features of context fusion from the dimensions of the fused text to the dimensions of the target text.

5. The method according to claim 3, wherein, The context fusion layer is a transformer encoder.

6. A named entity recognition method, comprising: Obtain the text to be identified, and the associated image that matches the text to be identified; Extract the image description text of the associated images; The text to be identified and the image description text are merged to obtain fused text, wherein the fused text includes the text to be identified, the image description text, and a preset symbol, and the preset symbol distinguishes the position of the text to be identified from the position of the image description text; The fused text is input into the named entity recognition model to obtain the named entity information of the text to be recognized. The named entity recognition model is used to fuse the contextual semantics of the fused text to obtain context-fused text features. The named entity recognition model is also used to extract the text features from the dimension of the fused text to the dimension of the text to be recognized using the preset symbol. The named entity recognition model is trained by the method according to any one of claims 1-5. The step of extracting the image description text of the associated image includes: The associated image is input into a pre-trained image description model to obtain the image description text, wherein each text in the image description text is ordered according to the context semantics.

7. A named entity recognition method, comprising: Obtain the product description text and the product image corresponding to the product description text; Extract the image description text from the product image; The product description text and the image description text are merged to obtain merged text, wherein the merged text includes the product description text, the image description text, and a preset symbol, and the preset symbol distinguishes the position of the product description text from the position of the image description text; The fused text is input into the named entity recognition model to obtain the named entity information of the product description text. The named entity recognition model is used to fuse the contextual semantics of the fused text to obtain context-fused text features. The named entity recognition model is also used to extract the text features from the dimension of the fused text to the dimension of the product description text using the preset symbol. The named entity recognition model is trained by the method according to any one of claims 1-5. The step of extracting the image description text of the product image includes: The product image is input into a pre-trained image description model to obtain the image description text, wherein each text in the image description text is ordered according to the context semantics.

8. A model training device, comprising: The acquisition module is used to acquire target text and image description text of associated images, wherein the associated images are matched with the target text; A fusion module is used to fuse the target text and the image description text to obtain fused text, wherein the fused text includes the target text, the image description text, and a preset symbol, and the preset symbol distinguishes the position of the target text from the position of the image description text; The training module is used to train a named entity recognition model based on the named entity tags of the fused text and the target text. The named entity recognition model is used to fuse the contextual semantics of the fused text to obtain contextually fused text features. The named entity recognition model is also used to extract the text features from the dimension of the fused text to the dimension of the target text using the preset symbols. The acquisition module is further configured to input the associated image into a pre-trained image description model to obtain the image description text, wherein each text in the image description text is sorted according to the context semantics.

9. A named entity recognition device, comprising: The acquisition module is used to acquire the text to be identified and the associated image that matches the text to be identified; The extraction module is used to extract the image description text of the associated images; A fusion module is used to fuse the text to be identified and the image description text to obtain fused text, wherein the fused text includes the text to be identified, the image description text, and a preset symbol, and the preset symbol distinguishes the position of the text to be identified from the position of the image description text; The recognition module is used to input the fused text into the named entity recognition model to obtain the named entity information of the text to be recognized. The named entity recognition model is used to fuse the contextual semantics of the fused text to obtain context-fused text features. The named entity recognition model is also used to extract the text features from the dimension of the fused text to the dimension of the text to be recognized using the preset symbol. The named entity recognition model is trained by the method according to any one of claims 1-5. The extraction module is further configured to input the associated image into a pre-trained image description model to obtain the image description text, wherein each text in the image description text is ordered according to the context semantics.

10. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-7.

11. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-7.

Images