Data processing method and computer readable storage medium

Abstract

The present disclosure provides a data processing method and a computer readable storage medium, the data processing method is used for training a named entity recognition model, comprising: obtaining a labeled training sample pair and an unlabeled training sample pair; for each training sample pair, obtaining the corresponding latent representation features of the corresponding training sample pair and fusing, and then obtaining a first prediction result of the labeled training sample pair and a second prediction result of the unlabeled training sample pair by performing named entity prediction; obtaining the reconstruction features of each training sample pair according to the latent representation features of each training sample pair; determining three loss functions based on the first prediction result, the second prediction result, each sample pair and the reconstruction features of each sample pair; and training the named entity recognition model according to the three loss functions. The present disclosure adopts semi-supervised training, which can reduce the training cost while ensuring the accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of Artificial Intelligence (AI), and in particular, to a data processing method and a computer readable storage medium. BACKGROUND

[0002] Multi-modal named entity recognition has become a basic task of extracting named entities from unstructured texts and images on social media. Compared with traditional named entity recognition, multi-modal named entity recognition on social media presents a unique challenge of narrowing the semantic gap between multiple modal data. The purpose of multi-modal named entity recognition is to identify the named entities carried therein by combining multiple types of information.

[0003] The current conventional way of training and identifying a multi-modal named entity recognition model is supervised training, that is, the training data of each modality needs to be labeled, and the cost of labeling is relatively high, resulting in a relatively high cost of training and identifying a multi-modal named entity recognition model. Moreover, the existing multi-modal named entity recognition usually only focuses on certain modal data, resulting in inaccurate recognition results. SUMMARY

[0004] The present disclosure provides a data processing method and a computer readable storage medium

[0005] In a first aspect, the present disclosure provides a data processing method for training a named entity recognition model, wherein the named entity recognition model comprises a named entity prediction network and at least two recognition networks corresponding to different modal data; the data processing method comprises:

[0006] obtaining a labeled training sample pair and an unlabeled training sample pair; wherein one training sample pair is composed of at least two modal data;

[0007] identifying the data of the corresponding modal in the labeled training sample pair and the unlabeled training sample pair by the recognition network corresponding to each modal data, respectively, to obtain the first latent representation feature of each modal data in the labeled training sample pair and the second latent representation feature of each modal data in the unlabeled training sample pair;

[0008] fusing and predicting the named entity of the first latent representation feature of each modal data in the labeled training sample pair by the named entity prediction network to obtain the first prediction result of the labeled training sample pair, and fusing and predicting the named entity of the second latent representation feature of each modal data in the unlabeled training sample pair by the named entity prediction network to obtain the second prediction result of the unlabeled training sample pair;

[0009] decoding the first latent representation feature of each modality data in the pair of labeled training samples to obtain a reconstruction feature of each modality data in the pair of labeled training samples, and performing reconstruction processing based on the second latent representation feature of each modality data in the pair of unlabeled training samples and the second prediction result to obtain a reconstruction feature of each modality data in the pair of unlabeled training samples;

[0010] determining a first loss function based on the first prediction result of the pair of labeled training samples and the label of the labeled training sample, determining a second loss function based on the pair of labeled training samples and the reconstruction features of the pair of labeled training samples, and determining a third loss function based on the pair of unlabeled training samples and the reconstruction features of the pair of unlabeled training samples;

[0011] training the named entity recognition model according to the first loss function, the second loss function, and the third loss function.

[0012] In a second aspect, the present disclosure provides a named entity recognition method, which is executed by invoking a named entity recognition model trained by the above data processing method. The method comprises:

[0013] inputting at least two kinds of modality data into the named entity recognition model, respectively; the at least two kinds of modality data have a semantic correlation relationship;

[0014] performing recognition processing on the corresponding modality data through the recognition network corresponding to each modality data in the named entity recognition model to obtain the latent representation feature corresponding to each modality data;

[0015] fusing the latent representation features of the at least two kinds of modality data through the named entity prediction network in the named entity recognition model;

[0016] performing named entity recognition based on the fused latent representation features to obtain the multi-modal named entity contained in the at least two kinds of modality data.

[0017] In a third aspect, the present disclosure provides a data processing apparatus, which comprises:

[0018] a training unit configured to train a named entity recognition model, wherein the named entity recognition model comprises a named entity prediction network and a recognition network corresponding to at least two kinds of modality data, respectively; and the training unit comprises:

[0019] a obtaining unit configured to obtain a pair of labeled training samples and a pair of unlabeled training samples; wherein one pair of training samples is composed of at least two kinds of modality data;

[0020] The recognition unit is configured to respectively recognize the data of the corresponding modal in the labeled training sample pair and the unlabeled training sample pair by an identification network corresponding to each modal data, to obtain a first latent representation feature corresponding to each modal data in the labeled training sample pair, and a second latent representation feature corresponding to each modal data in the unlabeled training sample pair;

[0021] The processing unit is configured to fuse and perform named entity prediction on the first latent representation feature of each modal data in the labeled training sample pair by the named entity prediction network, to obtain a first prediction result of the labeled training sample pair, and fuse and perform named entity prediction on the second latent representation feature of each modal data in the unlabeled training sample pair by the named entity prediction network, to obtain a second prediction result of the unlabeled training sample pair;

[0022] The reconstruction unit is configured to decode the first latent representation feature of each modal data in the labeled training sample pair to obtain a reconstructed feature of each modal data in the labeled training sample pair, and reconstruct based on the second latent representation feature of each modal data in the unlabeled training sample pair and the second prediction result to obtain a reconstructed feature of each modal data in the unlabeled training sample pair. The determination unit is configured to determine a first loss function based on the first prediction result of the labeled training sample pair and the label of the labeled training sample, determine a second loss function based on the labeled training sample pair and the reconstructed feature of the labeled training sample pair, and determine a third loss function based on the unlabeled training sample pair and the reconstructed feature of the unlabeled training sample pair.

[0023] The training unit is configured to train the named entity recognition model according to the first loss function, the second loss function and the third loss function.

[0024] In a fourth aspect, the present disclosure provides a named entity recognition device, which is called by a named entity recognition model and is executed. The named entity recognition model is trained by the above data processing method. The named entity recognition device comprises:

[0025] The input unit is configured to input at least two modal data into the named entity recognition model respectively.

[0026] The recognition unit is configured to respectively recognize the data of the corresponding modal in the labeled training sample pair and the unlabeled training sample pair by an identification network corresponding to each modal data, to obtain a first latent representation feature corresponding to each modal data in the labeled training sample pair, and a second latent representation feature corresponding to each modal data in the unlabeled training sample pair;

[0027] a fusion unit configured to fuse latent representation features of the at least two modal data by a named entity prediction network in the named entity recognition model; and the recognition unit is further configured to perform named entity recognition based on the fused latent representation features to obtain multi-modal named entities contained in the at least two modal data.

[0028] In a fifth aspect, the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the above-mentioned data processing method and named entity recognition method.

[0029] In a sixth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned data processing method and named entity recognition method.

[0030] In a seventh aspect, the present disclosure provides a computer program or a computer program product, the computer program product comprising a computer program stored in a computer-readable storage medium, and the computer program, when executed by a processor, implements the above-mentioned named entity recognition method.

[0031] In the embodiments provided by the present disclosure, the named entity recognition model to be trained includes a named entity prediction network and at least two identification networks corresponding to the recognition network of each modality data, and then a pair of labeled training samples and a pair of unlabeled training samples are obtained; wherein one training sample pair is composed of at least two kinds of modality data. Subsequently, the data of the corresponding modality in the labeled training sample pair and the unlabeled training sample pair are respectively identified by the identification network corresponding to each modality data to obtain the first latent representation feature corresponding to each modality data in the labeled training sample pair and the second latent representation feature corresponding to each modality data in the unlabeled training sample pair. The first latent representation feature of each modality data in the labeled training sample pair is fused and the named entity is predicted by the named entity prediction network to obtain the first prediction result of the labeled training sample pair, and the second latent representation feature of each modality data in the unlabeled training sample pair is fused and the named entity is predicted to obtain the second prediction result of the unlabeled training sample pair. The first latent representation feature of each modality data in the labeled training sample pair is decoded to obtain the reconstruction feature of each modality data in the labeled training sample pair, and the second latent representation feature of each modality data in the unlabeled training sample pair and the second prediction result are reconstructed to obtain the reconstruction feature of each modality data in the unlabeled training sample pair. Then, the first loss function is determined based on the first prediction result of the labeled training sample pair and the label of the labeled training sample, the second loss function is determined based on the labeled training sample pair and the reconstruction feature of the labeled training sample pair, and the third loss function is determined based on the unlabeled training sample pair and the reconstruction feature of the unlabeled training sample pair, and the named entity recognition model is trained according to the first loss function, the second loss function and the third loss function. The data used to train the named entity recognition model to be trained includes a pair of labeled training samples and a pair of unlabeled training samples. It can be seen that, compared with the traditional way of labeling all modality training data, part of the training samples involved in the embodiments of the present disclosure are labeled training samples, and the other part is labeled training samples, that is, the embodiments of the present disclosure do not need to label all multi-modal training samples, but only part of them. The training method of the embodiments of the present disclosure is semi-supervised training, which can reduce the training cost. In addition, in the embodiments of the present disclosure, the latent representation feature corresponding to each modality data is identified, and then the subsequent calculation and model training are performed based on the latent representation features of various modality data. It can be understood that the semantic features of each modality are mapped to the same dimension, which breaks the semantic cross-modal problem compared with the traditional way of directly fusing multi-modal semantic features, so that the alignment effect of the fused features on the semantic of each modality data is relatively better, and the granularity of the latent representation feature is relatively finer, thereby facilitating the optimization of the recognition effect of multi-modal named entities.

[0032] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings are included to provide a further understanding of the present disclosure and constitute a part of the specification, which together with the embodiments of the present disclosure serve to explain the present disclosure, and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent from the detailed description of exemplary embodiments described below, with reference to the accompanying drawings, in which:

[0034] Figure 1 A flowchart of a data processing method provided for an embodiment of the present disclosure;

[0035] Figure 2 A structural diagram of a to-be-trained named entity recognition model provided for an embodiment of the present disclosure;

[0036] Figure 3 A flowchart of a named entity recognition method provided for an embodiment of the present disclosure;

[0037] Figure 4A An exemplary to-be-trained model structural diagram provided for an embodiment of the present disclosure;

[0038] Figure 4B A corresponding recognition model structural diagram provided for an embodiment of the present disclosure; Figure 4A A corresponding recognition model structural diagram provided for an embodiment of the present disclosure;

[0039] Figure 5 A block diagram of a data processing apparatus provided for an embodiment of the present disclosure;

[0040] Figure 6 A block diagram of a named entity recognition apparatus provided for an embodiment of the present disclosure;

[0041] Figure 7 A block diagram of an electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] In order for those skilled in the art to better understand the technical solutions of the present disclosure, exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered only as exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, the description in the following description omits the description of well-known functions and structures.

[0043] In the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0044] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0045] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms “comprise” and / or “consist of,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The terms “connected” or “coupled” or similar terms are not limited to a physical or mechanical connection, but can include an electrical connection, whether direct or indirect.

[0046] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an overly literal or overly formal sense unless expressly so defined herein.

[0047] Embodiments of the present disclosure relate to the field of Nature Language processing (NLP), and are applied to a multi-modal named entity recognition technology scenario.

[0048] Among them, the Named Entity Recognition (NER) technology is also called named entity recognition, named entity, which refers to the recognition of entities with specific meanings in the text, such as including person (person, PER), location (location, LOC), organization (organization, ORG), special entity (miscellaneous, MISC) and non-entity (other, O), and special entities include entities representing time, quantity, currency, and proportion value.

[0049] The modality of data can be understood as a way of carrying data, and multi-modal data refers to data carried in multiple ways. The data of each modality can be one of text, image, audio, and video. Multi-modal named entity recognition refers to recognizing named entities in multi-modal data.

[0050] Generally in a multi-modal named entity recognition process, first the features of multiple modalities are fused, and then based on the fused features, the named entity is recognized based on the fused modalities. Based on this, an important part of training a named entity recognition model is to train the ability of the named entity recognition model to extract and fuse named entities in different modalities of data, therefore, the conventional way of training a named entity recognition model is to label all the training samples of various modalities, and then use the labeled training samples of various modalities to train the named entity recognition model, i.e. supervised training, but the cost of labeling all the training samples of various modalities is high, resulting in a high cost of the conventional way of training a named entity recognition model.

[0051] Therefore, the embodiments of the present disclosure provide a data processing method for training a named entity recognition model, the training samples used include labeled training sample pairs and unlabeled training sample pairs, wherein each training sample pair is composed of at least two modalities of data. For each training sample pair, the latent representation features of each modality of data in the corresponding training sample pair are obtained, the features are reconstructed based on the fused latent representation features of multiple modalities, and then the named entity recognition model is trained according to the loss between the reconstructed features and the training samples. It can be seen that by using the implementation manner of the embodiments of the present disclosure, part of the training samples are labeled training samples, i.e. the training method of the embodiments of the present disclosure is a "semi-supervised" training method, so that the performance of the model trained can be improved without reducing the labeling cost.

[0052] The data processing method according to the embodiments of the present disclosure can be executed by an electronic device, which can be a terminal device such as a vehicle-mounted device, a User Equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, or an electronic device which can be an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing. The method can be realized by calling the computer readable program instructions stored in the memory by the processor of the electronic device.

[0053] Figure 1 A flowchart of a data processing method according to the embodiments of the present disclosure is provided, which is used for training a named entity recognition model. In some implementations, the named entity recognition model can be as shown in Figure 2 The named entity recognition model includes a named entity prediction network and X recognition networks, and the X recognition networks correspond to one modality of data respectively. X is an integer greater than or equal to 2.

[0054] In combination withFigure 2 Referring to Figure 1 The method comprises:

[0055] In step S11, pairs of labeled training samples and pairs of unlabeled training samples are obtained.

[0056] One pair of training samples is composed of at least two modalities of data. Exemplarily, the specific value of the "at least two" can be X.

[0057] In some implementations, the at least two modalities of data in one pair of training samples are semantically related. For example, the X modalities of data included in the pair of labeled training samples can contain the same named entity, and the pair of labeled training samples includes the label of the corresponding named entity. The X modalities of data included in the pair of unlabeled training samples can contain the same named entity, but do not contain the label information of the named entity.

[0058] For example, the X modalities of data are text and image, the text in the pair of labeled training samples includes a person's name, for example, and the image included in the pair of labeled training samples includes image content of the corresponding person's name in the text, and the label of the named entity "PER". The text in the pair of unlabeled training samples includes a person's name, for example, and the image content can be an image of the person's back, for example.

[0059] In step S12, the data of the corresponding modality in the pair of labeled training samples and the pair of unlabeled training samples is identified by the identification network corresponding to each modality of data, respectively, to obtain the first latent representation feature corresponding to each modality of data in the pair of labeled training samples, and the second latent representation feature corresponding to each modality of data in the pair of unlabeled training samples.

[0060] As shown in Figure 2 , after the electronic device inputs each training sample into the named entity recognition model, each identification network in the named entity recognition model processes the data of the corresponding modality of data in the training sample, and outputs the latent representation feature of the corresponding modality of data.

[0061] Each latent representation feature is used to represent the latent feature factor of the semantic feature of the corresponding modality of data, and accordingly, the named entity recognition model can obtain the semantic feature of the corresponding modality of data, and then obtain the latent representation feature of the semantic feature according to the corresponding semantic feature.

[0062] Exemplarily, again referring to Figure 2The named entity recognition model can include at least two pre-processing networks corresponding to each modality data respectively, and each recognition network corresponding to each modality data can include an encoder. Further, the named entity recognition model can respectively perform feature extraction processing on the corresponding modality data in the labeled training sample pair and the unlabeled training sample pair by using the pre-processing network corresponding to each modality data, to obtain semantic features of each modality data in the labeled training sample pair and semantic features of each modality data in the unlabeled training sample pair. Then, the semantic features of the corresponding modality data in the labeled training sample pair are recognized by the encoder in the recognition network corresponding to each modality data, to obtain first latent representation features corresponding to each modality data in the labeled training sample pair. The semantic features of the corresponding modality data in the unlabeled training sample pair are recognized by the encoder in the recognition network corresponding to each modality data, to obtain second latent representation features corresponding to each modality data in the unlabeled training sample pair.

[0063] It should be noted that for each modality data, the named entity recognition model can deploy any conventional pre-processing network adapted to the corresponding modality to extract semantic features. For example, for text, a Bidirectional Encoder Representations from Transformers (Bert) model and / or a Bi-directional Long Short-Term Memory (BiLSTM) model can be used to extract semantic features of the text. For images, a Residual Network (ResNet) can be used to extract semantic features of the images.

[0064] In some implementations, the recognition network corresponding to each modality data constitutes a Variational Autoencoder (VAE) corresponding to each modality data, and the encoder (Ecoder) in the recognition network corresponding to each modality data is equivalent to an encoder part in the corresponding VAE. The VAE parameter configurations in the recognition networks corresponding to different modality data can be different, and the VAE parameter configurations in the recognition network corresponding to each modality data can be related to the corresponding modality data.

[0065] In some implementations, if the at least two modal data includes text, one text in each of the pair of labeled training samples and the pair of unlabeled training samples, and the text preprocessing network corresponding to the text includes a word vector obtaining model (e.g., the Bert model described above) and a word segment obtaining model (e.g., the Bi-LSTM model), the word vector obtaining model in the text preprocessing network is used to obtain word vectors of the text in the pair of labeled training samples and the pair of unlabeled training samples, respectively, and the obtained word vectors are input into the word segment obtaining model in the text preprocessing network. Then, the word segment obtaining model in the text preprocessing network is used to group words based on the word vectors of the text in the pair of labeled training samples to obtain a plurality of word segments, and the semantic features of each word segment are calculated based on the word vectors of the words in the word segment to obtain the semantic features of the word segment, and the semantic features of the word segments are combined to obtain the text semantic features of the text in the pair of labeled training samples. The word segment obtaining model in the text preprocessing network is also used to group words based on the word vectors of the text in the pair of unlabeled training samples to obtain a plurality of word segments, and the semantic features of each word segment are calculated based on the word vectors of the words in the word segment to obtain the semantic features of the word segment, and the semantic features of the word segments are combined to obtain the text semantic features of the text in the pair of unlabeled training samples.

[0066] In some other implementations, if the at least two modal data includes text, the text preprocessing network corresponding to the text can further include a word segment boundary feature obtaining model, which is used to obtain boundary features of each word segment. The boundary features of each word segment are used to represent the position information of the start word and the end word in the text in the pair of labeled training samples or the pair of unlabeled training samples. The word segment boundary feature obtaining model can be another Bi-LSTM model.

[0067] In this implementation, when the at least two modal data includes text, the text is split into word segments, and then subsequent operations are performed based on the word segments as semantic feature units. This can further improve the accuracy of named entity recognition by performing feature fusion and named entity recognition based on text semantic features with finer granularity.

[0068] In some implementations, the at least two modalities of data include images, one image in each of the pair of labeled training samples and the pair of unlabeled training samples, and the pre-processing network corresponding to the image is an image pre-processing network, e.g., including a ResNet and an average pooling layer. In this implementation, the named entity recognition model can extract features from the image in the pair of labeled training samples by using the image pre-processing network to obtain image features of the image in the pair of labeled training samples, and then the average pooling layer can be used to pool the image features of the image in the pair of labeled training samples to obtain image semantic features of the image in the pair of labeled training samples. Similarly, the named entity recognition model can extract features from the image in the pair of unlabeled training samples by using the image pre-processing network to obtain image features of the image in the pair of unlabeled training samples, and then the average pooling layer can be used to pool the image features of the image in the pair of unlabeled training samples to obtain image semantic features of the image in the pair of unlabeled training samples.

[0069] In step S13, the first latent representation features of each modality of data in the pair of labeled training samples are fused and named entity prediction is performed by the named entity prediction network to obtain a first prediction result of the pair of labeled training samples, and the second latent representation features of each modality of data in the pair of unlabeled training samples are fused and named entity prediction is performed by the named entity prediction network to obtain a second prediction result of the pair of unlabeled training samples.

[0070] Referring again to Figure 2 The X recognition networks obtain the latent representation features, and the obtained latent representation features are input into the named entity prediction network.

[0071] In some implementations, each latent representation feature includes an average vector and a variance vector. Correspondingly, the first latent representation feature can include an average vector and a variance vector, and the second latent representation feature can include an average vector and a variance vector. Then, fusing the first latent representation features of each modality of data in the pair of labeled training samples can include fusing the average vectors in the first latent representation features of each modality of data and fusing the variance vectors in the first latent representation features of each modality of data; and fusing the second latent representation features of each modality of data in the pair of unlabeled training samples can include fusing the average vectors in the second latent representation features of each modality of data and fusing the variance vectors in the second latent representation features of each modality of data.

[0072] For example, the named entity prediction network can use a product of experts (POE) algorithm to fuse the latent representation features.

[0073] In some implementations, the named entity prediction network can employ a feed-forward neural network (FFNN) to perform named entity prediction based on the fused features to obtain the corresponding prediction results. For example, the named entity prediction network can employ a FFNN to perform named entity prediction on the fused features of the first latent representation features of each modality data to obtain the first prediction results of the pair of labeled training samples; the named entity prediction network can employ a FFNN to perform named entity prediction on the fused features of the second latent representation features of each modality data to obtain the second prediction results of the pair of unlabeled training samples.

[0074] In some other implementations, when the text is included in any of the pair of training samples, according to the description of step S12, the latent representation features of the text are obtained based on the word segment features of the text. Based on this, after the latent representation features of each modality data in the corresponding pair of training samples are fused, the fused features can be concatenated with the boundary features of the text in the pair of training samples, and named entity prediction is performed on the concatenated features to obtain the prediction results of the corresponding pair of training samples. For example, when the pair of labeled training samples includes text, the named entity prediction network in the named entity recognition model fuses the first latent representation features of each modality data in the pair of labeled training samples, concatenates the fused features with the boundary features of the text in the pair of labeled training samples, and then performs named entity prediction on the concatenated features to obtain the first prediction results of the pair of labeled training samples. For another example, when the pair of unlabeled training samples includes text, the named entity prediction network in the named entity recognition model fuses the first latent representation features of each modality data in the pair of unlabeled training samples, concatenates the fused features with the boundary features of the text in the pair of unlabeled training samples, and then performs named entity prediction on the concatenated features to obtain the second prediction results of the pair of unlabeled training samples.

[0075] Since the semantic features of each modality data are related to the modality, and the latent representation features of each modality are features in the same dimension, and the latent representation features have a finer granularity of representation of the semantic features. Based on this, by using the present implementation, the named entity recognition model enables the fused features to not only break through the cross-modality problem, but also improve the alignment effect of the fused features on the semantics of each modality data, and improve the effect of the fused features on the detailed representation of each modality, thereby facilitating the improvement of the accuracy of multi-modal named entity recognition.

[0076] In step S14, the first latent representation features of each modality data in the pair of labeled training samples are decoded to obtain the reconstructed features of each modality data in the pair of labeled training samples, and the second latent representation features of each modality data in the pair of unlabeled training samples and the second prediction results are reconstructed to obtain the reconstructed features of each modality data in the pair of unlabeled training samples.

[0077] The recognition network corresponding to each modality data can decode the first latent representation feature of each modality data in the labeled training sample pair through the decoder (Dcoder) part of the VAE to obtain the reconstructed feature of each modality data in the labeled training sample pair, and reconstruct the second latent representation feature of each modality data in the unlabeled training sample pair and the second prediction result to obtain the reconstructed feature of each modality data in the unlabeled training sample pair.

[0078] It should be noted that the data of part of the modalities in the training sample can correspond to the named entity label, so as to identify the named entity in the corresponding modality data. For example, the text can correspond to the named entity label, and the data of other part of the modalities can not correspond to the named entity label. For example, the image can not correspond to the named entity label. Based on this, for the unlabeled training sample pair, the target modality data in the unlabeled training sample pair may, for example, need to be reconstructed in combination with the named entity label. Therefore, for the target modality data in the corresponding unlabeled training sample pair, the second latent representation feature of the target modality data and the second prediction result are spliced to obtain the reconstructed feature of the target modality data, and for other modality data in the unlabeled training sample pair except the target modality data, the second latent representation feature of the other modality data is decoded as the reconstructed feature of the other modality data.

[0079] In step S15, the first loss function is determined based on the first prediction result of the labeled training sample pair and the label of the labeled training sample, the second loss function is determined based on the labeled training sample pair and the reconstructed feature of the labeled training sample pair, and the third loss function is determined based on the unlabeled training sample pair and the reconstructed feature of the unlabeled training sample pair.

[0080] In some implementations, the second loss function is determined based on the labeled training sample pair and the reconstructed feature of the labeled training sample pair, including: obtaining the semantic feature of each modality data in the labeled training sample pair; performing difference operation on the semantic feature of each modality data in the labeled training sample pair and the reconstructed feature of the corresponding modality data to obtain a first difference operation result; performing divergence operation on the normal distribution to which the first latent representation feature of each modality data in the labeled training sample pair conforms to obtain a first divergence operation result; and performing addition operation on the first difference operation result and the first divergence operation result to obtain the second loss function.

[0081] In some implementations, the third loss function is determined based on the unlabeled training sample pair and the reconstructed features of the unlabeled training sample pair, including: obtaining semantic features of each modality data in the labeled training sample pair; performing difference operation on the semantic features of each modality data in the unlabeled training sample pair and the reconstructed features of the corresponding modality data to obtain a second difference operation result; performing divergence operation on a normal distribution to which the second latent representation features of each modality data in the unlabeled training sample pair conform to obtain a second divergence operation result; and performing addition operation on the second difference operation result and the second divergence operation result to obtain the third loss function.

[0082] In step S16, the named entity recognition model is trained according to the first loss function, the second loss function, and the third loss function.

[0083] The electronic device can obtain the hyperparameters corresponding to the first loss function, the second loss function, and the third loss function, respectively, and then perform addition operation on the first loss function, the second loss function, and the third loss function after multiplying the hyperparameters corresponding thereto to obtain a target loss function, and adjust the parameters of the named entity recognition model in the direction of reducing the value of the target loss function.

[0084] When the named entity recognition model reaches a usable state, the electronic device can use the named entity recognition model to perform multi-modal named entity recognition.

[0085] Referring to Figure 3 , Figure 3 A flowchart of a named entity recognition method provided by an embodiment of the present disclosure is provided, and the method includes:

[0086] In step S31, at least two modalities of data are input into the named entity recognition model.

[0087] The named entity recognition model can be trained by the above data processing method.

[0088] In step S32, the corresponding modality data is identified by the identification network corresponding to each modality data in the named entity recognition model to obtain the latent representation features corresponding to each modality data.

[0089] In step S33, the latent representation features of the at least two modalities of data are fused by the named entity prediction network in the named entity recognition model.

[0090] In step S34, the fused latent representation features are used to perform named entity recognition to obtain multi-modal named entities contained in the at least two modalities of data.

[0091] It should be understood that the technical features and processing procedures involved in steps S31 to S34 can refer to the related description in steps S11 to S13, and the present disclosure will not be described in detail here.

[0092] It can be seen that the embodiment of the present disclosure provides a named entity recognition method. The named entity recognition model obtains the latent representation features of the semantic features of each modality data, and then performs named entity recognition based on the fused latent representation features of each modality data. Since the semantic features in different modal dimensions are decoded, that is, the semantic features in different dimensions are mapped to the same dimension, the latent representation features not only represent semantic features with smaller granularity, but also break through the problem of cross-modal semantic features, which is conducive to improving the alignment effect of the fused features on the semantics of each modality data, thereby optimizing the recognition effect of multi-modal named entities.

[0093] It can be understood that, Figure 3 The multi-modal named entity recognition method can be applied to one of data classification, data recognition, data matching, etc. For example, different types of image-text information are classified into sports news, technology news and entertainment news, etc. according to the multi-modal named entities (such as names, face images, etc.) in the image content and text content of the image-text information; for example, by identifying the multi-modal named entities (such as names, place names, and place names representing landmark buildings, etc.) in the multimedia information (such as including images, texts and audios), it is determined whether the relevant multimedia information is legal; for example, according to the author image and the name contained in the literary works, the author and the corresponding author's literary works are matched through multi-modal named entity recognition, etc. Here, it will not be listed one by one.

[0094] The technical solutions of the embodiments of the present disclosure will be described below in conjunction with exemplary implementation scenarios.

[0095] Referring to Figure 4A , Figure 4A A structure diagram of a named entity recognition model to be trained is shown, taking text and image as an example of multi-modal data. For text and image, Figure 4AThe named entity recognition model can include a text preprocessing network 411, a text recognition network 412, an image preprocessing network 421, an image recognition network 422, and a named entity prediction network 431. Specifically, the text preprocessing network 411 can include a word vector acquisition model Bert, a word segment acquisition model BiLSTM1, and a word segment boundary feature acquisition model BiLSTM1. The text recognition network 412 is implemented as a text VAE, and accordingly includes an encoder and a decoder. The image preprocessing network 421 can specifically include an image feature extraction model ResNet model and an average pooling layer. The image recognition network 422 is implemented as an image VAE, and accordingly includes an encoder and a decoder. The named entity prediction network 431 includes a POE fusion model and an FFNN model, wherein the POE fusion model is used to perform the fusion of various types of latent representation features in the implementation process described above, and the FFNN model is used to perform the prediction of named entities in the implementation process described above.

[0096] The following is introduced from two stages of training a model and using a trained model.

[0097] Training a model:

[0098] In this example, the text training samples include labeled texts and unlabeled texts, wherein the number of labeled texts is less than or equal to the number of unlabeled texts, for example, the total number of text training samples is 8000, of which the number of labeled texts can be 100 and the number of unlabeled texts can be 7900. The image training samples are all unlabeled. The labeled training sample pair is represented as |D l |, |D l |= , wherein refers to the xth labeled training sample pair of text, refers to the image in the xth labeled training sample pair, refers to the label in the xth labeled training sample pair. contains the text of the named entity indicated by The image content of may be the graph of the corresponding named entity in The unlabeled training sample pair is represented as |D u |, |D u |= , wherein refers to the xth unlabeled training sample pair of text, refers to the image in the xth unlabeled training sample pair.

[0099] Any labeled training text can be represented as S={w1,w2,…,wNs}, where w1 to wNs are each word in the text S, and the label y of the text S can be represented as... ,in, It refers to the starting position of the word in the named entity in the labeled training text. It refers to the ending position of a word in a named entity in labeled training text. This refers to the named entity type. For example, given the labeled training text S = { Anyway, the, best, Benz, in, the, world}, and the label y = {(4,4,PER)}, the named entity type of the word "Benz" in the text S, which starts at position 4 and ends at position 4, is "PER".

[0100] The following uses labeled training sample pairs as an example to illustrate the processing of text and images respectively.

[0101] Text in labeled training sample pairs Figure 4A The flow of quantities includes:

[0102] Input the text from each labeled training sample pair Figure 4A In the text preprocessing network 411, the BERT model extracts features from the input text samples to obtain the character vector of each character in the text samples. ,in, The number of characters in a text sample S is represented by bi, which is the character vector of the i-th character in the text sample S.

[0103] Then, the BERT model will respectively The input is fed into two BiLSTM models in the text preprocessing network 411. Among them, the BiLSTM model 1, for example, performs word grouping based on each character to enumerate all possible word segments and obtain the semantic features of each word segment. ,in, This refers to the character vector of each character from the i-th to j-th characters in the text sample S. For example, the BiLSTM model 2, which obtains the boundary features of each segment based on the position of characters in each segment within the text S, can be used to acquire these features. .in, It can characterize the semantic features of text samples.

[0104] Furthermore, the BiLSTM model 1, which extracts semantic features from text samples, further utilizes these features. Input is used in the text recognition network 412 (i.e., the text VAE model). The encoder in the text VAE... latent representation feature of the text sample , wherein , denotes an average vector in the latent representation feature of the text sample, denotes a variance vector in the latent representation feature of the text sample.

[0105] After that, the latent representation feature corresponding to the text sample There are two processing logics. In one processing logic, the decoder in the text VAE reconstructs the text based on the latent representation feature to obtain the reconstructed text . The other processing logic of the latent representation feature corresponding to the text sample is to input the latent representation feature into the POE fusion model in the named entity prediction network 431, and fuse the latent representation feature of the text sample with the latent representation feature of the image sample through the deployed POE algorithm in the POE fusion model. The process of obtaining the latent representation feature of the image sample is described as follows.

[0106] The quantity flow of the image in the labeled training sample in Figure 4A includes:

[0107] The image in each pair of labeled training samples is input into Figure 4A the image feature extraction model ResNet in the schematic image preprocessing network 421, which extracts features from the input image sample to obtain , denotes the vector of the i-th image block in the M image blocks, and M is used to represent the total number of image blocks, and then the ResNet inputs into the average pooling layer in the image preprocessing network 421, and after average pooling, obtains the vector of the image sample .

[0108] After that, the image preprocessing network 421 inputs the vector of the image sample into the image recognition network 422 (i.e., the image VAE model). The encoder in the image VAE encodes to obtain the latent representation feature of the image sample , wherein , denotes an average vector in the latent representation feature of the image sample, denotes a variance vector in the latent representation feature of the image sample.

[0109] After that, the latent representation feature corresponding to the image sample There are two processing logics. In one processing logic, the decoder in the image VAE decodes and reconstructs the image based on the latent representation feature to obtain a reconstructed image .

[0110] the latent representation feature corresponding to the image sample Another processing logic is to input the latent representation feature into the POE fusion model in the named entity prediction network 431, and pass the latent representation feature of the image sample through the POE fusion model to obtain the latent representation feature of the image sample and the latent representation feature of the text sample .

[0111] Further, Figure 4A The POE fusion model in the named entity prediction network 431 fuses the latent representation feature of the text sample and the latent representation feature of the image sample according to the algorithm to obtain a multimodal latent representation feature of the text and the image , wherein is a parameter greater than 0 and less than 1. Then, the multimodal latent representation feature is concatenated with the boundary feature of the word segment input into the word segment boundary feature acquisition model BiLSTM model 2 , and the concatenated feature is input into the FFNN in the named entity prediction network 431 for multimodal named entity prediction to obtain the named entity type of each feature . , wherein is the prediction result of the corresponding named entity type of the labeled training sample pair.

[0112] It should be understood that the above is an example of a labeled training sample pair. For the text and image in the unlabeled training sample pair, Figure 4A the processing process is similar to the above processing flow, which will not be described in detail here.

[0113] It should be noted that when processing the text in the unlabeled training sample pair, since the text in the unlabeled training sample pair has no corresponding label, when the text VAE decodes and reconstructs the text in the unlabeled training sample pair, the label feature processed from the unlabeled training sample pair can be concatenated for decoding and reconstruction.

[0114] Further, the electronic device can calculate the following three loss values based on the above data:

[0115] a loss value of the label in the labeled training sample pair and a predicted result of the named entity type corresponding to the labeled training sample pair ;

[0116] a loss value between the labeled training sample pair and the reconstructed feature of the labeled training sample pair:

[0117]

[0118] wherein, in the labeled training sample pair denotes a difference operation result between the semantic feature of the text in the labeled training sample pair and the reconstructed feature of the corresponding text; denotes a difference operation result between the semantic feature of the image in the labeled training sample pair and the reconstructed feature of the corresponding image; denotes a normal distribution divergence operation result of the latent representation feature of the text in the labeled training sample pair; denotes a normal distribution divergence operation result of the latent representation feature of the image in the labeled training sample pair.

[0119] a loss value between the unlabeled training sample pair and the reconstructed feature of the unlabeled training sample pair:

[0120] ,

[0121] wherein, in the unlabeled training sample pair denotes a difference operation result between the semantic feature of the text in the unlabeled training sample pair and the reconstructed feature of the corresponding text, denotes a difference operation result between the semantic feature of the image in the unlabeled training sample pair and the reconstructed feature of the corresponding image, denotes a normal distribution divergence operation result of the latent representation feature of the text in the unlabeled training sample pair, denotes a normal distribution divergence operation result of the latent representation feature of the image in the unlabeled training sample pair.

[0122] Further, the electronic device can calculate Figure 4A a loss value of the schematic model: , denotes a hyperparameter for balancing and and If L converges, it is considered that the schematic model has been trained and can be used to identify multi-modal named entities; if L diverges, it is considered that the schematic model still needs to be trained. Figure 4A Figure 4A

[0123] Using the trained model:

[0124] ​​According to the description of the above process of training the model, Figure 4A The decoder of the Chinese text VAE and the decoder of the image VAE are used to reconstruct the features in the training process to calculate the loss value. When the model can be used to identify the multi-modal named entity, Figure 4A The decoder of the Chinese text VAE and the decoder of the image VAE can be removed from the trained identification model, and the identification model as shown in Figure 4B is obtained.

[0125] Correspondingly, Figure 4B The multi-modal named entity contained in the to-be-identified text and the to-be-identified image can be identified, and in the process of identifying the multi-modal named entity contained in the to-be-identified text and the to-be-identified image, Figure 4B The functions and executed actions of each module in the foregoing embodiments can be referred to the related description of the foregoing embodiments, and the disclosure will not be described in detail here.

[0126] It can be understood that, Figure 4A and Figure 4B This is only an illustrative description, and does not constitute a limitation on the model involved in the embodiments of the disclosure. In other implementations, the to-be-trained model and the identification model can each contain more or fewer functional modules, or the deployment manner of each functional module can be different from the above illustration, and the embodiments of the disclosure will not be described in detail.

[0127] As can be seen, by using the named entity identification method provided by the embodiments of the disclosure, the to-be-trained model is trained in a semi-supervised manner, and in the case of reducing the labeling cost, a multi-modal named entity identification model with better identification performance can be trained. Further, the corresponding identification model sets the VAE matched with each modality data, and in the identification process, the semantic features of each modality data can be respectively mapped to the same dimension, and the feature factors in the same dimension can more finely represent the semantic features of each modality data. Based on this, the fused features not only can break through the cross-modal problem of semantic features, align the semantics of each modality data, but also represent the granularity of semantic features more finely, so as to optimize the identification effect of the multi-modal named entity.

[0128] It can be understood that the above-mentioned various method embodiments mentioned in the disclosure can be combined with each other to form combined embodiments without violating the principle logic. Limited by the length, the disclosure will not be described in detail. Those skilled in the art can understand that in the above-mentioned method of the specific implementation, the specific execution order of each step should be determined according to its function and possible internal logic.

[0129] In addition, the data processing apparatus, the electronic device, and the computer-readable storage medium provided by the present disclosure can be used to implement any one of the data processing methods or named entity recognition methods provided by the present disclosure. The corresponding technical solutions and descriptions are the same as those in the method part, and will not be repeated here.

[0130] Figure 5 A block diagram of a data processing apparatus provided by an embodiment of the present disclosure is provided.

[0131] With reference to Figure 5 The data processing apparatus provided by the embodiment of the present disclosure is used for training a named entity recognition model, and the named entity recognition model comprises a named entity prediction network and at least two recognition networks corresponding to different modal data. The data processing apparatus comprises an acquisition unit 51, an identification unit 52, a processing unit 53, a reconstruction unit 54, a determination unit 55, and a training unit 56.

[0132] The acquisition unit 51 is configured to acquire a labeled training sample pair and an unlabeled training sample pair; wherein one training sample pair is composed of at least two modal data;

[0133] The identification unit 52 is configured to identify the data of the corresponding modal in the labeled training sample pair and the unlabeled training sample pair by the recognition network corresponding to each modal data, respectively, to obtain the first latent representation feature corresponding to each modal data in the labeled training sample pair, and the second latent representation feature corresponding to each modal data in the unlabeled training sample pair;

[0134] The processing unit 53 is configured to fuse and perform named entity prediction on the first latent representation feature of each modal data in the labeled training sample pair by the named entity prediction network, to obtain a first prediction result of the labeled training sample pair, and fuse and perform named entity prediction on the second latent representation feature of each modal data in the unlabeled training sample pair by the named entity prediction network, to obtain a second prediction result of the unlabeled training sample pair;

[0135] The reconstruction unit 54 is configured to decode the first latent representation feature of each modal data in the labeled training sample pair to obtain a reconstructed feature of each modal data in the labeled training sample pair, and perform reconstruction processing based on the second latent representation feature of each modal data in the unlabeled training sample pair and the second prediction result, to obtain a reconstructed feature of each modal data in the unlabeled training sample pair;

[0136] The determining unit 55 is configured to determine a first loss function based on the first prediction result of the pair of labeled training samples and the label of the pair of labeled training samples, determine a second loss function based on the pair of labeled training samples and the reconstructed features of the pair of labeled training samples, and determine a third loss function based on the pair of unlabeled training samples and the reconstructed features of the pair of unlabeled training samples.

[0137] The training unit 56 is configured to train the named entity recognition model according to the first loss function, the second loss function, and the third loss function.

[0138] In an embodiment, the named entity recognition model further comprises at least two pre-processing networks corresponding to the at least two modal data respectively, and each of the at least two recognition networks corresponding to the at least two modal data comprises an encoder. When the recognition unit 52 respectively identifies the data of the corresponding modal in the pair of labeled training samples and the pair of unlabeled training samples through the at least two recognition networks corresponding to the at least two modal data to obtain the first latent representation feature corresponding to each of the at least two modal data in the pair of labeled training samples and the second latent representation feature corresponding to each of the at least two modal data in the pair of unlabeled training samples, the following steps are performed:

[0139] The data of the corresponding modal in the pair of labeled training samples and the pair of unlabeled training samples are respectively subjected to feature extraction processing through the at least two pre-processing networks corresponding to the at least two modal data to obtain the semantic features of each of the at least two modal data in the pair of labeled training samples and the semantic features of each of the at least two modal data in the pair of unlabeled training samples.

[0140] The semantic features of the corresponding modal data in the pair of labeled training samples are subjected to identification processing through the encoder in the at least two recognition networks corresponding to the at least two modal data to obtain the first latent representation feature corresponding to each of the at least two modal data in the pair of labeled training samples. The semantic features of the corresponding modal data in the pair of unlabeled training samples are subjected to identification processing through the encoder in the at least two recognition networks corresponding to the at least two modal data to obtain the second latent representation feature corresponding to each of the at least two modal data in the pair of unlabeled training samples.

[0141] In an embodiment, the at least two modal data comprise any two or more of text, image, and audio.

[0142] In an embodiment, if the at least two modal data include texts, the pre-processing network corresponding to each of the at least two modal data includes a text pre-processing network, each of the labeled training sample pair and the unlabeled training sample pair includes a text, the semantic feature of each modal data in the labeled training sample pair includes a text semantic feature of the text in the labeled training sample pair, and the semantic feature of each modal data in the unlabeled training sample pair includes a text semantic feature of the text in the unlabeled training sample pair.

[0143] When the identification unit 52 performs feature extraction processing on the data of the corresponding modal in the labeled training sample pair and the unlabeled training sample pair respectively by using the pre-processing network corresponding to each modal data, the following steps are performed:

[0144] The word vector of the text in the labeled training sample pair and the unlabeled training sample pair is obtained by a word vector obtaining model in the text pre-processing network, and the obtained word vector is input into a word segment obtaining model in the text pre-processing network.

[0145] The word segment obtaining model in the text pre-processing network is used to obtain a plurality of word segments based on the word vector of the text in the labeled training sample pair, and the semantic feature of each word segment is calculated according to the word vector of the word in each word segment, and the semantic features of the word segments are combined to obtain the text semantic feature of the text in the labeled training sample pair.

[0146] The word segment obtaining model in the text pre-processing network is used to obtain a plurality of word segments based on the word vector of the text in the labeled training sample pair, and the semantic feature of each word segment is calculated according to the word vector of the word in each word segment, and the semantic features of the word segments are combined to obtain the text semantic feature of the text in the labeled training sample pair.

[0147] In an embodiment, the obtaining unit 51 is further configured to obtain the boundary feature of each word segment by using a word segment boundary feature obtaining model in the text pre-processing network, and the boundary feature of each word segment is used to represent the position information of the start word and the end word in the text in the labeled training sample pair or the unlabeled training sample pair.

[0148] In an embodiment, if the at least two modalities of data include images, the pre-processing network corresponding to each modality of data includes an image pre-processing network, each of the pair of labeled training samples and the pair of unlabeled training samples includes an image, the semantic feature of each modality of data in the pair of labeled training samples includes an image semantic feature of the image in the pair of labeled training samples, and the semantic feature of each modality of data in the pair of unlabeled training samples includes an image semantic feature of the image in the pair of unlabeled training samples.

[0149] When the identification unit 52 performs feature extraction on the data of the corresponding modality in the pair of labeled training samples and the pair of unlabeled training samples respectively by using the pre-processing network corresponding to each modality of data, the identification unit 52 performs the following steps:

[0150] The image pre-processing network is used to perform feature extraction on the image in the pair of labeled training samples and the image in the pair of unlabeled training samples respectively, to obtain an image feature of the image in the pair of labeled training samples and an image feature of the image in the pair of unlabeled training samples.

[0151] An average pooling layer is used to perform pooling processing on the image feature of the image in the pair of labeled training samples and the image feature of the image in the pair of unlabeled training samples respectively, to obtain an image semantic feature of the image in the pair of labeled training samples and an image semantic feature of the image in the pair of unlabeled training samples.

[0152] In an embodiment, the first latent representation feature includes an average vector and a variance vector, and the second latent representation feature includes an average vector and a variance vector.

[0153] When the processing unit 53 performs fusion of the first latent representation feature of each modality of data in the pair of labeled training samples and named entity prediction by using the named entity prediction network in the named entity recognition model, to obtain the first prediction result of the pair of labeled training samples, the processing unit 53 performs the following steps:

[0154] After the fusion of the first latent representation feature of each modality of data in the pair of labeled training samples, the first latent representation feature is spliced with the boundary feature of the text in the pair of labeled training samples.

[0155] The spliced feature is subjected to named entity prediction, to obtain the first prediction result of the pair of labeled training samples.

[0156] In an embodiment, the reconstructing unit 54 performs the following steps when reconstructing each modality data in the unlabeled training sample pair based on the second latent representation feature of each modality data in the unlabeled training sample pair and the second prediction result to obtain the reconstructed feature of each modality data in the unlabeled training sample pair:

[0157] For the target modality data in the unlabeled training sample pair, the second latent representation feature of the target modality data and the second prediction result are spliced to obtain the reconstructed feature of the target modality data; the target modality data refers to any one of the at least two modalities of data;

[0158] For the other modality data in the unlabeled training sample pair except the target modality data, the second latent representation feature of the other modality data is decoded as the reconstructed feature of the other modality data.

[0159] In an embodiment, the determining unit 55 performs the following steps when determining the second loss function based on the labeled training sample pair and the reconstructed feature data of the labeled training sample pair:

[0160] Obtain the semantic feature of each modality data in the labeled training sample pair;

[0161] Differences are operated between the semantic feature of each modality data in the labeled training sample pair and the reconstructed feature of the corresponding modality data to obtain a first difference operation result;

[0162] The first latent representation feature of each modality data in the labeled training sample pair conforms to a normal distribution, and divergence operation is performed to obtain a first divergence operation result;

[0163] The first difference operation result and the first divergence operation result are added to obtain a second loss function.

[0164] In an embodiment, the determining unit 55 performs the following steps when determining the third loss function based on the unlabeled training sample pair and the reconstructed feature of the unlabeled training sample pair:

[0165] Obtain the semantic feature of each modality data in the labeled training sample pair;

[0166] Differences are operated between the semantic feature of each modality data in the labeled training sample pair and the reconstructed feature of the corresponding modality data to obtain a first difference operation result;

[0167] The second latent representation feature of each modality data in the unlabeled training sample pair conforms to a normal distribution, and divergence operation is performed to obtain a second divergence operation result;

[0168] The second difference operation result is added to the second divergence operation result to obtain a third loss function.

[0169] In one embodiment, the named entity is one of the following entity types: a person name, a place name, an organization name, a special entity, and a non-entity.

[0170] In one embodiment, the training unit 56 performs the following steps when training the named entity recognition model according to the first loss function, the second loss function, and the third loss function: obtaining hyperparameters corresponding to the first loss function, the second loss function, and the third loss function, respectively;

[0171] The first loss function, the second loss function, and the third loss function are multiplied by the respective corresponding hyperparameters and then added to obtain a target loss function.

[0172] The parameters of the named entity recognition model are adjusted in a direction that reduces the value of the target loss function.

[0173] It can be understood that the division of the above modules is only a logical functional division. In actual implementation, the above modules can be integrated into hardware implementation. For example, the function of the obtaining unit 51 in the above implementation mode can be integrated into an I / O interface, and the functions of the recognition unit 52, the processing unit 53, the reconstruction unit 54, the determination unit 55, and the training unit 56 can be integrated into a processor.

[0174] Referring to Figure 6 , Figure 6 is a structural schematic diagram of a named entity recognition device provided by an embodiment of the present disclosure. The device includes:

[0175] An input unit 61 is configured to input at least two modal data into the named entity recognition model, respectively.

[0176] A recognition unit 62 is configured to perform recognition processing on corresponding modal data by using a recognition network corresponding to each modal data in the named entity recognition model, to obtain latent representation features corresponding to each modal data.

[0177] A fusion unit 63 is configured to fuse the latent representation features of the at least two modal data by using a named entity prediction network in the named entity recognition model.

[0178] The recognition unit 62 is further configured to perform named entity recognition based on the fused latent representation features, to obtain multi-modal named entities contained in the at least two modal data.

[0179] It can be understood that the division of the above modules is only a logical function division, and each of the above modules can be integrated into hardware implementation in actual implementation.

[0180] With reference to Figure 7 , Figure 7 The electronic device provided by the embodiment of the present disclosure comprises at least one processor 601, at least one memory 602, and one or more I / O interfaces 603 connected between the processor 601 and the memory 602; wherein the memory 602 stores one or more computer programs executable by the at least one processor 601, and the one or more computer programs are executed by the at least one processor 601 to enable the at least one processor 601 to perform the naming entity recognition method described above.

[0181] The embodiment of the present disclosure further provides a computer readable storage medium, which can be a volatile or non-volatile computer readable storage medium, and the computer readable storage medium stores a computer program, wherein the computer program is executed by the processor 601 to perform the following steps:

[0182] The application relates to a method for training a named entity recognition model, the named entity recognition model comprising a named entity prediction network and at least two identification networks corresponding to two kinds of modal data; specifically: obtaining a labeled training sample pair and an unlabeled training sample pair; wherein one training sample pair is composed of at least two kinds of modal data; identifying the data of the corresponding modal in the labeled training sample pair and the unlabeled training sample pair through the identification network corresponding to each kind of modal data, obtaining the first latent representation feature corresponding to each kind of modal data in the labeled training sample pair, and the second latent representation feature corresponding to each kind of modal data in the unlabeled training sample pair; fusing and performing named entity prediction on the first latent representation feature of each kind of modal data in the labeled training sample pair through the named entity prediction network, obtaining the first prediction result of the labeled training sample pair, and fusing and performing named entity prediction on the second latent representation feature of each kind of modal data in the unlabeled training sample pair through the named entity prediction network, obtaining the second prediction result of the unlabeled training sample pair; decoding the first latent representation feature of each kind of modal data in the labeled training sample pair to obtain the reconstruction feature of each kind of modal data in the labeled training sample pair, and performing reconstruction processing based on the second latent representation feature of each kind of modal data in the unlabeled training sample pair and the second prediction result, obtaining the reconstruction feature of each kind of modal data in the unlabeled training sample pair; determining a first loss function based on the first prediction result of the labeled training sample pair and the label of the labeled training sample, determining a second loss function based on the labeled training sample pair and the reconstruction feature of the labeled training sample pair, and determining a third loss function based on the unlabeled training sample pair and the reconstruction feature of the unlabeled training sample pair; and training the named entity recognition model according to the first loss function, the second loss function and the third loss function.

[0183] The named entity recognition model further comprises a preprocessing network corresponding to each of the at least two modal data, and each of the recognition networks corresponding to the modal data comprises an encoder; in some embodiments, the processor 601 is further configured to implement: performing feature extraction on the data of the corresponding modal in the pair of labeled training samples and the pair of unlabeled training samples respectively by using the preprocessing network corresponding to each of the modal data, to obtain semantic features of each of the modal data in the pair of labeled training samples and semantic features of each of the modal data in the pair of unlabeled training samples; performing recognition on the semantic features of the corresponding modal data in the pair of labeled training samples by using the encoder in the recognition network corresponding to each of the modal data, to obtain first latent representation features corresponding to each of the modal data in the pair of labeled training samples; and performing recognition on the semantic features of the corresponding modal data in the pair of unlabeled training samples by using the encoder in the recognition network corresponding to each of the modal data, to obtain second latent representation features corresponding to each of the modal data in the pair of unlabeled training samples.

[0184] If the at least two modal data comprise text, the preprocessing network corresponding to each of the at least two modal data comprises a text preprocessing network, and each of the pair of labeled training samples and the pair of unlabeled training samples comprises a text, the semantic features of each of the modal data in the pair of labeled training samples comprise text semantic features of the text in the pair of labeled training samples, and the semantic features of each of the modal data in the pair of unlabeled training samples comprise text semantic features of the text in the pair of unlabeled training samples; in some embodiments, the processor 601 is further configured to implement: obtaining word vectors of the text in the pair of labeled training samples and the pair of unlabeled training samples respectively by using a word vector acquisition model in the text preprocessing network, and inputting the obtained word vectors into a word segment acquisition model in the text preprocessing network; obtaining a plurality of word segments by using the word segment acquisition model in the text preprocessing network based on the word vectors of the text in the pair of labeled training samples, calculating semantic features of each word segment based on the word vectors of the words in the word segment, and combining the semantic features of the word segments to obtain the text semantic features of the text in the pair of labeled training samples; and obtaining a plurality of word segments by using the word segment acquisition model in the text preprocessing network based on the word vectors of the text in the pair of unlabeled training samples, calculating semantic features of each word segment based on the word vectors of the words in the word segment, and combining the semantic features of the word segments to obtain the text semantic features of the text in the pair of unlabeled training samples.

[0185] In some embodiments, the processor 601 is further configured to implement: obtaining boundary features of each word segment by using a word segment boundary feature acquisition model in the text preprocessing network, and the boundary features of each word segment are used to represent position information of a start word and an end word in the text in the pair of labeled training samples or the pair of unlabeled training samples.

[0186] If the at least two modal data include images, the pre-processing network corresponding to each of the at least two modal data includes an image pre-processing network, and each of the labeled training sample pair and the unlabeled training sample pair includes an image, the semantic feature of each modal data in the labeled training sample pair includes an image semantic feature of the image in the labeled training sample pair, and the semantic feature of each modal data in the unlabeled training sample pair includes an image semantic feature of the image in the unlabeled training sample pair; in some embodiments, the processor 601 is further configured to implement: performing feature extraction on the image in the labeled training sample pair and the image in the unlabeled training sample pair respectively by using the image pre-processing network, to obtain an image feature of the image in the labeled training sample pair and an image feature of the image in the unlabeled training sample pair; and performing pooling processing on the image feature of the image in the labeled training sample pair and the image feature of the image in the unlabeled training sample pair respectively by using an average pooling layer, to obtain the image semantic feature of the image in the labeled training sample pair and the image semantic feature of the image in the unlabeled training sample pair.

[0187] The first latent representation feature includes an average vector and a variance vector, and the second latent representation feature includes an average vector and a variance vector; in some embodiments, the processor 601 is further configured to implement: fusing the average vector in the first latent representation feature of each modal data, and fusing the variance vector in the first latent representation feature of each modal data; and the fusing the second latent representation feature of each modal data in the unlabeled training sample pair includes: fusing the average vector in the second latent representation feature of each modal data, and fusing the variance vector in the second latent representation feature of each modal data.

[0188] In some embodiments, the processor 601 is further configured to implement: after fusing the first latent representation feature of each modal data in the labeled training sample pair, splicing the boundary feature of the text in the labeled training sample pair; and performing named entity prediction on the spliced feature, to obtain a first prediction result of the labeled training sample pair.

[0189] In some embodiments, the processor 601 is further configured to implement: for the target modal data in the unlabeled training sample pair, splicing the second latent representation feature of the target modal data and the second prediction result, to obtain a reconstructed feature of the target modal data; the target modal data refers to any one of the at least two modal data; and for the modal data other than the target modal data in the unlabeled training sample pair, taking the second latent representation feature of the other modal data after decoding as the reconstructed feature of the other modal data.

[0190] In some embodiments, the processor 601 is further configured to implement: obtaining semantic features of each modality data in the pair of labeled training samples; performing difference operation on the semantic features of each modality data in the pair of labeled training samples and the reconstructed features of the corresponding modality data to obtain a first difference operation result; performing divergence operation on a normal distribution to which the first latent representation features of each modality data in the pair of labeled training samples conform to obtain a first divergence operation result; and performing addition operation on the first difference operation result and the first divergence operation result to obtain a second loss function.

[0191] In some embodiments, the processor 601 is further configured to implement: obtaining semantic features of each modality data in the pair of labeled training samples; performing difference operation on the semantic features of each modality data in the pair of labeled training samples and the reconstructed features of the corresponding modality data to obtain a first difference operation result; performing divergence operation on a normal distribution to which the first latent representation features of each modality data in the pair of labeled training samples conform to obtain a first divergence operation result; and performing addition operation on the first difference operation result and the first divergence operation result to obtain a second loss function.

[0192] In some embodiments, the processor 601 is further configured to implement: obtaining hyperparameters corresponding to the first loss function, the second loss function, and the third loss function respectively; performing multiplication operation on the first loss function, the second loss function, and the third loss function and the respective corresponding hyperparameters, and then performing addition operation to obtain a target loss function; and adjusting the parameters of the named entity recognition model in a direction of reducing the value of the target loss function.

[0193] In some embodiments, the processor 601 is further configured to implement: inputting at least two modalities of data into the named entity recognition model respectively; performing identification processing on the respective modalities of data by the identification network corresponding to each modality of data in the named entity recognition model to obtain latent representation features corresponding to each modality of data; fusing the latent representation features of the at least two modalities of data by the named entity prediction network in the named entity recognition model; and performing named entity recognition based on the fused latent representation features to obtain multi-modal named entities contained in the at least two modalities of data.

[0194] The embodiments of the present disclosure further provide a computer program product, including computer readable code or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-mentioned data processing method or named entity recognition method.

[0195] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0196] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0197] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0198] Computer readable program instructions for carrying out operations of the present disclosure can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or any combination of one or more of the above in any combination, written in any combination of one or more programming languages, including object oriented programming languages such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0199] The computer program product described herein can be embodied in a specific manner by hardware, software, or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), and the like.

[0200] The various aspects of the present disclosure are described herein with reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer readable program instructions.

[0201] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage cycles that change state. The instructions can be executed by one or more processors of a computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process such that the instructions which execute via the one or more processors of the computer or other programmable data processing devices create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0202] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0203] The flow and block diagrams in the drawings show architectural, functional, and operational aspects of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of instructions which comprise one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may

[0204] Example embodiments have been disclosed and, although a specific terminology is employed, it is merely for the convenience of the reader and is not intended to be limiting. In some instances, specific details have been included for the purpose of providing a thorough understanding of the embodiments presented, but it will be apparent to those skilled in the art that embodiments of the application can be practiced without these specific details. In some instances, features, characteristics and / or elements described in connection with a particular embodiment can be used in conjunction with, or switched to, features, characteristics and / or elements described in connection with other embodiments, unless otherwise explicitly stated. Accordingly, it will be understood that various changes in form and details can be made without departing from the scope of the disclosure as set forth in the appended claims.

Claims

1. A data processing method, characterized in that, The data processing method is used to train a named entity recognition model, the named entity recognition model including a named entity prediction network and recognition networks corresponding to at least two modalities of data; the data processing method includes: Obtain labeled training sample pairs and unlabeled training sample pairs; wherein, a training sample pair consists of at least two modalities of data; the at least two modalities of data are semantically related; The recognition network corresponding to each modality data is used to identify the data of the corresponding modality in the labeled training sample pair and the unlabeled training sample pair, respectively, to obtain the first latent representation feature corresponding to each modality data in the labeled training sample pair and the second latent representation feature corresponding to each modality data in the unlabeled training sample pair; The first latent representation features of each modality data in the labeled training sample pair are fused and named entity prediction is performed by the named entity prediction network to obtain the first prediction result of the labeled training sample pair; and the second latent representation features of each modality data in the unlabeled training sample pair are fused and named entity prediction is performed by the named entity prediction network to obtain the second prediction result of the unlabeled training sample pair. The first latent representation feature of each modality data in the labeled training sample pair is decoded to obtain the reconstructed feature of each modality data in the labeled training sample pair, and the second latent representation feature and the second prediction result are reconstructed to obtain the reconstructed feature of each modality data in the unlabeled training sample pair. A first loss function is determined based on the first prediction result of the labeled training sample pair and the label of the labeled training sample; a second loss function is determined based on the labeled training sample pair and the reconstruction features of the labeled training sample pair; and a third loss function is determined based on the unlabeled training sample pair and the reconstruction features of the unlabeled training sample pair. The named entity recognition model is trained based on the first loss function, the second loss function, and the third loss function.

2. The data processing method according to claim 1, characterized in that, The named entity recognition model also includes preprocessing networks corresponding to at least two modalities of data, and the recognition network corresponding to each modality of data includes an encoder; The step of using a recognition network corresponding to each modality of data to recognize and process the data of the corresponding modality in the labeled training sample pairs and the unlabeled training sample pairs, respectively, to obtain a first latent representation feature corresponding to each modality of data in the labeled training sample pairs and a second latent representation feature corresponding to each modality of data in the unlabeled training sample pairs, includes: The preprocessing network corresponding to each modality of data is used to perform feature extraction processing on the data of the corresponding modality in the labeled training sample pairs and the unlabeled training sample pairs, respectively, to obtain the semantic features of each modality of data in the labeled training sample pairs and the semantic features of each modality of data in the unlabeled training sample pairs; The semantic features of the corresponding modal data in the labeled training sample pair are identified by the encoder in the recognition network corresponding to each modal data, so as to obtain the first latent representation feature corresponding to each modal data in the labeled training sample pair. The semantic features of the corresponding modal data in the unlabeled training sample pair are identified by the encoder in the recognition network corresponding to each modal data, thereby obtaining the second latent representation feature corresponding to each modal data in the unlabeled training sample pair.

3. The method according to claim 2, characterized in that, The at least two modal data include any two or more of text, images, and audio.

4. The data processing method according to claim 3, characterized in that, If the at least two modal data include text, the preprocessing network corresponding to the at least two modal data respectively includes a text preprocessing network, each of the labeled training sample pair and the unlabeled training sample pair includes a text, the semantic features of each modal data in the labeled training sample pair include the text semantic features of the text in the labeled training sample pair, and the semantic features of each modal data in the unlabeled training sample pair include the text semantic features of the text in the unlabeled training sample pair; The preprocessing network corresponding to each modality of data is used to perform feature extraction processing on the data of the corresponding modality in the labeled training sample pairs and the unlabeled training sample pairs, respectively, to obtain the semantic features of each modality of data in the labeled training sample pairs and the semantic features of each modality of data in the unlabeled training sample pairs, including: The text vectors of the labeled training sample pairs and the unlabeled training sample pairs are obtained by the character vector acquisition model in the text preprocessing network, and the obtained character vectors are input into the word segment acquisition model of the text preprocessing network. The word segment acquisition model of the text preprocessing network obtains multiple word segments by combining the character vectors of the Chinese text with the labeled training samples. The semantic features of the corresponding word segments are calculated based on the character vectors of the characters that make up each word segment. The semantic features of each word segment are combined to form the text semantic features of the Chinese text with the labeled training samples. The text preprocessing network's word segment acquisition model uses the unlabeled training samples to group words from the Chinese text's character vectors to obtain multiple word segments. The semantic features of each word segment are calculated based on the character vectors of the characters that make up each word segment. The semantic features of each word segment are then combined to form the text semantic features of the unlabeled training samples for the Chinese text.

5. The data processing method according to claim 4, characterized in that, Also includes: The boundary features of each word segment are obtained through the word segment boundary feature acquisition model in the text preprocessing network. The boundary features of each word segment are used to characterize the position information of the start word and the end word in the text of the labeled training sample pair or the unlabeled training sample pair.

6. The data processing method according to claim 3, characterized in that, If the at least two modal data include images, the preprocessing networks corresponding to the at least two modal data respectively include image preprocessing networks, each of the labeled training sample pairs and the unlabeled training sample pairs includes an image, the semantic features of each modal data in the labeled training sample pairs include the image semantic features of the images in the labeled training sample pairs, and the semantic features of each modal data in the unlabeled training sample pairs include the image semantic features of the images in the unlabeled training sample pairs; The preprocessing network corresponding to each modality of data is used to perform feature extraction processing on the data of the corresponding modality in the labeled training sample pairs and the unlabeled training sample pairs, respectively, to obtain the semantic features of each modality of data in the labeled training sample pairs and the semantic features of each modality of data in the unlabeled training sample pairs, including: The image preprocessing network is used to extract features from the labeled training sample centering image and the unlabeled training sample centering image, respectively, to obtain the image features of the labeled training sample centering image and the image features of the unlabeled training sample centering image. An average pooling layer is used to pool the image features of the labeled training sample pairs and the image features of the unlabeled training sample pairs, respectively, to obtain the image semantic features of the labeled training sample pairs and the image semantic features of the unlabeled training sample pairs.

7. The data processing method according to claim 2, characterized in that, The first latent representation feature includes a mean vector and a variance vector, and the second latent representation feature includes a mean vector and a variance vector; The step of fusing the first latent representation features of each modality data in the labeled training sample pair includes: fusing the average vector in the first latent representation features of each modality data, and fusing the variance vector in the first latent representation features of each modality data; The step of fusing the second latent representation features of each modality data in the unlabeled training sample pair includes: fusing the average vector in the second latent representation features of each modality data, and fusing the variance vector in the second latent representation features of each modality data.

8. The data processing method according to claim 5, characterized in that, The step of fusing and predicting the first latent representation features of each modality data in the labeled training sample pair through the named entity prediction network in the named entity recognition model to obtain the first prediction result of the labeled training sample pair includes: After fusing the first latent representation features of each modality data in the labeled training sample pair, it is concatenated with the boundary features of the text in the labeled training sample pair. Named entity prediction is performed on the concatenated features to obtain the first prediction result of the labeled training sample pair.

9. The method according to claim 7, characterized in that, The reconstruction process, based on the second latent representation features of each modality data in the unlabeled training sample pair and the second prediction result, yields the reconstructed features of each modality data in the unlabeled training sample pair; including: For the target modality data in the unlabeled training sample pair, the second latent representation feature of the target modality data and the second prediction result are concatenated to obtain the reconstructed feature of the target modality data; the target modality data refers to any one of at least two modality data. For the modal data other than the target modal data in the unlabeled training sample pair, the second latent representation features of the other modal data are decoded and used as the reconstructed features of the other modal data.

10. The method according to claim 7, characterized in that, The step of determining the second loss function based on the labeled training sample pairs and the reconstructed feature data of the labeled training sample pairs includes: Obtain the semantic features of each modality data in the labeled training sample pairs; The semantic features of each modality data in the labeled training sample pair are compared with the reconstructed features of the corresponding modality data to obtain the first difference operation result; A divergence operation is performed on the normal distribution that the first latent representation features of each modality data in the labeled training sample pair conform to, to obtain the first divergence operation result; The first difference calculation result is added to the first divergence calculation result to obtain the second loss function.

11. The method according to claim 7, characterized in that, The step of determining the third loss function based on the unlabeled training sample pairs and their reconstructed features includes: Obtain the semantic features of each modality data in the labeled training sample pairs; The semantic features of each modality data in the unlabeled training sample pair are compared with the reconstructed features of the corresponding modality data to obtain the second difference operation result; A divergence operation is performed on the normal distribution that the second latent representation features of each modality data in the unlabeled training sample pair conform to, to obtain the second divergence operation result; The second difference calculation result is added to the second divergence calculation result to obtain the third loss function.

12. The data processing method according to any one of claims 1-11, characterized in that, The named entity is one of the following entity types: person name, place name, organization name, special entity, or non-entity.

13. The data processing method according to claim 1, characterized in that, Training the named entity recognition model based on the first loss function, the second loss function, and the third loss function includes: Obtain the hyperparameters corresponding to the first loss function, the second loss function, and the third loss function respectively; The target loss function is obtained by multiplying the first loss function, the second loss function, and the third loss function by their respective hyperparameters and then adding them together. The parameters of the named entity recognition model are adjusted in the direction of reducing the value of the target loss function.

14. A named entity recognition method, characterized in that, The method involves invoking a named entity recognition model, which is trained using the data processing methods described in claims 1 to 13, the methods comprising: At least two modalities of data are input into the named entity recognition model; the at least two modalities of data are semantically related. The corresponding modal data is processed by the recognition network corresponding to each modal data in the named entity recognition model to obtain the potential representation features corresponding to each modal data. The latent representation features of the at least two modality data are fused through the named entity prediction network in the named entity recognition model. Named entity recognition is performed based on the fused latent representation features to obtain multimodal named entities contained in the at least two modal data.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1-13 or the named entity recognition method as described in claim 14.

Images