Text processing method and device, computer equipment and computer readable storage medium

Abstract

The invention discloses a text processing method and device, computer equipment and a computer readable storage medium. The method comprises the steps of obtaining to-be-processed text information; determining multi-modal feature information based on the to-be-processed text information; and decoding the multi-modal feature information to obtain target text information. By the adoption of the method, noise introduced by irrelevant information can be reduced, and the processing quality of the to-be-processed text is improved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a text processing method, apparatus, computer device and computer-readable storage medium. Background Art

[0002] Multimodal machine translation refers to the task of using images and text for machine translation. By using image content as the visual information context to assist text in machine translation, it can translate closer to the scene, thereby improving the accuracy of the translation task. Therefore, it is widely used in application scenarios such as popular science video subtitle translation and radio station subtitle translation. However, existing multimodal machine translation usually uses the source sentence to retrieve related images in a preset sentence-image set, and then uses the retrieved images as the visual context to assist the source sentence in machine translation. This method has the problem that the matching degree between the retrieved image and the source sentence is not high. Summary of the invention

[0003] The embodiments of the present application provide a text processing method, apparatus, computer device and computer-readable storage medium, which can reduce noise introduced by irrelevant information and improve the processing quality of text information to be processed.

[0004] The technical solution adopted by the present invention to solve the problem is as follows:

[0005] On the one hand, the present application provides a text processing method, comprising:

[0006] Get the text information to be processed;

[0007] Determining multimodal feature information based on the text information to be processed;

[0008] The multimodal feature information is decoded to obtain target text information.

[0009] In some embodiments of the present application, multimodal feature information is determined based on the text information to be processed, including:

[0010] Encode the text information to be processed to obtain text encoding information of the text information to be processed;

[0011] Perform regional image retrieval and feature extraction on the text information to be processed to obtain general visual features;

[0012] Multimodal aggregation is performed on general visual features and text encoding information to obtain multimodal feature information.

[0013] In some implementation schemes of the present application, regional image retrieval and feature extraction are performed on the text information to be processed to obtain general visual features, including:

[0014] Extract noun phrases from the text information to be processed to obtain the first noun phrase;

[0015] Retrieving a pre-constructed regional image set based on the first noun phrase to obtain a target regional image of the first noun phrase, wherein the target regional image corresponds to the target noun phrase;

[0016] Feature extraction is performed on the target area image based on the target noun phrase to obtain universal visual features.

[0017] In some embodiments of the present application, the regional image set includes a plurality of first regional images, the plurality of first regional images correspond to a plurality of second noun phrases, and the pre-constructed regional image set is retrieved based on the first noun phrase to obtain the target regional image of the first noun phrase, including:

[0018] encoding the first noun phrase to obtain first encoding information, and encoding the second noun phrase corresponding to each first regional image in the regional image set to obtain second encoding information;

[0019] Determining similarity information between the first noun phrase and the second noun phrase corresponding to each first region image based on the first coding information and the second coding information;

[0020] Based on the similarity information, a target region image of the first noun phrase is determined from the plurality of first region images.

[0021] In some embodiments of the present application, before searching a pre-constructed regional image set based on the first noun phrase to obtain a target regional image of the first noun phrase, the process includes:

[0022] Obtain a preset text image set, which includes multiple images and description text of each image;

[0023] Extract noun phrases from the description text to obtain multiple second noun phrases corresponding to multiple images;

[0024] Performing region extraction on each image based on the second noun phrase corresponding to each image to obtain a plurality of first region images corresponding to the plurality of second noun phrases;

[0025] A regional image set is constructed based on the multiple first regional images.

[0026] In some embodiments of the present application, feature extraction is performed on the target area image based on the target noun phrase to obtain common visual features, including:

[0027] Extract features from the target area image to obtain initial visual features;

[0028] Determine mean information and variance information based on the target noun phrase and the initial visual features;

[0029] Determine latent variable information based on mean information and variance information;

[0030] Perform linear operations on the initial visual features and latent variable information to obtain filtered initial visual features;

[0031] Based on the filtered initial visual features, a common visual feature is determined.

[0032] In some embodiments of the present application, multimodal aggregation is performed on general visual features and text encoding information to obtain multimodal feature information, including:

[0033] Based on the text encoding information and the common visual features, determine the noun phrase representation of the text information to be processed;

[0034] Adding the noun phrase representation to the text encoding information to obtain the text encoding information with the noun phrase representation added;

[0035] The text encoding information represented by the added noun phrase is fused with the text encoding information to obtain multimodal feature information.

[0036] In a second aspect, an embodiment of the present invention further provides a text processing device, comprising:

[0037] A text acquisition unit, used to acquire text information to be processed;

[0038] An information determination unit, used to determine multimodal feature information based on the text information to be processed;

[0039] The text processing unit is used to decode the multimodal feature information to obtain target text information.

[0040] In a third aspect, the present application further provides a computer device, the computer device comprising:

[0041] one or more processors;

[0042] Memory; and

[0043] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement any one of the text processing methods in the first aspect.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is loaded by a processor to execute the steps in any one of the text processing methods in the first aspect.

[0045] The beneficial effects of the present invention are as follows: based on the text information to be processed, multimodal feature information is determined, and feature information irrelevant to the text information to be processed is filtered out from the obtained multimodal feature information. The target text information is determined based on the multimodal feature information, which can reduce the noise introduced by irrelevant information and improve the processing quality of the text information to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 is a flowchart of a text processing method provided by an embodiment of the present invention;

[0048] Figure 2 It is a flowchart of a specific embodiment of the text processing method provided by an embodiment of the present invention;

[0049] Figure 3 is a structural diagram of a text processing model provided by an embodiment of the present invention;

[0050] Figure 4 is a principle block diagram of a text processing device provided by an embodiment of the present invention;

[0051] Figure 5 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0053] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, which are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third", and "fourth" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", "third", and "fourth" may explicitly or implicitly include one or more features. In the description of the present application, "multiple" means two or more, and "several" means one or more, unless otherwise clearly and specifically defined.

[0054] In this application, the word "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0055] It should be noted that since the method of the embodiment of the present application is executed in a computer device, the processing objects of each computer device exist in the form of data or information. For example, time is actually time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data for processing by the computer device. The details will not be repeated here.

[0056] Multimodal machine translation refers to the task of using images and text for machine translation. The inventors have found through research that traditional multimodal machine translation usually inputs images and source sentences to be translated in pairs, and then uses the image content as a visual information context to assist the source sentence in machine translation. However, in practice, there are very few such sentence-image pairs, which makes traditional multimodal machine translation tasks difficult to perform. In order to solve the problem of insufficient sentence-image pairs, some existing methods use source sentences to retrieve related images in a preset sentence-image set, and then use the retrieved images as visual context to assist the source sentence in machine translation. However, since this method directly retrieves related images in the sentence-image set through the source sentence, the retrieved images often do not match the source sentence to a high degree.

[0057] Based on this, in an embodiment of the present application, text information to be processed is obtained; based on the text information to be processed, multimodal feature information is determined; the multimodal feature information is decoded to obtain target text information. Since feature information irrelevant to the text information to be processed is filtered out in the multimodal feature information, the target text information is determined based on the multimodal feature information, which can reduce the noise introduced by irrelevant information and improve the processing quality of the text information to be processed.

[0058] The content of this application is further illustrated below through the description of embodiments in conjunction with the accompanying drawings.

[0059] This embodiment provides a text processing method, such as Figure 1 and Figure 2 As shown in , the method includes:

[0060] Step S10: Obtain text information to be processed.

[0061] Specifically, the text information to be processed is text that needs to be translated. The text information to be processed can be a sentence or a paragraph input by a user through a computer device (for example, a smart phone), or it can be text data obtained by converting speech to text of collected user conversations through a computer device.

[0062] Step S20: Determine multimodal feature information based on the text information to be processed.

[0063] The multimodal feature information is an aggregated feature obtained by multimodally aggregating the text encoding information of the text information to be processed and the general visual features of the noun phrases in the text information to be processed. The multimodal feature information filters out the feature information that is irrelevant to the text information to be processed. Determining the target text information based on the multimodal feature information can reduce the noise introduced by irrelevant information and improve the processing quality of the text information to be processed.

[0064] In a specific implementation, Figure 2 As shown, step S20 includes:

[0065] Step S21, encoding the text information to be processed to obtain text encoding information of the text information to be processed;

[0066] Step S22, performing regional image retrieval and feature extraction on the text information to be processed to obtain general visual features;

[0067] Step S23: performing multimodal aggregation on the general visual features and text encoding information to obtain multimodal feature information.

[0068] The text encoding information is the encoding representation of the text information to be processed obtained by encoding the text information to be processed. The text encoding information can be expressed as E = (e1, e2, ..., e n ), where e n Represents the text representation of the nth word in the text information to be processed.

[0069] In a specific implementation, the text processing method is applied to a text processing model, such as Figure 3 As shown, the text processing model includes an encoder. Accordingly, the steps of encoding the text information to be processed and obtaining the text encoding information of the text information to be processed may specifically include: performing word embedding processing on the text information to be processed to obtain a first word embedding representation of the text information to be processed, adding preset position information to the first word embedding representation to obtain a first word embedding representation with added position information, inputting the first word embedding representation with added position information into the encoder, and outputting the text encoding information of the text information to be processed through the encoder.

[0070] Further, continue to refer to Figure 3 As shown, the encoder includes a first attention layer, a first residual connection layer, a first fully connected layer and a second residual connection layer cascaded in sequence, then the first word embedding representation with added position information is input into the encoder, and the step of outputting the text encoding information of the text information to be processed through the encoder can specifically include: inputting the first word embedding representation with added position information into the first attention layer, outputting the first feature through the first attention layer, inputting the first feature and the first word embedding representation with added position information into the first residual connection layer for fusion and normalization processing to obtain the second feature, inputting the second feature into the first fully connected layer for position transformation processing to obtain the second feature after position transformation, and inputting the second feature after position transformation and the second feature into the second residual connection layer for fusion and normalization processing to obtain the text encoding information. Among them, the first fully connected layer can adopt a feedforward neural network, and the first attention layer can adopt a multi-head attention layer.

[0071] By performing regional image retrieval on the text information to be processed, regional images in the image that match the noun phrases in the text information to be processed can be retrieved, and the general visual features are features extracted from the regional images that match the noun phrases in the text information to be processed. In this embodiment, regional image retrieval and feature extraction are performed on the text information to be processed, and feature information that is not related to the noun phrases in the text information to be processed is filtered out from the obtained general visual features. The text information to be processed is processed based on the general visual features, which can improve the processing quality of the text information to be processed.

[0072] After obtaining the general visual features, multimodal aggregation is performed on the general visual features and text encoding information to obtain multimodal feature information. Since the general visual features filter out feature information that is not related to the noun phrases in the text information to be processed, the multimodal feature information obtained based on the general visual features also filters out feature information that is not related to the noun phrases in the text information to be processed. Processing the text information to be processed based on the multimodal feature information can reduce the noise introduced by irrelevant information and improve the processing quality of the text information to be processed.

[0073] In a specific implementation, continue to refer to Figure 2 As shown, step S22 includes:

[0074] Step S221, extracting noun phrases from the text information to be processed to obtain a first noun phrase;

[0075] Step S222: searching a pre-constructed region image set based on the first noun phrase to obtain a target region image of the first noun phrase, wherein the target region image corresponds to the target noun phrase;

[0076] Step S223: extracting features of the target area image based on the target noun phrase to obtain common visual features.

[0077] The first noun phrase is a noun phrase in the text information to be processed. A noun phrase refers to a type of phrase whose grammatical function is equivalent to a noun. A noun phrase can generally serve as a subject, object, or attributive in a sentence. Noun phrases include attributive phrases centered on nouns (such as "the great motherland" and "these children"), joint phrases composed of nouns (such as "workers and farmers"), complex phrases (such as "the capital Beijing"), directional phrases (such as "on the desktop" and "in front of the building"), and "of" phrases (such as "night watchman"), etc.

[0078] In a specific implementation, continue to refer to Figure 3As shown, the text processing model includes a visual representation retrieval module, and the visual representation retrieval module includes an extraction submodule. Accordingly, the step of extracting noun phrases from the text information to be processed to obtain the first noun phrase may specifically include: inputting the text information to be processed into the extraction submodule, and extracting noun phrases from the text information to be processed by the extraction submodule to obtain the first noun phrase. For example, the extraction submodule may use the open source library spaCy to identify noun phrases in the text information to be processed to obtain the first noun phrase Where t represents the number of noun phrases identified from the text information to be processed, and each first noun phrase It consists of a sequence of words. l i First noun phrase The index of the first word of , d i First noun phrase Length.

[0079] The region image set is a pre-constructed region image set, which includes a plurality of first region images, each of which corresponds to a plurality of second noun phrases, and is obtained by extracting regions from images in a preset text image set based on the corresponding second noun phrases. For example, the plurality of first region images include Multiple first area images Short with multiple second nouns Corresponding respectively, based on multiple second noun short For image I i By performing region extraction, multiple first region images can be obtained.

[0080] The target region image is the first region image most relevant to the first noun phrase in the region image set. The target region image may be one first region image most relevant to the first noun phrase in the region image set, or may be multiple first region images most relevant to the first noun phrase in the region image set. For example, the target region image is the K first region images most relevant to the first noun phrase in the region image set, where K ≥ 2 and K is an integer.

[0081] In a specific implementation, continue to refer to Figure 3 As shown, the visual representation retrieval module also includes a retrieval submodule. Accordingly, the step of retrieving a pre-constructed regional image set based on a first noun phrase to obtain a target regional image of the first noun phrase may specifically include: inputting the first noun phrase into the retrieval submodule, and retrieving a pre-constructed regional image set based on the first noun phrase through the retrieval submodule to obtain the target regional image of the first noun phrase.

[0082] It should be noted that the existing method directly uses the source sentence to retrieve relevant images in a preset sentence-image set. The above step S222 has at least the following two improvements compared with the existing technology: (1) image retrieval is performed through the noun phrase (i.e., the first noun phrase) of the text information to be processed, rather than directly performing image retrieval based on the text information to be processed; (2) instead of directly retrieving images from the preset sentence-image set, a regional image set is first constructed based on the preset sentence-image set, and then regional images are retrieved from the regional image set, that is, instead of directly retrieving images, regions in the images are retrieved. In this way, only image regions related to the text information to be processed can be searched, and image regions in the image that are not related to the text information to be processed can be filtered out, so that the retrieved target regional image can be more consistent with the expression of the source sentence, thereby improving the processing quality of the text information to be processed.

[0083] In the above steps, it is mentioned that the target area image is the first area image corresponding to the K second noun phrases that are most similar to the first noun phrase among multiple second noun phrases, and the target noun phrase corresponding to the target area image is the noun phrase corresponding to the target area image among multiple second noun phrases. After the traditional multimodal machine translation retrieves the image related to the source sentence, it directly performs feature extraction on the retrieved image. The extracted features will contain a lot of feature information (such as background, texture, etc.) that is irrelevant to the text information to be processed, resulting in poor translation quality of the text information to be processed. This embodiment performs feature extraction on the target area image based on the target noun phrase, and the obtained general visual features filter out the feature information that is irrelevant to the target noun phrase. The text information to be processed is processed based on the general visual features, which can improve the processing quality of the text information to be processed.

[0084] In a specific implementation, continue to refer to Figure 3 As shown, the visual representation retrieval module also includes a visual representation submodule. Accordingly, the step of extracting features of the target area image based on the target noun phrase to obtain the general visual features specifically includes: inputting the target noun phrase and the target area image into the visual representation submodule, and extracting features of the target area image based on the target noun phrase through the visual representation submodule to obtain the general visual features.

[0085] In a specific implementation, the region image set includes a plurality of first region images, and the plurality of first region images correspond to a plurality of second noun phrases. Step S222 includes:

[0086] Step S2221, encoding the first noun phrase to obtain first encoding information, and encoding the second noun phrase corresponding to each first regional image in the regional image set to obtain second encoding information;

[0087] Step S2222: determining similarity information between the first noun phrase and the second noun phrase corresponding to each first region image based on the first coding information and the second coding information;

[0088] Step S2223: determining a target region image of the first noun phrase from the plurality of first region images based on the similarity information.

[0089] The first encoding information is the context information of each word in the first noun phrase obtained by encoding the first noun phrase, that is, the first encoding information contains the context information of each word in the first noun phrase. For example, the first noun phrase can be encoded using a pre-trained BERT, and the first encoding information can be expressed as: Among them, a l Indicates the first noun phrase The context information of the lth word in Indicates the first noun phrase The lth word in the sentence, l represents the first noun phrase The number of words in .

[0090] The regional image set includes multiple first regional images, and the multiple first regional images correspond to multiple second noun phrases respectively. The second encoding information is the context information of each word in the second noun phrase obtained by encoding the second noun phrase corresponding to each first regional image in the regional image set. That is, the second encoding information contains the context information of each word in the second noun phrase. For example, the pre-trained BERT can be used to encode the second noun phrase, and the second encoding information can be expressed as: b1, b2, …, b m =BERT(p1,p2,…,p m ), where b m Indicates the second noun phrase P i The context information of the mth word in m Indicates the second noun phrase P i The mth word in the sentence, m represents the second noun phrase P i The number of words in .

[0091] The similarity information is used to characterize the degree of similarity between the first noun phrase and the second noun phrase corresponding to each first region image. After obtaining the first coding information and the second coding information, the embodiment determines the similarity information between the first noun phrase and the second noun phrase corresponding to each first region image based on the first coding information and the second coding information, and then selects the second noun phrase most similar to the first noun phrase from multiple second noun phrases based on the similarity information, and determines the first region image corresponding to the most similar second noun phrase as the target region image corresponding to the first noun phrase.

[0092] In a specific implementation, when the target region image is the K first region images most relevant to the first noun phrase in the region image set, based on the similarity information, the K second noun phrases most similar to the first noun phrase are screened out from the plurality of second noun phrases, and the K first region images corresponding to the K most similar second noun phrases are determined as the target region images corresponding to the first noun phrase. For example, the similarity scores of the second noun phrase A, the second noun phrase B, the second noun phrase C, the second noun phrase D, and the second noun phrase E with the first noun phrase A are 0.5, 0.7, 0.6, 0.3, and 0.9, respectively. Based on the similarity scores, the three second noun phrases most similar to the first noun phrase A (i.e., the second noun phrase E, the second noun phrase B, and the second noun phrase C) are screened out, and the first region images corresponding to the second noun phrase E, the second noun phrase B, and the second noun phrase C are determined as the target region images corresponding to the first noun phrase A.

[0093] In a specific implementation, the step of determining similarity information between the first noun phrase and the second noun phrase corresponding to each first region image based on the first coding information and the second coding information may specifically include: determining third coding information based on the first coding information, and determining fourth coding information based on the second coding information, and determining similarity information between the first noun phrase and the second noun phrase corresponding to each first region image based on the third coding information and the fourth coding information. The calculation formula of the similarity information is: First noun phrase and the second noun phrase P i The similarity information between First noun phrase The third coded information, enco(P i ) indicates the second noun phrase P i The fourth encoded information.

[0094] The third encoding information is the average value of the context information of each word in the first noun phrase. In the above step, it is mentioned that the first encoding information contains the context information of each word in the first noun phrase. Based on the first encoding information, the third encoding information can be determined. The third encoding information can be expressed as: in, First noun phrase The third coded information, a j First noun phrase The context information of the jth word in , l represents the first noun phrase The number of words in .

[0095] Furthermore, the fourth encoding information is an average value of the context information of each word in the second noun phrase. In the aforementioned step, it is mentioned that the second encoding information includes the context information of each word in the second noun phrase. Based on the second encoding information, the fourth encoding information can be determined, wherein the fourth encoding information can be expressed as: enco(P i ) indicates the second noun phrase P i The fourth coded information, b k Indicates the second noun phrase P i The context information of the kth word in , m represents the second noun phrase P i The number of words in .

[0096] In a specific implementation, before step S222, the following steps are included:

[0097] Step S01, obtaining a preset text image set, wherein the text image set includes multiple images and a description text of each image;

[0098] Step S02, extracting noun phrases from the description text to obtain a plurality of second noun phrases corresponding to the plurality of images;

[0099] Step S03, performing region extraction on each image based on the second noun phrase corresponding to each image, to obtain a plurality of first region images corresponding to a plurality of second noun phrases;

[0100] Step S04: construct a regional image set based on the multiple first regional images.

[0101] The text image set is a pre-set text image set, which includes multiple images and description text of each image. The text image set can be expressed as: G = {(X i ,I i )}, i∈(1,N), G represents the text image set, I i represents the i-th image in the text image set, X i represents the description text of the i-th image, and N represents the number of images in the text image set. The second noun phrase is a noun phrase extracted from the description texts of multiple images. For example, the description text X is identified using the open source library spaCy. i The noun phrase in the second noun phrase Among them, t i Indicates the description text X i The number of second noun phrases extracted from the

[0102] The first region image is a region image of each image obtained by performing region extraction on each image based on the second noun phrase corresponding to each image. Each noun phrase in For the i-th image I i Performing region extraction can extract multiple noun phrases A plurality of first region images corresponding to each other The regional image set is a set of multiple first regional images corresponding to multiple second noun phrases. The regional image set can be expressed as {R i}, t i Represents the description text X of the i-th image i , where N represents the number of images in the text image set.

[0103] Existing multimodal machine translation usually directly uses the source sentence to retrieve relevant images in a preset sentence-image set. There will be many image areas irrelevant to the source sentence in the retrieved images. The irrelevant image areas will introduce noise during the source sentence translation process, resulting in poor translation quality of the source sentence. In order to solve the above problem, this embodiment will pre-extract noun phrases from the description text of each image in the text image set to obtain multiple second noun phrases corresponding to multiple images, and then perform region extraction on each image based on the second noun phrases corresponding to each image to obtain multiple first region images corresponding to multiple second noun phrases, and then construct a region image set based on the multiple first region images. Text processing is performed based on the region image set, and only image areas related to the text information to be processed can be searched, reducing the noise introduced by irrelevant image areas and improving the quality of text processing.

[0104] In a specific implementation, step S223 includes:

[0105] Step S2231, extracting features from the target area image to obtain initial visual features;

[0106] Step S2232, determining mean information and variance information based on the target noun phrase and the initial visual features;

[0107] Step S2233, determining latent variable information based on mean information and variance information;

[0108] Step S2234: performing linear operations on the initial visual features and latent variable information to obtain filtered initial visual features;

[0109] Step S2235: Determine the common visual features based on the filtered initial visual features.

[0110] The initial visual features are visual features extracted by directly extracting features from the target area image. For example, the target area image can be input into a pre-trained FasterNet, and features of the target area image can be extracted through the pre-trained FasterNet to obtain the initial visual features of the first noun phrase.

[0111] Taking into account that the initial visual features contain a lot of visual information that is not mentioned in the first noun phrase, such as shape, texture, etc., in order to prevent redundant visual information from interfering with the text processing task, after obtaining the initial visual features, in this embodiment, the initial visual features are filtered based on the target noun phrase to obtain the filtered initial visual features, and then based on the filtered initial visual features, the common visual features of the first noun phrase are determined.

[0112] As mentioned in the above steps, the target region image may be a first region image that is most relevant to the first noun phrase in the region image set, or may be a plurality of first region images that are most relevant to the first noun phrase in the region image set. In a specific implementation, when the target region image is a first region image that is most relevant to the first noun phrase in the region image set, the step of determining the common visual features of the first noun phrase based on the filtered initial visual features may specifically include: determining the filtered initial visual features as the common visual features of the first noun phrase.

[0113] In another specific implementation, when the target area image is a plurality of first area images that are most relevant to the first noun phrase in the area image set, feature extraction is performed on the plurality of target area images to obtain a plurality of initial visual features of the first noun phrase, and then a plurality of filtered initial visual features can be obtained. Then, based on the filtered initial visual features, the step of determining the general visual features of the first noun phrase may specifically include: obtaining similarity information between the target noun phrase corresponding to each target area image and the first noun phrase, and determining the general visual features of the first noun phrase based on the similarity information and the plurality of filtered initial visual features. The formula for determining the general visual features is: u represents the universal visual feature, and K represents the first noun phrase The number of target area images, Indicates the first noun phrase The target noun phrase corresponding to the kth target region image Similarity information of Represents the filtered initial visual features obtained from the kth target area image.

[0114] It should be noted that when the target area image is a plurality of first area images that are most relevant to the first noun phrase in the area image set, the common visual features of the first noun phrase are combined with the visual information of multiple target area images, which can avoid the deviation caused by a single target area image and further improve the processing quality of the text information to be processed.

[0115] In order to filter out the visual information not mentioned in the target noun phrase from the initial visual features, this embodiment introduces a latent variable model based on visual feature extraction. The latent variable model includes an encoding layer and a decoding layer. After the target noun phrase and the initial visual features are input into the encoding layer of the latent variable model, the encoding layer will output mean information and variance information. The mean information and variance information can be expressed as: represents mean information, represents variance information, P represents the target noun phrase, represents the initial visual features extracted from the kth target area image, RNN(·) represents the final hidden state of a single-layer unidirectional recurrent neural network, and Linear(·) represents a linear change.

[0116] After determining the mean information and variance information, the latent variable information can be determined based on the mean information and variance information, and a linear operation is performed based on the initial visual features and the latent variable information through the decoding layer in the latent variable model to obtain the filtered initial visual features. Among them, the decoding layer in the latent variable model can adopt a single-layer unidirectional recurrent neural network (RNN), and the calculation formula of the latent variable information and the filtered initial visual features can be expressed as: z represents latent variable information, μ q represents mean information, σ p represents variance information, α represents random noise, N(0,I) represents standard normal distribution, represents element-wise multiplication, Represents the filtered initial visual features obtained from the kth target area image.

[0117] Furthermore, the latent variable model of this embodiment adopts the Conditional Variational AutoEncoder (CVAE) framework to maximize the conditional marginal log likelihood logp(P|v)=log∫p(P|z,v)p(z|v)dz by maximizing the evidence lower bound (ELBO). Specifically, the latent variable model is trained using the following loss function: Among them, P represents a noun phrase, v represents the initial visual feature, and D KL represents KL divergence, p ω (z|v) represents the prior probability, q φ (z|P,v) represents the approximate posterior probability, p θ (P|z,v) represents the decoder, p ω (z|v)=N(z;μ p (v),σ p (v) 2 I), q φ (z|P,v)=N(z;μ q (P,v),σ q (P,v) 2 I), μ p (v)=Linear(v),σ p (v)=Linear(v),μ q (P, v) = Linear ((RNN (P), v)), σ q (P,v)=Linear((RNN(P),v)). The latent variable model trained by the above loss function can pay more attention to the semantic features mentioned in the noun phrase, while filtering out the information in the initial visual feature v that is irrelevant to the noun phrase.

[0118] In a specific implementation, continue to refer to Figure 2 As shown, step S23 includes:

[0119] Step S231, determining the noun phrase representation of the text information to be processed based on the text encoding information and the general visual features;

[0120] Step S232, adding the noun phrase representation to the text encoding information to obtain the text encoding information with the noun phrase representation added;

[0121] Step S233: Fusing the text encoding information represented by the added noun phrase with the text encoding information to obtain multimodal feature information.

[0122] For each first noun phrase The corresponding word text representation can be found in the text encoding information The noun phrase representation of the text information to be processed can be recorded as M = (m1, m2, .., m t ), m i Indicates the first noun phrase The universal visual representation u i With the first noun phrase The word text representation is fused to obtain the fusion information.

[0123] In a specific implementation, continue to refer to Figure 3 As shown, the text processing model also includes a multimodal aggregation module, and the multimodal aggregation module includes an aggregation submodule. Accordingly, the step of determining the noun phrase representation of the text information to be processed based on the text encoding information and the general visual features can specifically include: inputting the text encoding information and the general visual features into the aggregation submodule, determining the word text representation of the first noun phrase based on the text encoding information through the aggregation submodule, and fusing the general visual features of the first noun phrase and the word text representation of the first noun phrase to obtain the noun phrase representation of the text information to be processed. Among them, m i First noun phrase The general visual representation u i With the first noun phrase The word text represents the fusion information obtained by fusion, W1 and W2 represent the preset first fusion weight and second fusion weight, e j First noun phrase The text representation of the jth word in , represents element-by-element multiplication, l i First noun phrase The index of the first word in , d i First noun phrase Length.

[0124] Continue to refer to Figure 3 As shown, the multimodal aggregation module also includes a second attention layer, and the step of adding the noun phrase representation to the text encoding information may specifically include: inputting the noun phrase representation and the text encoding information into the second attention layer, adding the noun phrase representation to the text encoding information through the second attention layer, and obtaining the text encoding information with the noun phrase representation added. For example, the second attention layer may use a multi-head attention mechanism to add the noun phrase representation to the text encoding information, and the process of the second attention layer adding the noun phrase representation to the text encoding information may be expressed as: in, Indicates the text encoding information of adding noun phrase representation, E represents text encoding information, and M represents noun phrase representation.

[0125] Continue to refer to Figure 3 As shown, the multimodal aggregation module also includes an information fusion layer, and the step of fusing the text encoding information represented by the added noun phrase with the text encoding information can specifically include: inputting the text encoding information represented by the added noun phrase and the text encoding information into the information fusion layer, and fusing the text encoding information represented by the added noun phrase with the text encoding information through the information fusion layer to obtain the target fusion information.

[0126] Specifically, the information fusion layer can use a gating mechanism to fuse the text encoding information represented by the added noun phrase with the text encoding information, and the step of fusing the text encoding information represented by the added noun phrase with the text encoding information can further include: fusing the text encoding information represented by the added noun phrase with the text encoding information to obtain first fusion information, applying the first fusion information to a preset activation function, and multiplying the first fusion information after the activation function with the text encoding information represented by the added noun phrase to obtain second fusion information, and finally fusing the second fusion information with the text encoding information to obtain target fusion information. Among them, the activation function can be set as needed. For example, when the activation function is a sigmoid function, the calculation process of the target fusion information can be expressed as: Wherein, S represents the target fusion information, E represents the text encoding information, W3 and W4 represent the preset third fusion weight and fourth fusion weight, Indicates the text encoding information of adding noun phrase representation, Represents element-wise multiplication.

[0127] Step S30: Decode the multimodal feature information to obtain target text information.

[0128] The target text information is the translated text of the text information to be processed. After obtaining the multimodal feature information, this embodiment decodes the multimodal feature information to obtain the translated text of the text information to be processed. Since the feature information irrelevant to the text information to be processed is filtered out in the multimodal feature information, the target text information is determined based on the multimodal feature information, which can reduce the noise introduced by irrelevant information and improve the processing quality of the text information to be processed.

[0129] In a specific implementation, continue to refer to Figure 3 As shown, the text processing model also includes a decoder, which decodes the multimodal feature information. The step of obtaining the target text information may specifically include: inputting the multimodal feature information into the decoder, and outputting the target text information through the decoder.

[0130] Continue to refer to Figure 3As shown, the decoder includes a third attention layer, a third residual connection layer, a fourth attention layer, a fourth residual connection layer, a second fully connected layer and a fifth residual connection layer which are cascaded in sequence. Among them, the output item of the third attention layer and the input item of the third attention layer are the input items of the third residual connection layer, the output item of the third residual connection layer and the output item of the multimodal aggregation module are the input items of the fourth attention layer, the output item of the fourth attention layer and the output item of the third residual connection layer are the input items of the fourth residual connection layer, the output item of the fourth residual connection layer is the input item of the second fully connected layer, and the output item of the second fully connected layer and the output item of the fourth residual connection layer are the input items of the fifth residual connection layer. Among them, the third attention layer can adopt a masked multi-head attention layer, the fourth attention layer can adopt a multi-head attention layer, and the second fully connected layer can adopt a feedforward neural network.

[0131] In a specific implementation, before step S20, it also includes: obtaining a training text set, which includes training text and real text corresponding to the training text; inputting the training text and the real text into a preset network model, and outputting the probability of each word in the predicted text corresponding to the training text belonging to each word in the real text through the preset network model; determining the loss value based on the probability and the loss function of the preset network model; when the loss value does not meet the preset conditions, correcting the model parameters of the preset network model based on the preset parameter learning rate, and continuing to perform the step of inputting the training text and the real text into the preset network model until the loss value meets the preset conditions to obtain the text processing model. Among them, the calculation formula of the loss value is: represents the loss value, represents the i-th word in the real text, X represents the training text, and y i Represents the i-th word in the predicted text corresponding to the training text, Represents the i-th word y in the predicted text corresponding to the training text i belong probability.

[0132] Continue to refer to Figure 3As shown, during the training process of the preset network model, when the real text is input into the preset network model, the real text is specifically processed by word embedding to obtain a second word embedding representation of the real text, the preset position information is added to the second word embedding representation, and the second word embedding representation with the added position information is input into the third attention layer for processing to obtain a third feature, and then the third feature and the second word embedding representation with the added position information are input into the third residual connection layer for fusion and normalization to obtain a fourth feature, and then the fourth feature and the visual representation output by the multimodal aggregation module are input into the fourth attention layer for processing to obtain a fifth feature, and then the fifth feature and the fourth feature are input into the fourth residual connection layer for fusion and normalization to obtain a sixth feature, and then the sixth feature is processed by the second fully connected layer and the fifth residual connection layer in turn, and the processing result is sent to the linear activation layer to obtain the probability that each word in the predicted text corresponding to the training text belongs to each word in the real text.

[0133] In order to better implement the text processing method in the embodiment of the present application, based on the text processing method, the embodiment of the present application also provides a text processing device, such as Figure 4 As shown, the text processing device includes:

[0134] A text acquisition unit 410, used to acquire text information to be processed;

[0135] An information determining unit 420, configured to determine multimodal feature information based on the text information to be processed;

[0136] The text processing unit 430 is used to decode the multimodal feature information to obtain target text information.

[0137] In an embodiment of the present application, multimodal feature information is determined based on the text information to be processed, and feature information irrelevant to the text information to be processed is filtered out from the obtained multimodal feature information. The target text information is determined based on the multimodal feature information, which can reduce the noise introduced by irrelevant information and improve the processing quality of the text information to be processed.

[0138] In some embodiments of the present application, the information determination unit 420 is specifically used to:

[0139] Encode the text information to be processed to obtain text encoding information of the text information to be processed;

[0140] Perform regional image retrieval and feature extraction on the text information to be processed to obtain general visual features;

[0141] Multimodal aggregation is performed on general visual features and text encoding information to obtain multimodal feature information.

[0142] In some embodiments of the present application, the information determination unit 420 is further configured to:

[0143] Extract noun phrases from the text information to be processed to obtain the first noun phrase;

[0144] Retrieving a pre-constructed regional image set based on the first noun phrase to obtain a target regional image of the first noun phrase, wherein the target regional image corresponds to the target noun phrase;

[0145] Feature extraction is performed on the target area image based on the target noun phrase to obtain the general visual features.

[0146] In some embodiments of the present application, the region image set includes a plurality of first region images, and the plurality of first region images correspond to a plurality of second noun phrases. The information determination unit 420 is further configured to:

[0147] encoding the first noun phrase to obtain first encoding information, and encoding the second noun phrase corresponding to each first regional image in the regional image set to obtain second encoding information;

[0148] Determining similarity information between the first noun phrase and the second noun phrase corresponding to each first region image based on the first coding information and the second coding information;

[0149] Based on the similarity information, a target region image of the first noun phrase is determined from the plurality of first region images.

[0150] In some embodiments of the present application, the information determination unit 420 is further configured to:

[0151] Obtain a preset text image set, which includes multiple images and description text of each image;

[0152] Extract noun phrases from the description text to obtain multiple second noun phrases corresponding to multiple images;

[0153] Performing region extraction on each image based on the second noun phrase corresponding to each image to obtain a plurality of first region images corresponding to the plurality of second noun phrases;

[0154] A regional image set is constructed based on the multiple first regional images.

[0155] In some embodiments of the present application, the information determination unit 420 is further configured to:

[0156] Extract features from the target area image to obtain initial visual features;

[0157] Determine mean information and variance information based on the target noun phrase and the initial visual features;

[0158] Determine latent variable information based on mean information and variance information;

[0159] Perform linear operations on the initial visual features and latent variable information to obtain filtered initial visual features;

[0160] Based on the filtered initial visual features, a common visual feature is determined.

[0161] In some embodiments of the present application, the information determination unit 420 is further configured to:

[0162] Multimodal aggregation is performed on general visual features and text encoding information to obtain multimodal feature information, including:

[0163] Based on the text encoding information and the common visual features, determine the noun phrase representation of the text information to be processed;

[0164] Adding the noun phrase representation to the text encoding information to obtain the text encoding information with the noun phrase representation added;

[0165] The text encoding information represented by the added noun phrase is fused with the text encoding information to obtain multimodal feature information.

[0166] The embodiment of the present application further provides a computer device, which integrates any text processing device provided in the embodiment of the present application, and the computer device includes:

[0167] one or more processors;

[0168] Memory; and

[0169] One or more applications, wherein the one or more applications are stored in a memory and are configured to execute, by a processor, the steps of the text processing method in any of the above text processing method embodiments.

[0170] The present application also provides a computer device that integrates any text processing device provided in the present application. Figure 5 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0171] The computer device may include one or more processing core processors 501, one or more computer-readable storage media memories 502, a power supply 503, an input unit 504 and other components. Those skilled in the art will appreciate that Figure 5 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0172] The processor 501 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 502 and calling data stored in the memory 502, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501.

[0173] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0174] The computer device also includes a power supply 503 for supplying power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 503 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0175] The computer device may further include an input unit 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0176] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 501 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 502 according to the following instructions, and the processor 501 will run the application programs stored in the memory 502, thereby realizing various functions, as follows:

[0177] Get the text information to be processed;

[0178] Determine multimodal feature information based on the text information to be processed;

[0179] The multimodal feature information is decoded to obtain the target text information.

[0180] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0181] To this end, the embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any text processing method provided in the embodiment of the present application. For example, the computer program is loaded by a processor to execute the following steps:

[0182] Get the text information to be processed;

[0183] Determine multimodal feature information based on the text information to be processed;

[0184] The multimodal feature information is decoded to obtain the target text information.

[0185] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0186] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0187] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0188] The above is a detailed introduction to a text processing method, device, computer equipment and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A text processing method, characterized in that: include: Get the text information to be processed; Determining multimodal feature information based on the text information to be processed; The multimodal feature information is decoded to obtain target text information.

2. The method according to claim 1, characterized in that Determining multimodal feature information based on the text information to be processed includes: Encoding the text information to be processed to obtain text encoding information of the text information to be processed; Performing regional image retrieval and feature extraction on the text information to be processed to obtain general visual features; The general visual features and the text encoding information are multimodally aggregated to obtain multimodal feature information.

3. The method according to claim 2, characterized in that The performing of regional image retrieval and feature extraction on the text information to be processed to obtain general visual features includes: Extracting noun phrases from the text information to be processed to obtain a first noun phrase; Retrieving a pre-constructed regional image set based on the first noun phrase to obtain a target regional image of the first noun phrase, wherein the target regional image corresponds to the target noun phrase; Feature extraction is performed on the target area image based on the target noun phrase to obtain a common visual feature.

4. The method according to claim 3, characterized in that The regional image set includes a plurality of first regional images, and the plurality of first regional images correspond to a plurality of second noun phrases. The searching of the pre-constructed regional image set based on the first noun phrase to obtain the target regional image of the first noun phrase includes: encoding the first noun phrase to obtain first encoding information, and encoding the second noun phrase corresponding to each of the first regional images in the regional image set to obtain second encoding information; Determining similarity information between the first noun phrase and a second noun phrase corresponding to each of the first region images based on the first encoding information and the second encoding information; Based on the similarity information, a target region image of the first noun phrase is determined from a plurality of the first region images.

5. The method according to claim 4, characterized in that Before searching the pre-built regional image set based on the first noun phrase to obtain the target regional image of the first noun phrase, the method includes: Acquire a preset text image set, wherein the text image set includes a plurality of images and a description text of each of the images; Extracting noun phrases from the description text to obtain a plurality of second noun phrases corresponding to the plurality of images; performing region extraction on each of the images based on the second noun phrase corresponding to each of the images to obtain a plurality of first region images corresponding to a plurality of the second noun phrases; A regional image set is constructed based on the plurality of first regional images.

6. The method according to claim 3, characterized in that The step of extracting features from the target area image based on the target noun phrase to obtain a common visual feature includes: Extracting features from the target area image to obtain initial visual features; Determining mean information and variance information based on the target noun phrase and the initial visual features; Determining latent variable information based on the mean information and the variance information; Performing a linear operation on the initial visual feature and the latent variable information to obtain a filtered initial visual feature; Based on the filtered initial visual features, a common visual feature is determined.

7. The method according to claim 2, characterized in that The multimodal aggregation of the general visual features and the text encoding information to obtain multimodal feature information includes: Determining a noun phrase representation of the text information to be processed based on the text encoding information and the general visual features; Adding the noun phrase representation to the text encoding information to obtain text encoding information with the noun phrase representation added; The text encoding information represented by the added noun phrase is fused with the text encoding information to obtain multimodal feature information.

8. A text processing device, characterized in that: include: A text acquisition unit, used to acquire text information to be processed; An information determination unit, configured to determine multimodal feature information based on the text information to be processed; The text processing unit is used to decode the multimodal feature information to obtain target text information.

9. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the text processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the text processing method according to any one of claims 1 to 7.

Images