Data processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to a data processing method, apparatus, electronic device, and storage medium. The method comprises: obtaining a business image to be processed and its associated business text; obtaining a visual cue vector based on extracted visual features of the business image to be processed; encoding the visual cue vector and the business text to obtain a coding vector; the coding vector comprising a visual coding vector corresponding to the visual cue vector and a text coding vector corresponding to the business text; determining an entity cue vector based on entity coding vectors corresponding to each named entity in the visual coding vector and the text coding vector; and decoding the coding vector based on the visual cue vector and the entity cue vector to obtain a description text corresponding to the business image to be processed. The present disclosure improves the accuracy of generating named entities in the description text.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, device, electronic device, and storage medium. Background Art

[0002] Currently, news articles published on the Internet often contain news images, and each news image is also accompanied by corresponding descriptive text. In order to improve the publishing efficiency of news articles on the Internet, descriptive text is usually automatically generated for each news image in the news article.

[0003] Considering that news scenarios involve a large number of named entities, such as names of people, places, and organizations, and these named entities often exceed the vocabulary of the language model (Out Of Vocabulary, abbreviated as OOV), in order to automatically generate descriptive text containing named entities, related technologies combine manually written entity-level description templates on the basis of language models. However, since the manually written entity-level description templates cover a limited scope and can only perceive local entity words, the generation of named entities is not accurate, which in turn reduces the accuracy of the descriptive text corresponding to the news image. Summary of the Invention

[0004] The present disclosure provides a data processing method, apparatus, electronic device, and storage medium to at least address the problem of poor accuracy in automatically generating descriptive text corresponding to news images in related technologies. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, including:

[0006] Acquire a business image to be processed and a business text associated with the business image to be processed; the business text includes a named entity;

[0007] Obtaining a visual cue vector based on the extracted visual features of the business image to be processed;

[0008] Encoding the visual prompt vector and the business text to obtain an encoding vector; the encoding vector includes a visual encoding vector corresponding to the visual prompt vector and a text encoding vector corresponding to the business text;

[0009] Determining an entity prompt vector based on the entity encoding vector corresponding to each of the named entities in the visual encoding vector and the text encoding vector;

[0010] The encoding vector is decoded based on the visual prompt vector and the entity prompt vector to obtain a description text corresponding to the business image to be processed.

[0011] In an exemplary embodiment, determining the entity prompt vector based on the entity encoding vector corresponding to each of the named entities in the visual encoding vector and the text encoding vector includes:

[0012] Determining a correlation between the visual encoding vector and each entity encoding vector in the text encoding vector;

[0013] A target entity encoding vector is selected from the entity encoding vectors based on the correlation degree to form an entity prompt vector.

[0014] In an exemplary embodiment, selecting a target entity encoding vector from the entity encoding vectors based on the relevance to form an entity prompt vector includes:

[0015] Determine the entity encoding vector corresponding to the maximum correlation degree to obtain the key entity encoding vector;

[0016] Determining a degree of dependence between the key entity code vector and each remaining entity code vector; the remaining entity code vector refers to an entity code vector other than the key entity code vector;

[0017] Determining a target remaining entity encoding vector based on the dependency degree; wherein the dependency degree corresponding to the target remaining entity encoding vector is greater than a preset dependency degree threshold;

[0018] The key entity encoding vector and the target remaining entity encoding vector are concatenated as a target entity encoding vector to obtain an entity prompt vector.

[0019] In an exemplary embodiment, determining the degree of dependency between the key entity encoding vector and each remaining entity encoding vector includes:

[0020] Using the key entity encoding vector as the initial hidden state of a bidirectional long short-term memory network;

[0021] Inputting the remaining entity encoding vector into the bidirectional long short-term memory network to obtain a state vector corresponding to the remaining entity encoding vector;

[0022] Normalization processing is performed based on the state vector corresponding to the remaining entity coding vector to obtain a normalized result corresponding to the remaining entity coding vector; the normalized result represents the degree of dependence between the key entity coding vector and the corresponding remaining entity coding vector.

[0023] In an exemplary embodiment, decoding the encoding vector based on the visual prompt vector and the entity prompt vector to obtain the description text corresponding to the service image to be processed includes:

[0024] concatenating the visual cue vector and the entity cue vector to obtain a multimodal cue vector;

[0025] Autoregressive decoding is performed based on the multimodal prompt vector and the encoding vector to obtain a description text corresponding to the service image to be processed.

[0026] In an exemplary embodiment, obtaining a visual prompt vector based on the extracted visual features of the business image to be processed includes:

[0027] Inputting the business image to be processed into a vision-language pre-training model for feature extraction to obtain extracted visual features;

[0028] Mapping the visual features to the input space of the target language model based on a multi-layer perception network to obtain a visual cue vector;

[0029] The target language model is used to perform the encoding process and the decoding process.

[0030] In an exemplary embodiment, the method is implemented based on a data processing model, and the method further comprises the step of training the data processing model:

[0031] Acquire a sample business image text pair and a corresponding reference description text; the sample business image text pair includes a sample business image and a sample business text associated with the sample business image, wherein the sample business text includes a named entity;

[0032] Extracting visual features of the sample business image based on the vision-language pre-trained model, and mapping the visual features of the sample business image to the input space of the pre-trained language model based on an initial multi-layer perception network to obtain a sample visual cue vector;

[0033] Inputting the sample visual cue vector and the sample business text into an encoder of a pre-trained language model for encoding processing to obtain a sample encoding vector; the sample encoding vector includes a sample visual vector corresponding to the sample visual cue vector and a sample text encoding vector corresponding to the sample business text;

[0034] Determining a sample entity prompt vector based on the sample visual encoding vector and the sample entity encoding vector corresponding to each of the named entities in the sample text encoding vector;

[0035] The sample visual prompt vector and the sample entity prompt vector are concatenated to obtain a sample multimodal prompt vector; the sample multimodal prompt vector and the sample encoding vector are input into the decoder of the pre-trained language model for decoding processing to obtain a predicted description text;

[0036] The model parameters are adjusted based on the difference between the predicted description text and the reference description text until the training is terminated when a preset training termination condition is reached, thereby obtaining the data processing model.

[0037] In an exemplary embodiment, adjusting the model parameters based on the difference between the predicted description text and the reference description text includes:

[0038] Determining a loss value based on a difference between the predicted description text and the reference description text;

[0039] The model parameters of the vision-language pre-training model are fixed, and the parameters of the initial multi-layer perception network and the pre-training language model are adjusted based on the loss value.

[0040] In an exemplary embodiment, determining the sample entity prompt vector based on the sample entity encoding vector corresponding to each named entity in the sample visual encoding vector and the sample text encoding vector includes:

[0041] Determining a correlation between the sample visual encoding vector and each sample entity encoding vector in the sample text encoding vector;

[0042] A target sample entity encoding vector is selected from the sample entity encoding vectors based on the sample relevance to form a sample entity prompt vector.

[0043] In an exemplary embodiment, selecting a target sample entity encoding vector from the sample entity encoding vectors based on the sample relevance to form a sample entity prompt vector includes:

[0044] Determine the sample entity encoding vector corresponding to the maximum sample correlation degree to obtain the key sample entity encoding vector;

[0045] Determining a sample dependency between the key sample entity coding vector and each remaining sample entity coding vector; the remaining sample entity coding vector refers to a sample entity coding vector other than the key sample entity coding vector;

[0046] Determining a target remaining sample entity coding vector based on the sample dependency; the sample dependency corresponding to the target remaining sample entity coding vector is greater than a preset dependency threshold;

[0047] The key sample entity encoding vector and the target remaining sample entity encoding vector are concatenated as a target sample entity encoding vector to obtain a sample entity prompt vector.

[0048] In an exemplary embodiment, determining the sample dependency between the key sample entity encoding vector and each remaining sample entity encoding vector includes:

[0049] Using the key sample entity encoding vector as the initial hidden state of a bidirectional long short-term memory network;

[0050] Inputting the remaining sample entity encoding vector into the bidirectional long short-term memory network to obtain a state vector corresponding to the remaining sample entity encoding vector;

[0051] Normalization processing is performed based on the state vector corresponding to the remaining sample entity coding vector to obtain a sample normalization result corresponding to the remaining sample entity coding vector; the sample normalization result represents the degree of sample dependence between the key sample entity coding vector and the remaining sample entity coding vector.

[0052] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing apparatus, including:

[0053] A data acquisition unit is configured to acquire a business image to be processed and business text associated with the business image to be processed; the business text includes a named entity;

[0054] a visual cue vector determining unit, configured to obtain a visual cue vector based on the extracted visual features of the service image to be processed;

[0055] an encoding unit configured to perform encoding processing on the visual cue vector and the business text to obtain an encoding vector; the encoding vector includes a visual encoding vector corresponding to the visual cue vector and a text encoding vector corresponding to the business text;

[0056] an entity hint vector determining unit, configured to determine an entity hint vector based on the entity encoding vector corresponding to each of the named entities in the visual encoding vector and the text encoding vector;

[0057] The decoding unit is configured to perform decoding processing on the encoding vector based on the visual prompt vector and the entity prompt vector to obtain a description text corresponding to the business image to be processed.

[0058] In an exemplary embodiment, the entity hint vector determining unit includes:

[0059] A first correlation determination unit is configured to determine the correlation between the visual encoding vector and each entity encoding vector in the text encoding vector;

[0060] The entity prompt vector determining subunit is configured to select a target entity encoding vector from the entity encoding vectors based on the correlation degree to form an entity prompt vector.

[0061] In an exemplary embodiment, the entity hint vector determination subunit includes:

[0062] A first key entity determining unit is configured to determine an entity encoding vector corresponding to the maximum correlation degree to obtain a key entity encoding vector;

[0063] A first dependency determination unit is configured to determine the dependency between the key entity code vector and each remaining entity code vector; the remaining entity code vector refers to an entity code vector other than the key entity code vector;

[0064] A first determining unit is configured to determine a target remaining entity encoding vector based on the dependency degree; the dependency degree corresponding to the target remaining entity encoding vector is greater than a preset dependency degree threshold;

[0065] The first construction subunit is configured to execute concatenation of the key entity encoding vector and the target remaining entity encoding vector as a target entity encoding vector to obtain an entity prompt vector.

[0066] In an exemplary embodiment, the first dependency level determining unit includes:

[0067] A first initialization unit is configured to execute the key entity encoding vector as an initial hidden state of a bidirectional long short-term memory network;

[0068] A first state vector determining unit is configured to input the remaining entity encoding vector into the bidirectional long short-term memory network to obtain a state vector corresponding to the remaining entity encoding vector;

[0069] The first normalization unit is configured to perform normalization processing based on the state vector corresponding to the remaining entity coding vector to obtain a normalized result corresponding to the remaining entity coding vector; the normalized result represents the degree of dependence between the key entity coding vector and the remaining entity coding vector.

[0070] In an exemplary embodiment, the decoding unit includes:

[0071] a multimodal cue vector determining unit, configured to concatenate the visual cue vector and the entity cue vector to obtain a multimodal cue vector;

[0072] The decoding subunit is configured to perform autoregressive decoding processing based on the multimodal prompt vector and the encoding vector to obtain a description text corresponding to the service image to be processed.

[0073] In an exemplary embodiment, the visual cue vector determination unit includes:

[0074] a visual feature extraction unit configured to input the service image to be processed into a vision-language pre-training model for feature extraction to obtain extracted visual features;

[0075] A first mapping unit is configured to perform mapping of the visual features to an input space of a target language model based on a multi-layer perception network to obtain a visual cue vector;

[0076] The target language model is used to perform the encoding process and the decoding process.

[0077] In an exemplary embodiment, the apparatus further comprises a training unit, wherein the training unit comprises:

[0078] A sample acquisition unit is configured to acquire a sample business image text pair and a corresponding reference description text; the sample business image text pair includes a sample business image and a sample business text associated with the sample business image, and the sample business text includes a named entity;

[0079] a sample visual cue vector determination unit configured to extract visual features of the sample business image based on the visual-language pre-trained model, and map the visual features of the sample business image to the input space of the pre-trained language model based on an initial multi-layer perception network to obtain a sample visual cue vector;

[0080] a sample encoding unit configured to perform encoding processing by inputting the sample visual cue vector and the sample business text into an encoder of a pre-trained language model to obtain a sample encoding vector; the sample encoding vector includes a sample visual vector corresponding to the sample visual cue vector and a sample text encoding vector corresponding to the sample business text;

[0081] a sample entity prompt vector determining unit, configured to determine a sample entity prompt vector based on the sample entity encoding vector corresponding to each of the named entities in the sample visual encoding vector and the sample text encoding vector;

[0082] a sample multimodal suggestive vector determining unit configured to concatenate the sample visual prompt vector and the sample entity prompt vector to obtain a sample multimodal prompt vector; input the sample multimodal prompt vector and the sample encoding vector into a decoder of the pre-trained language model for decoding processing to obtain a predicted description text;

[0083] The parameter adjustment unit is configured to adjust the model parameters based on the difference between the predicted description text and the reference description text until the preset training end condition is reached to end the training and obtain the data processing model.

[0084] In an exemplary embodiment, the parameter adjustment unit includes:

[0085] a loss determining unit configured to determine a loss value based on a difference between the predicted description text and the reference description text;

[0086] A parameter adjustment subunit is configured to fix the model parameters of the vision-language pre-training model and adjust the parameters of the initial multi-layer perception network and the pre-training language model based on the loss value.

[0087] In an exemplary embodiment, the sample entity hint vector determining unit includes:

[0088] A second correlation determination unit is configured to determine the correlation between the sample visual encoding vector and each sample entity encoding vector in the sample text encoding vector;

[0089] The sample entity prompt vector determining subunit is configured to select a target sample entity encoding vector from the sample entity encoding vectors based on the sample correlation degree to form a sample entity prompt vector.

[0090] In an exemplary embodiment, the sample entity prompt vector determining subunit includes:

[0091] A second key entity determining unit is configured to determine a sample entity encoding vector corresponding to the maximum sample correlation degree to obtain a key sample entity encoding vector;

[0092] A second dependency determination unit is configured to determine a sample dependency between the key sample entity code vector and each remaining sample entity code vector; the remaining sample entity code vector refers to a sample entity code vector other than the key sample entity code vector;

[0093] The second determining unit is configured to determine a target remaining sample entity coding vector based on the sample dependency; the sample dependency corresponding to the target remaining sample entity coding vector is greater than a preset dependency threshold;

[0094] The second construction subunit is configured to execute concatenation of the key sample entity encoding vector and the target remaining sample entity encoding vector as a target sample entity encoding vector to obtain a sample entity prompt vector.

[0095] In an exemplary embodiment, the second dependency level determining unit includes:

[0096] A second initialization unit is configured to execute the key sample entity encoding vector as the initial hidden state of a bidirectional long short-term memory network;

[0097] A second state vector determining unit is configured to input the remaining sample entity encoding vector into the bidirectional long short-term memory network to obtain a state vector corresponding to the remaining sample entity encoding vector;

[0098] The second normalization unit is configured to perform normalization processing based on the state vector corresponding to the remaining sample entity coding vector to obtain a sample normalization result corresponding to the remaining sample entity coding vector; the sample normalization result represents the degree of sample dependence between the key sample entity coding vector and the remaining sample entity coding vector.

[0099] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0100] processor;

[0101] a memory for storing instructions executable by the processor;

[0102] The processor is configured to execute the instructions to implement the data processing method of the first aspect mentioned above.

[0103] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data processing method of the first aspect described above.

[0104] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program / instruction, which implements the data processing method of the first aspect when executed by a processor.

[0105] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0106] A visual cue vector is obtained based on the extracted visual features of the business image to be processed, and the visual cue vector and the business text associated with the business image to be processed are encoded to obtain a coding vector, which includes a visual coding vector corresponding to the visual cue vector and a text coding vector corresponding to the business text. Then, an entity cue vector is determined based on the entity coding vectors corresponding to each named entity in the visual coding vector and the text coding vector, and the coding vector is decoded based on the visual cue vector and the entity cue vector to obtain a description text corresponding to the business image to be processed, so that more attention can be paid to the named entities related to the content of the business image to be processed during the decoding process, thereby improving the accuracy of named entity generation and thus improving the accuracy of the description text.

[0107] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0109] Figure 1 is a schematic diagram of an application environment of a data processing method according to an exemplary embodiment;

[0110] Figure 2 is a flow chart showing a data processing method according to an exemplary embodiment;

[0111] Figure 3 is a flowchart illustrating a method for constructing an entity prompt vector according to an exemplary embodiment;

[0112] Figure 4 is a flowchart showing another method of constructing an entity prompt vector according to an exemplary embodiment;

[0113] Figure 5 is a structural diagram of a data processing model according to an exemplary embodiment;

[0114] Figure 6 According to an exemplary embodiment, a fine-tuning pre-training model is shown to obtain Figure 5 Schematic diagram of the data processing model;

[0115] Figure 7 is a structural block diagram of a data processing device according to an exemplary embodiment;

[0116] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0117] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0118] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0119] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0120] News image description generation technology aims to automatically generate descriptive text for images in news articles. Since news image description generation requires attention to both the visual subject in the image and the background information contained in the article, traditional image description generation methods are limited to image input and cannot process long text information. In addition, news scenes involve a large number of named entities, such as names of people, places, and organizations. These entity words are often out of vocabulary (OOV) of language models, making traditional image description generation methods unable to solve the problem of text generation containing entity words.

[0121] In related technologies, some methods adopt a template-based approach, replacing all entity words in the description sentence with slots representing entity words, and then matching and filling in the blanks with the input at the entity level. The subdivision can include two-stage generation and one-stage generation. The two-stage generation is to first generate a template sentence without entity words and then predict entity words. The one-stage generation is to consider the distribution probability of non-entity words and entity words when generating each word to achieve end-to-end dynamic prediction. However, the above methods have the following defects: 1) Entity word generation is heavily dependent on the quality of manual templates at the entity level. In actual applications, image description sentences in news scenes are usually natural and rich in information. Therefore, defining an optimal template is very time-consuming and difficult to cover all manually written description sentences; 2) Current text generation is limited to template-based supervised training methods. Multi-stage iterative generation and template filling will increase the complexity of the model and cause a certain amount of error accumulation. The OOV problem is difficult to alleviate in one-stage generation, and the accuracy of entity word generation is poor.

[0122] In order to ensure the accuracy of entity words, other methods in the related art consider updating detailed entity word templates as supervision signals, that is, classifying entity words according to their grammatical components in the sentence to introduce entity word type labels to further constrain the entity word generation space. However, this supervised training based on entity-level templates can only perform local entity word perception, which not only destroys the semantic coherence of language structure and context, but also lacks perception of the internal relationship of global entity words in the language space, resulting in inaccurate entity words in the final generated image description text.

[0123] In view of this, an embodiment of the present disclosure provides a data processing method, which obtains a visual prompt vector based on the extracted visual features of the business image to be processed, and encodes the visual prompt vector and the business text associated with the business image to be processed to obtain a coding vector, which includes a visual coding vector corresponding to the visual prompt vector and a text coding vector corresponding to the business text, and then determines the entity prompt vector based on the entity coding vector corresponding to each named entity in the visual coding vector and the text coding vector, and decodes the coding vector based on the visual prompt vector and the entity prompt vector to obtain the description text corresponding to the business image to be processed, so that more attention can be paid to the named entities related to the content of the business image to be processed during the decoding process, thereby improving the accuracy of named entity generation, and thus improving the accuracy of the image description text.

[0124] See also Figure 1 , which is a schematic diagram of an application environment of a data processing method according to an exemplary embodiment. The application environment may include a terminal 110 and a server 120. The terminal 110 and the server 120 may be connected via a wired network or a wireless network.

[0125] Terminal 110 may be, but is not limited to, a smartphone, tablet computer, laptop computer, or desktop computer. Terminal 110 may be installed with client software, such as an application (App), that provides data processing functionality. This application may be a standalone application or a subroutine within an application. For example, the application may include a news application, such as an application with a news publishing function. Users of terminal 110 may log in to the application using pre-registered user information, which may include an account number and password.

[0126] Server 120 may be a server that provides background services for applications in terminal 110. Specifically, the background services may be services that process images to generate corresponding description text for the images. Server 120 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0127] The technical solutions of the embodiments of the present disclosure are described in detail below. Figure 2 is a flow chart showing a data processing method according to an exemplary embodiment, which can be applied to Figure 1 In the electronic device shown, Figure 2 As shown, the following steps are included:

[0128] In step S201, a business image to be processed and business text associated with the business image to be processed are obtained.

[0129] The business text associated with the business image to be processed includes named entities, such as names of people, places, and organizations, etc. For example, the business image to be processed may be a news image, and the business text associated with the business image to be processed may be a news article corresponding to the news image.

[0130] In step S203, a visual prompt vector is obtained based on the extracted visual features of the business image to be processed.

[0131] The visual features of the business image to be processed refer to features obtained by extracting features from the business image to be processed.

[0132] A visual cue vector is a vector obtained by performing a dimensional transformation on a visual feature. This dimensional transformation is used to establish a connection between visual concepts and language priors. In a specific implementation, the dimensional transformation can be performed based on the input dimension of the subsequent encoding processing of the text information.

[0133] In step S205, the visual prompt vector and the business text are encoded to obtain an encoding vector.

[0134] The encoding vector includes a visual encoding vector corresponding to the visual prompt vector and a text encoding vector corresponding to the business text.

[0135] Specifically, the business text may be segmented first, each word may be taken as a token to obtain a word sequence, and then the visual cue vector and the word sequence may be concatenated to obtain a sequence to be encoded, and then the sequence to be encoded may be encoded.

[0136] In the embodiment of the present disclosure, in order to utilize context information to simultaneously encode the business image and business text to be processed based on the bidirectional self-attention mechanism, the above-mentioned sequence to be encoded is encoded based on the bidirectional self-attention mechanism. For example, the encoding process can be expressed as: C = SA (p v ,x A ).

[0137] Among them, SA() represents the self-attention mechanism (Self-attention); p v represents the visual cue vector; x A Represents business text; C represents the encoding vector. Among them, C includes C I and C A , C I represents the visual encoding vector, C A Represents the text encoding vector; where C A Including entity encoding vector C E , that is, C E ∈C A .

[0138] In step S207, an entity prompt vector is determined based on the entity encoding vector corresponding to each of the named entities in the visual encoding vector and the text encoding vector.

[0139] The entity encoding vector is the vector corresponding to the named entity in the text encoding vector.

[0140] In a specific implementation, a natural language processing toolkit (such as the SpaCy toolkit) can be used to mark all named entities in the business text, so that the entity encoding vector corresponding to each named entity can be extracted from the encoding vector based on the marking information.

[0141] In an exemplary embodiment, in order to enable the entity hint vector to more accurately reflect the content of the image to be processed, the above step S207 may include:

[0142] Determining a correlation between the visual encoding vector and each entity encoding vector in the text encoding vector;

[0143] A target entity encoding vector is selected from the entity encoding vectors based on the correlation degree to form an entity prompt vector.

[0144] In a specific implementation, the relevance score corresponding to each entity encoding vector can be calculated by the following formula to represent the degree of relevance:

[0145]

[0146] Among them, C I represents the visual encoding vector; represents weighted average; represents the entity encoding vector i, τ represents a hyperparameter, which can be set based on actual experience; <> represents dot product calculation; s i represents the relevance score of entity encoding vector i.

[0147] Then we can get the correlation score set S E , which includes the correlation score corresponding to each entity encoding vector. Based on the above correlation score, a target entity encoding vector whose correlation with the visual encoding vector meets the preset conditions can be selected from the entity encoding vector to form an entity prompt vector.

[0148] In an exemplary embodiment, when selecting a target entity encoding vector from the entity encoding vectors based on the correlation degree to form an entity prompt vector, the following can be done: Figure 3 The following steps are shown:

[0149] In step S301, the entity coding vector corresponding to the maximum correlation degree is determined to obtain a key entity coding vector.

[0150] In the specific implementation, the key entity encoding vector C key It can be expressed by the following formula:

[0151]

[0152] In step S303, the degree of dependency between the key entity code vector and each remaining entity code vector is determined.

[0153] The remaining entity coding vectors refer to entity coding vectors other than the key entity coding vectors.

[0154] In an exemplary embodiment, in order to obtain entity-level contextual hint vectors to improve the accuracy of entity words in the generated description text, as shown in FIG. Figure 4As shown, the above step S303 may include:

[0155] In step S401, the key entity encoding vector is used as the initial hidden state of a bidirectional long short-term memory network.

[0156] In step S403, the remaining entity encoding vector is input into the bidirectional long short-term memory network to obtain a state vector corresponding to the remaining entity encoding vector.

[0157] In step S405 , normalization processing is performed based on the state vector corresponding to the remaining entity code vector to obtain a normalization result corresponding to the remaining entity code vector.

[0158] The normalization result represents the degree of dependence between the key entity encoding vector and the corresponding remaining entity encoding vectors.

[0159] The bidirectional long short-term memory network is initialized in step S401 so that the initial hidden state of the initialized bidirectional long short-term memory network is the key entity coding vector, and then in step S403, the remaining entity coding vector is input into the initialized bidirectional long short-term memory network for processing to obtain the state vector corresponding to the remaining entity coding vector.

[0160] In a specific implementation, the state vector can be expressed by the following formula:

[0161] C prompt =LSTM(C key ,C′ E )

[0162] Correspondingly, the normalized result can be expressed by the following formula:

[0163] S prompt =σ(fc(C prompt ))

[0164] Among them, LSTM represents a bidirectional long short-term memory network, C key As the initial hidden state of the LSTM; C′ E represents the remaining entity encoding vector; C prompt represents the state vector of the network output; fc() represents a single linear layer; σ() represents the activation function; S prompt Indicates the normalized result, which is a numerical value that can reflect the score.

[0165] The above implementation method uses the key entity encoding vector as the initial hidden state of the bidirectional LSTM, so that the potential dependency relationship between the key entity encoding vector and other entity encoding vectors can be modeled based on context information, thereby improving the accuracy of determining the dependency relationship.

[0166] In step S305, a target remaining entity encoding vector is determined based on the dependency level.

[0167] Among them, the dependency degree corresponding to the target residual entity coding vector is greater than a preset dependency degree threshold.

[0168] The preset dependency threshold can be set according to actual needs.

[0169] In specific implementation, if (preset dependency threshold), then you can The corresponding entity encoding vector Determine the target residual entity encoding vector, so that we can get

[0170] In step S307 , the key entity encoding vector and the target remaining entity encoding vector are concatenated as a target entity encoding vector to obtain an entity prompt vector.

[0171] In the specific implementation, the entity hint vector p μ It can be expressed by the following formula:

[0172]

[0173] The above implementation achieves the screening and aggregation of important entity encoding vectors (ie, target entity encoding vectors) from entity encoding vectors based on context information, and generates an accurate, global entity-level contextual prompt vector.

[0174] In step S209, the encoding vector is decoded based on the visual prompt vector and the entity prompt vector to obtain a description text corresponding to the business image to be processed.

[0175] Specifically, the visual prompt vector and the entity prompt vector can be spliced ​​to obtain a multimodal prompt vector, and then the encoding vector and the multimodal prompt vector can be used for decoding through the cross-attention mechanism to obtain the descriptive text corresponding to the business image to be processed.

[0176] It can be seen from the technical solution of the embodiment of the present disclosure that by constructing a visual prompt vector and an entity prompt vector, and combining these two prompt vectors to decode the encoding vector to obtain the description text of the business image to be processed, more attention can be paid to the named entities related to the business image content during decoding, thereby improving the accuracy of the named entities generated in the description text, and thus improving the accuracy of the generated description text.

[0177] In an exemplary embodiment, to improve the accuracy of the generated description text, the decoding process in step S209 may be an autoregressive decoding process, that is, the prediction result of the current time step needs to rely on the prediction results of the past time steps. Therefore, the above step S209 may include the following when implemented:

[0178] concatenating the visual cue vector and the entity cue vector to obtain a multimodal cue vector;

[0179] Autoregressive decoding is performed based on the multimodal prompt vector and the encoding vector to obtain a description text corresponding to the service image to be processed.

[0180] Specifically, the multimodal prompt vector P can be expressed as P = [p v ;p μ ]=[p1,…,p k ], where k represents the length of the multimodal prompt vector. In the embodiment of the present disclosure, the length k of the multimodal prompt vector is a fixed length, and the specific length value can be set according to actual experience.

[0181] t represents the time step of autoregressive decoding, y t Represents the word element generated at time step t. When performing autoregressive decoding based on the multimodal prompt vector and the encoding vector, for time step t, S is first processed based on S = [P; y] and the self-attention mechanism to obtain the output vector SelfAtt(S), where y = {y1,…,y t-1}, then perform cross attention processing based on SelfAtt(S) and the encoding vector C, and then based on the cross attention processing result h t To predict the word corresponding to time step t.

[0182] It should be noted that, in the embodiment of the present disclosure, when S is processed based on S=[P;y] and the self-attention mechanism to obtain the output vector SelfAtt(S), let the "query Q" input be y and let the "key-value pair KV" input be S, then:

[0183]

[0184] Among them, Q, K, V are obtained by mapping y; K P ,V P Obtained by P mapping; [;] indicates splicing; d H Represents the feature dimension of the self-attention mechanism.

[0185] In the embodiment of the present disclosure, when performing cross attention processing based on SelfAtt(D) and the encoding vector C, the “query Q” input is the above-mentioned SelfAtt(S), and the “key-value pair KV” input is the encoding vector C. t To predict the word corresponding to time step t, from h t The vector corresponding to the sequence after extracting the multimodal prompt vector in S is used to calculate the distribution probability of the word unit. The specific distribution probability is The calculation formula can be expressed as follows:

[0186]

[0187] in, Represents a matrix vector mapping to the logical distribution over the vocabulary; [|P idx |:] indicates the sequence following the multimodal hint vector.

[0188] The above embodiment performs autoregressive decoding processing based on the multimodal prompt vector and the encoding vector, and uses the multimodal prompt vector to guide the generation of entity words and non-entity words in each decoding step, so that the entity words in the generated description text are more accurate.

[0189] In an exemplary embodiment, in order to enhance the influence of the visual cue vector on semantic understanding so as to more accurately focus on entity words related to the image content, step S203 may include:

[0190] Input the business image to be processed into the vision-language pre-training model for feature extraction to obtain extracted visual features;

[0191] Mapping the visual features to the input space of the target language model based on a multi-layer perception network to obtain a visual cue vector;

[0192] The target language model is used to perform the encoding process and the decoding process.

[0193] Specifically, the image to be processed can be encoded using an image encoder based on a vision-language pre-training model to obtain encoded features, which are used as the extracted visual features. The vision-language pre-training model can be a CLIP (Contrastive Language-Image Pre-training) model, which is a pre-trained model obtained by contrastive learning on large-scale image and text datasets.

[0194] The target language model is obtained by fine-tuning the pre-trained language model based on the sample business image and the sample business text corresponding to the sample business image. The target language model is used to implement the encoding and decoding processes in the embodiment of the present disclosure.

[0195] Exemplarily, the pre-trained language model includes a Transformer-based encoder and a Transformer-based decoder. The Transformer-based encoder can be an encoder with bidirectional encoding capabilities. For example, the pre-trained language model can be BART (Bidirectional and Auto-Regressive Transformers). BART uses a standard Transformer-based neural machine translation architecture and can be considered a generalized form of pre-trained models such as BERT (bidirectional encoder) and GPT (left-to-right decoder).

[0196] The above implementation method extracts visual features through a visual-language pre-training model, thereby utilizing the high-level image semantic understanding capabilities of the visual-language pre-training model to obtain visual features that are more consistent with semantic understanding, which is conducive to improving the influence of the visual cue vector obtained based on this on semantic understanding, and thus improving the accuracy of entity words in the description text.

[0197] As can be seen from the above embodiments, the data processing method of the embodiment of the present disclosure can be implemented based on a data processing model, which is composed of a vision-language pre-training model, a multi-layer perception network, a target language model and an entity prompt vector building module, such as Figure 5 The figure shows a structural diagram of the data processing model provided by an embodiment of the present disclosure, wherein the target language model is obtained by fine-tuning the pre-trained language model, including a Transformer-based encoder and a Transformer-based decoder.

[0198] In a specific implementation, the business image to be processed is input into a vision-language pre-trained model. The image encoder of the vision-language pre-trained model performs encoding processing to obtain visual features as output. This visual feature serves as the input of a multi-layer perceptual network, which is then mapped to the input space of a target language model to obtain a visual cue vector. This visual cue vector is then concatenated with the business text associated with the business image to be processed and used as the input of the encoder in the target language model. The target language model encoder then performs encoding processing to obtain an encoding vector. This encoding vector includes a visual encoding vector corresponding to the visual cue vector and a text encoding vector corresponding to the business text.

[0199] The entity encoding vector corresponding to each named entity in the visual encoding vector and the text encoding vector is input into the entity prompt vector construction model. The entity prompt vector construction model selects the target entity encoding vector from the entity encoding vector based on the degree of correlation between the visual prompt vector and each entity encoding vector to form an entity prompt vector.

[0200] After the visual cue vector and the entity cue vector are concatenated, they form the input of the decoder in the target language model together with the encoding vector output by the encoder. The decoder may include a stack of multiple Transformer decoding modules, each of which includes two attention layers. The first attention layer uses the concatenation of the visual cue vector and the entity cue vector plus the prediction result of the historical moment as input for processing based on the self-attention mechanism. The second attention layer uses the processing result of the first attention layer and the encoding vector output by the encoder for autoregressive decoding processing based on the cross-attention mechanism. The self-attention mechanism processing method of the first attention layer can be found in the relevant content of the aforementioned step S209 and will not be repeated here.

[0201] Based on this, in an exemplary embodiment, the embodiment of the present disclosure may further include the step of training a data processing model. Specifically, the training data processing model may include:

[0202] A sample business image text pair and a corresponding reference description text are obtained, wherein the sample business image text pair includes a sample business image and a sample business text associated with the sample business image, and the sample business text includes a named entity.

[0203] The visual features of the sample business image are extracted based on the vision-language pre-training model, and the visual features of the sample business image are mapped to the input space of the pre-trained language model based on the initial multi-layer perception network to obtain a sample visual prompt vector.

[0204] The sample visual cue vector and the sample business text are input into an encoder of a pre-trained language model for encoding to obtain a sample encoding vector, wherein the sample encoding vector includes a sample visual vector corresponding to the sample visual cue vector and a sample text encoding vector corresponding to the sample business text.

[0205] A sample entity prompt vector is determined based on the sample visual encoding vector and the sample entity encoding vector corresponding to each of the named entities in the sample text encoding vector.

[0206] The sample visual prompt vector and the sample entity prompt vector are concatenated to obtain a sample multimodal prompt vector; the sample multimodal prompt vector and the sample encoding vector are input into the decoder of the pre-trained language model for decoding processing to obtain a predicted description text.

[0207] The model parameters are adjusted based on the difference between the predicted description text and the reference description text until the training is terminated when a preset training termination condition is reached, thereby obtaining a data processing model.

[0208] Among them, the preset training end condition can be set according to actual needs, for example, the number of iterations reaches a preset iteration threshold, or the loss value reaches a preset loss threshold, etc.

[0209] In the above embodiment, by utilizing a large-scale visual-language pre-training model to convert sample business images into visual prompt vectors, and combining with sample business texts to construct entity prompt vectors, and then based on the multimodal prompt vectors composed of images and texts, combined with the prompt learning mechanism, the pre-trained language model is fine-tuned, which can simultaneously unify the entity-level representations in business texts and business images, realize the joint learning of the two pre-training models, and thus obtain the data processing model of the embodiment of the present disclosure.

[0210] In an exemplary embodiment, to improve the efficiency of fine-tuning, the visual-language pre-trained model can be fixed and not participate in parameter updates, and only the mapping network (i.e., the initial multi-layer perceptron network) and the pre-trained language model are fine-tuned together. Therefore, when adjusting the model parameters based on the difference between the predicted description text and the reference description text, the above-mentioned method can include:

[0211] Determining a loss value based on a difference between the predicted description text and the reference description text;

[0212] The model parameters of the vision-language pre-training model are fixed, and the parameters of the initial multi-layer perception network and the pre-training language model are adjusted based on the loss value.

[0213] The loss value can be obtained based on a preset loss function, which can be a one-way language modeling loss based on cross entropy. Specifically, the preset loss function L can be expressed as follows:

[0214]

[0215] Among them, x A Indicates sample business text; x I represents a sample business image; P represents a multimodal prompt vector, P = [p v ;p μ ]=[p1,…,p k ];y τ＜t represents the predicted word before time step t, y τ＜t =y1,…,y t-1 ;p θ() represents the likelihood function; l represents the sum of k and the number of words contained in the reference description text.

[0216] In an exemplary embodiment, determining a sample entity prompt vector based on the sample visual encoding vector and the sample entity encoding vector corresponding to each of the named entities in the sample text encoding vector may include:

[0217] Determining a sample correlation degree between the sample visual encoding vector and each sample entity encoding vector in the sample text encoding vector;

[0218] A target sample entity encoding vector is selected from the sample entity encoding vectors based on the sample relevance to form a sample entity prompt vector.

[0219] The specific method for determining the degree of sample correlation can refer to the aforementioned embodiment of the present disclosure. Figure 2 The method for determining the degree of correlation in the illustrated embodiment will not be described in detail here.

[0220] The above implementation enables the sample entity hint vector to more accurately reflect the content of the sample image, thereby facilitating improved model training results.

[0221] In an exemplary embodiment, selecting a target sample entity encoding vector from the sample entity encoding vectors based on the sample relevance to form a sample entity hint vector includes:

[0222] Determine the sample entity encoding vector corresponding to the maximum sample correlation degree to obtain the key sample entity encoding vector;

[0223] Determining a sample dependency between the key sample entity coding vector and each remaining sample entity coding vector; the remaining sample entity coding vector refers to a sample entity coding vector other than the key sample entity coding vector;

[0224] Determining a target remaining sample entity coding vector based on the sample dependency; the sample dependency corresponding to the target remaining sample entity coding vector is greater than a preset dependency threshold;

[0225] The key sample entity encoding vector and the target remaining sample entity encoding vector are concatenated as a target sample entity encoding vector to obtain a sample entity prompt vector.

[0226] The specific method for determining the sample dependency and the target residual sample entity encoding vector can be found in the aforementioned embodiment of the present disclosure. Figure 3 The relevant descriptions in the method shown will not be repeated here.

[0227] The above implementation method filters and aggregates important entities based on context information and generates entity-level contextual hint vectors, which is beneficial to improving the entity accuracy of the model prediction results.

[0228] In an exemplary embodiment, determining the sample dependency between the key sample entity encoding vector and each remaining sample entity encoding vector includes:

[0229] Using the key sample entity encoding vector as the initial hidden state of a bidirectional long short-term memory network;

[0230] Inputting the remaining sample entity encoding vector into the bidirectional long short-term memory network to obtain a state vector corresponding to the remaining sample entity encoding vector;

[0231] Normalization processing is performed based on the state vector corresponding to the remaining sample entity coding vector to obtain a sample normalization result corresponding to the remaining sample entity coding vector; the sample normalization result represents the degree of sample dependence between the key sample entity coding vector and the remaining sample entity coding vector.

[0232] The above implementation method uses the key sample entity encoding vector as the initial hidden state of the bidirectional LSTM, thereby modeling the potential dependency relationship between the key sample entity encoding vector and other sample entity encoding vectors based on context information, thereby improving the accuracy of determining the dependency relationship.

[0233] In order to more clearly illustrate the technical solution of the embodiment of the present disclosure, the following Figure 6 Taking the news business as an example, where the visual-language pre-training model is CLIP and the pre-trained language model is BART, the fine-tuning process of the embodiment of the present disclosure is introduced to obtain the data processing model of the embodiment of the present disclosure.

[0234] like Figure 6 As shown, for a given dataset of paired sample news images and sample news texts in, Represent the sample news text and the corresponding sample news image respectively, and the sample news image The input is sent to the CLIP module of the data processing model, and the sample news image is processed by the image encoder of the CLIP module. Feature extraction is performed to obtain visual features, which are recorded as

[0235] The visual features are input into the initial multi-layer perception network MLP of the data model, and the visual cue vector is obtained by mapping it to the input space of BART through the initial multi-layer perception network, which is recorded as Among them, MLP stands for Multilayer Perceptron, pv represents the visual cue vector.

[0236] The visual cue vector p v and sample news text as the input sequence of BART encoder, denoted as Input to the BART encoder and act on it through the bidirectional attention mechanism And based on the hidden layer state vector in the last encoder layer, the sample encoding vector C is obtained. Among them, C includes p v The corresponding visual encoding vector C I and The corresponding sample text encoding vector C A ,like Figure 6 (a) and (b) in .

[0237] Input the sample encoding vector C into the context entity prompt building module for sequence modeling to obtain the entity prompt vector p μ , and p v With p μ The multimodal prompt vector P is obtained by splicing, which is denoted as P = [p v ;p μ ]=[p1,…,p k ].

[0238] The contextual entity hint construction module automatically learns the association between visual hints and text representations from the latent semantic space, constructs a global entity hint vector sequence, and guides the generation of entity tokens and non-entity tokens in each decoding step. In the specific implementation, the SpaCy toolkit is used to mark all named entities E in the sample news text, and then the sample text encoding vector C is obtained. A Extract the corresponding hidden state vectors C of these entity words E (C E ∈C A ), and finally the visual encoding vector C is transformed into I Perform weighted averaging and calculate the relevance score of each entity word based on the average result to obtain the relevance score set S E ,like Figure 6 As shown in (c), S E Each element in represents the importance score of a named entity. During training, the entity encoding vector with the highest score is taken out as the key entity encoding vector C through the argmax maximum value operation. key (like Figure 6 in ). Then, as Figure 6 As shown in (d), use C key As the initial hidden state of the bidirectional LSTM model to model the potential dependencies between the key entity and other entities (i.e. Figure 6 The score in C is then combined with the preset dependency threshold η to obtain the target remaining entity encoding vector. key It is concatenated with the target remaining entity encoding vector to construct the entity prompt vector p μ .

[0239] The multimodal hint vector P is added to the input sequence of the BART decoder, denoted as [P; y], and input into the BART decoder. Through the cross-attention mechanism, C and P are used to perform autoregressive iteration in [P; y] to generate the predicted description text (i.e., the predicted description text word-units are obtained in sequence). The self-attention layer of the BART decoder can be expressed as:

[0240]

[0241] Among them, Q, K, V are obtained by mapping y; K P ,V P Obtained by P mapping; [;] indicates splicing; d H Represents the feature dimension of the self-attention mechanism.

[0242] If you use P idx Represents the index number of the multimodal prompt vector P in the entire decoder input sequence, |P idx | represents the length of P, then the probability distribution of the output at time step t is: in, Represents a matrix vector mapping to the logical distribution over the vocabulary; [|P idx |:] indicates the sequence following the multimodal hint vector.

[0243] After obtaining the predicted description text, based on the predicted description text and The cross-entropy loss is calculated based on the difference between the corresponding reference description texts. When adjusting the model parameters, the CLIP module is frozen, and the parameters of the initial MLP and BART are adjusted based on the cross-entropy loss until the cross-entropy loss reaches a preset minimum, ending the training. This results in a data processing model that includes a vision-language pre-trained model, a fine-tuned MLP (i.e., a multi-layer perceptron network), a fine-tuned BART (i.e., a target language model), and a contextual entity prompt building module.

[0244] Figure 7 FIG. 1 is a structural block diagram of a data processing device according to an exemplary embodiment. Figure 7 , the data processing device 700 includes:

[0245] The data acquisition unit 710 is configured to acquire a business image to be processed and business text associated with the business image to be processed; the business text includes a named entity;

[0246] A visual cue vector determining unit 720 is configured to obtain a visual cue vector based on the extracted visual features of the service image to be processed;

[0247] The encoding unit 730 is configured to perform encoding processing on the visual cue vector and the business text to obtain an encoding vector; the encoding vector includes a visual encoding vector corresponding to the visual cue vector and a text encoding vector corresponding to the business text;

[0248] An entity hint vector determining unit 740 is configured to determine an entity hint vector based on the entity encoding vector corresponding to each of the named entities in the visual encoding vector and the text encoding vector;

[0249] The decoding unit 750 is configured to perform decoding processing on the encoding vector based on the visual prompt vector and the entity prompt vector to obtain a description text corresponding to the business image to be processed.

[0250] In an exemplary embodiment, the entity hint vector determining unit 740 includes:

[0251] A first correlation determination unit is configured to determine the correlation between the visual encoding vector and each entity encoding vector in the text encoding vector;

[0252] The entity prompt vector determining subunit is configured to select a target entity encoding vector from the entity encoding vectors based on the correlation degree to form an entity prompt vector.

[0253] In an exemplary embodiment, the entity hint vector determination subunit includes:

[0254] A first key entity determining unit is configured to determine an entity encoding vector corresponding to the maximum correlation degree to obtain a key entity encoding vector;

[0255] A first dependency determination unit is configured to determine the dependency between the key entity code vector and each remaining entity code vector; the remaining entity code vector refers to an entity code vector other than the key entity code vector;

[0256] A first determining unit is configured to determine a target remaining entity encoding vector based on the dependency degree; the dependency degree corresponding to the target remaining entity encoding vector is greater than a preset dependency degree threshold;

[0257] The first construction subunit is configured to execute concatenation of the key entity encoding vector and the target remaining entity encoding vector as a target entity encoding vector to obtain an entity prompt vector.

[0258] In an exemplary embodiment, the first dependency level determining unit includes:

[0259] A first initialization unit is configured to execute the key entity encoding vector as an initial hidden state of a bidirectional long short-term memory network;

[0260] A first state vector determining unit is configured to input the remaining entity encoding vector into the bidirectional long short-term memory network to obtain a state vector corresponding to the remaining entity encoding vector;

[0261] The first normalization unit is configured to perform normalization processing based on the state vector corresponding to the remaining entity coding vector to obtain a normalized result corresponding to the remaining entity coding vector; the normalized result represents the degree of dependence between the key entity coding vector and the remaining entity coding vector.

[0262] In an exemplary embodiment, the decoding unit 750 includes:

[0263] a multimodal cue vector determining unit, configured to concatenate the visual cue vector and the entity cue vector to obtain a multimodal cue vector;

[0264] The decoding subunit is configured to perform autoregressive decoding processing based on the multimodal prompt vector and the encoding vector to obtain a description text corresponding to the service image to be processed.

[0265] In an exemplary embodiment, the visual cue vector determining unit 720 includes:

[0266] a visual feature extraction unit configured to input the service image to be processed into a vision-language pre-training model for feature extraction to obtain extracted visual features;

[0267] A first mapping unit is configured to perform mapping of the visual features to an input space of a target language model based on a multi-layer perception network to obtain a visual cue vector;

[0268] The target language model is used to perform the encoding process and the decoding process.

[0269] In an exemplary embodiment, the apparatus further comprises a training unit, wherein the training unit comprises:

[0270] A sample acquisition unit is configured to acquire a sample business image text pair and a corresponding reference description text; the sample business image text pair includes a sample business image and a sample business text associated with the sample business image, and the sample business text includes a named entity;

[0271] a sample visual cue vector determination unit configured to extract visual features of the sample business image based on the visual-language pre-trained model, and map the visual features of the sample business image to the input space of the pre-trained language model based on an initial multi-layer perception network to obtain a sample visual cue vector;

[0272] a sample encoding unit configured to perform encoding processing by inputting the sample visual cue vector and the sample business text into an encoder of a pre-trained language model to obtain a sample encoding vector; the sample encoding vector includes a sample visual vector corresponding to the sample visual cue vector and a sample text encoding vector corresponding to the sample business text;

[0273] a sample entity prompt vector determining unit, configured to determine a sample entity prompt vector based on the sample entity encoding vector corresponding to each of the named entities in the sample visual encoding vector and the sample text encoding vector;

[0274] a sample multimodal suggestive vector determining unit configured to concatenate the sample visual prompt vector and the sample entity prompt vector to obtain a sample multimodal prompt vector; input the sample multimodal prompt vector and the sample encoding vector into a decoder of the pre-trained language model for decoding processing to obtain a predicted description text;

[0275] The parameter adjustment unit is configured to adjust the model parameters based on the difference between the predicted description text and the reference description text until the preset training end condition is reached to end the training and obtain the data processing model.

[0276] In an exemplary embodiment, the parameter adjustment unit includes:

[0277] a loss determining unit configured to determine a loss value based on a difference between the predicted description text and the reference description text;

[0278] A parameter adjustment subunit is configured to fix the model parameters of the vision-language pre-training model and adjust the parameters of the initial multi-layer perception network and the pre-training language model based on the loss value.

[0279] In an exemplary embodiment, the sample entity hint vector determining unit includes:

[0280] A second correlation determination unit is configured to determine the correlation between the sample visual encoding vector and each sample entity encoding vector in the sample text encoding vector;

[0281] The sample entity prompt vector determining subunit is configured to select a target sample entity encoding vector from the sample entity encoding vectors based on the sample correlation degree to form a sample entity prompt vector.

[0282] In an exemplary embodiment, the sample entity prompt vector determining subunit includes:

[0283] A second key entity determining unit is configured to determine a sample entity encoding vector corresponding to the maximum sample correlation degree to obtain a key sample entity encoding vector;

[0284] A second dependency determination unit is configured to determine a sample dependency between the key sample entity code vector and each remaining sample entity code vector; the remaining sample entity code vector refers to a sample entity code vector other than the key sample entity code vector;

[0285] The second determining unit is configured to determine a target remaining sample entity coding vector based on the sample dependency; the sample dependency corresponding to the target remaining sample entity coding vector is greater than a preset dependency threshold;

[0286] The second construction subunit is configured to execute concatenation of the key sample entity encoding vector and the target remaining sample entity encoding vector as a target sample entity encoding vector to obtain a sample entity prompt vector.

[0287] In an exemplary embodiment, the second dependency level determining unit includes:

[0288] A second initialization unit is configured to execute the key sample entity encoding vector as the initial hidden state of a bidirectional long short-term memory network;

[0289] A second state vector determining unit is configured to input the remaining sample entity encoding vector into the bidirectional long short-term memory network to obtain a state vector corresponding to the remaining sample entity encoding vector;

[0290] The second normalization unit is configured to perform normalization processing based on the state vector corresponding to the remaining sample entity coding vector to obtain a sample normalization result corresponding to the remaining sample entity coding vector; the sample normalization result represents the degree of sample dependence between the key sample entity coding vector and the remaining sample entity coding vector.

[0291] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0292] In an exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing processor-executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, it implements any data processing method provided in the embodiments of the present disclosure.

[0293] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Figure 8 is a structural block diagram of an electronic device for data processing according to an exemplary embodiment. Figure 8 As shown, the server 800 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 810 (the processor 810 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 830 for storing data, and one or more storage media 820 (such as one or more mass storage devices) for storing application programs 823 or data 822. Among them, the memory 830 and the storage medium 820 can be temporary storage or permanent storage. The program stored in the storage medium 820 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the central processing unit 810 can be configured to communicate with the storage medium 820 to execute a series of instruction operations in the storage medium 820 on the server 800. The server 800 may also include one or more power supplies 860, one or more wired or wireless network interfaces 850, one or more input and output interfaces 840, and / or one or more operating systems 821, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0294] The input / output interface 840 can be used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communication provider of the server 800. In one embodiment, the input / output interface 840 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one embodiment, the input / output interface 840 can be a radio frequency (RF) module for wirelessly communicating with the Internet.

[0295] It can be understood by those skilled in the art that Figure 8 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 8 More or fewer components than shown, or with Figure 8 Different configurations shown.

[0296] In an exemplary embodiment, a computer-readable storage medium including instructions is further provided, such as a memory 830 including instructions, which can be executed by the processor 810 of the apparatus 800 to perform the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0297] In an exemplary embodiment, a computer program product is further provided, including a computer program / instruction, which implements any one of the data processing methods provided in the embodiments of the present disclosure when executed by a processor.

[0298] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0299] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: include: Acquire a business image to be processed and a business text associated with the business image to be processed; the business text includes a named entity; Obtaining a visual cue vector based on the extracted visual features of the business image to be processed; Encoding the visual prompt vector and the business text to obtain an encoding vector; the encoding vector includes a visual encoding vector corresponding to the visual prompt vector and a text encoding vector corresponding to the business text; Determining an entity prompt vector based on the entity encoding vector corresponding to each of the named entities in the visual encoding vector and the text encoding vector; The encoding vector is decoded based on the visual prompt vector and the entity prompt vector to obtain a description text corresponding to the business image to be processed.

2. The method according to claim 1, characterized in that The determining of the entity prompt vector based on the entity encoding vector corresponding to each of the named entities in the visual encoding vector and the text encoding vector includes: Determining a correlation between the visual encoding vector and each entity encoding vector in the text encoding vector; A target entity encoding vector is selected from the entity encoding vectors based on the correlation degree to form an entity prompt vector.

3. The method according to claim 2, characterized in that The step of selecting a target entity encoding vector from the entity encoding vectors based on the correlation to form an entity prompt vector includes: Determine the entity encoding vector corresponding to the maximum correlation degree to obtain the key entity encoding vector; Determining a degree of dependence between the key entity code vector and each remaining entity code vector; the remaining entity code vector refers to an entity code vector other than the key entity code vector; Determining a target remaining entity encoding vector based on the dependency degree; wherein the dependency degree corresponding to the target remaining entity encoding vector is greater than a preset dependency degree threshold; The key entity encoding vector and the target remaining entity encoding vector are concatenated as a target entity encoding vector to obtain an entity prompt vector.

4. The method according to claim 3, characterized in that Determining the degree of dependence between the key entity encoding vector and each remaining entity encoding vector includes: Using the key entity encoding vector as the initial hidden state of a bidirectional long short-term memory network; Inputting the remaining entity encoding vector into the bidirectional long short-term memory network to obtain a state vector corresponding to the remaining entity encoding vector; Normalization processing is performed based on the state vector corresponding to the remaining entity coding vector to obtain a normalized result corresponding to the remaining entity coding vector; the normalized result represents the degree of dependence between the key entity coding vector and the remaining entity coding vector.

5. The method according to claim 1, characterized in that The decoding process of the encoding vector based on the visual prompt vector and the entity prompt vector to obtain the description text corresponding to the service image to be processed includes: concatenating the visual cue vector and the entity cue vector to obtain a multimodal cue vector; Autoregressive decoding is performed based on the multimodal prompt vector and the encoding vector to obtain a description text corresponding to the service image to be processed.

6. The method according to claim 1, characterized in that The step of obtaining a visual prompt vector based on the extracted visual features of the business image to be processed includes: Inputting the business image to be processed into a vision-language pre-training model for feature extraction to obtain extracted visual features; Mapping the visual features to the input space of the target language model based on a multi-layer perception network to obtain a visual cue vector; The target language model is used to perform the encoding process and the decoding process.

7. The method according to any one of claims 1 to 6, characterized in that The method is implemented based on a data processing model, and further comprises the step of training the data processing model: Acquire a sample business image text pair and a corresponding reference description text; the sample business image text pair includes a sample business image and a sample business text associated with the sample business image, wherein the sample business text includes a named entity; Extracting visual features of the sample business image based on the vision-language pre-trained model, and mapping the visual features of the sample business image to the input space of the pre-trained language model based on an initial multi-layer perception network to obtain a sample visual cue vector; Inputting the sample visual cue vector and the sample business text into an encoder of a pre-trained language model for encoding processing to obtain a sample encoding vector; the sample encoding vector includes a sample visual encoding vector corresponding to the sample visual cue vector and a sample text encoding vector corresponding to the sample business text; Determining a sample entity prompt vector based on the sample visual encoding vector and the sample entity encoding vector corresponding to each of the named entities in the sample text encoding vector; The sample visual prompt vector and the sample entity prompt vector are concatenated to obtain a sample multimodal prompt vector; the sample multimodal prompt vector and the sample encoding vector are input into the decoder of the pre-trained language model for decoding processing to obtain a predicted description text; The model parameters are adjusted based on the difference between the predicted description text and the reference description text until the training is terminated when a preset training termination condition is reached, thereby obtaining the data processing model.

8. The method according to claim 7, characterized in that The adjusting of model parameters based on the difference between the predicted description text and the reference description text includes: Determining a loss value based on a difference between the predicted description text and the reference description text; The model parameters of the vision-language pre-training model are fixed, and the parameters of the initial multi-layer perception network and the pre-training language model are adjusted based on the loss value.

9. The method according to claim 7, characterized in that The determining of the sample entity prompt vector based on the sample entity encoding vector corresponding to each of the named entities in the sample visual encoding vector and the sample text encoding vector includes: Determining a sample correlation degree between the sample visual encoding vector and each sample entity encoding vector in the sample text encoding vector; A target sample entity encoding vector is selected from the sample entity encoding vectors based on the sample relevance to form a sample entity prompt vector.

10. The method according to claim 9, characterized in that The step of selecting a target sample entity encoding vector from the sample entity encoding vectors based on the sample correlation to form a sample entity prompt vector includes: Determine the sample entity encoding vector corresponding to the maximum sample correlation degree to obtain the key sample entity encoding vector; Determining a sample dependency between the key sample entity coding vector and each remaining sample entity coding vector; the remaining sample entity coding vector refers to a sample entity coding vector other than the key sample entity coding vector; Determining a target remaining sample entity coding vector based on the sample dependency; the sample dependency corresponding to the target remaining sample entity coding vector is greater than a preset dependency threshold; The key sample entity encoding vector and the target remaining sample entity encoding vector are concatenated as a target sample entity encoding vector to obtain a sample entity prompt vector.

11. The method according to claim 10, characterized in that The determining of the sample dependency between the key sample entity encoding vector and each remaining sample entity encoding vector includes: Using the key sample entity encoding vector as the initial hidden state of a bidirectional long short-term memory network; Inputting the remaining sample entity encoding vector into the bidirectional long short-term memory network to obtain a state vector corresponding to the remaining sample entity encoding vector; Normalization processing is performed based on the state vector corresponding to the remaining sample entity coding vector to obtain a sample normalization result corresponding to the remaining sample entity coding vector; the sample normalization result represents the degree of sample dependence between the key sample entity coding vector and the remaining sample entity coding vector.

12. A data processing device, characterized in that: include: A data acquisition unit is configured to acquire a business image to be processed and business text associated with the business image to be processed; the business text includes a named entity; a visual cue vector determining unit, configured to obtain a visual cue vector based on the extracted visual features of the service image to be processed; an encoding unit configured to perform encoding processing on the visual cue vector and the business text to obtain an encoding vector; the encoding vector includes a visual encoding vector corresponding to the visual cue vector and a text encoding vector corresponding to the business text; an entity hint vector determining unit, configured to determine an entity hint vector based on the entity encoding vector corresponding to each of the named entities in the visual encoding vector and the text encoding vector; The decoding unit is configured to perform decoding processing on the encoding vector based on the visual prompt vector and the entity prompt vector to obtain a description text corresponding to the business image to be processed.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data processing method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

Images