Image text processing method and device, readable medium and electronic equipment

Abstract

The present disclosure relates to an image text processing method and device, readable medium and electronic equipment, and relates to the technical field of electronic information processing. The method comprises: acquiring a target image and a question text corresponding to the target image, performing feature extraction on the target image to obtain image features, performing feature extraction on the question text according to the image features to obtain visual text features, determining an answer text corresponding to the question text according to the visual text features, the answer text being used to describe a target object in the target image, determining identification information according to the visual text features and the image features, the identification information being used to identify a region where the target object is located in the target image. The present disclosure uses image features to assist in feature extraction of the question text, can effectively mine key information in the question text, and uses visual text features to assist in identification of the image features, can accurately identify the target object in the target image, and realizes high-accuracy and high-efficiency visual language question and answer landing.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of electronic information processing, and in particular, to an image text processing method and device, readable medium and electronic equipment. BACKGROUND

[0002] With the continuous development of artificial intelligence related technologies, image recognition technologies and natural language processing technologies have been widely applied, effectively improving people's quality of life. For example, visual language question answering (English: Visual Question Answering, abbreviated: VQA) can effectively help the visually impaired understand visual information. Users can capture visual content through terminal devices, and then ask questions about the visual content in the terminal device through language, and the terminal device answers in natural language by recognizing the visual content. However, only giving answers in natural language provides very limited assistance, and without visual clues as a guide, users cannot zoom in on details or judge whether the answer is reliable based on visual clues, limiting the application of visual language question answering. SUMMARY

[0003] This summary is provided to introduce a selection of concepts, which will be described in more detail below in the detailed description section. This summary is not intended to identify key or essential features of the claimed technology, nor is it intended to limit the scope of the claimed technology.

[0004] In a first aspect, the present disclosure provides an image text processing method, comprising:

[0005] obtaining a target image and a question text corresponding to the target image;

[0006] performing feature extraction on the target image to obtain image features;

[0007] performing feature extraction on the question text according to the image features to obtain visual text features;

[0008] determining an answer text corresponding to the question text according to the visual text features, the answer text being used to describe a target object in the target image;

[0009] determining identification information according to the visual text features and the image features, the identification information being used to identify a region in the target image where the target object is located.

[0010] In a second aspect, the present disclosure provides an image text processing device, comprising:

[0011] obtain a target image and a question text corresponding to the target image;

[0012] extract a feature of the target image to obtain an image feature;

[0013] extract a feature of the question text according to the image feature to obtain a visual text feature;

[0014] determine an answer text corresponding to the question text according to the visual text feature, the answer text being used to describe a target object in the target image;

[0015] determine identification information according to the visual text feature and the image feature, the identification information being used to identify a region in which the target object is located in the target image.

[0016] In a third aspect, the present disclosure provides a computer readable medium having a computer program stored thereon, the program being executed by a processing device to implement the steps of the method of the first aspect of the present disclosure.

[0017] In a fourth aspect, the present disclosure provides an electronic device, comprising:

[0018] a storage device having a computer program stored thereon;

[0019] a processing device configured to execute the computer program in the storage device to implement the steps of the method of the first aspect of the present disclosure.

[0020] According to the above technical solution, the present disclosure first obtains a target image and a question text corresponding to the target image, then extracts a feature of the target image to obtain an image feature, and extracts a feature of the question text according to the image feature to obtain a visual text feature. Then, according to the visual text feature, an answer text corresponding to the question text is determined, the answer text being used to describe a target object in the target image. Finally, according to the visual text feature and the image feature, identification information capable of identifying a region in which the target object is located in the target image is determined. The present disclosure utilizes the image feature to assist in the feature extraction of the question text, which can effectively mine key information in the question text. The present disclosure also utilizes the visual text feature to assist in the recognition of the image feature, which can accurately recognize the target object in the target image, and realizes high-accuracy, high-efficiency, and end-to-end visual language question answering landing.

[0021] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings. The same or similar components have the same or similar reference numbers regardless of the figure number. It should be understood that the drawings are not necessarily to scale, with emphasis instead being placed upon illustrating the principles of the embodiments of the present disclosure. In the drawings:

[0023] Figure 1 is a flowchart of an image text processing method according to an exemplary embodiment;

[0024] Figure 2 is a schematic diagram of a processing model according to an exemplary embodiment;

[0025] Figure 3 is a flowchart of another image text processing method according to an exemplary embodiment;

[0026] Figure 4 is a flowchart of training a visual encoder, a visual text encoder, a text decoder, and a text visual decoder according to an exemplary embodiment;

[0027] Figure 5 is a flowchart of another training a visual encoder, a visual text encoder, a text decoder, and a text visual decoder according to an exemplary embodiment;

[0028] Figure 6 is a block diagram of an image text processing apparatus according to an exemplary embodiment;

[0029] Figure 7 is a block diagram of another image text processing apparatus according to an exemplary embodiment;

[0030] Figure 8 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While several embodiments of the present disclosure have been illustrated and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the present disclosure. It is to be understood that the drawings and descriptions are illustrative and explanatory only, and are not intended to be limiting to the present disclosure.

[0032] It should be understood that various steps in the method embodiments of the present disclosure can be performed in a different order and / or concurrently. Furthermore, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0033] The term "include" and variations thereof as used herein mean "to include, without limitation." The term "based on" means "based at least in part on." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The term "some embodiments" means "at least some embodiments." Related terms shall be construed accordingly.

[0034] It should be noted that the terms "first", "second", etc. mentioned in the disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0035] It should be noted that the modification of "one" or "multiple" mentioned in the disclosure is illustrative and not limiting, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0036] The names of the messages or information exchanged between the plurality of devices in the embodiments of the disclosure are only for illustrative purposes, and are not used to limit the scope of the messages or information.

[0037] It can be understood that before using the technical solutions disclosed in the embodiments of the disclosure, the type, use range, use scenario, etc. of the personal information involved in the disclosure should be informed to the user and the authorization of the user should be obtained in a proper manner according to relevant laws and regulations.

[0038] For example, in response to receiving the active request of the user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the electronic device, application program, server or storage medium, etc. software or hardware performing the operation of the technical solutions of the disclosure according to the prompt information.

[0039] As an optional but non-limiting implementation manner, in response to receiving the active request of the user, the manner of sending prompt information to the user may, for example, be a pop-up window manner, and the prompt information may, for example, be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0040] It can be understood that the above notification and user authorization process is only illustrative, and does not limit the implementation manner of the disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation manner of the disclosure.

[0041] Meanwhile, it can be understood that the data (including but not limited to the data itself, the acquisition or use of the data) involved in the technical solution should comply with the requirements of the corresponding laws, regulations and relevant provisions.

[0042] Figure 1 is a flow chart of an image text processing method according to an exemplary embodiment, as shown in Figure 1 , the method can include:

[0043] Step 101, obtaining a target image and a question text corresponding to the target image.

[0044] For example, the execution subject of the present disclosure can be a terminal device, which can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc., which are not specifically limited by the present disclosure.

[0045] When a user needs to understand visual information with the help of a terminal device, the terminal device can be used to obtain a target image, which can be an image collected by the user using an image collection device (such as a camera) provided on the terminal device, an image selected by the user in the local storage of the terminal device, or an image obtained by the user using the terminal device on the network. Then, the user can ask a question about the target image, and the user can enter the question in text mode using the terminal device, and the terminal device obtains the question text. The user can also enter the question in voice mode using the terminal device, and the terminal device can recognize the voice and convert it into text to obtain the question text after obtaining the voice. For example, the target image is a road image, and the corresponding question text can be "What color is the traffic light?".

[0046] Step 102, feature extraction is performed on the target image to obtain image features.

[0047] Step 103, feature extraction is performed on the question text according to the image features to obtain visual text features.

[0048] The image feature can be understood as a feature vector or a feature map (English: Feature Map) for representing the target image. Specifically, the ResNet network or the Encoder in the Transformer can be used to extract the image feature, and the present disclosure does not make specific limitations thereto. After obtaining the image feature, the image feature can be taken as a reference to extract the visual text feature of the question text based on the image feature. In this way, the visual text feature can include information of two modalities (text modality and image modality). It can also be understood that the image feature and the question text are fused into the question text with image modality information, thereby extracting the visual text feature of the text modality. It can also be understood that the image feature is taken as an explanation to assist in mining the question text, and the visual text feature is obtained.

[0049] In step 104, the answer text corresponding to the question text is determined according to the visual text feature, and the answer text is used to describe the target object in the target image.

[0050] In step 105, the identification information is determined according to the visual text feature and the image feature, and the identification information is used to identify the region where the target object is located in the target image.

[0051] The visual text feature can be decoded to obtain the answer text corresponding to the question text. Specifically, the Decoder in the Bert (English: Bidirectional Encoder Representations from Transformers) can be used to decode the visual text feature. The answer text is used to describe the target object in the target image, and the answer text corresponds to the question text, including the answer to the question described by the question text, that is, the answer text is used to answer the question described by the question text. For example, the target image is a road image, the corresponding question text is “What color is the traffic light?”, and the target object is the traffic light, and the answer text can be “green”. For another example, the target image is a medicine image, the corresponding question text is “How many pills are in the medicine box?”, and the target object is the pill, and the answer text can be “12”. After determining the answer text, the answer text can be directly displayed on the display screen of the terminal device, or the answer text can be converted into audio and played through the playing device (for example, a loudspeaker) of the terminal device, and the present disclosure does not make specific limitations thereto.

[0052] To further assist the user, the visual text feature can be used as a reference to decode the image feature to obtain identification information that can identify the region of the target object in the target image. It can also be understood that the visual text feature is mapped to the image space and fused with the image feature to obtain an image feature with text modal information, thereby decoding the identification information. It can also be understood that the visual text feature is used as evidence, and the image feature focuses on the region related to the evidence, thereby capturing the region of the target object in the target image. The form of the identification information can be a mask image (English: mask), which is superimposed on the target image to identify the region of the target object in the target image. The form of the identification information can also be a coordinate range used to identify the position of the target object in the target image. The identification information can identify the target object, so the user can zoom in on the region of the target object to view the details of the target object, or can determine whether the answer text is reliable based on the target object in the target image. Since the image feature is used to assist in extracting the features of the question text, the key information in the question text can be effectively mined, and the visual text feature is used to assist in identifying the image feature, which can accurately identify the target object in the target image, realize visual language question answer grounding (English: Visual Question Answer Grounding, abbreviated: VQAG), that is, while answering the question, the evidence supporting the answer is given, which improves the accuracy and efficiency of visual language question answer.

[0053] In summary, the present disclosure first acquires a target image and a question text corresponding to the target image, then extracts features from the target image to obtain image features, and extracts features from the question text according to the image features to obtain visual text features. Then, according to the visual text features, the answer text corresponding to the question text is determined, which describes the target object in the target image, and finally, according to the visual text features and the image features, identification information that can identify the region of the target object in the target image is determined. The present disclosure uses image features to assist in extracting features of the question text, which can effectively mine key information in the question text, and uses visual text features to assist in identifying image features, which can accurately identify the target object in the target image, and realizes high-accuracy, high-efficiency, end-to-end visual language question answer grounding.

[0054] In an application scenario, a processing model can be pre-trained to implement the image text processing method provided by the present disclosure. The structure of the processing model can include a visual encoder (English: Visual Encoder, abbreviated as VE), a visual text encoder (English: Visual-based Linguistic Encoder, abbreviated as VLE), a text decoder (English: Linguistic Decoder, abbreviated as LD), and a text visual decoder (English: Linguistic-based Visual Decoder, abbreviated as LVD). As shown in Figure 2 , the input of the visual encoder and the input of the visual text encoder serve as the input of the processing model, the output of the visual encoder serves as the input of the visual text encoder and the text visual decoder, the output of the visual text encoder serves as the input of the text visual decoder and the text decoder, and the output of the text visual decoder and the output of the text decoder serve as the output of the processing model.

[0055] Correspondingly, step 102 can include:

[0056] The target image is input into the visual encoder for encoding to obtain image features.

[0057] Step 103 can include:

[0058] The image features and the question text are input into the visual text encoder for encoding to obtain visual text features.

[0059] As shown in Figure 2 , the structure of the VE may, for example, be an Encoder in a Transformer, which includes a plurality of encoding layers connected in sequence (only one encoding layer is shown in the figure, and the plurality of encoding layers are not shown), each encoding layer including a self-attention sublayer (denoted as Self-Attention), a feed-forward unit (denoted as Feed Forward), and a residual unit. The target image is input into the self-attention sublayer as the input of the self-attention sublayer, and the target image is used as the query vector (English: Query, abbreviated as Q), the key vector (English: Key, abbreviated as K), and the value vector (English: Value, abbreviated as V) of the self-attention mechanism. The target image is encoded by using the self-attention mechanism, and then the output of the self-attention sublayer is superimposed on the target image by using the residual unit. The superimposition result is input into the feed-forward unit, and then the output of the feed-forward unit is superimposed on the superimposition result by using the residual unit to obtain the image features.

[0060] The structure of the VLE is as shown in Figure 2As shown, the LD includes a self-attention layer (denoted as: Self-Attention), a cross-attention layer (denoted as: Cross-Attention), a feed forward unit (denoted as: Feed Forward), and a residual unit. The question text can be input into the self-attention layer, the output of the self-attention layer is superimposed with the question text by using the residual unit to obtain a first superimposed result, and the first superimposed result is input into the cross-attention layer together with the image feature. The first superimposed result is superimposed with the output of the cross-attention layer by using the residual unit to obtain a second superimposed result, and finally, the second superimposed result is input into the feed forward unit. The output of the feed forward unit is superimposed with the second superimposed result by using the residual unit to obtain the visual text feature.

[0061] Step 104 can include:

[0062] The visual text feature is input into the text decoder for decoding to obtain the answer text.

[0063] Step 105 can include:

[0064] The visual text feature and the image feature are input into the text-visual decoder for decoding to obtain the identification information.

[0065] For example, the structure of the LD is as shown in Figure 2 As shown, for example, the decoder in Bert can be used, which includes a self-attention layer (denoted as: Self-Attention), a feed forward unit (denoted as: Feed Forward), a residual unit, and an MLP (English: Multilayer Perceptron, Chinese: Multilayer Perceptron). The visual text feature can be input into the self-attention layer, and the visual text feature is used as the query vector, the key vector, and the value vector of the self-attention mechanism at the same time. The visual text feature is decoded by using the self-attention mechanism, and then the output of the self-attention layer is superimposed with the visual text feature by using the residual unit. The superimposed result is input into the feed forward unit, the output of the feed forward unit is superimposed with the superimposed result by using the residual unit, and finally, the answer text is obtained through the MLP.

[0066] As shown in Figure 2As shown, the structure of the LVD includes a cross-attention layer (denoted as Cross-Attention), a feed forward unit (denoted as Feed Forward), a residual unit, and a convolution segmentation layer (denoted as Conv). The visual text feature and the image feature can be input into the cross-attention layer, and then the output of the cross-attention layer is superimposed with the image feature by using the residual unit. The superimposed result is input into the feed forward unit, and then the output of the feed forward unit is superimposed with the superimposed result by using the residual unit. Finally, the identification information is obtained through the convolution segmentation layer. In this way, the end-to-end visual language question and answer landing is realized through a unified framework, and the effect of giving evidence supporting the answer while answering the question is achieved.

[0067] Figure 3 is a flowchart of another image text processing method according to an example embodiment, as shown in Figure 3 As shown, step 103 can include:

[0068] Step 1031, input the question text into the self-attention layer in the visual text encoder, encode the question text by using the self-attention mechanism, and obtain the text feature.

[0069] Step 1032, input the text feature and the image feature into the cross-attention layer in the visual text encoder, encode the text feature and the image feature by using the cross-attention mechanism, and obtain the visual text feature.

[0070] For example, the question text can be input into the self-attention layer in the visual text encoder, and the question text is used as the query vector, the key vector, and the value vector of the self-attention mechanism at the same time. The question text is encoded by using the self-attention mechanism, and the text feature is obtained. The text feature only contains the information in the question text. On this basis, the text feature and the image feature can be input into the cross-attention layer in the visual text encoder. The text feature is used as the query vector of the cross-attention mechanism, and the image feature is used as the key vector and the value vector of the cross-attention mechanism. The text feature and the image feature are fused by using the cross-attention mechanism, and the visual text feature containing the information of two modalities (text modality and image modality) is obtained.

[0071] Specifically, the text feature and the question text can be superimposed by using a residual unit to obtain a first superimposition result, and the first superimposition result and the image feature are taken as inputs of the cross-attention layer. The first superimposition result is taken as a query vector of the cross-attention mechanism, and the image feature is taken as a key vector and a value vector of the cross-attention mechanism, and the output of the cross-attention layer is obtained by using the cross-attention mechanism. Then, the first superimposition result and the output of the cross-attention layer are superimposed by using a residual unit to obtain a second superimposition result. Finally, the second superimposition result is input into the feedforward unit, and the output of the feedforward unit and the second superimposition result are superimposed by using a residual unit to obtain the visual-text feature.

[0072] In another implementation, step 105 can include:

[0073] The visual-text feature and the image feature are input into the cross-attention layer of the text-visual decoder, the image feature is taken as a query vector of the cross-attention mechanism, and the visual-text feature is taken as a key vector and a value vector of the cross-attention mechanism, and the identification information is obtained by using the cross-attention mechanism.

[0074] Specifically, the visual-text feature and the image feature are input into the cross-attention layer of the text-visual decoder, the image feature is taken as a query vector of the cross-attention mechanism, and the visual-text feature is taken as a key vector and a value vector of the cross-attention mechanism, and the identification information is obtained by using the cross-attention mechanism. Specifically, the output of the cross-attention layer and the image feature can be superimposed by using a residual unit, the superimposition result is input into the feedforward unit, the output of the feedforward unit and the superimposition result are superimposed by using a residual unit, and finally the identification information is obtained through the convolution segmentation layer.

[0075] Figure 4 is a flowchart of training a visual encoder, a visual-text encoder, a text decoder and a text-visual decoder according to an example embodiment, as shown in Figure 4 As shown in the figure, the visual encoder, the visual-text encoder, the text decoder and the text-visual decoder are trained in the following way:

[0076] Step A, obtaining a sample input set and a sample output set, the sample input set including a plurality of sample inputs, the sample input including a training image and a training question text corresponding to the training image, and the sample output set including a sample output corresponding to each sample input, each sample output including a training answer text corresponding to the training question text and training identification information, the training answer text being used to describe a training object in the training image, and the training identification information being used to identify a region where the training object is located in the training image.

[0077] For example, before training the visual encoder, the visual-text encoder, the text decoder and the text-visual decoder, a sample input set and a sample output set for training are needed, the sample input set includes a plurality of sample inputs, and the sample output set includes a sample output corresponding to each sample input. The sample input can include a training image and a training question text corresponding to the training image, and the training image and the training question text can be preprocessed. For example, random augmentation can be performed on the training image: random scaling, cropping, random color jitter (brightness, contrast, etc.). The training question text that is too long can also be truncated, and the training question text that is too short can be zero-padded. Correspondingly, each sample output includes a training answer text corresponding to the training question text and training identification information, wherein the training answer text is used to describe a training object in the training image, and the training identification information is used to identify a region where the training object is located in the training image.

[0078] Step B, for each sample input, input the training image in the sample input into the visual encoder for encoding to obtain a training image feature.

[0079] Step C, input the training image feature and the training question text in the sample input into the visual-text encoder for encoding to obtain a training visual-text feature.

[0080] Step D, input the training visual-text feature into the text decoder for decoding to obtain a predicted answer text.

[0081] Step E, input the training visual-text feature and the training image feature into the text-visual decoder for decoding to obtain predicted identification information.

[0082] Step F, according to the predicted answer text, the predicted identification information and the sample output corresponding to the sample input, training the visual encoder, the visual-text encoder, the text decoder and the text-visual decoder.

[0083] For example, the manner of obtaining the predicted answer text and the predicted identification information is the same as the manner of obtaining the answer text and the identification information, which will not be described here. Finally, the visual encoder, the visual text encoder, the text decoder, and the text visual decoder are trained according to the predicted answer text, the predicted identification information, and the sample output corresponding to the sample input. For example, the text loss can be determined according to the predicted answer text and the training answer text in the corresponding sample output, the identification loss can be determined according to the predicted identification information and the training identification information in the corresponding sample output, the total loss can be determined according to the text loss and the identification loss, and finally the parameters of the neurons in the visual encoder, the visual text encoder, the text decoder, and the text visual decoder, such as the weights and biases of the neurons, are corrected by using the back propagation algorithm to reduce the total loss. The above steps are repeated until the total loss meets the preset condition, for example, the total loss is less than the preset loss threshold or the total loss converges, to achieve the purpose of training the visual encoder, the visual text encoder, the text decoder, and the text visual decoder.

[0084] Specifically, the visual encoder, the visual text encoder, and the text decoder can be pre-trained using a pre-training data set, and then the pre-trained visual encoder, the visual text encoder, and the text decoder are jointly trained with the text visual decoder, which can improve the training efficiency and effectively reduce the training calculation amount. The visual encoder is used for feature extraction of images, so a large amount of images can be obtained from the network as the pre-training data set to pre-train the visual encoder. Similarly, a large amount of text can be obtained from the network as the pre-training data set to pre-train the text decoder. A large amount of text and corresponding description text can be obtained from the network as the pre-training data set to pre-train the visual text encoder.

[0085] It should be noted that in the above training process, the initial learning rate can be set to 5e-5, the optimizer can be selected as AdamW, the batch size can be set to 16, the number of rounds can be set to 30, the size of the training image can be 512x512, and the length of the training question text can be 20.

[0086] Figure 5 is another flowchart of training the visual encoder, the visual text encoder, the text decoder, and the text visual decoder according to an example embodiment, as shown in Figure 5 As shown in the figure, the manner of training the visual encoder, the visual text encoder, the text decoder, and the text visual decoder also includes:

[0087] Step G, training the visual encoder, the visual text encoder, the text decoder and the text visual decoder for multiple rounds using the sample input set, and storing the parameter set of each round of the visual encoder, the visual text encoder, the text decoder and the text visual decoder as a set of parameter copy sets.

[0088] Step H, determining an inference parameter set according to the multiple sets of parameter copy sets.

[0089] Step I, updating the visual encoder, the visual text encoder, the text decoder and the text visual decoder according to the inference parameter set.

[0090] For example, when training the visual encoder, the visual text encoder, the text decoder and the text visual decoder, multiple rounds of training are performed, and one round of training can be understood as completing steps B to F for each sample input in the sample input set. After completing one round of training, the parameter set (including the weights and bias of the neurons) of the visual encoder, the visual text encoder, the text decoder and the text visual decoder obtained in this round can be stored as a set of parameter copy sets. In order to avoid local optimization of the training result, after completing multiple rounds (for example, 30 rounds) of training, an inference parameter set can be determined according to the multiple sets of parameter copy sets corresponding to the multiple rounds. Specifically, the multiple sets of parameter copy sets can be smoothed, and the obtained result can be used as the inference parameter set. The smoothing can be an exponential moving average or other moving average processing, which is not limited in the present disclosure. Finally, the visual encoder, the visual text encoder, the text decoder and the text visual decoder can be updated according to the inference parameter set to complete the training process. In this way, the visual encoder, the visual text encoder, the text decoder and the text visual decoder can be more robust and more universal.

[0091] In summary, the present disclosure first acquires a target image and a problem text corresponding to the target image, then performs feature extraction on the target image to obtain image features, and performs feature extraction on the problem text according to the image features to obtain visual text features. Then, according to the visual text features, an answer text corresponding to the problem text is determined, which is used to describe a target object in the target image. Finally, according to the visual text features and the image features, region identification information capable of identifying the target object in the target image is determined. The present disclosure uses image features to assist in feature extraction of the problem text, which can effectively mine key information in the problem text, and uses visual text features to assist in identification of image features, which can accurately identify the target object in the target image, and realizes high-accuracy, high-efficiency and end-to-end visual language question and answer landing.

[0092] Figure 6 is a block diagram of an image text processing device according to an exemplary embodiment, as shown inFigure 6 As shown in the figure, the apparatus 200 can include:

[0093] The acquisition module 201 is configured to acquire a target image and a question text corresponding to the target image.

[0094] The first extraction module 202 is configured to perform feature extraction on the target image to obtain image features.

[0095] The second extraction module 203 is configured to perform feature extraction on the question text according to the image features to obtain visual text features.

[0096] The first determination module 204 is configured to determine an answer text corresponding to the question text according to the visual text features, the answer text being used to describe a target object in the target image.

[0097] The second determination module 205 is configured to determine identification information according to the visual text features and the image features, the identification information being used to identify a region where the target object is located in the target image.

[0098] In an implementation manner, the first extraction module 202 can be configured to:

[0099] input the target image into a visual encoder to perform encoding to obtain the image features.

[0100] The second extraction module 203 can be configured to:

[0101] input the image features and the question text into a visual text encoder to perform encoding to obtain the visual text features.

[0102] The first determination module 204 can be configured to:

[0103] input the visual text features into a text decoder to perform decoding to obtain the answer text.

[0104] The second determination module 205 can be configured to:

[0105] input the visual text features and the image features into a text-visual decoder to perform decoding to obtain the identification information.

[0106] Figure 7 is a block diagram of another image-text processing apparatus according to an example embodiment, as shown in the figure, the second extraction module 203 can include: Figure 7

[0107] The self-attention extraction sub-module 2031 is configured to input the question text into a self-attention layer in the visual text encoder, encode the question text by using a self-attention mechanism to obtain text features.

[0108] ​The cross-attention extraction submodule 2032 is configured to input the text feature and the image feature into a cross-attention layer in the visual-textual encoder, encode the text feature and the image feature by using the cross-attention mechanism, and obtain a visual-textual feature.

[0109] In an implementation manner, the cross-attention extraction submodule 2032 can be configured to:

[0110] input the text feature as a query vector of the cross-attention mechanism and the image feature as a key vector and a value vector of the cross-attention mechanism, so as to obtain the visual-textual feature by using the cross-attention mechanism.

[0111] In another implementation manner, the second determination module 205 can be configured to:

[0112] input the visual-textual feature and the image feature into a cross-attention layer of the textual-visual decoder, input the image feature as a query vector of the cross-attention mechanism and the visual-textual feature as a key vector and a value vector of the cross-attention mechanism, and obtain the identification information by using the cross-attention mechanism.

[0113] In an implementation manner, the visual encoder, the visual-textual encoder, the textual decoder, and the textual-visual decoder are trained in the following manner:

[0114] Step A, obtaining a sample input set and a sample output set, the sample input set including a plurality of sample inputs, each sample input including a training image and a training question text corresponding to the training image, and the sample output set including a sample output corresponding to each sample input, each sample output including a training answer text corresponding to the training question text and training identification information, the training answer text being used to describe a training object in the training image, and the training identification information being used to identify a region where the training object is located in the training image.

[0115] Step B, for each sample input, inputting the training image in the sample input into the visual encoder to obtain a training image feature.

[0116] Step C, inputting the training image feature and the training question text in the sample input into the visual-textual encoder to obtain a training visual-textual feature.

[0117] Step D, inputting the training visual-textual feature into the textual decoder to obtain a predicted answer text.

[0118] Step E, inputting the training visual-textual feature and the training image feature into the textual-visual decoder to obtain predicted identification information.

[0119] Step F, training the visual encoder, the visual text encoder, the text decoder and the text visual decoder according to the predicted answer text, the predicted identification information and the sample output corresponding to the sample input.

[0120] In another implementation, the manner of training the visual encoder, the visual text encoder, the text decoder and the text visual decoder further comprises:

[0121] Step G, training the visual encoder, the visual text encoder, the text decoder and the text visual decoder for multiple rounds using the set of sample inputs, and taking the parameter set of the visual encoder, the visual text encoder, the text decoder and the text visual decoder of each round as a set of parameter replicas.

[0122] Step H, determining an inference parameter set according to the multiple sets of parameter replicas.

[0123] Step I, updating the visual encoder, the visual text encoder, the text decoder and the text visual decoder according to the inference parameter set.

[0124] As to the apparatus in the above-mentioned embodiments, the specific manners in which the respective modules perform operations have been described in detail in the embodiments related to the method, and thus will not be described in detail here.

[0125] To sum up, the present disclosure first acquires a target image and a problem text corresponding to the target image, then performs feature extraction on the target image to obtain image features, and performs feature extraction on the problem text according to the image features to obtain visual text features. Then, according to the visual text features, the answer text corresponding to the problem text is determined, which is used to describe the target object in the target image. Finally, according to the visual text features and the image features, the identification information capable of identifying the region where the target object in the target image is located is determined. The present disclosure uses image features to assist in feature extraction of the problem text, which can effectively mine key information in the problem text, and uses visual text features to assist in identification of image features, which can accurately identify the target object in the target image, and realizes high-accuracy, high-efficiency and end-to-end visual language question answering landing.

[0126] Reference will now be made to the following description Figure 8 which shows a structural schematic diagram of an electronic device (e.g., an execution subject of an embodiment of the present disclosure) 300 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 8The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0127] like Figure 8 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0128] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0129] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of embodiments of this disclosure.

[0130] It should be noted that the computer-readable medium described above in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagated data signal, in which a computer-readable program code is embodied. Such a propagated data signal can take many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to, wire, cable, RF (radio frequency), etc., or any suitable combination thereof.

[0131] In some embodiments, the terminal device, the server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication of any form or medium (e.g., a communication network). Examples of the communication network include a local area network ("LAN"), a wide area network ("WAN"), an internetwork (e.g., the Internet), and an end-to-end network (e.g., an ad hoc end-to-end network), as well as any currently known or future developed network.

[0132] The above computer-readable medium can be included in the above electronic device; or can exist separately without being assembled into the electronic device.

[0133] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: acquire a target image and problem text corresponding to the target image; perform feature extraction on the target image to obtain image features; perform feature extraction on the problem text according to the image features to obtain visual text features; determine answer text corresponding to the problem text according to the visual text features, the answer text being used to describe a target object in the target image; and determine identification information according to the visual text features and the image features, the identification information being used to identify a region in which the target object is located in the target image.

[0134] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0135] The flow and block diagrams in the drawings show architectural, functional, and operational representations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0136] The modules described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself, for example, the acquisition module can also be described as a "module for acquiring target images and question texts".

[0137] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that can be used include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0138] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media can include one or more lines of electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0139] According to one or more embodiments of the present disclosure, example 1 provides an image text processing method, comprising: acquiring a target image and question text corresponding to the target image; performing feature extraction on the target image to obtain image features; performing feature extraction on the question text according to the image features to obtain visual text features; determining answer text corresponding to the question text according to the visual text features, the answer text being used to describe a target object in the target image; and determining identification information according to the visual text features and the image features, the identification information being used to identify a region where the target object is located in the target image.

[0140] According to one or more embodiments of the present disclosure, example 2 provides the method of example 1, wherein the feature extraction on the target image to obtain image features comprises: inputting the target image into a visual encoder to obtain the image features; the feature extraction on the question text according to the image features to obtain visual text features comprises: inputting the image features and the question text into a visual text encoder to obtain the visual text features; the determination of the answer text corresponding to the question text according to the visual text features comprises: inputting the visual text features into a text decoder to obtain the answer text; and the determination of the identification information according to the visual text features and the image features comprises: inputting the visual text features and the image features into a text visual decoder to obtain the identification information.

[0141] According to one or more embodiments of the present disclosure, example 3 provides the method of example 2, wherein the inputting the image features and the question text into a visual text encoder to obtain the visual text features comprises: inputting the question text into a self-attention layer in the visual text encoder to encode the question text by using a self-attention mechanism to obtain text features; and inputting the text features and the image features into a cross-attention layer in the visual text encoder to encode the text features and the image features by using a cross-attention mechanism to obtain the visual text features.

[0142] According to one or more embodiments of the present disclosure, example 4 provides the method of example 3, wherein the encoding the text features and the image features by using a cross-attention mechanism to obtain the visual text features comprises: taking the text features as a query vector of the cross-attention mechanism, and taking the image features as a key vector and a value vector of the cross-attention mechanism to obtain the visual text features by using the cross-attention mechanism.

[0143] According to one or more embodiments of the present disclosure, example 5 provides the method of example 2, wherein the inputting the visual text features and the image features into a text visual decoder to obtain the identification information comprises: inputting the visual text features and the image features into a cross-attention layer of the text visual decoder, taking the image features as a query vector of a cross-attention mechanism, and taking the visual text features as a key vector and a value vector of the cross-attention mechanism to obtain the identification information by using the cross-attention mechanism.

[0144] According to one or more embodiments of the present disclosure, example 6 provides the method of example 1 to example 5, the visual encoder, the visual-text encoder, the text decoder and the text-visual decoder are trained by: obtaining a sample input set and the sample output set, the sample input set comprising a plurality of sample inputs, the sample inputs comprising training images and training question texts corresponding to the training images, the sample output set comprising sample outputs corresponding to each of the sample inputs, each of the sample outputs comprising training answer texts corresponding to the training question texts and training identification information, the training answer texts being used to describe training objects in the training images, and the training identification information being used to identify regions in the training images where the training objects are located; for each of the sample inputs, inputting the training image in the sample input into the visual encoder for encoding to obtain training image features; inputting the training image features and the training question text in the sample input into the visual-text encoder for encoding to obtain training visual-text features; inputting the training visual-text features into the text decoder for decoding to obtain predicted answer texts; inputting the training visual-text features and the training image features into the text-visual decoder for decoding to obtain predicted identification information; and training the visual encoder, the visual-text encoder, the text decoder and the text-visual decoder according to the predicted answer texts, the predicted identification information and the sample output corresponding to the sample input.

[0145] According to one or more embodiments of the present disclosure, example 7 provides the method of example 6, and the manner of training the visual encoder, the visual-text encoder, the text decoder and the text-visual decoder further comprises: performing multiple rounds of training on the visual encoder, the visual-text encoder, the text decoder and the text-visual decoder using the sample input set, and taking a parameter set of the visual encoder, the visual-text encoder, the text decoder and the text-visual decoder in each round as a set of parameter copies; determining an inference parameter set according to multiple sets of the parameter copies; and updating the visual encoder, the visual-text encoder, the text decoder and the text-visual decoder according to the inference parameter set.

[0146] According to one or more embodiments of the present disclosure, example 8 provides an image text processing apparatus, comprising: an acquisition module configured to acquire a target image and a question text corresponding to the target image; a first extraction module configured to perform feature extraction on the target image to obtain image features; a second extraction module configured to perform feature extraction on the question text according to the image features to obtain visual text features; a first determination module configured to determine an answer text corresponding to the question text according to the visual text features, the answer text being used to describe a target object in the target image; and a second determination module configured to determine identification information according to the visual text features and the image features, the identification information being used to identify a region in which the target object is located in the target image.

[0147] According to one or more embodiments of the present disclosure, example 9 provides a computer readable medium having stored thereon a computer program, which, when executed by a processing apparatus, implements the steps of the method described in examples 1 to 7.

[0148] According to one or more embodiments of the present disclosure, example 10 provides an electronic device, comprising: a storage device having stored thereon a computer program; and a processing apparatus configured to execute the computer program in the storage device to implement the steps of the method described in examples 1 to 7.

[0149] The above description is merely exemplary of the disclosure and the application of the principles thereof and it is not intended to limit the scope of the disclosure to the specific forms set forth. The disclosure is susceptible to numerous modifications and variations once the scope of the disclosure is understood. For example, the specific sequences of operations described can be modified in various ways. The recitation of one feature does not exclude others. The specific features described can be combined in any suitable manner. The disclosure is not limited to the specific embodiments described herein, but rather, is only limited by the claims that follow.

[0150] Furthermore, while operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order depicted. On the contrary, many of the operations can be performed in a different order or concurrently than was depicted in the examples. In addition, while a particular feature can have been disclosed with respect to only one of the embodiments, such feature can be combined with other features of other embodiments that are disclosed and that can be modified and used in combination with the various embodiments. Also, while particular features can have been disclosed with respect to certain embodiments, features of other embodiments can include, but are not limited to, the features expressly mentioned or potentially apparent from the description.

[0151] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With respect to the devices in the above-described embodiments, in which various modules perform operations, the specific manner in which the various modules perform the operations has been described in detail in the embodiments relating to the method. Here, no detailed explanation will be given.

Claims

1. An image text processing method characterized by, The method comprises: acquiring a target image and problem text corresponding to the target image; performing feature extraction on the target image to obtain image features; performing feature extraction on the problem text according to the image features to obtain visual text features; determining answer text corresponding to the problem text according to the visual text features, the answer text being used to describe a target object in the target image; determining identification information according to the visual text features and the image features, the identification information being used to identify a region in which the target object is located in the target image.

2. The method of claim 1, wherein, The performing feature extraction on the target image to obtain image features comprises: inputting the target image into a visual encoder to obtain the image features; The performing feature extraction on the problem text according to the image features to obtain visual text features comprises: inputting the image features and the problem text into a visual text encoder to obtain the visual text features; The determining answer text corresponding to the problem text according to the visual text features comprises: inputting the visual text features into a text decoder to obtain the answer text; The determining identification information according to the visual text features and the image features comprises: inputting the visual text features and the image features into a text visual decoder to obtain the identification information.

3. The method of claim 2, wherein, The inputting the image features and the problem text into a visual text encoder to obtain the visual text features comprises: inputting the problem text into a self-attention layer in the visual text encoder, and encoding the problem text by using a self-attention mechanism to obtain text features; inputting the text features and the image features into a cross-attention layer in the visual text encoder, and encoding the text features and the image features by using a cross-attention mechanism to obtain the visual text features.

4. The method of claim 3, wherein, The encoding the text features and the image features by using a cross-attention mechanism to obtain the visual text features comprises: taking the text features as query vectors of the cross-attention mechanism, and taking the image features as key vectors and value vectors of the cross-attention mechanism, so as to obtain the visual text features by using the cross-attention mechanism.

5. The method of claim 2, wherein, The inputting the visual text features and the image features into a text visual decoder to obtain the identification information comprises: inputting the visual text features and the image features into a cross-attention layer of the text visual decoder, taking the image features as query vectors of the cross-attention mechanism, and taking the visual text features as key vectors and value vectors of the cross-attention mechanism, so as to obtain the identification information by using the cross-attention mechanism.

6. The method according to any one of claims 2-5, characterized in that, The visual encoder, the visual text encoder, the text decoder and the text visual decoder are obtained by training in the following manner: obtaining a sample input set and a sample output set, the sample input set comprising a plurality of sample inputs, each sample input comprising a training image and a training question text corresponding to the training image, the sample output set comprising a sample output corresponding to each sample input, each sample output comprising a training answer text corresponding to the training question text and training identification information, the training answer text being used to describe a training object in the training image, the training identification information being used to identify a region in the training image where the training object is located; for each sample input, inputting the training image in the sample input into the visual encoder to obtain a training image feature; inputting the training image feature and the training question text in the sample input into the visual text encoder to obtain a training visual text feature; inputting the training visual text feature into the text decoder to obtain a predicted answer text; inputting the training visual text feature and the training image feature into the text visual decoder to obtain predicted identification information; training the visual encoder, the visual text encoder, the text decoder and the text visual decoder according to the predicted answer text, the predicted identification information and the sample output corresponding to the sample input.

7. The method of claim 6, wherein, The manner of training the visual encoder, the visual text encoder, the text decoder and the text visual decoder further comprises: training the visual encoder, the visual text encoder, the text decoder and the text visual decoder for multiple rounds using the sample input set, and taking a parameter set of the visual encoder, the visual text encoder, the text decoder and the text visual decoder in each round as a set of parameter copies; determining an inference parameter set according to multiple sets of the parameter copies; updating the visual encoder, the visual text encoder, the text decoder and the text visual decoder according to the inference parameter set.

8. An image text processing apparatus characterized by comprising: The apparatus comprises: an obtaining module configured to obtain a target image and a question text corresponding to the target image; a first extracting module configured to perform feature extraction on the target image to obtain an image feature; a second extracting module configured to perform feature extraction on the question text according to the image feature to obtain a visual text feature; a first determining module configured to determine an answer text corresponding to the question text according to the visual text feature, the answer text being used to describe a target object in the target image; a second determining module configured to determine identification information according to the visual text feature and the image feature, the identification information being used to identify a region in the target image where the target object is located.

9. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by the processing apparatus to implement the steps of the method of any one of claims 1-7.

10. An electronic device, comprising: comprise: a storage device having a computer program stored thereon; a processing device configured to execute the computer program in the storage device to implement the steps of the method of any one of claims 1-7.

Images