Text and image combined error correction method and device, storage medium and computer equipment

Abstract

The application discloses a text-image joint error correction method and device, a storage medium and a computer device. The method comprises the following steps: obtaining text data and image data to be processed, wherein the text data comprises target text, and the image data comprises a scene picture; extracting a text vector representation of the text data, wherein the text vector representation comprises text information of the target text; extracting a picture vector representation of the image data, wherein the picture vector representation comprises image information of the scene picture; calculating a multi-modal vector representation comprising the text information and the image information according to the text vector representation and the picture vector representation; when the target text is used to express the scene picture, determining an error correction result for the target text and the scene picture according to the multi-modal vector representation, so as to correct the text-image according to the error correction result, realize text-image joint error correction, and improve the error correction capability.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, storage medium and computer equipment for joint error correction of images and texts. Background Art

[0002] Most of the current systems for text error correction are based on language models, and then use pinyin and other constraints to correct the text. The most common one is an algorithm based on statistical language models (N-gram), which scores words and sentences, and then determines whether the current sentence has obvious word usage errors. It then combines pinyin and a list of commonly confused words to comprehensively determine whether there are word and syntactic errors in the sentence. The problems they address are basically single-modal text input information, and no multi-modal error correction systems have been found so far. N-grams are generally obtained through statistics of large-scale corpora, and there are also related works using masked language models (MLMs), but in essence, they all correct errors on pure text, without the help of relevant information such as images. For the same sentence, the results of error detection and correction are generally fixed because the score calculated for the current sentence through the language model is fixed. Although the system can sometimes be adjusted by adjusting the error detection and correction thresholds, the current relevant multimodal text error correction systems can only process single-modal text input when solving the error detection and correction problems in text writing, and cannot correct network data containing pictures and texts. In addition, some error correction systems can only correct typos, and some can only correct grammatical errors, resulting in insufficient error correction capabilities. Summary of the Invention

[0003] The embodiments of the present application provide a method, apparatus, storage medium, and computer equipment for joint error correction of images and texts, which can implement joint error correction of images and texts and improve error correction capabilities.

[0004] On the one hand, a method for joint error correction of text and images is provided, the method comprising: obtaining text data and image data to be processed, the text data comprising target text, and the image data comprising scene pictures; extracting a text vector representation of the text data, the text vector representation comprising text information of the target text; extracting a picture vector representation of the image data, the picture vector representation comprising image information of the scene picture; calculating a multimodal vector representation comprising text information and image information based on the text vector representation and the picture vector representation; and when the target text is used to express the scene picture, determining error correction results for the target text and the scene picture based on the multimodal vector representation.

[0005] Optionally, extracting the text vector representation of the text data includes: converting each word in the text data into a serial number corresponding to each word in the vocabulary through a vocabulary, and searching for the text vector representation of the text data according to the serial number.

[0006] Optionally, extracting the picture vector representation of the image data includes: performing target detection and feature extraction on the scene picture according to a target detection model to obtain the picture vector representation, wherein the picture vector representation includes the image information vector representation of each image target in the scene picture and the image information vector representation of the entire picture.

[0007] Optionally, the calculating of a multimodal vector representation containing text information and image information based on the text vector representation and the picture vector representation includes: processing the text vector representation and the picture vector based on a self-attention model to obtain global interaction information between the text information of the target text and the image information of the scene picture; normalizing the global interaction information to obtain first normalized information; and determining a multimodal vector representation containing text information and image information based on the global interaction information and the first normalized information.

[0008] Optionally, the text vector representation and the picture vector are processed based on the self-attention model to obtain global interaction information between the text information of the target text and the image information of the scene picture, including: inputting the embedding vector representation determined according to the text vector representation and the picture vector representation into the self-attention model, and calculating a matching matrix according to the product between the embedding vector representation and the transposed matrix of the embedding vector representation; and determining the global interaction information between the text information of the target text and the image information of the scene picture according to the product of the matching matrix and the embedding vector representation.

[0009] Optionally, determining a multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information includes: adding the global interaction information and the first normalization information to obtain first sum information; inputting the first sum information into a fully connected layer for processing, and then normalizing the output result of the fully connected layer to obtain second normalized information; adding the first sum information and the second normalization information to obtain the multimodal vector representation containing text information and image information.

[0010] Optionally, the method further includes: obtaining a position vector representation and a type vector representation, wherein the position vector representation is used to mark the position of each word in the text data, and the type vector representation is used to distinguish between text type and image type;

[0011] The calculating of a multimodal vector representation containing text information and image information based on the text vector representation and the picture vector representation includes: calculating a multimodal vector representation containing text information and image information based on the text vector representation, the picture vector representation, the position vector representation and the type vector representation.

[0012] Optionally, determining the error correction results for the target text and the scene image based on the multimodal vector representation includes: connecting the multimodal vector representation to a fully connected layer to obtain a first feature vector, the first feature vector containing vector representations corresponding to each word in the target text and each image target in the scene image; determining the error correction operations corresponding to each word in the target text and each image target in the scene image based on the first feature vector; calculating the error correction result based on the error correction operation and the first feature vector, so as to perform error correction processing on the target text and / or the scene image based on the error correction result.

[0013] Optionally, determining the error correction result based on the error correction operation and the first feature vector also includes: if the error correction operation is error-free, determining the error correction result to output the word corresponding to the error-free error correction operation; or if the error correction operation is a deletion operation, determining the error correction result to delete the word corresponding to the deletion operation in the target text; or if the error correction operation is a modification operation, determining the error correction result to modify the word corresponding to the modification operation in the target text to a predicted word, or modifying the image target corresponding to the modification operation in the scene picture to a predicted image target.

[0014] On the other hand, a combined image and text error correction device is provided, the device comprising:

[0015] An acquisition unit, configured to acquire text data and image data to be processed, wherein the text data includes target text and the image data includes scene images;

[0016] a first extraction unit, configured to extract a text vector representation of the text data, wherein the text vector representation includes text information of the target text;

[0017] a second extraction unit, configured to extract a picture vector representation of the image data, wherein the picture vector representation includes image information of the scene picture;

[0018] a computing unit, configured to compute a multimodal vector representation containing text information and image information based on the text vector representation and the image vector representation;

[0019] A determination unit is used to determine an error correction result for the target text and the scene picture according to the multimodal vector representation when the target text is used to express the scene picture.

[0020] On the other hand, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the joint text-graphic error correction method as described in any of the above embodiments.

[0021] On the other hand, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to execute the steps of the joint text and image error correction method as described in any of the above embodiments by calling the computer program stored in the memory.

[0022] On the other hand, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the method for joint error correction of graphics and text as described in any of the above embodiments.

[0023] The embodiment of the present application obtains text data and image data to be processed, wherein the text data includes a target text and the image data includes a scene picture; extracts a text vector representation of the text data, wherein the text vector representation includes text information of the target text; extracts a picture vector representation of the image data, wherein the picture vector representation includes image information of the scene picture; calculates a multimodal vector representation including text information and image information based on the text vector representation and the picture vector representation; when the target text is used to express the scene picture, determines the error correction result for the target text and the scene picture based on the multimodal vector representation. The embodiment of the present application realizes multimodal joint error correction of image and text input through the Transformer model. The model simultaneously inputs image data and text data including the target text, and calculates a multimodal vector representation including text information and image information through the attention mechanism inside the Transformer model to filter out useful image information and text information, and then detects error information in the image based on the text information in the multimodal vector representation, determines the error correction result for the target text and the scene picture, and corrects the image and text based on the error correction result, thereby realizing joint error correction of image and text and improving the error correction capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 This is a structural framework diagram of the image-text joint error correction system provided in an embodiment of the present application.

[0026] Figure 2 This is a schematic diagram of the first process flow of the image-text joint error correction method provided in an embodiment of the present application.

[0027] Figure 3 Schematic diagram of the first application scenario of the combined image and text error correction method provided in an embodiment of the present application.

[0028] Figure 4 A second flow chart of the combined image and text error correction method provided in an embodiment of the present application.

[0029] Figure 5 Schematic diagram of a second application scenario of the combined image and text error correction method provided in an embodiment of the present application.

[0030] Figure 6 This is a schematic diagram of the third process of the combined image and text error correction method provided in an embodiment of the present application.

[0031] Figure 7 Schematic diagram of the third application scenario of the combined image and text error correction method provided in an embodiment of the present application.

[0032] Figure 8 A schematic diagram of the structure of the image-text joint error correction device provided in an embodiment of the present application.

[0033] Figure 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0035] The embodiments of the present application provide a method, apparatus, computer device and storage medium for joint error correction of images and texts. Specifically, the method for joint error correction of images and texts in the embodiments of the present application can be executed by a computer device, wherein the computer device can be a terminal or a server or other device. The terminal can be a smart phone, a tablet computer, a laptop computer, a smart TV, a smart speaker, a wearable smart device, a personal computer (PC) or other device, and the terminal can also include a client, which can be a video client, a browser client or an instant messaging client, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0036] The embodiments of the present application can be applied to various scenarios such as artificial intelligence, speech recognition, and smart transportation.

[0037] First, some nouns or terms that appear in the description of the embodiments of this application are explained as follows:

[0038] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0039] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0040] Deep learning (DL), a branch of machine learning, is an algorithm that attempts to achieve high-level abstraction of data using multiple processing layers that include complex structures or multiple nonlinear transformations. Deep learning learns the inherent patterns and representational hierarchies of training sample data. The information gained from this learning process is highly helpful in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to acquire human-like analytical learning capabilities and recognize data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far surpass previous technologies.

[0041] Neural Network (NN) is a deep learning model in the field of machine learning and cognitive science that imitates the structure and function of biological neural networks.

[0042] The Transformer model is a classic NLP (natural language processing) model. It relies entirely on attention to encode inputs and compute outputs, rather than relying on sequence-aligned recurrent neural networks or convolutional neural networks. Instead of adopting the sequential structure of RNNs, the Transformer model uses a self-attention mechanism, enabling parallel training and the ability to incorporate global information.

[0043] The current text error correction systems have the following shortcomings when solving the error detection and correction problems of text writing:

[0044] 1. The system can only process data with text as input. Currently, almost all text error correction systems can only handle single-mode text input and are unable to correct web data containing images and text. As a result, the current web data containing images and text cannot be correctly input and processed by error correction systems.

[0045] 2. Current error correction systems are all single-modal and cannot leverage multimodal information to improve error correction effectiveness. Error correction itself is a multi-factor processing system. Although current text error correction systems can correct errors based on different texts and different application scenarios, they are essentially single-modal, pure text scenarios and cannot utilize additional information outside of the text to assist in error correction.

[0046] 3. Poor system transferability. A correction system is fundamentally based on a language model, which is derived from large-scale text corpora through statistical or MLM training. Once targeted, it cannot be seamlessly transferred to other fields. For example, a language model trained on judicial text will not perform well in everyday conversations.

[0047] 4. Error correction problems are divided into very specific categories. Some error correction systems can only correct typos, while others can only correct grammatical errors. Many error correction systems isolate the problem and address each type of problem separately. This can lead to inappropriate error detection and correction processes, or the same error can be detected by multiple modules, resulting in inconsistent correction results.

[0048] Traditional error correction is based on text, but in real-life scenarios, the demand for multimodal joint error correction is increasing. For example, consider the sentence "A person is eating an apple." The corresponding scene is a person playing with an iPhone. If a traditional text-based error correction system were used to correct this text, it would fail to detect the error or would instead interpret it as "A person is looking down at his iPhone." Alternatively, a traditional text-based error correction system might correct it as "A person is looking down at his iPhone," rather than "A person is looking down at his iPhone." In these cases, multimodal joint image-text error correction is needed. With the rapid development of the internet, information containing both images and text is increasing. Therefore, joint image-text error correction is becoming increasingly valuable. Traditional single-modal error correction is limited in some scenarios and cannot assist with error correction based on specific information such as images. Therefore, single-modal error correction inherently has significant limitations.

[0049] The embodiment of the present application further introduces image information on the basis of single text to perform joint error correction. This not only solves the error correction of more complex scenarios, but also improves the ability to detect and correctly correct errors through mutual enhancement of image and text information.

[0050] The multimodal image-text joint error correction system based on neural networks and deep learning models proposed in the embodiments of the present application can overcome the above problems. First, the embodiments of the present application can input images and texts into the error correction system at the same time, and use the image and text information for error correction. Nowadays, a lot of Internet data is a mixture of images and texts. A system that only corrects texts will gradually fail to keep up with the trend of the times. Secondly, the information of the two modes of image and text can be used to improve the overall error correction effect, and avoid some errors that cannot be judged by text alone from being corrected. In addition, the embodiments of the present application will detect erroneous information in the picture based on the text information, and then improve the quality of the text and picture in the error correction process, and can also correct the images and texts at the same time.

[0051] See also Figure 1 , Figure 1 A structural framework diagram of the text-image joint error correction system provided in an embodiment of the present application. The text-image joint error correction system includes a multimodal Transformer model and an error correction module. First, text data of the text modality is input, which includes the target text. Then, image data of the image modality is input, and the image data includes a scene picture. Then, a multimodal Transformer model is input. A multimodal vector representation that combines text information and image information can be calculated through the Transformer model. The multimodal vector representation is then input into an error correction module to calculate information that may have errors in the text and image.

[0052] By simultaneously inputting the signals of the target text and scene image, the Transformer model's internal attention mechanism filters out useful image and text information. Then, based on this filtered information, the error correction module calculates whether there are errors in each character (or each word) and each image target in the scene image.

[0053] The feature representations of the two modalities, text and image, are learned when images and text are input simultaneously. The model can accurately obtain useful representation information. Finally, it accurately determines whether the scene image contains any errors in the current target text. If the target text contains any errors, the errors are corrected. If there are errors in the scene image, the scene image is corrected or the problematic part of the image is output. For example, each detection target corresponding to text information in the scene image can be used as an image target. The degree of match between the image information and the text information in the image target can be used to determine whether the image target contains any errors, that is, to determine where the errors are in the scene image.

[0054] Since the entire joint image and text error correction system directly completes multimodal image and text error correction through a single model, the overall structure is relatively simple and the performance is relatively good.

[0055] It should be noted that the order of description of the following embodiments does not limit the priority order of the embodiments.

[0056] Each embodiment of the present application provides a method for joint error correction of images and texts, which can be executed by a terminal or a server, or jointly by a terminal and a server; the embodiments of the present application illustrate the method for joint error correction of images and texts by an example in which the method is executed by a server.

[0057] See also Figures 2 to 7 , Figure 2 、 Figure 4 and Figure 6 These are all flowcharts of the image-text joint error correction method provided in the embodiments of the present application. Figure 3 、 Figure 5 and Figure 7 These are schematic diagrams of application scenarios of the image-text joint error correction method provided in the embodiments of the present application. The method includes:

[0058] Step 110 : Acquire text data and image data to be processed, wherein the text data includes target text, and the image data includes scene images.

[0059] For example, the text data includes a target text. For example, the target text to be processed is "The beautiful woman in the picture is wearing glasses." The target text can also be used as a question.

[0060] For example, the image data includes a scene picture, which is a picture provided for the scene described in the target text. For example, when answering, the user can answer the question of whether the target text (question) is a correct expression of the scene picture by observing the scene picture.

[0061] Step 120: extracting a text vector representation of the text data, wherein the text vector representation includes text information of the target text.

[0062] Optionally, extracting a text vector representation of the text data includes:

[0063] Each word in the text data is converted into a sequence number corresponding to each word in the vocabulary through a vocabulary, and a text vector representation of the text data is searched according to the sequence number.

[0064] First, the text data can be represented by vectorization to map each word in the target text to a specific space to obtain a text vector representation of the text data.

[0065] For example, combined with Figure 3To illustrate, the image-text joint error correction system may include a multimodal data input processing module, a multimodal feature extraction module and an error correction module, wherein the feature extraction module may adopt a Transformer model.

[0066] For example, the TXT-formatted text data (i.e., the original text corresponding to the TXT-formatted target text) is input into the data input processing module. The original text is converted into a sequence number (ID) in the vocabulary corresponding to each word. The original text is then searched for the embedding vector representation of each word in the vocabulary using the ID. For example, the target text "The beautiful woman in the picture is wearing glasses" is converted into the IDs [1, 4, 3, 6, 7, 0, 12, 87, 98] using the vocabulary. Since "picture" has an ID of 1 in the vocabulary, it is converted to 1. The vector representation corresponding to each word (w1, w2, w3, w4, w5, w6, w7, w8, w9, w10) is then found using the ID, resulting in a sequence of word vectors. This sequence of word vectors is then used as an input parameter for the Transformer model and can be defined as a text vector representation. This text vector representation is a matrix of seq_len times hid_size, where seq_len represents the length of the text and hid_size represents the size of the word vector.

[0067] Step 130: extract a picture vector representation of the image data, wherein the picture vector representation includes image information of the scene picture.

[0068] Optionally, extracting a picture vector representation of the image data includes:

[0069] Target detection and feature extraction are performed on the scene image according to the target detection model to obtain the image vector representation, wherein the image vector representation includes the image information vector representation of each image target in the scene image and the image information vector representation of the entire image.

[0070] For example, the image data fed into the data processing module is first extracted through the object detection model (FastRCNN) to extract physical numerical information. Then, through model learning, a FastRCNN model is obtained that can correctly extract physical information. Fast RCNN (Fast Regions with CNN features) is a fast region-based convolutional network method for object detection.

[0071] For example, combined with Figure 3To explain, input image data containing scene pictures, first use Fast-RCNN to perform target detection and feature extraction on the scene pictures in the image data, and obtain the image information vector representation of each image target in the picture and the image information vector representation of the entire picture. Among them, the image target corresponds to the object that needs to be paid attention to in the text data. For example, if the text data mentions "the beautiful woman in the picture", then we need to pay attention to the people in the scene picture, such as the man and woman in the picture. Among them, the image information vector representation of the entire picture will be applied to each text word separately, and the image information vector representation of a single image target will correspond to the text vector that specifically represents the image information, that is, Figure 3 The img text vector is shown. The final image data, as the input parameter of the Transformer model, is also a matrix of seq_len and hid_size, which can be defined as a picture vector representation.

[0072] Step 140 : Calculate a multimodal vector representation containing text information and image information based on the text vector representation and the image vector representation.

[0073] For example, the embedding vector representation containing an image and text is input into the Transformer model to calculate a multimodal vector representation that combines text and image information. A multimodal vector representation is a vector that contains both text and image information.

[0074] Optionally, the method further includes: obtaining a position vector representation and a type vector representation, wherein the position vector representation is used to mark the position of each word in the text data, and the type vector representation is used to distinguish between text type and image type;

[0075] The calculating, based on the text vector representation and the image vector representation, a multimodal vector representation containing text information and image information includes:

[0076] A multimodal vector representation containing text information and image information is calculated based on the text vector representation, the picture vector representation, the position vector representation, and the type vector representation.

[0077] The position vector is used to mark the position of each word in the text data. The position vector represents a matrix of size seq_len times hid_size. The type vector is used to distinguish between text and image types. The type vector represents a matrix of size seq_len times hid_size. For example, text type is represented by 0 and image type is represented by 1.

[0078] For example, Figure 3As shown, the input parameters of the final input Transformer model may include an embedding vector representation Embedding composed of a text vector representation plus an image vector representation plus a position vector representation and a type vector representation, denoted as E. For example, the Embedding vector representation of the target text and the scene image is input into the Transformer model to calculate the multimodal vector representation between the text and the image. The multimodal vector representation is a vector that contains both text information and image information, that is, the vector representation of each word in the target text and the vector representation of each image target in the scene image are calculated internally by the Transformer model, and the optimal feature vector is extracted by means of each group of co-occurrences. Finally, this multimodal representation is output to the error correction module. In this joint text-image error correction system, the error correction module is used to calculate whether each word in the target text needs to be modified, how to modify it, whether each image target in the scene image has errors, and whether it needs to be modified.

[0079] For example, combined with Figure 3 To illustrate, the text vector representation and the image vector representation can be processed by a multimodal feature extraction module to calculate a multimodal vector representation containing text information and image information, wherein the feature extraction module can use a Transformer model. The main function of the feature extraction module is to calculate a multimodal vector representation that combines text information and image information. Figure 3 As shown in the figure, after processing by the Transformer model, a multimodal vector representation that combines text information and image information is finally obtained. This multimodal vector representation can be used by the error correction module to calculate whether there are errors in the target text and scene images, and whether "addition", "deletion", "modification" and other processing are required. Figure 3 As shown in the figure, the calculation is to determine whether the expression "woman" matches the image and whether the image shows a woman wearing glasses. This module is designed to calculate the association between image and text and output the matching feature matrix.

[0080] Optional, such as Figure 4 As shown, step 140 can be implemented through steps 141 to 143, specifically:

[0081] Step 141: Process the text vector representation and the image vector based on the self-attention model to obtain global interaction information between the text information of the target text and the image information of the scene image.

[0082] Optionally, the text vector representation and the picture vector are processed based on the self-attention model to obtain global interaction information between the text information of the target text and the image information of the scene picture, including: inputting the embedding vector representation determined according to the text vector representation and the picture vector representation into the self-attention model, and calculating a matching matrix according to the product between the embedding vector representation and the transposed matrix of the embedding vector representation; and determining the global interaction information between the text information of the target text and the image information of the scene picture according to the product of the matching matrix and the embedding vector representation.

[0083] The length dimension of the global interaction information is the same as that of the embedding vector representation.

[0084] For example, see Figure 5 The input to the feature extraction module is: vector representations of the image and text, whose size is a matrix of seq_len times hid_size. For example, the vector representation of the image and text can be an embedding vector representation consisting of the text vector representation plus the image vector representation, denoted as E. The vector representation of the image and text can also be an embedding vector representation consisting of the text vector representation plus the image vector representation plus the position vector representation and the type vector representation, denoted as E.

[0085] The output of the feature extraction module is a multimodal vector representation that combines image information and all text information, with a size of seq_len times hid_size.

[0086] For example, see Figure 5 , the matching matrix is ​​calculated by the self-attention model inside the feature extraction module. The input is the embedding vector representation E of seq_len times hid_size. The embedding vector representation is the vector representation of the image and text; the output is the global interaction information H s , H s The size is seq_len times hid_size. In the specific calculation process, self_attention means that the attention is calculated by itself, and the E matrix is ​​multiplied by E T , get the matching matrix M, where the size of the matching matrix M is seq_len times seq_len, and then multiply the M matrix by E to get H s , H s The size of E is seq_len times hid_size. T is the transposed matrix of the E matrix.

[0087] Step 142: normalize the global interaction information to obtain first normalized information.

[0088] For example, see Figure 5 , the global interaction information H output by the self_attention model s Normalize, i.e. norm, and get the first normalized information H n , the first normalized information H n The size of is seq_len times hid_size. Normalization does not affect the matrix size. The length dimension of the global interaction information is the same as the length dimension of the embedding vector representation, and the size of the word vector of the global interaction information is the same as the size of the word vector represented by the embedding vector.

[0089] Step 143: Determine a multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information.

[0090] Optionally, determining a multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information includes: adding the global interaction information and the first normalization information to obtain first sum information; inputting the first sum information into a fully connected layer for processing, and then normalizing the output result of the fully connected layer to obtain second normalized information; adding the first sum information and the second normalization information to obtain the multimodal vector representation containing text information and image information.

[0091] For example, see Figure 5 , the input of the fully connected layer is H s +H n , which is the global interaction information H s With the first normalized information H n The first summation information is added and then input into the fully connected layer for processing, where the output of the fully connected layer is seq_len multiplied by hid_size. The output of the fully connected layer is then normalized (norm) to obtain the second normalized information, and the second normalized information is added to the first summation information again to obtain the multimodal vector representation H. nn . Due to H nn The self_attention model can have multiple layers, the same size as the input E matrix, and can generally be set to 12 or 24 layers.

[0092] For example, the multimodal vector representation H nnFinally, by directly outputting output, the size of output is a matrix of seq_len times hid_size, and the multimodal vector representation is a multimodal vector representation that integrates image information and all text information. Among them, the length dimension of the multimodal vector representation is the same as the length dimension of the embedding vector representation.

[0093] This feature extraction module calculates a multimodal vector representation that fuses text and images. This multimodal vector representation contains highly abstract semantic matching relationships between the text and images, providing rich information for subsequent modules to detect and correct errors based on the semantics of this matching information. It also transforms text matching, which previously involved simple character-level matching, into matching between vector spaces. This feature extraction module elevates text matching to the level of semantic space.

[0094] Step 150 : When the target text is used to express the scene image, determine an error correction result for the target text and the scene image according to the multimodal vector representation.

[0095] For example, the output result of the feature extraction module (a multimodal vector representation that integrates text information and image information) can be processed by an error correction module to calculate whether each word in the target text needs to be modified and how to modify it, as well as whether each image target in the scene image has errors and whether it needs to be modified.

[0096] Among them, the error correction module mainly processes the output results of the feature extraction module to correct each word in the target text and each image target part in the scene picture. Figure 3 For example, consider the target text "The beautiful woman in the picture is wearing glasses." While the target text itself contains no grammatical errors, and the scene image itself contains no problems, errors occur when the target text is used to represent the scene image. This means there's an error message indicating a mismatch between the scene image and the target text. The error correction module's primary task is to detect and correct errors.

[0097] Optional, such as Figure 6 As shown, step 150 can be implemented through steps 151 to 153, specifically:

[0098] Step 151: Connect the multimodal vector representation to a fully connected layer to obtain a first feature vector, where the first feature vector includes vector representations corresponding to each word in the target text and each image target in the scene image.

[0099] For example, see Figure 7, a multimodal vector representation that combines text information and image information, followed by a full connection, is used to obtain the first eigenvector H = [h1,h2,h3,…,h 15 ], for example, seq_len = 15, H contains 15 vectors corresponding to all words in the target text and all image targets in the scene image.

[0100] Step 152: Determine, based on the first feature vector, the error correction operations corresponding to each word in the target text and each image object in the scene image.

[0101] For example, see Figure 7 , the first feature vector H is input into the recognition and correction mode module, through which the error correction operation O corresponding to each word and each image target on the length dimension seq_len can be obtained. 15 ], the error correction operations mainly include three types: no error, this error-free operation means that no error correction is needed; deletion operation, this deletion operation means that the current word is redundant and needs to be deleted; modification operation, this modification operation means that the current word is an incorrect word and needs to be corrected; or it means that the current image target is an incorrect image target and needs to be corrected.

[0102] Step 153: Calculate an error correction result according to the error correction operation and the first feature vector, and perform error correction processing on the target text and / or the scene image according to the error correction result.

[0103] Optionally, determining an error correction result according to the error correction operation and the first feature vector further includes:

[0104] If the error correction operation is correct, determining the error correction result to output a word corresponding to the correct error correction operation; or

[0105] If the error correction operation is a deletion operation, determining the error correction result is to delete the word corresponding to the deletion operation in the target text; or

[0106] If the error correction operation is a modification operation, determining the error correction result is to modify the word corresponding to the modification operation in the target text into a predicted word, or to modify the image target corresponding to the modification operation in the scene image into a predicted image target.

[0107] For example, see Figure 7, according to the error correction operation O and the first feature vector H, the error correction result is finally calculated. For example, "no error" directly outputs the word corresponding to the error correction operation without error, or if the error correction operations corresponding to all words in the target text are all no error, the target text can be directly output. "Delete", the word corresponding to the deletion operation is output as empty. "Modify", the current h i (like Figure 7 The example is h1) connected to the full connection to predict a new word (or character) output, if the current h i is an image target. A new image target is output through full connectivity at the location of this image target in the scene image. For example, for the input target text "The beautiful woman in the picture is wearing glasses", the correction result is "漂亮、亮、女" (beautiful, bright, female) and the corresponding correction operation is modification. The predicted new words are "帅、气、男" (handsome, energetic, male). When the target text is corrected based on the correction result, the output is "帅、气、男" (handsome man in the picture is wearing glasses).

[0108] The image-text joint error correction system provided in the embodiment of the present application can not only correct the text during the error correction process, but also simultaneously identify problematic areas in the image. If there is an erroneous image, a corrected image will be generated as a modification suggestion.

[0109] Before using the image-text joint error correction system, sufficient cross-modal data can be provided to perform model learning. After learning, the entire image-text joint error correction system can automatically correct inputs containing both images and text, and can also process text or images separately.

[0110] The image-text joint error correction method provided in the embodiment of the present application uses image information to assist text error correction, and uses text information to discover some common sense errors in images. It pioneered the design of a set of error correction systems that combine image and text information, and promoted error correction from a single modal task to a multimodal and hypermodal level. It innovatively applied multimodal technology to the error correction task, and it is a multimodal joint error correction. Although the embodiment of the present application only describes the joint error correction of images and texts in detail, in fact, more other modal information can be added to assist in error correction, such as voice, video, etc. The innovation of the image-text error correction method defines error correction as two stages. The first stage identifies the methods and means to be taken for error correction, and the second stage uses the corresponding entanglement results under the specific error correction method for error correction. The complex error correction system is designed as an end-to-end model method, which is very conducive to later maintenance and deployment. The error correction method of multiple modalities reinforcing each other and multimodal signal processing are the inevitable trends in the future development of artificial intelligence. The forward-looking multimodal image-text joint error correction method of this system elevates intelligent error correction to a new level, can solve problems that conventional pure text error correction cannot solve, and can help machines learn knowledge and attempts from multiple modalities, which is also very helpful for single-modality error correction. The embodiment of the present application can detect inappropriate expressions in text based on images. At the same time, some pictures containing errors can also be corrected with the help of text information.

[0111] The embodiments of the present application provide a multimodal joint text and image error correction method and a joint text and image error correction system. Compared with previous methods that can only correct text errors, the embodiments of the present application can correct text and images at the same time. While correcting the text, corresponding arguments can also be found on the image, thereby increasing the comprehensibility of the correction.

[0112] All of the above technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0113] The embodiment of the present application obtains text data and image data to be processed, wherein the text data includes a target text and the image data includes a scene picture; extracts a text vector representation of the text data, wherein the text vector representation includes text information of the target text; extracts a picture vector representation of the image data, wherein the picture vector representation includes image information of the scene picture; calculates a multimodal vector representation including text information and image information based on the text vector representation and the picture vector representation; when the target text is used to express the scene picture, determines the error correction result for the target text and the scene picture based on the multimodal vector representation. The embodiment of the present application realizes multimodal joint error correction of image and text input through the Transformer model. The model simultaneously inputs image data and text data including the target text, and calculates a multimodal vector representation including text information and image information through the attention mechanism inside the Transformer model to filter out useful image information and text information, and then detects error information in the image based on the text information in the multimodal vector representation, determines the error correction result for the target text and the scene picture, and corrects the image and text based on the error correction result, thereby realizing joint error correction of image and text and improving the error correction capability.

[0114] In order to better implement the image-text joint error correction method of the embodiment of the present application, the embodiment of the present application also provides an image-text joint error correction device. Figure 8 , Figure 8 A schematic diagram of the structure of a combined image and text error correction device provided in an embodiment of the present application. The combined image and text error correction device 200 may include:

[0115] An acquisition unit 201 is configured to acquire text data and image data to be processed, wherein the text data includes target text and the image data includes scene images;

[0116] A first extraction unit 202 is configured to extract a text vector representation of the text data, wherein the text vector representation includes text information of the target text;

[0117] A second extraction unit 203 is configured to extract a picture vector representation of the image data, wherein the picture vector representation includes image information of the scene picture;

[0118] a calculation unit 204 configured to calculate a multimodal vector representation containing text information and image information based on the text vector representation and the image vector representation;

[0119] The determination unit 205 is configured to determine, when the target text is used to express the scene image, an error correction result for the target text and the scene image based on the multimodal vector representation.

[0120] Optionally, the first extraction unit 202 may be configured to convert each word in the text data into a sequence number corresponding to each word in the vocabulary through a vocabulary, and search for a text vector representation of the text data according to the sequence number.

[0121] Optionally, the second extraction unit 203 can be used to perform target detection and feature extraction on the scene image according to the target detection model to obtain the image vector representation, wherein the image vector representation includes the image information vector representation of each image target in the scene image and the image information vector representation of the entire image.

[0122] Optionally, the computing unit 204 can be specifically used to: process the text vector representation and the image vector based on a self-attention model to obtain global interaction information between the text information of the target text and the image information of the scene image; normalize the global interaction information to obtain first normalized information; and determine a multimodal vector representation containing text information and image information based on the global interaction information and the first normalized information.

[0123] Optionally, when the computing unit 204 processes the text vector representation and the picture vector based on the self-attention model to obtain the global interaction information between the text information of the target text and the image information of the scene picture, it can be used to: input the embedding vector representation determined according to the text vector representation and the picture vector representation into the self-attention model, calculate the matching matrix according to the product between the embedding vector representation and the transposed matrix of the embedding vector representation; and determine the global interaction information between the text information of the target text and the image information of the scene picture according to the product of the matching matrix and the embedding vector representation.

[0124] Optionally, when determining the multimodal vector representation containing text information and image information based on the global interaction information and the first normalization information, the calculation unit 204 can be used to: add the global interaction information and the first normalization information to obtain first sum information; after inputting the first sum information into the fully connected layer for processing, normalize the output result of the fully connected layer to obtain second normalized information; add the first sum information and the second normalization information to obtain the multimodal vector representation containing text information and image information.

[0125] Optionally, the acquisition unit 201 may also be used to acquire a position vector representation and a type vector representation, wherein the position vector representation is used to mark the position of each word in the text data, and the type vector representation is used to distinguish between text type and image type;

[0126] The calculation unit 204 may be configured to calculate a multimodal vector representation containing text information and image information based on the text vector representation, the image vector representation, the position vector representation, and the type vector representation.

[0127] Optionally, the determination unit 205 can be specifically used to: connect the multimodal vector representation to a fully connected layer to obtain a first feature vector, wherein the first feature vector contains vector representations corresponding to each word in the target text and each image target in the scene picture; determine the error correction operations corresponding to each word in the target text and each image target in the scene picture according to the first feature vector; calculate the error correction result according to the error correction operation and the first feature vector, and perform error correction processing on the target text and / or the scene picture according to the error correction result.

[0128] Optionally, when determining the error correction result based on the error correction operation and the first feature vector, the determination unit 205 can also be used to: if the error correction operation is error-free, determine the error correction result to output the word corresponding to the error-free error correction operation; or if the error correction operation is a deletion operation, determine the error correction result to delete the word corresponding to the deletion operation in the target text; or if the error correction operation is a modification operation, determine the error correction result to modify the word corresponding to the modification operation in the target text to a predicted word, or modify the image target corresponding to the modification operation in the scene picture to a predicted image target.

[0129] It should be noted that the functions of each module in the image-text joint error correction device 200 in the embodiment of the present application can correspond to the specific implementation methods of any embodiment in the above-mentioned method embodiments, and will not be repeated here.

[0130] Each unit in the above-mentioned combined image and text error correction device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above-mentioned units.

[0131] The image-text joint error correction device 200 can be integrated into a terminal or server that has a storage device and a processor installed and has computing power, or the image-text joint error correction device 200 is the terminal or server. The terminal can be a smart phone, tablet computer, laptop computer, smart TV, smart speaker, wearable smart device, personal computer (PC) and other devices. The terminal can also include a client, which can be a video client, browser client or instant messaging client, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0132] Figure 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown in FIG. Figure 9 As shown, computer device 300 may include: a communication interface 301, a memory 302, a processor 303, and a communication bus 304. Communication interface 301, memory 302, and processor 303 communicate with each other via communication bus 304. Communication interface 301 is used for data communication between device 300 and external devices. Memory 302 may be used to store software programs and modules. Processor 303 executes the software programs and modules stored in memory 302, such as the software programs for the corresponding operations in the aforementioned method embodiments.

[0133] Optionally, the processor 303 can call the software program and module stored in the memory 302 to perform the following operations: obtain text data and image data to be processed, the text data including the target text, and the image data including the scene picture; extract the text vector representation of the text data, the text vector representation including the text information of the target text; extract the picture vector representation of the image data, the picture vector representation including the image information of the scene picture; calculate a multimodal vector representation including text information and image information based on the text vector representation and the picture vector representation; when the target text is used to express the scene picture, determine the error correction results for the target text and the scene picture based on the multimodal vector representation.

[0134] Optionally, the computer device 300 is the terminal or server. The terminal may be a smartphone, tablet computer, laptop computer, smart TV, smart speaker, wearable smart device, personal computer, or other device. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0135] Optionally, the present application also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0136] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding process of the image-text joint error correction method in the embodiment of this application. For the sake of brevity, it is not further described here.

[0137] This application also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding process of the combined image and text error correction method in the embodiments of this application. For the sake of brevity, the details are not repeated here.

[0138] This application also provides a computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding process of the combined image and text error correction method in the embodiments of this application. For the sake of brevity, the detailed description is omitted here.

[0139] It should be understood that the processor of the embodiments of the present application may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly implemented as a hardware decoding processor, or can be implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0140] It is understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0141] It should be understood that the above-mentioned memories are exemplary but not restrictive. For example, the memories in the embodiments of the present application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM RAM (DR RAM), etc. In other words, the memories in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0142] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0143] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, each functional unit in the embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0147] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0148] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for joint error correction of images and texts, characterized in that: include: Acquire text data and image data to be processed, wherein the text data includes target text and the image data includes scene images; extracting a text vector representation of the text data, wherein the text vector representation includes text information of the target text; extracting a picture vector representation of the image data, the picture vector representation comprising image information of the scene picture; Obtaining a position vector representation and a type vector representation, wherein the position vector representation is used to mark the position of each word in the text data, and the type vector representation is used to distinguish between text type and image type; Calculating a multimodal vector representation containing text information and image information based on the text vector representation and the image vector representation, including: calculating a multimodal vector representation containing text information and image information based on a self-attention model according to the text vector representation, the image vector representation, the position vector representation, and the type vector representation, wherein the multimodal vector representation contains a semantic matching relationship between the text information and the image information; When the target text is used to express the scene picture, an error correction result for the target text and the scene picture is determined according to the multimodal vector representation, including: determining an error correction result corresponding to an error correction operation according to the multimodal vector representation through a fully connected layer and an error correction mode identification module in an error correction module, if the error correction operation is error-free, the error correction result is outputting a word corresponding to the error-free error correction operation; if the error correction operation is a deletion operation, the error correction result is deleting the word corresponding to the deletion operation in the target text; if the error correction operation is a modification operation, the error correction result is changing the word corresponding to the modification operation in the target text to a predicted word, or changing the image target corresponding to the modification operation in the scene picture to a predicted image target; Assist error correction based on voice and video modal information.

2. The image-text joint error correction method according to claim 1, characterized in that: The extracting the text vector representation of the text data includes: Each word in the text data is converted into a sequence number corresponding to each word in the vocabulary through a vocabulary, and a text vector representation of the text data is searched according to the sequence number.

3. The image-text joint error correction method according to claim 1, characterized in that: The extracting a picture vector representation of the image data comprises: Target detection and feature extraction are performed on the scene image according to the target detection model to obtain the image vector representation, wherein the image vector representation includes the image information vector representation of each image target in the scene image and the image information vector representation of the entire image.

4. The image-text joint error correction method according to claim 1, wherein: The calculating, based on the text vector representation and the picture vector representation, a multimodal vector representation containing text information and image information includes: Processing the text vector representation and the image vector based on a self-attention model to obtain global interaction information between the text information of the target text and the image information of the scene image; Normalizing the global interaction information to obtain first normalized information; A multimodal vector representation including text information and image information is determined based on the global interaction information and the first normalization information.

5. The image-text joint error correction method according to claim 4, characterized in that: The processing of the text vector representation and the picture vector based on the self-attention model to obtain global interaction information between the text information of the target text and the image information of the scene picture includes: Inputting an embedding vector representation determined according to the text vector representation and the image vector representation into a self-attention model, and calculating a matching matrix according to the product between the embedding vector representation and the transposed matrix of the embedding vector representation; Global interaction information between the text information of the target text and the image information of the scene picture is determined based on the product of the matching matrix and the embedding vector representation.

6. The image-text joint error correction method according to claim 4, characterized in that: The determining, based on the global interaction information and the first normalization information, a multimodal vector representation including text information and image information includes: Adding the global interaction information and the first normalized information to obtain first summed information; After inputting the first summation information into a fully connected layer for processing, normalizing the output result of the fully connected layer to obtain second normalized information; The first summation information and the second normalization information are added to obtain the multimodal vector representation containing text information and image information.

7. The image-text joint error correction method according to any one of claims 1 to 6, characterized in that: Determining the error correction results for the target text and the scene image according to the multimodal vector representation includes: Connecting the multimodal vector representation to a fully connected layer to obtain a first feature vector, wherein the first feature vector includes a vector representation corresponding to each word in the target text and each image object in the scene image; Determining, based on the first feature vector, an error correction operation corresponding to each word in the target text and each image object in the scene image; An error correction result is calculated according to the error correction operation and the first feature vector, so as to perform error correction processing on the target text and / or the scene image according to the error correction result.

8. A combined image and text error correction device, characterized in that: The device comprises: An acquisition unit, configured to acquire text data and image data to be processed, wherein the text data includes target text and the image data includes scene images; a first extraction unit, configured to extract a text vector representation of the text data, wherein the text vector representation includes text information of the target text; a second extraction unit, configured to extract a picture vector representation of the image data, wherein the picture vector representation includes image information of the scene picture; The acquisition unit is further configured to acquire a position vector representation and a type vector representation, wherein the position vector representation is used to mark the position of each word in the text data, and the type vector representation is used to distinguish between text type and image type; a computing unit, configured to compute a multimodal vector representation containing text information and image information based on the text vector representation and the image vector representation, comprising: computing the multimodal vector representation containing text information and image information based on a self-attention model based on the text vector representation, the image vector representation, the position vector representation, and the type vector representation, wherein the multimodal vector representation contains a semantic matching relationship between the text information and the image information; a determination unit, configured to determine, when the target text is used to express the scene image, an error correction result for the target text and the scene image according to the multimodal vector representation, including: determining, through a fully connected layer and an error correction mode identification module in the error correction module, an error correction result corresponding to an error correction operation according to the multimodal vector representation, if the error correction operation is error-free, the error correction result is outputting a word corresponding to the error-free error correction operation; if the error correction operation is a deletion operation, the error correction result is deleting the word corresponding to the deletion operation in the target text; if the error correction operation is a modification operation, the error correction result is changing the word corresponding to the modification operation in the target text to a predicted word, or changing the image target corresponding to the modification operation in the scene image to a predicted image target; The determination unit is also used to assist in error correction based on voice and video modality information.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps of the image-text joint error correction method according to any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to execute the steps of the image-text joint error correction method according to any one of claims 1 to 7 by calling the computer program stored in the memory.

11. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the image-text joint error correction method according to any one of claims 1 to 7 are implemented.

Images