Answer text generation method and device and storage medium

Abstract

The invention provides an answer text generation method and device and a storage medium, relates to the technical field of artificial intelligence, and can execute a text information extraction strategy matched with a shielding degree according to the shielding degree of a to-be-recognized image, and extract an answer text for answering a question according to text information of the to-be-recognized image. The method comprises the following steps: acquiring a to-be-recognized image and a question related to the content of the to-be-recognized image; determining a corresponding text information extraction strategy according to the shielding degree of the to-be-recognized image; and based on the text information extraction strategy, identifying the to-be-identified image to obtain text information of the to-be-identified image, and extracting an answer text for answering the question from the text information.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device and storage medium for generating an answer text. Background Art

[0002] With the development of artificial intelligence technology, image question answering technology, as an important technology in the intersection of computer vision and natural language processing, is increasingly attracting widespread attention from academia and industry. This technology can automatically generate answers that are consistent with the context and / or text information contained in the image by understanding the image content and the natural language associated with it.

[0003] At present, image question answering technology usually needs to rely on relatively simple image processing algorithms. Therefore, when faced with complex scenes or objects, or when there are seals or watermarks on the pictures, the existing image question answering technology will generate overly scattered or fragmented text in the process of converting the visual information in the pictures into text information, making the recognition results inaccurate. Summary of the invention

[0004] The present application provides a method, device and storage medium for generating an answer text, which can perform a text information extraction strategy that matches the occlusion degree of the image to be identified, and obtain an answer text for answering the question based on the text information extraction, thereby improving the accuracy of the answer text.

[0005] In order to achieve the above purpose, this application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a method for generating an answer text, the method comprising: obtaining an image to be identified and a question related to the content of the image to be identified; determining a corresponding text information extraction strategy based on the degree of occlusion of the image to be identified; based on the text information extraction strategy, identifying the image to be identified, obtaining text information of the image to be identified, and extracting an answer text for answering the question from the text information.

[0007] According to the above technical means, the present application can determine different text information extraction strategies according to the degree of occlusion of the image to be identified, thereby implementing a text information extraction strategy that is suitable for the image to be identified. According to different text information extraction strategies, simple images that are not occluded and complex images that are partially occluded or completely occluded can be distinguished and processed, avoiding an overly simple processing flow that reduces the accuracy of complex images and an overly complex processing flow that makes simple images too time-consuming.

[0008] In a possible implementation, the text information extraction strategy includes at least one of the following: a text information extraction strategy corresponding to an unobstructed image; a text information extraction strategy corresponding to a partially obstructed image; and a text information extraction strategy corresponding to a completely obstructed image.

[0009] According to the above technical means, the present application can distinguish images into unobstructed images, partially obstructed images, and completely obstructed images according to the degree of occlusion, so that the text information extraction strategy can fully cover various different images, thereby executing a more appropriate text information extraction process for different images.

[0010] In a possible implementation, text recognition is performed on the image to be recognized to obtain first text information; based on the correlation between any two characters in the first text information, each character in the first text information is sorted to obtain second text information corresponding to the image to be recognized; wherein the correlation is used to reflect the coherence between the two characters.

[0011] According to the above technical means, the present application can adjust the order of the first text information to obtain the second text information. The second text information is the most fluent arrangement of the first text information, making the results in the subsequent use process more accurate and improving the readability of the text.

[0012] In a possible implementation, the relative positions of the target text pair and other characters in the first text information except the target text pair are determined in sequence according to the Viterbi algorithm based on the target text alignment center to obtain the second text information.

[0013] According to the above technical means, the present application can implement the adjustment of text information according to the Viterbi algorithm, and provides a complete process for the order adjustment of text information.

[0014] In one possible implementation, based on the threshold processing technology, the occluded image of the image to be identified is removed to obtain the image to be identified after the occlusion is removed; through the preset multimodal artificial intelligence model, text recognition is performed on the image to be identified after the occlusion is removed to obtain the text information of the image to be identified.

[0015] According to the above technical means, the present application can remove the blocked part of the image to be recognized, thereby improving the accuracy of the text recognition results.

[0016] In a possible implementation, the image to be identified is split to obtain a completely occluded partial image and an unoccluded partial image; based on a text information extraction strategy corresponding to the unoccluded image, the completely occluded partial image is processed to obtain third text information; based on a text information extraction strategy corresponding to the completely occluded image, the unoccluded partial image is processed to obtain fourth text information; based on the third text information and the fourth text information, the text information corresponding to the image to be identified is determined.

[0017] According to the above technical means, the present application can split the partially obscured image into a completely obscured partial image and an unobscured partial image. This allows the complex image part in the image to be identified to undergo a more complex and time-consuming text information extraction strategy, while the simple graphics undergo a relatively simple text information extraction strategy. This reduces resource utilization while improving the accuracy of the extraction results.

[0018] In one possible implementation, when the occlusion degree is unobstructed, the type of information required for the question is determined, and based on the artificial intelligence model, the text information of the image to be recognized is extracted according to the information type to obtain the answer text for answering the question; when the occlusion degree is complete occlusion, the text information of the image to be recognized is extracted according to the question through a preset multimodal artificial intelligence model to obtain the answer text for answering the question.

[0019] According to the above technical means, this application can

[0020] In a second aspect, the present application provides a device for generating an answer text, the device comprising: a communication unit and a processing unit.

[0021] A communication unit, used to obtain an image to be identified and questions related to the content of the image to be identified;

[0022] A processing unit, used to determine a corresponding text information extraction strategy according to the occlusion degree of the image to be identified;

[0023] The processing unit is also used to identify the image to be identified based on the text information extraction strategy, obtain the text information of the image to be identified, and extract the answer text for answering the question from the text information.

[0024] In a possible implementation, the text information extraction strategy includes at least one of the following: a text information extraction strategy corresponding to an unobstructed image; a text information extraction strategy corresponding to a partially obstructed image; and a text information extraction strategy corresponding to a completely obstructed image.

[0025] In one possible implementation, the processing unit is further used to perform text recognition on the image to be recognized to obtain first text information; based on the correlation between any two characters in the first text information, each character in the first text information is sorted to obtain second text information corresponding to the image to be recognized; wherein the correlation is used to reflect the coherence between the two characters.

[0026] In a possible implementation, the processing unit is further used to determine the relative positions of other characters in the first text information except the target character pair and the target character pair in sequence according to the Viterbi algorithm with the target character alignment center to obtain the second text information.

[0027] In one possible implementation, the processing unit is also used to remove the occluded image of the image to be identified based on threshold processing technology to obtain the image to be identified with the occlusion removed; and to perform text recognition on the image to be identified with the occlusion removed by using a preset multimodal artificial intelligence model to obtain text information of the image to be identified.

[0028] In one possible implementation, the processing unit is further used to split the image to be identified to obtain a completely occluded partial image and an unoccluded partial image; based on a text information extraction strategy corresponding to the unoccluded image, the completely occluded partial image is processed to obtain third text information; based on a text information extraction strategy corresponding to the completely occluded image, the unoccluded partial image is processed to obtain fourth text information; based on the third text information and the fourth text information, the text information corresponding to the image to be identified is determined.

[0029] In one possible implementation, the processing unit is also used to determine the type of information required for the question when the occlusion degree is unobstructed, and based on the artificial intelligence model, extract the text information of the image to be recognized according to the information type to obtain the answer text for answering the question; when the occlusion degree is complete occlusion, the text information of the image to be recognized is extracted according to the question through a preset multimodal artificial intelligence model to obtain the answer text for answering the question.

[0030] In a third aspect, the present application provides an electronic device comprising: a processor and a communication interface; the communication interface and the processor are coupled, and the processor is used to run a computer program or instructions to implement a method for generating an answer text as described in the first aspect and any possible implementation of the first aspect.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal, the terminal executes the method for generating an answer text as described in the first aspect and any possible implementation of the first aspect.

[0032] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute a method for generating an answer text as described in the first aspect and any possible implementation of the first aspect.

[0033] In a sixth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run a computer program or instructions to implement a method for generating an answer text as described in the first aspect and any possible implementation of the first aspect.

[0034] Specifically, the chip provided in the present application also includes a memory for storing computer programs or instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A process for generating an answer text provided in an embodiment of the present application Figure 1 ;

[0036] Figure 2 A process for generating an answer text provided in an embodiment of the present application Figure 2 ;

[0037] Figure 3 A schematic diagram of a method for generating an answer text provided in an embodiment of the present application Figure 1 ;

[0038] Figure 4 A schematic diagram of a method for generating an answer text provided in an embodiment of the present application Figure 2 ;

[0039] Figure 5 A schematic diagram of a method for generating an answer text provided in an embodiment of the present application Figure 3 ;

[0040] Figure 6 A schematic diagram of a method for generating an answer text provided in an embodiment of the present application Figure 4 ;

[0041] Figure 7 A process for generating an answer text provided in an embodiment of the present application Figure 3 ;

[0042] Figure 8 A schematic diagram of the structure of a device for generating an answer text provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following is a detailed description of the answer text generation method, device and storage medium provided in the embodiments of the present application in conjunction with the accompanying drawings.

[0044] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0045] The terms "first" and "second" and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0046] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0047] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0048] In the description of the present application, unless otherwise specified, “plurality” means two or more.

[0049] The following explains the terms involved in the embodiments of the present application to facilitate the reader's understanding.

[0050] (1) Image question answering technology: It combines knowledge and methods from multiple fields and disciplines, such as natural language understanding, computer vision, and reasoning. It processes image information through image recognition technology, and combines it with natural language processing technology to understand user questions. It uses deep learning and other technologies to perform reasoning and obtain answers described in natural language.

[0051] (2) Viterbi algorithm: A dynamic programming algorithm used to find the most likely hidden state sequence given a given observation sequence. It is widely used in the decoding problem of hidden Markov models (HMMs), such as speech recognition and natural language processing. The algorithm calculates the probability of all possible state transitions at each time step and retains the optimal path to each state, ultimately finding the global optimal hidden state sequence.

[0052] The above is a brief introduction to some concepts involved in the embodiments of this application.

[0053] At present, image question answering technology usually needs to rely on relatively simple image processing algorithms. Therefore, when faced with complex scenes or objects, or when there are seals or watermarks on the pictures, the existing image question answering technology will generate overly scattered or fragmented text in the process of converting the visual information in the pictures into text information, making the recognition results inaccurate.

[0054] In view of this, an embodiment of the present application provides a method for generating answer text, which is used to determine a text extraction strategy that matches the image to be identified based on the degree of occlusion of the image, and extract the answer text used to answer the question based on the text information, thereby improving the accuracy of the image question and answer results.

[0055] The following describes the method for generating the answer text provided in the embodiments of the present application in conjunction with the accompanying drawings. Among them, the actions, terms, etc. involved in the various embodiments of the present application can refer to each other without limitation. The message name or parameter name in the message exchanged between the various devices in the embodiments of the present application is only an example, and other names can also be used in the specific implementation without limitation. The actions involved in the various embodiments of the present application are only an example, and other names can also be used in the specific implementation, such as: "included in" in the embodiments of the present application can also be replaced by "carried on" or "carried in", etc.

[0056] like Figure 1 As shown, the embodiment of the present application proposes a method for generating an answer text, the method comprising:

[0057] S101: Obtain an image to be identified and questions related to the content of the image to be identified.

[0058] The image to be identified includes one or more text information, and one or more text information includes text information for answering the question.

[0059] In one application scenario, the image to be recognized is a photo of a diary, and the question is "What is the date of this diary?" The photo contains multiple text information such as the body of the diary, the title, and the date, and the text information used to answer the question is the text information corresponding to the date.

[0060] S102: Determine a corresponding text information extraction strategy according to the occlusion degree of the image to be recognized.

[0061] The degree of occlusion includes completely occluded, partially occluded, and unoccluded.

[0062] In one example, the image to be identified is identified through image processing technology and adversarial networks to determine the degree of occlusion. According to the degree of occlusion of the image to be identified, a corresponding text information extraction strategy is determined.

[0063] The text information extraction strategy includes at least one of the following: a text information extraction strategy corresponding to an unobstructed image, a text information extraction strategy corresponding to a partially obstructed image, and a text information extraction strategy corresponding to a completely obstructed image.

[0064] In one application scenario, an image to be identified is identified through an open source computer vision library (OpenCV) and an adversarial network to determine the degree of occlusion of the image to be identified.

[0065] Among them, OpenCV is an image processing library that provides image processing and computer vision algorithms, as well as functions and classes for image and video processing. Adversarial network is a neural network architecture based on deep learning. It is trained through competition and adversarial methods and can perform data enhancement, image editing and conversion, data repair and other operations on images.

[0066] S103: Based on the text information extraction strategy, the image to be identified is identified to obtain text information of the image to be identified, and answer text for answering the question is extracted from the text information.

[0067] In one example, according to different degrees of occlusion, the text information extraction strategy can be divided into Case 1 to Case 3. According to different text information extraction strategies, text extraction is performed on the image to be recognized to obtain the text information of the image to be recognized, and the answer text for answering the question is extracted from the text information.

[0068] Case 1: The occlusion degree is not occluded.

[0069] Based on the optical character recognition technology, the image to be recognized is subjected to text recognition to obtain the first text information. Based on the correlation between any two characters in the first text information, each character in the first text information is sorted to obtain the second text information corresponding to the image to be recognized, that is, the text information of the image to be recognized. According to the text information of the image to be recognized, the answer text for answering the question is extracted. For details, refer to the description of the second embodiment, which will not be repeated here.

[0070] The correlation degree is used to reflect the coherence between two characters. The first text information is the text information contained in the image to be recognized.

[0071] Case 2: The degree of occlusion is complete occlusion.

[0072] Based on the threshold processing technology, the occlusion of the recognition image is removed to obtain the image to be recognized after the occlusion is removed. By presetting the multimodal artificial intelligence model, the text recognition is performed on the image to be recognized after the occlusion is removed to obtain the first text information, that is, the text information of the image to be recognized. According to the text information of the image to be recognized, the answer text for answering the question is extracted. The specific description of the reference embodiment 3 is not repeated here.

[0073] Among them, the multimodal artificial intelligence model is an artificial intelligence model that can simultaneously process and fuse multiple data types (such as text, images, audio, video, etc.), and can realize functions such as data preprocessing, feature extraction, cross-modal association learning, fusion and generation. It has stronger understanding ability, more accurate generation ability, and better generalization ability.

[0074] Case 3: The degree of occlusion is partial occlusion.

[0075] The image to be identified is split to obtain a completely blocked partial image and an unblocked partial image. Based on the text information extraction strategy corresponding to the unblocked image, the completely blocked partial image is processed to obtain third text information. Based on the text information extraction strategy corresponding to the completely blocked image, the unblocked partial image is processed to obtain fourth text information. Answer texts for answering questions are extracted from the third text information and the fourth text information respectively, and the two answer texts for answering questions are structurally integrated to obtain answer texts for answering questions.

[0076] According to the above technical means, the present application can determine different text information extraction strategies according to the degree of occlusion of the image to be identified, thereby implementing a text information extraction strategy that is suitable for the image to be identified. According to different text information extraction strategies, simple images that are not occluded and complex images that are partially occluded or completely occluded can be distinguished and processed, avoiding an overly simple processing flow that reduces the accuracy of complex images and an overly complex processing flow that makes simple images too time-consuming.

[0077] In another embodiment (Example 2), as Figure 2 As shown, a method for generating an answer text is provided for an embodiment of the present application, which is used to extract text from an image to be identified based on a text information extraction strategy when the occlusion degree is not occluded, obtain text information of the image to be identified, and extract an answer text for answering the question from the text information. The method includes S201 to S203.

[0078] S201: Perform text recognition on an image to be recognized to obtain first text information.

[0079] In one example, text recognition is performed on the image to be recognized based on optical character recognition (OCR) technology to obtain first text information.

[0080] S202: Sort each character in the first text information based on the correlation between any two characters in the first text information to obtain second text information corresponding to the image to be recognized.

[0081] Among them, the correlation is used to reflect the fluency between two characters.

[0082] In one example, the relative positions of other characters in the first text information except the target character pair and the target character pair are determined in sequence according to the Viterbi algorithm with the target character pair aligned at the center to obtain the second text information.

[0083] In one application scenario, the first text information is used as an observation sequence O = (o1, o2, ..., o t ), correspondingly, the attributes, part of speech, named entity, and semantic role of each character in the first text information are used as a hidden state sequence Q = (q1, q2, ..., q t ), model parameters λ=(A, B, π). According to the recursive formula of the Viterbi algorithm, the correlation between any two characters is calculated. After the recursive calculation of the characters contained in the first text information is completed, the two characters with the highest correlation among all the characters can be obtained. Based on the two characters with the highest correlation, the recursive calculation process is backtracked, and the characters with the highest correlation among the remaining characters are added. After all the characters are backtracked, the second text information is obtained, which is the most fluent text information.

[0084] The observation sequence represents the original arrangement order of the first text information, o1, o2, ..., o t Represent each word and / or character respectively. The hidden state sequence represents various attributes or features of each word, such as part of speech, named entity (e.g., name of a person, place name, organization name, etc.), semantic role (e.g., subject, predicate, object, etc.) and emotional emotion, etc. A hidden state in the hidden sequence corresponds to a word in the observation sequence. In the model parameters λ=(A,B,π), A is the state transition probability matrix, B is the observation probability matrix, and π is the initial state probability distribution. The initial value is usually set to 0. For details, please refer to the prior art and will not be elaborated here.

[0085] The recursive formula includes formula 1 and formula 2.

[0086]

[0087] Among them, δ t (i) represents the maximum probability of being in state i at time t, which can be understood as the maximum probability of adding the next word based on the existing observation sequence. N represents the number of hidden states, a ji is a part of the model parameter A, which indicates the probability of transitioning from state j to state i. The greater the probability of transitioning from state j to state i, the greater the degree of association between the text added to state i and state j compared to state j. i (o t ) is part of the model parameter B, indicating the generation of observation value o in state it The probability of can be understood as the probability of state i generating the corresponding text information. Time t is the time point in the recursive process, which can also be understood as the number of recursive iterations. It indicates the optimal state at time t-1 before the state i at time t. max indicates the maximum value of the function, and argmax indicates the value of each parameter when the function is at the maximum value.

[0088] Furthermore, through ψ t (i) Used to record the path to facilitate backtracking to find the optimal hidden state sequence. t (i) indicates that the state is i at time t.

[0089] like Figure 3 As shown, S is the initial state, A1, A2, A3 are the states of the second step, B1, B2, B3 are the states of the third step, C1, C2, C3 are the states of the fourth step, and E is the state with the highest probability at the end of the recursive process. According to state E, backtracking is performed, and each time backtracking to the state with the highest current probability, the state with the highest overall probability can be obtained after the backtracking is completed.

[0090] The state represents the state of the text arrangement order or the state of the hidden sequence corresponding to the text arrangement order.

[0091] The termination condition of the above recursive process is to obtain the maximum δ at the final time T. T (i) is the final maximum probability, that is, the probability between the two characters with the greatest correlation. T (i) and ψ t (i) Backtracking the recursive process to obtain the optimal hidden state sequence, and based on the optimal hidden state sequence, obtain the observation sequence, i.e., the second text information.

[0092] S203: Extract answer text for answering the question based on the text information of the image to be recognized.

[0093] In one example, the information type (key) required for a question is determined through prompt engineering (prompt), and based on an artificial intelligence model, the text information of the image to be recognized is extracted according to the information type to obtain the answer text (value) used to answer the question.

[0094] Among them, prompt word engineering is used to guide artificial intelligence to accurately understand task requirements and intentions, thereby generating outputs that meet expectations and improving the performance and performance of the model. Prompt word engineering can be used to quickly extract key-value pairs to obtain answer text for answering questions.

[0095] In another example, through the prefix tuning method, before inputting tokens into each layer of the artificial intelligence model, a section of virtual tokens related to the question is constructed as a prefix, providing the artificial intelligence model with guidance information related to the question, reducing the artificial intelligence model's attention to irrelevant information, thereby more accurately extracting content related to the question from the text and obtaining an answer text that better meets the requirements of the question.

[0096] In one application scenario, the artificial intelligence model can be an enhanced representation through knowledge integration (ERNIE) pre-training model. During the pre-training phase of ERNIE, the network structure of ERNIE is modified, and the characteristic of ERNIE network sharing weights is used to solve the problem of catastrophic data forgetting in the pre-training process of conventional multi-task models. Random noise is added to the embedding layer, and tokens are replaced, etc. to modify the ERNIE network structure and obtain an adversarial learning pre-training framework.

[0097] like Figure 4The figure shows the network structure of ERNIE. Large-scale text data and knowledge graph are used as input sources and input into the embedding layer. After being processed by the embedding layer, a comprehensive vector representation that integrates text semantics, location, and knowledge graph-related knowledge is obtained. The comprehensive vector is input into transformer block 1 for continuous learning, and then outputs a universal representation at transformer block k, which is then input into transformer block k+1. The process from transformer block k+1 to transformer block n is the process of building a language understanding network or a language generation network. After completing the construction of the language understanding network or the language production network, the ERNIE network structure can be fine-tuned, zero-short learned, or few-shot learned through the language understanding network or the language generation network to obtain an artificial intelligence model suitable for this field.

[0098] Based on the above ERNIE network structure, the ERNIE network structure is modified to obtain the adversarial learning pre-training framework. In the working process of the embedding layer, the embedding operation is performed, and the embedding operation includes the process of task identifier embedding, position embedding, sentence embedding, and token embedding. Taking text information as an example, before the embedding process, the text information is segmented. For example, the text information is "Today's weather is really good", which is divided into three tokens: "Today", "Weather", and "Really good". Multiple embedding operations are performed based on tokens.

[0099] like Figure 5 As shown, after the embedding operation is completed, the disturbance, that is, the noise, in the adversarial learning process is determined based on the feature vector after the embedding operation. The embedding operation includes the following operations 1 to 4. That is, the operation is performed in the order of operation 1, operation 2, operation 3, and operation 4.

[0100] Operation 1: Embed the task ID.

[0101] In one example, when an artificial intelligence model faces multiple different natural language processing tasks, in order to distinguish the specific task to which the current input data belongs, the task identifier is converted into a vector form and embedded into the input data, so that the artificial intelligence model can adapt to different task goals in the subsequent processing process. task .

[0102] Operation 2: Position embedding.

[0103] In one example, multiple word units embedded with task identifiers need to distinguish the position of each word unit in the original text information, so the position information of each word unit is encoded into a vector form and embedded into the word unit to obtain E0, E1, E2, and E3.

[0104] Operation 3: Sentence embedding.

[0105] In one example, the word units belonging to the same sentence are determined among multiple word units. For example, in the position embedding process, E0 and E1 are one sentence, and E2 and E3 are one sentence. Then, E0 and E1 are obtained by sentence embedding. A , E2 and E3 are embedded into E B .

[0106] Operation 4: word embedding.

[0107] In one example, the vector information after word embedding needs to be formatted according to the input requirements of the artificial intelligence model, and special tags are added to assist the subsequent artificial intelligence model processing and understanding of sentence structure. For example, the cls tag is added to represent the overall characteristics of the text information, and the sep tag is used to distinguish different sentences. A Belong to the same sentence, need to add E in front of it cls To mark the beginning of a sentence, add E at the end. sep As a marker for the end of a sentence.

[0108] Further, such as Figure 6As shown, noise is added to the feature vector after the embedding operation, and the feature vector with noise is input into the ERNIE model based on the transformer architecture to perform an adversarial multi-head attention mechanism. After being processed by the adversarial multi-head attention mechanism, an optimized and integrated feature vector is output. The optimized and integrated feature vector is input into the feedforward neural network for multi-level processing such as linear transformation and nonlinear activation to obtain multiple feature vectors H0, H1...H5. The multiple feature vectors integrate the features transmitted from the adversarial multi-head attention mechanism and the features mined and optimized by the feedforward neural network itself. Based on multiple feature vectors, the loss can be calculated, so that the artificial intelligence model can be trained and optimized according to the loss.

[0109] According to the above technical means, the present application can improve the accuracy of text extraction results and answer text extraction results for unobstructed images through the Viterbi algorithm and adversarial learning technology.

[0110] In another embodiment (Example 3), as Figure 7 As shown, a method for generating an answer text is provided for an embodiment of the present application, which is used to remove the occlusion when the occlusion degree is completely occluded, extract the text information of the image to be recognized according to the question by presetting a multimodal artificial intelligence model, and obtain the answer text for answering the question. The method includes S701 to S703.

[0111] S701 : Based on the threshold processing technology, remove the occluded image of the image to be recognized, so as to obtain the image to be recognized with the occlusion removed.

[0112] In one example, the color of the occluding image is determined, and the occluding image is removed by using a threshold processing technique using a color channel of the occluding image.

[0113] S702: Perform text recognition on the image to be identified after removing the occlusions by using a preset multimodal artificial intelligence model to obtain text information of the image to be identified.

[0114] Among them, the multimodal artificial intelligence model can be one or more of the existing multimodal artificial intelligence models, which is not limited here.

[0115] S703: Perform text recognition on the image to be identified after removing the occlusions by using a preset multimodal artificial intelligence model to obtain first text information, that is, text information of the image to be identified.

[0116] According to the above technical means, the present application can identify text information of the image to be identified that is completely occluded, and extract the answer to the question according to the question. Through the strong performance and computing power of the multimodal artificial intelligence model, the accuracy of the recognition and extraction results of the completely occluded image to be identified can be improved.

[0117] It is understandable that the above-mentioned method for generating the answer text can be implemented by a device for generating the answer text. In order to realize the above-mentioned functions, the device for generating the answer text includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed herein, the embodiments disclosed in the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments disclosed in the present application.

[0118] The embodiments disclosed in the present application can divide the functional modules of the device for generating the answer text generated by the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments disclosed in the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0119] Figure 8 A schematic diagram of a device for generating an answer text provided in an embodiment of the present application. Figure 8 As shown, the answer text generating device 80 can be used to perform Figure 1-Figure 7 The answer text generation method shown in FIG. 8 includes a communication unit 801 and a processing unit 802 .

[0120] The communication unit 801 is used to obtain the image to be identified and questions related to the content of the image to be identified.

[0121] The processing unit 802 is used to determine a corresponding text information extraction strategy according to the occlusion degree of the image to be recognized.

[0122] The processing unit 802 is further used to identify the image to be identified based on the text information extraction strategy, obtain the text information of the image to be identified, and extract the answer text for answering the question from the text information.

[0123] In a possible implementation, the text information extraction strategy includes at least one of the following: a text information extraction strategy corresponding to an unobstructed image, a text information extraction strategy corresponding to a partially obstructed image, and a text information extraction strategy corresponding to a completely obstructed image.

[0124] In a possible implementation, the processing unit 802 is further configured to perform text recognition on the image to be recognized to obtain first text information. Based on the correlation between any two characters in the first text information, each character in the first text information is sorted to obtain second text information corresponding to the image to be recognized. The correlation is used to reflect the coherence between the two characters.

[0125] In a possible implementation, the processing unit 802 is further configured to determine the relative positions of other characters in the first text information except the target character pair and the target character pair in sequence according to the Viterbi algorithm with the target character alignment center to obtain the second text information.

[0126] In a possible implementation, the processing unit 802 is further configured to remove the occluded image of the image to be identified based on a threshold processing technique to obtain the image to be identified after the occlusion is removed. By presetting a multimodal artificial intelligence model, text recognition is performed on the image to be identified after the occlusion is removed to obtain text information of the image to be identified.

[0127] In a possible implementation, the processing unit 802 is further used to split the image to be identified to obtain a completely blocked partial image and an unblocked partial image. Based on the text information extraction strategy corresponding to the unblocked image, the completely blocked partial image is processed to obtain third text information. Based on the text information extraction strategy corresponding to the completely blocked image, the unblocked partial image is processed to obtain fourth text information. Based on the third text information and the fourth text information, the text information corresponding to the image to be identified is determined.

[0128] In a possible implementation, the processing unit 802 is further used to determine the type of information required for the question when the occlusion degree is not occluded, and extract the text information of the image to be recognized according to the information type based on the artificial intelligence model to obtain the answer text for answering the question. When the occlusion degree is completely occluded, the text information of the image to be recognized is extracted according to the question through a preset multimodal artificial intelligence model to obtain the answer text for answering the question.

[0129] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0130] The present disclosure also provides a computer-readable storage medium having instructions stored thereon. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for generating an answer text provided in the above-mentioned embodiment of the present disclosure.

[0131] The embodiments of the present disclosure also provide a computer program product containing instructions, which, when executed on an electronic device, enables the electronic device to execute the method for generating an answer text provided by the embodiments of the present disclosure.

[0132] Among them, the computer readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a register, a hard disk, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above, or any other form of computer readable storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (ASIC). In the embodiments of the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0133] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for generating an answer text, characterized in that: The method comprises: Obtaining an image to be identified and questions related to the content of the image to be identified; Determining a corresponding text information extraction strategy according to the occlusion degree of the image to be identified; Based on the text information extraction strategy, the image to be identified is identified to obtain text information of the image to be identified, and an answer text for answering the question is extracted from the text information.

2. The method according to claim 1, characterized in that: The text information extraction strategy includes at least one of the following: Text information extraction strategy corresponding to the unoccluded image; Strategies for extracting text information corresponding to partially occluded images; Text information extraction strategy corresponding to completely occluded images.

3. The method according to claim 2, characterized in that The occlusion degree is that the image is not occluded, and the image to be identified is identified based on the text information extraction strategy to obtain text information of the image to be identified, including: Performing text recognition on the image to be recognized to obtain first text information; Based on the correlation between any two characters in the first text information, each character in the first text information is sorted to obtain second text information corresponding to the image to be recognized; wherein the correlation is used to reflect the coherence between the two characters.

4. The method according to claim 3, characterized in that The step of sorting each character in the first text information based on the correlation between any two characters in the first text information includes: Based on the target text alignment center, the relative positions of other characters in the first text information except the target text pair and the target text pair are determined in sequence according to the Viterbi algorithm to obtain the second text information.

5. The method according to claim 2, characterized in that: The occlusion degree is completely occluded, and the image to be identified is identified based on the text information extraction strategy to obtain text information of the image to be identified, including: Based on the threshold processing technology, the occluded image of the image to be identified is removed to obtain the image to be identified with the occluded image removed; By presetting a multimodal artificial intelligence model, text recognition is performed on the image to be identified after the occlusion is removed to obtain text information of the image to be identified.

6. The method according to any one of claims 2 to 5, characterized in that: The occlusion degree is partial occlusion, and the image to be identified is identified based on the text information extraction strategy to obtain text information of the image to be identified, including: Splitting the image to be identified to obtain a completely blocked portion of the image and an unblocked portion of the image; Based on the text information extraction strategy corresponding to the unobstructed image, the completely obstructed partial image is processed to obtain third text information; Based on the text information extraction strategy corresponding to the completely blocked image, the unblocked portion of the image is processed to obtain fourth text information; Based on the third text information and the fourth text information, text information corresponding to the image to be recognized is determined.

7. The method according to any one of claims 1 to 5, characterized in that The degree of shielding is not shielded or completely shielded, and the extracting of answer text for answering the question from the text information includes: In the case where the occlusion degree is not occluded, determining the type of information required for the question, and extracting text information of the image to be identified according to the information type based on an artificial intelligence model to obtain an answer text for answering the question; When the occlusion degree is complete occlusion, the preset multimodal artificial intelligence model is used to extract the text information of the image to be identified according to the question to obtain an answer text for answering the question.

8. A device for generating an answer text, characterized in that: The device comprises: a communication unit and a processing unit; The communication unit is used to obtain an image to be identified and questions related to the content of the image to be identified; The processing unit is used to determine a corresponding text information extraction strategy according to the occlusion degree of the image to be identified; The processing unit is further used to identify the image to be identified based on the text information extraction strategy, obtain the text information of the image to be identified, and extract the answer text for answering the question from the text information.

9. An electronic device, characterized in that: include: A memory and a processor; the memory and the processor are coupled; The memory is used to store instructions executable by the processor; When the processor executes the instructions, it performs the method for generating the answer text as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the method for generating an answer text as described in any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises computer program instructions, which, when executed by a processor, implement the method for generating an answer text as described in any one of claims 1 to 7.

Images