Content recognition method, apparatus, device, and storage medium

Abstract

The present disclosure provides a content recognition method and device, equipment and storage medium, relates to the technical field of artificial intelligence, and can be applied to various scenes such as cloud technology, artificial intelligence and vehicle-mounted devices. The method comprises the following steps: obtaining a question image and an answer text; processing the answer text and the corresponding question image by a content recognition model to obtain content recognition sequence information corresponding to the question image; and determining a content recognition result based on the content recognition sequence information corresponding to the question image. Thus, the generation of misjudgments is reduced, and the accuracy of content recognition is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computers, and particularly relates to a content recognition method and device, equipment and a storage medium. BACKGROUND

[0002] With the development of artificial intelligence, artificial intelligence technology is applied to more and more fields. For example, in the field of education, intelligent correction technology for student homework or test papers has appeared, which can reduce the pressure of teachers correcting homework and exams, and can also assist students in learning.

[0003] In related technologies, the OCR (optical character recognition) technology is mainly used to recognize written content, and then the recognized text is compared with the standard answer, and the intelligent correction is performed according to the comparison result. However, when the writing of the written content is not standardized, the intelligent correction based on the OCR technology is easy to produce misjudgment, which reduces the accuracy of content recognition. SUMMARY

[0004] The present disclosure provides a content recognition method, device, equipment and storage medium to solve at least one technical problem in the prior art.

[0005] In one aspect, the present disclosure provides a content recognition method, comprising:

[0006] obtaining a to-be-recognized image, the to-be-recognized image comprising a stem image and a problem-solving image corresponding to the stem image;

[0007] obtaining an answer text matching a question description content in the stem image;

[0008] processing the answer text and the corresponding problem-solving image through a content recognition model to obtain content recognition sequence information corresponding to the problem-solving image; the content recognition sequence information representing a correspondence relationship between each text character in the answer text and at least one pixel block in the corresponding problem-solving image based on a character dimension;

[0009] determining a content recognition result based on the content recognition sequence information corresponding to the problem-solving image.

[0010] In another aspect, a content recognition device is also provided, comprising:

[0011] a first obtaining module configured to obtain a to-be-recognized image, the to-be-recognized image comprising a stem image and a problem-solving image corresponding to the stem image;

[0012] a second obtaining module configured to obtain an answer text matching a question description content in the stem image;

[0013] The identification processing module is configured to process the answer text and the corresponding question image by using a content identification model to obtain content identification sequence information corresponding to the question image; the content identification sequence information represents a correspondence relationship between each text character in the answer text and at least one pixel block in the corresponding question image based on a character dimension.

[0014] The identification result determination module is configured to determine a content identification result based on the content identification sequence information corresponding to the question image.

[0015] In another aspect, an electronic device is provided, which includes a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or the at least one program is loaded and executed by the processor to implement any of the above-described methods.

[0016] In another aspect, a computer-readable storage medium is provided. The storage medium stores at least one instruction or at least one program. The at least one instruction or the at least one program is loaded and executed by a processor to implement any of the above-described methods.

[0017] In another aspect, a computer program product or computer program is provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium. The processor executes the computer instructions to cause the computer device to perform any of the above-described methods.

[0018] The content identification method, device, equipment and storage medium provided by the present disclosure have the following technical effects:

[0019] The embodiment of the disclosure acquires a to-be-recognized image, the to-be-recognized image includes a stem image and a problem-solving image corresponding to the stem image; acquires an answer text matching a question description content in the stem image; processes the answer text and the corresponding problem-solving image through a content recognition model to obtain content recognition sequence information corresponding to the problem-solving image; the content recognition sequence information represents a corresponding relationship between each text character in the answer text and at least one pixel block in the corresponding problem-solving image based on a character dimension; and determines a content recognition result based on the content recognition sequence information corresponding to the problem-solving image. Since two modal input data, including the answer text and the corresponding problem-solving image, are input into the content recognition model, the text information of the answer text participates in each discrimination stage of content recognition, which helps to assist in improving the matching ability of the model for the problem-solving content in the problem-solving image and the answer text, reduces the generation of misjudgment, and improves the accuracy of content recognition. In addition, the content recognition sequence information corresponding to the problem-solving image is obtained by processing through the content recognition model, the content recognition sequence information represents the corresponding relationship between each text character in the answer text and at least one pixel block in the corresponding problem-solving image based on the character dimension, thereby modeling the original two-classification task model into a more fine-grained sequence labeling problem, and further improving the accuracy of content recognition. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art and the advantages thereof, a brief introduction will be given to the drawings needed in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0021] Figure 1 is an application environment schematic diagram of a content recognition method provided by the embodiment of the present disclosure;

[0022] Figure 2 is a flowchart schematic diagram of a content recognition method provided by the embodiment of the present disclosure;

[0023] Figure 3 is an exemplary handwritten picture provided by the embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of an exemplary content recognition provided by the embodiment of the present disclosure;

[0025] Figure 5 is a partial flowchart schematic diagram of a content recognition method provided by the embodiment of the present disclosure;

[0026] Figure 6 is a structure schematic diagram of a content recognition model provided by the embodiment of the present disclosure;

[0027] Figure 7 is a structural block diagram of a content recognition device provided by an embodiment of the present disclosure.

[0028] Figure 8 is a structural block diagram of a content recognition device provided by an embodiment of the present disclosure.

[0029] Figure 9 is a hardware structure schematic diagram of a device for implementing the method provided by the embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order for those skilled in the art to better understand the present disclosure scheme, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present disclosure.

[0031] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the embodiments of the present disclosure will be further described in detail below in conjunction with the drawings.

[0032] In order to facilitate the understanding of the above technical solutions and the technical effects generated by the embodiments of the present disclosure, the terms involved in the embodiments of the present disclosure are briefly introduced:

[0033] Artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0034] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0035] Computer Vision (CV) Computer vision is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further process images to make them more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, and tries to establish artificial intelligence systems that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0036] Nature Language processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, the research in this field will involve natural language, i.e. the language used in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph, etc.

[0037] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0038] Autonomous driving technology usually includes high-precision map, environment perception, behavior decision, path planning, motion control, etc. Autonomous driving technology has a wide application prospect.

[0039] With the research and progress of artificial intelligence technology, artificial intelligence technology is researched and applied in many fields, such as common smart home, smart wearable device, virtual assistant, smart speaker, smart marketing, unmanned driving, autonomous driving, unmanned aerial vehicle, robot, smart medical treatment, smart customer service, and the like. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0040] The scheme provided by the embodiments of the present disclosure relates to computer vision technology, machine learning and other technologies of artificial intelligence, which is specifically explained as follows:

[0041] The content recognition method provided by the present disclosure can be applied in an application environment as shown in Figure 1 As shown in Figure 1 The hardware environment can at least include a terminal 110 and a server 120.

[0042] The terminal 110 includes but is not limited to a mobile phone, a computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, and the like.

[0043] The server 120 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present disclosure. It should be noted that the server 120 can be implemented as a cloud server in the cloud.

[0044] In some embodiments, the server 120 can also be implemented as a node in a blockchain system. Blockchain is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. Blockchain is essentially a decentralized database, which is a series of data blocks associated using cryptography. Each data block contains information about a batch of network transactions, which is used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0045] It should be noted that in actual application, the content recognition method provided by the embodiments of the present disclosure can be implemented in the server or the terminal, or jointly implemented by the terminal and the server.

[0046] Of course, the method provided by the embodiments of the present disclosure is not limited to be used in Figure 1 the hardware environment as shown in the figure, but can also be used in other possible hardware environments, which are not limited by the embodiments of the present disclosure. For Figure 1The functions that can be achieved by the various devices of the illustrated hardware environment will be described in subsequent method embodiments, and will not be described in detail here.

[0047] Figure 2 is a flowchart of a content recognition method provided by an embodiment of the present disclosure. The present disclosure provides method operation steps as described in the embodiments or flowcharts, but can include more or fewer operation steps based on conventional or non-inventive labor. The order of steps listed in the embodiments is only one of the many execution orders, and does not represent the only execution order. The execution subject of the content recognition method can be a content recognition device provided by an embodiment of the present disclosure, or a server integrated with the content recognition device, wherein the content recognition device can be realized in the form of hardware or software. Taking the server in the execution subject as an example, as shown in Figure 1 , the method can include: Figure 2

[0048] S201: Obtain an image to be recognized, the image to be recognized including a stem image and a corresponding problem-solving image corresponding to the stem image.

[0049] For example, in the field of education, the image to be recognized can be a partial area image of a test paper, a partial area image of a practice question, etc. The image to be recognized includes a stem image and a corresponding problem-solving image corresponding to the stem image. The type of the question involved in the stem image can be a Chinese fill-in-the-blank question, a Chinese answer question, an English fill-in-the-blank question, an English fill-in-the-blank question, or other subjective questions. The problem-solving image contains the problem-solving content written by the answerer. The number of stem images and problem-solving images can be one or more, and each stem image can correspond to at least one problem-solving image.

[0050] Optionally, an initial image containing problem-solving content is obtained, which can be written by the answerer through a tool (such as a pen, an electronic pen, etc.). The initial image can be uploaded by a terminal, or obtained from a local or other storage location, which is not limited in the present disclosure. Then, the initial image is subjected to region recognition to identify the stem region and the problem-solving region. Next, the paired stem image and problem-solving image are extracted from the stem region and the problem-solving region in the initial image, respectively, as the image to be recognized. Then, the image to be recognized can be uploaded to the server through an application client or a browser running on the terminal, so that the server can execute corresponding content recognition on the obtained image to be recognized.

[0051] S203: Obtain an answer text matching the question description content in the stem image.

[0052] Optionally, after obtaining the stem image, the question description content in the stem image can be subjected to question recognition, and an answer text matching the recognized question can be obtained from, for example, a problem bank.​

[0053] For example, for the stem image a1, the description of the question is “The author of ‘Strategist’s Oath’ is __”, and the matching answer text is “Zhuge Liang”. For the stem image a2, the description of the question is “The original name of Mencius is __”, and the matching answer text is “Meng Ke”.

[0054] S205: processing the answer text and the corresponding question image by the content recognition model to obtain the content recognition sequence information corresponding to the question image.

[0055] The content recognition model is used to identify and process each pixel block in the question image and the answer text to obtain the content recognition sequence information corresponding to the question image. The content recognition sequence information represents the correspondence between each text character in the answer text and at least one pixel block in the corresponding question image based on the character dimension.

[0056] Optionally, the answer text and the corresponding question image can be input into the content recognition model for processing, and each character in the answer text is sequentially labeled based on at least one pixel block in the question image to obtain the content recognition sequence information corresponding to the question image.

[0057] For example, if the answer text m is “Tingtingjingzhi”, which contains four text characters: Ting, Ting, Jing, and Zhi, and the corresponding question image n includes pixel blocks 1, 2, 3, 4, 5, 6, 7, and 8, which can have the same pixel width. The content recognition model is called to process the answer text m and the corresponding question image n, and each of the four text characters is sequentially labeled based on at least one pixel block in the question image n. Optionally, the correspondence between the pixel blocks in the question image n and the text characters in the answer text m can be represented as {(pixel block 1 + pixel block 2, Ting), (pixel block 3, Ting), (pixel block 4 + pixel block 5, Jing), (pixel block 6 + pixel block 7 + pixel block 8, Zhi)}, and each of the four text characters is sequentially labeled based on the correspondence between the pixel blocks and the text characters to obtain the content recognition sequence information corresponding to the question image.

[0058] Optionally, in some embodiments, at least one pixel block in the question image corresponding to each character in the answer text can have pixel block overlap. For example, character a corresponds to “pixel block 1 + pixel block 2”, character b corresponds to “pixel block 2 + pixel block 3”, and characters a and b correspond to pixel block 3.

[0059] Optionally, the content recognition sequence information may include a tag sequence. For example, the tag sequence p corresponding to the problem-solving image n can be represented as {tag1, tag2, tag3, tag4}. Each sequence tag in the tag sequence p corresponds to the four text characters "ting", "ting", "jing", and "zhi" in the answer text m in sequence. That is, "tag1" corresponds to the text character "ting", "tag2" corresponds to the text character "ting", "tag3" corresponds to the text character "jing", and "tag4" corresponds to the text character "zhi".

[0060] S207: Determine a content recognition result based on the content recognition sequence information corresponding to the problem-solving image.

[0061] Among them, the content recognition result can be used to characterize the matching degree between the problem-solving content in the problem-solving image and the corresponding answer text.

[0062] Optionally, after obtaining the content recognition sequence information corresponding to the problem-solving image, it can be converted according to the recognition result to convert the content recognition sequence information into the content recognition result of the problem-solving content in the problem-solving image. Exemplarily, taking the content recognition result as the marking result, if the content recognition sequence information indicates that the problem-solving content in the problem-solving image completely matches the corresponding answer text, it means that the answer text is exactly the same as the handwritten content in the problem-solving image, then the corresponding marking result is "answer correct"; if the two do not completely match, it means that the answer text is not exactly the same as the handwritten content in the problem-solving image, then the corresponding marking result is "answer wrong".

[0063] In the related art, in the field of education, mainly by inputting the handwritten text image of the student's answer, using OCR to recognize the text in the handwritten image, and finally comparing it with the standard answer, and realizing intelligent marking according to the comparison result. However, when the writing content is not standardized, for example, for some handwritten cursive characters, similar characters, or non-standard writings that are difficult to directly identify, it is easy to produce misjudgments using the existing intelligent marking technology, reducing the accuracy of intelligent marking. Specifically, as Figure 3 shown, the text corresponding to this handwritten picture 300 should be "Ke", but due to the lack of strict writing, if the handwritten picture 300 is directly recognized as the two characters "che" and "ke" through the OCR model, it will be judged as a wrong question in the subsequent marking link, resulting in misjudgment. And through the content recognition model of the embodiments of the present disclosure, it can be accurately recognized as "Ke", reducing misjudgment.

[0064] In an embodiment of the present disclosure, by obtaining an image to be recognized, the image to be recognized includes a question stem image and a problem-solving image corresponding to the question stem image; obtaining an answer text that matches the question description content in the question stem image; processing the answer text and the corresponding problem-solving image through a content recognition model to obtain content recognition sequence information corresponding to the problem-solving image; the content recognition sequence information represents the corresponding relationship based on the character dimension between each text character in the answer text and at least one pixel block in the corresponding problem-solving image; determining a content recognition result based on the content recognition sequence information corresponding to the problem-solving image. Since the input to the content recognition model simultaneously includes two types of modal input data, namely the answer text and the corresponding problem-solving image, the text information of the answer text participates in each discrimination stage of content recognition, which helps to assist in improving the matching ability of the problem-solving content in the problem-solving image and the answer text, reduce the occurrence of misjudgment, and improve the accuracy of content recognition. In addition, through the processing of the content recognition model, the content recognition sequence information corresponding to the problem-solving image is obtained, and the content recognition sequence information represents the corresponding relationship based on the character dimension between each text character in the answer text and at least one pixel block in the corresponding problem-solving image, thereby modeling the original binary classification task model into a more fine-grained sequence annotation problem, further improving the accuracy of content recognition, that is, improving the accuracy of intelligent marking and the error question recognition rate.

[0065] In an optional embodiment, the dimension difference between the sequence dimension of the label sequence corresponding to the content recognition sequence information and the text dimension of the corresponding answer text is a preset value. The preset value can be a natural number. Exemplarily, as Figure 4 shown, continuing with the above answer text m being "standing upright and pure", for example, a <blk>The placeholder 401, then, the adjusted answer text m can be expressed as <blk>If the content recognition sequence information corresponds to the tag sequence 402, then the tag sequence 402 can be {tag 0, tag 1, tag 2, tag 3, tag 4}, where "tag 0" corresponds to... <blk>Placeholder, and the other four tags respectively correspond to four text characters in the answer text m. At this time, the sequence dimension of the tag sequence corresponding to the content recognition sequence information is 5, while the text dimension of the answer text m is 4, so the dimension difference between the two is 1, that is, the preset value is 1.

[0066] It should be understood that in other embodiments, in addition to adding a placeholder indicating the start of the answer, a placeholder indicating the end of the answer or several other placeholders can also be added.

[0067] In this way, by adding at least one placeholder in the answer text, the dimension difference between the sequence dimension of the tag sequence corresponding to the content recognition sequence information and the text dimension of the corresponding answer text is the preset value, which can prevent situations such as answer missing (such as missing at the beginning or end of the answer), and improve the accuracy of content recognition.

[0068] In an optional implementation manner, each tag in the tag sequence corresponding to the content recognition sequence information at least includes at least one of a first type of tag (such as a del tag) for representing a deleted character, a second type of tag (such as a sub tag) for representing a replaced character, a third type of tag (such as an add tag) for representing an added character, and a fourth type of tag (such as an O tag) for representing an unchanged character. Optionally, the first three types of sequence tags can be used in combination with the two symbols "B" and "I", for example, to form combined tags such as "B-del", "B-add", "I-sub", etc. Among them, "B" represents the start of this type of tag, and "I" represents the continuation of this type of tag.

[0069] Exemplarily, continuing as Figure 4 shown, the problem-solving image n is a handwritten picture, and its corresponding written text is four characters "亭", "亭", "净", "直", while the answer text m is four characters "亭", "亭", "净", "植". Taking the handwritten picture as a reference, sequence annotation is performed on the answer text m. After comparison, the last character "直" corresponding to the handwritten picture is inconsistent with "植" in the answer text, so replacing "植" with "直" can obtain the written content in the handwritten picture. Therefore, the sequence tag of the character "植" in the answer text is "B-sub", and the other three characters in the answer text characters are consistent with the handwritten image, so the corresponding sequence tags are "O". Compared with the handwritten picture, there is no missing character before the first character of the answer text, so in the answer text <blk>The placeholder corresponds to the sequence label "O", and thus the label sequence 403 for the answer text m is [O, O, O, O, B-sub]. In other embodiments, if a character is missing before the first character of the answer text, the answer text m is <blk>The sequence label corresponding to the placeholder is "B-add". That is, instead of predicting "correct" or "wrong" by the content recognition model, sequence labeling is performed on the answer text to infer in fine granularity which answer character is different from the handwritten content.

[0070] In the above embodiments, since the label sequence corresponding to the content recognition sequence information includes at least one type of label used to represent the editing operation from the answer text to the question content in the question image, the answer text is labeled by these labels, the matching calculation amount of the answer text and the question image is reduced, and the efficiency of content recognition is improved.

[0071] In an optional embodiment, determining the content recognition result based on the content recognition sequence information corresponding to the question image comprises:

[0072] In the case that each sequence label in the content recognition sequence information corresponding to the question image is a preset type of label, the content recognition result is determined as a first recognition result; the first recognition result represents that the question image matches the corresponding answer text;

[0073] In the case that at least one sequence label in the content recognition sequence information corresponding to the question image is different from the preset type of label, the content recognition result is determined as a second recognition result; the second recognition result represents that the question image does not match the corresponding answer text.

[0074] The preset type of label can be at least one of the above four types of sequence labels. For example, the preset type of label can be the fourth type of label representing character invariance, such as the O label.

[0075] Optionally, in the case that each sequence label in the content recognition sequence information corresponding to the question image is a preset type of label, for example, the sequence labels corresponding to the question image are all O labels, the content recognition result is determined as the first recognition result representing that the question image matches the corresponding answer text, and the corresponding correction result is "correct answer". In the case that at least one sequence label in the content recognition sequence information corresponding to the question image is different from the preset type of label, for example Figure 4 The label sequence 403 of the question image n in the question image set 400 is [O, O, O, O, B-sub], wherein the last sequence label is not an O label, the content recognition result is determined as the second recognition result representing that the question image does not match the corresponding answer text, and the corresponding correction result is "wrong answer".

[0076] The above embodiment determines the corresponding content recognition result by detecting whether each sequence label in the content recognition sequence information corresponding to the question image is the preset class label, so that the content recognition problem is adjusted from a binary classification problem to a sequence level problem, the fine granularity of content recognition is realized, and the accuracy of content recognition is improved. At the same time, the content recognition calculation amount is also reduced, and the content recognition efficiency is further improved.

[0077] As shown in Figure 5 and Figure 6 , the answer text and the corresponding question image are processed by the content recognition model to obtain the content recognition sequence information corresponding to the question image, including:

[0078] S501: The feature extraction module in the content recognition model processes the answer text and the corresponding question image respectively to obtain the answer text feature and the corresponding question image feature.

[0079] In an optional implementation, the feature extraction module includes a text feature extraction layer, an image feature extraction layer, and a linear conversion layer. At this time, the feature extraction module in the content recognition model processes the answer text and the corresponding question image respectively to obtain the answer text feature and the corresponding question image feature, including:

[0080] S5011: The text feature extraction layer processes the answer text to obtain the initial answer text feature;

[0081] S5013: The image feature extraction layer processes each pixel block in the answer text corresponding question image to obtain the initial question image feature;

[0082] S5015: The linear conversion layer processes the initial answer text feature and the initial question image feature respectively to obtain the answer text feature and the corresponding question image feature with the same embedding dimension.

[0083] The text feature extraction layer refers to a network for extracting features of the text, and the image feature extraction layer refers to a network for extracting features of the image. The linear conversion layer refers to linear mapping processing of the extracted text feature and image feature.

[0084] Optionally, in the process of extracting the text feature, an embedding layer can also be added in the feature extraction module. The answer text is input into the embedding layer for embedding representation processing to obtain a word vector sequence corresponding to the answer text. Then, the word vector sequence is input into the text feature extraction layer for data processing to obtain an initial answer text feature. In the process of extracting the image feature, the exam question image is input into the image feature extraction layer, and the image feature extraction layer performs data processing on each pixel block in the exam question image to obtain an initial exam question image feature. For example, the text feature extraction layer can be a linear mapping layer or a natural language processing model for extracting an embedding representation with stronger expression capability. The image feature extraction layer can be a deep residual network (ResNet) or the like. The pixel width of each pixel block can be equal.

[0085] After obtaining the initial answer text feature and the initial exam question image feature, linear mapping processing can be performed on both by a linear conversion layer. Specifically, the number of linear conversion layers can be 2, and the two linear conversion layers perform linear mapping processing on the initial answer text feature and the initial exam question image feature, respectively, to obtain an answer text feature and a corresponding exam question image feature with the same embedding dimension. For example, after processing by the linear conversion layer, the tensor size of the obtained exam question image feature is (num_blocks, hidden_size), and the tensor size of the obtained answer text feature is (num_tokens, hidden_size), where num_blocks is the number of pixel blocks of the exam question image feature, num_tokens is the number of word vectors of the answer text feature, and hidden_size is the embedding dimension size of the corresponding hidden state of the exam question image feature and the answer text feature, i.e., the embedding dimensions of both are consistent.

[0086] In the above embodiment, the linear conversion layer processes the extracted initial answer text feature and the initial exam question image feature, respectively, to obtain an answer text feature and a corresponding exam question image feature with the same embedding dimension. In this way, attention calculation can be directly performed based on the answer text feature and the corresponding exam question image feature, which realizes the combination of multi-modal feature data and the attention mechanism, facilitates the improvement of information interaction between multi-modal data, and further improves the accuracy of content recognition.

[0087] S503: Process the exam question image feature by the image attention module in the content recognition model to obtain a target exam question image feature.

[0088] Optionally, the question image feature can be input into an image attention module in the content recognition model for encoding processing to obtain a target question image feature. The image attention module can include an image attention layer. The attention calculation formula of the image attention layer can be represented as:

[0089]

[0090] wherein Q, K and V correspond to query, key and value matrices respectively, and d is an embedding dimension (for example, d = 768).

[0091] In addition, the image attention module can further include a linear transformation layer connected to the image attention layer, and the linear transformation layer can include two fully connected layers, wherein only the first fully connected layer uses Relu as an activation function. In addition, for the self-attention layer and the linear transformation layer, both can introduce a residual structure, which can include a residual module and a normalization module, to alleviate the model degradation problem by performing residual connection and normalization operation on the data respectively.

[0092] For example, if the question image feature is ix, the attention calculation through the image attention module can be represented as S i = Attention(ix, ix, ix), that is, the attention S i Q, K and V in the attention S

[0093] S505: The answer text feature and the target question image feature are fused by a fusion module in the content recognition model to obtain a fusion feature.

[0094] Optionally, the fusion module can be used to mine the interaction between modalities. The answer text feature and the target question image feature can be input into the fusion module in the content recognition model for fusion processing to obtain a fusion feature. The fusion feature is used to represent the interaction feature between the answer text feature and the target question image feature.

[0095] In an optional implementation, the fusion module includes a text attention layer and a cross-attention layer. At this time, the fusion of the answer text feature and the target question image feature by the fusion module in the content recognition model to obtain the fusion feature includes:

[0096] processing the answer text feature by the text attention layer to obtain a target answer text feature;

[0097] processing the target answer text feature and the target question image feature by the cross-attention layer to obtain the fusion feature.

[0098] Optionally, the answer text feature is input into a text attention layer for encoding processing to obtain a target answer text feature. For example, if the answer text feature is tx, attention calculation through the text attention layer can be represented as S t = Attention(tx, tx, tx), that is, attention S t Q, K and V in the attention S

[0099] After obtaining the target answer text feature, the target answer text feature and the target question image feature are input into a cross-attention layer for fusion processing to obtain a fusion feature. The target answer text feature and the target question image feature interact with each other through the cross-attention layer to enable the model to learn the matching question image content and the answer text.

[0100] For example, if the question image feature is ix and the answer text feature is tx, attention calculation through the cross-attention layer can be represented as C i,t = Attention(tx, ix, ix), that is, attention C i,t Q in the attention C

[0101] Optionally, the fusion module can further include a linear transformation layer connected to the cross-attention layer. The linear transformation layer can include two fully connected layers, and only the first fully connected layer can use Relu as an activation function. In addition, the text attention layer, the cross-attention layer and the linear transformation layer can introduce a residual structure, which can include a residual module and a normalization module. The residual structure can alleviate the model degradation problem by performing residual connection and normalization operations on the data, respectively.

[0102] In the above embodiment, the answer text feature is processed through the text attention layer to obtain a target answer text feature, and the target answer text feature and the target question image feature are fused through the cross-attention layer to obtain a fusion feature. By combining the text attention layer and the cross-attention layer, the text and visual information are interacted to assist the answer text to seek corresponding matching information from each pixel block of the question image. Each answer character can pay attention to all effective pixel blocks without being limited by the order of language modeling. The matching ability of the model for the question image content and the answer text is improved, and the content recognition accuracy is further improved.

[0103] S507: The fusion feature is recognized through an output layer in the content recognition model to obtain content recognition sequence information corresponding to the question image.

[0104] Optionally, an output layer in the content recognition model is configured to map the text representation space to a label space to obtain an output result of the content recognition model. The output result can include content recognition sequence information corresponding to the question image.

[0105] By way of example only, the output layer can include a fully connected layer and an activation layer, and the content recognition sequence information corresponding to the question image is obtained by sequentially inputting the fused features into the fully connected layer and the activation layer of the content recognition model for recognition processing. The content recognition sequence information represents a correspondence relationship between each text character in the answer text and at least one pixel block in the corresponding question image based on the character dimension. Specifically, the content recognition sequence information can be used to reflect the editing operation required when converting the answer text into handwritten content.

[0106] In the above embodiments, the image attention module is used to capture the target question image features representing the context information in the visual modality, and the fusion module is used to mine the interaction between the modalities. The content recognition model can better understand the handwriting, ligature, and similar characters in the question image, thereby improving the matching ability of the content recognition model for the question image and the answer text, and further improving the content recognition accuracy.

[0107] In an optional implementation, as shown in Figure 6 The content recognition model is constructed based on a pre-trained text recognition model. The text recognition model is a model for recognizing text content in an image, and can be an OCR model. The text recognition model can include a recognition feature extraction layer for extracting image features of an input image, and a recognition linear layer for performing text mapping processing on the image features output by the image extraction layer.

[0108] Optionally, the content recognition model includes a text feature extraction layer and an image feature extraction layer. The structure parameters of the image feature extraction layer are the same as the structure parameters of the recognition feature extraction layer in the text recognition model. The structure parameters of the text feature extraction layer and the structure parameters of the recognition linear layer in the text recognition model are transposed to each other, that is, in the text recognition model, the recognition linear layer is used to project a high-dimensional vector into text; and in the content recognition model, the text feature extraction layer is used to map the text back to the same high-dimensional vector space, and the parameter matrix of the text feature extraction layer and the parameter matrix of the recognition linear layer are transposed to each other.

[0109] The above embodiment constructs the content recognition model by using a pre-trained character recognition model. The content recognition model has a text feature extraction layer and an image feature extraction layer, which are respectively associated with a recognition linear layer and a recognition feature extraction layer in the character recognition model. Since the structural parameters in the recognition linear layer are learned in the pre-training process to convert from an image feature space to a text space, the recognition linear layer serves as an interactive bridge between the spaces, so that the two modal data of the answer text and the question image are placed in the same space by the content recognition model, facilitating the information interaction of the fusion module.

[0110] In an optional embodiment, the content recognition model is trained by the following method:

[0111] A training sample set is obtained, which includes question sample images, corresponding answer sample texts, and recognition label sequences.

[0112] The content recognition model is used to predict the answer sample texts and the corresponding question sample images to obtain prediction results corresponding to the question sample images. The content recognition model is constructed based on a pre-trained character recognition model.

[0113] The content recognition model is trained based on the training loss determined by the prediction results and the corresponding recognition label sequences to obtain a trained content recognition model.

[0114] Optionally, for the training sample set, the question sample images with correct answers, the corresponding answer texts, and the recognition label sequences can be obtained from the data set as positive samples, and the question sample images with incorrect answers, the corresponding answer texts, and the recognition label sequences can be obtained from the data set as negative samples. In order to keep the number of training samples balanced, data augmentation can be used to expand the negative samples. For example, for each positive sample, the question sample image can be kept unchanged, and the corresponding answer text can be modified, which can include random character insertion, deletion, and replacement. Specifically, in the replacement character modification method, negative samples can be constructed by replacing original characters with similar characters. Experimental results show that expanding negative samples by this negative sample data augmentation method improves the performance of the content recognition model.

[0115] Before training the content recognition model, the character recognition model can be pre-trained. As shown in Figure 6 In the pre-training process, the negative samples can be expanded by the above data augmentation method to improve the recognition accuracy of the character recognition model. Then, the content recognition model is constructed by using the trained character recognition model, and the network parameters of the corresponding modules in the constructed content recognition model are initialized by using the model parameters of the trained character recognition model.

[0116] Optionally, the text recognition model is a model for recognizing text content in an image, which can be an OCR model. The text recognition model can include a recognition feature extraction layer for extracting image features of an input image and a recognition linear layer for text mapping processing of the image features output by the image extraction layer. The content recognition model includes a text feature extraction layer and an image feature extraction layer. The recognition feature extraction layer and the image feature extraction layer can be ResNet networks, etc. The structural parameters of the image feature extraction layer are the same as the structural parameters of the recognition feature extraction layer in the text recognition model. The structural parameters of the text feature extraction layer and the structural parameters of the recognition linear layer in the text recognition model are transposed to each other, that is, in the text recognition model, the recognition linear layer is used to project a high-dimensional vector to text; and in the content recognition model, the text feature extraction layer is used to map the text back to the same high-dimensional vector space, and the parameter matrix of the text feature extraction layer and the parameter matrix of the recognition linear layer are transposed to each other.

[0117] Then, the answer sample text and the corresponding question-answering sample image are predicted by the content recognition model to obtain a prediction result corresponding to the question-answering sample image. The prediction result represents the probability that a certain text character in the answer sample text is marked as a certain sequence of sample labels. The prediction result can include content recognition sequence sample information. Then, the prediction result and the corresponding recognition label sequence are used to determine a training loss (such as a cross-entropy function, etc.). The training content recognition model is based on the training loss until a training end condition is reached, and a trained content recognition model is obtained.

[0118] In the above embodiment, the content recognition model is trained based on the question-answering sample image, the corresponding answer sample text, and the recognition label sequence as a training sample set, so that the training task is not a simple "correct" or "incorrect" binary classification problem, but a more fine-grained sequence labeling problem, which is beneficial to improve the accuracy of the trained model.

[0119] In addition, since the content recognition model is constructed based on the pre-trained text recognition model, and the text feature extraction layer and the image feature extraction layer of the content recognition model are respectively associated with the recognition linear layer and the recognition feature extraction layer of the text recognition model, the structural parameters in the recognition linear layer learn the conversion from the image feature space to the text space in the pre-training process. Through the recognition linear layer as an interactive bridge between spaces, the answer text and the question-answering image are placed in the same space through the content recognition model, which is beneficial to the information interaction of the two modalities by the fusion module.

[0120] The embodiment of the present disclosure, in view of the limitations of the existing OCR automatic correction technology, uses a multi-modal method to enhance the matching ability of the content recognition model for image handwritten content and answer text, solving the problem of easy misrecognition of non-standard writing such as handwritten cursive and easily confused handwritten characters. At the same time, the model output is modeled as a more fine-grained sequence labeling problem, realizing a more fine-grained content recognition task, and improving the accuracy of correction and the recognition ability of wrong questions.

[0121] Through verification, we evaluate the model based on real data generated in the business scenario, and the results show that compared with pure OCR automatic correction technology, the correction accuracy of the content recognition method provided by the embodiment of the present disclosure is improved by more than 10%, the F1 of wrong question recognition is improved by 4%, and the precision is improved by 9%.

[0122] The following is an apparatus embodiment of the present disclosure, which can be used to execute the method embodiments of the present disclosure. For details not disclosed in the apparatus embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0123] Please refer to Figure 7 , which shows a structural block diagram of a content recognition apparatus provided by an embodiment of the present disclosure. The apparatus has the functions in the above-mentioned method examples, which can be realized by hardware or corresponding software executed by hardware. The content recognition apparatus can include:

[0124] The first obtaining module 710 is configured to obtain a to-be-recognized image, wherein the to-be-recognized image includes a stem image and a question image corresponding to the stem image;

[0125] The second obtaining module 720 is configured to obtain an answer text matched with the description content of the question in the stem image;

[0126] The recognition processing module 730 is configured to process the answer text and the corresponding question image by using a content recognition model to obtain content recognition sequence information corresponding to the question image; the content recognition sequence information represents a correspondence relationship between each text character in the answer text and at least one pixel block in the corresponding question image based on a character dimension;

[0127] The recognition result determination module 740 is configured to determine a content recognition result based on the content recognition sequence information corresponding to the question image.

[0128] In an optional implementation, as shown in Figure 8 , the recognition processing module 730 includes:

[0129] The first processing sub-module 731 is configured to process the answer text and the corresponding question image by using a feature extraction module in the content recognition model respectively to obtain an answer text feature and a corresponding question image feature respectively;

[0130] The second processing sub-module 732 is configured to process the question image features by an image attention module in the content recognition model to obtain target question image features.

[0131] The third processing sub-module 733 is configured to perform fusion processing on the answer text features and the target question image features by a fusion module in the content recognition model to obtain fusion features.

[0132] The fourth processing sub-module 734 is configured to perform recognition processing on the fusion features by an output layer in the content recognition model to obtain content recognition sequence information corresponding to the question image.

[0133] In an optional implementation, the feature extraction module includes a text feature extraction layer, an image feature extraction layer, and a linear conversion layer. The first processing sub-module 731 is specifically configured to:

[0134] perform processing on the answer text by the text feature extraction layer to obtain initial answer text features;

[0135] perform processing on each pixel block in the question image corresponding to the answer text by the image feature extraction layer to obtain initial question image features;

[0136] perform processing on the initial answer text features and the initial question image features respectively by the linear conversion layer to obtain answer text features and corresponding question image features with the same embedding dimension.

[0137] In an optional implementation, the fusion module includes a text attention layer and a cross-attention layer. The third processing sub-module 733 is specifically configured to:

[0138] perform processing on the answer text features by the text attention layer to obtain target answer text features;

[0139] perform fusion processing on the target answer text features and the target question image features by the cross-attention layer to obtain fusion features.

[0140] In an optional implementation, each label in a label sequence corresponding to the content recognition sequence information includes at least one of a first type of label for representing a deleted character, a second type of label for representing a replaced character, a third type of label for representing an added character, and a fourth type of label for representing an unchanged character.

[0141] In an optional implementation, a dimension difference between a sequence dimension of a label sequence corresponding to the content recognition sequence information and a text dimension of the corresponding answer text is a preset value. The recognition result determination module 740 is specifically configured to:

[0142] In a case where each sequence label in the content recognition sequence information corresponding to the question image is a preset class label, the content recognition result is determined as a first recognition result; the first recognition result indicates that the question image matches the corresponding answer text.

[0143] In a case where at least one sequence label in the content recognition sequence information corresponding to the question image is different from the preset class label, the content recognition result is determined as a second recognition result; the second recognition result indicates that the question image does not match the corresponding answer text.

[0144] In an optional implementation, the content recognition model is obtained by training in the following manner:

[0145] A training sample set is obtained, the training sample set including a question sample image, a corresponding answer sample text, and a recognition label sequence;

[0146] The content recognition model is used to predict the answer sample text and the corresponding question sample image to obtain a prediction result corresponding to the question sample image; the content recognition model is constructed based on a pre-trained text recognition model;

[0147] The content recognition model is trained based on a training loss determined based on the prediction result and the corresponding recognition label sequence to obtain a trained content recognition model.

[0148] In an optional implementation, the text recognition model includes a recognition feature extraction layer for extracting image features of an input image and a recognition linear layer for performing text mapping processing on the image features output by the image extraction layer;

[0149] The content recognition model includes a text feature extraction layer and an image feature extraction layer, the structure parameters of the text feature extraction layer and the structure parameters of the recognition linear layer are in a transposed relationship with each other, and the structure parameters of the image feature extraction layer are the same as the structure parameters of the recognition feature extraction layer.

[0150] The apparatuses provided in the above embodiments can execute the corresponding methods in the embodiments of the present disclosure, and have the corresponding function modules and advantages of executing the methods. Technical details not described in detail in the above embodiments can be referred to the methods provided in any of the embodiments of the present disclosure.

[0151] The embodiments of the present disclosure provide a computer device, which can include a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the method according to any of the above method embodiments.

[0152] This disclosure also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded by a processor and executed by any of the methods described in the above method embodiments.

[0153] This disclosure also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the methods described in this embodiment.

[0154] Furthermore, Figure 9 A schematic diagram of a hardware structure for implementing the methods provided in the embodiments of this disclosure is shown. The device may be a computer terminal, a mobile terminal, or other device, and may also participate in or include the apparatus provided in the embodiments of this disclosure. Figure 9 As shown, the computer terminal 11 may include one or more processors 112 (shown as 112a, 112b, ..., 112n in the figure) 112 (processor 112 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 114 for storing data, and a transmission device 116 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 9 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 11 may also include... Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.

[0155] It should be noted that the aforementioned one or more processors 112 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 11 (or mobile device). As per the embodiments of this disclosure, the data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0156] The memory 114 can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the method described in the embodiments of the present disclosure, and the processor 112 can execute various functional applications and data processing, i.e., implement the above-mentioned neural network processing method, by running the software programs and modules stored in the memory 104. The memory 114 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 114 can further include a memory remotely arranged with respect to the processor 112, which can be connected to the computer terminal 11 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0157] The transmission device 116 is used to receive or send data via a network. The specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the computer terminal 11. In one example, the transmission device 116 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to be able to communicate with the Internet. In one example, the transmission device 116 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

[0158] The display can be, for example, a touch screen type liquid crystal display (LCD), which can enable a user to interact with the user interface of the computer terminal 11 (or a mobile device).

[0159] It should be noted that the above-mentioned sequence of the embodiments of the present disclosure is only for description, and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present disclosure are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0160] The various embodiments in the present disclosure are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment mainly explains the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0161] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by a program to complete the related hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0162] The above only describes the preferred embodiments of the present disclosure and is not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.< / blk> < / blk> < / blk> < / blk> < / blk>

Claims

1. A content recognition method, characterized in that, include: Obtain the image to be identified, which includes the question stem image and the corresponding question image; Obtain the answer text that matches the question description in the question stem image; The answer text and the corresponding question-answering image are processed by the feature extraction module in the content recognition model to obtain the features of the answer text and the corresponding question-answering image, respectively. The image attention module in the content recognition model processes the features of the question-answering image to obtain the features of the target question-answering image; The fusion module in the content recognition model performs fusion processing on the answer text features and the target question image features to obtain fused features; The fused features are processed by the output layer of the content recognition model to obtain the content recognition sequence information corresponding to the question image. The content recognition sequence information represents the character-based correspondence between each text character in the answer text and at least one pixel block in the corresponding question image; Based on the content recognition sequence information corresponding to the question image, the content recognition result is determined.

2. The method according to claim 1, characterized in that, The feature extraction module includes a text feature extraction layer, an image feature extraction layer, and a linear transformation layer; The step of processing the answer text and the corresponding question-answering image through the feature extraction module in the content recognition model to obtain the answer text features and the corresponding question-answering image features respectively includes: The answer text is processed by the text feature extraction layer to obtain initial answer text features; The image feature extraction layer processes each pixel block in the question image corresponding to the answer text to obtain the initial question image features; The initial answer text features and the initial question-answering image features are processed by the linear transformation layer to obtain answer text features and corresponding question-answering image features with the same embedding dimension.

3. The method according to claim 1, characterized in that, The fusion module includes a text attention layer and a cross attention layer; The step of fusing the answer text features and the target question image features through the fusion module in the content recognition model to obtain fused features includes: The text attention layer is used to process the text features of the answer to obtain the text features of the target answer. The cross-attention layer is used to fuse the target answer text features and the target question-answering image features to obtain fused features.

4. The method according to any one of claims 1-3, characterized in that, The tag sequence corresponding to the content recognition sequence information includes at least one of the following: a first type of tag for representing deleted characters, a second type of tag for representing replaced characters, a third type of tag for representing added characters, and a fourth type of tag for representing unchanged characters.

5. The method according to any one of claims 1-3, characterized in that, The dimensional difference between the sequence dimension of the tag sequence corresponding to the content recognition sequence information and the text dimension of the corresponding answer text is a preset value; The determination of the content recognition result based on the content recognition sequence information corresponding to the question image includes: If each sequence label in the content recognition sequence information corresponding to the question image is a preset class label, the content recognition result is determined as the first recognition result; the first recognition result indicates that the question image matches the corresponding answer text. If at least one sequence label in the content recognition sequence information corresponding to the question image is different from the preset class label, the content recognition result is determined as the second recognition result; the second recognition result indicates that the question image does not match the corresponding answer text.

6. The method according to any one of claims 1-3, characterized in that, The content recognition model was trained in the following way: Obtain a training sample set, which includes question sample images, corresponding answer sample texts, and recognition label sequences; The answer sample text and the corresponding question sample image are predicted by a content recognition model to obtain the prediction result corresponding to the question sample image; The content recognition model is constructed based on a pre-trained text recognition model; The content recognition model is trained based on the prediction results and the corresponding recognition label sequence, and the training loss is determined to obtain the trained content recognition model.

7. The method according to claim 6, characterized in that, The text recognition model includes a recognition feature extraction layer and a recognition linear layer for extracting image features from the input image. The recognition linear layer is used to perform text mapping processing on the image features output by the image extraction layer. The content recognition model includes a text feature extraction layer and an image feature extraction layer. The structural parameters of the text feature extraction layer are transposes of the structural parameters of the recognition linear layer, and the structural parameters of the image feature extraction layer are the same as those of the recognition feature extraction layer.

8. A content recognition device, characterized in that, The device includes: The first acquisition module is used to acquire the image to be identified, the image to be identified including the question stem image and the question-answering image corresponding to the question stem image; The second acquisition module is used to acquire the answer text that matches the question description content in the question stem image; The recognition processing module is used to process the answer text and the corresponding question image separately through the feature extraction module in the content recognition model to obtain the answer text features and the corresponding question image features respectively; to process the question image features through the image attention module in the content recognition model to obtain the target question image features; to fuse the answer text features and the target question image features through the fusion module in the content recognition model to obtain fused features; and to recognize the fused features through the output layer in the content recognition model to obtain the content recognition sequence information corresponding to the question image; the content recognition sequence information represents the character-dimensional correspondence between each text character in the answer text and at least one pixel block in the corresponding question image. The recognition result determination module is used to determine the content recognition result based on the content recognition sequence information corresponding to the question image.

9. The apparatus according to claim 8, characterized in that, The feature extraction module includes a text feature extraction layer, an image feature extraction layer, and a linear transformation layer; The first processing submodule of the identification processing module is specifically used for: The answer text is processed by the text feature extraction layer to obtain initial answer text features; The image feature extraction layer processes each pixel block in the question image corresponding to the answer text to obtain the initial question image features; The initial answer text features and the initial question-answering image features are processed by the linear transformation layer to obtain answer text features and corresponding question-answering image features with the same embedding dimension.

10. The apparatus according to claim 8, characterized in that, The fusion module includes a text attention layer and a cross attention layer; The third processing submodule of the identification processing module is specifically used for: The text attention layer is used to process the text features of the answer to obtain the text features of the target answer. The cross-attention layer is used to fuse the target answer text features and the target question-answering image features to obtain fused features.

11. The apparatus according to any one of claims 8-10, characterized in that, The tag sequence corresponding to the content recognition sequence information includes at least one of the following: a first type of tag for representing deleted characters, a second type of tag for representing replaced characters, a third type of tag for representing added characters, and a fourth type of tag for representing unchanged characters.

12. The apparatus according to any one of claims 8-10, characterized in that, The dimensional difference between the sequence dimension of the tag sequence corresponding to the content recognition sequence information and the text dimension of the corresponding answer text is a preset value; The identification result determination module is specifically used for: If each sequence label in the content recognition sequence information corresponding to the question image is a preset class label, the content recognition result is determined as the first recognition result; the first recognition result indicates that the question image matches the corresponding answer text. If at least one sequence label in the content recognition sequence information corresponding to the question image is different from the preset class label, the content recognition result is determined as the second recognition result; the second recognition result indicates that the question image does not match the corresponding answer text.

13. The apparatus according to any one of claims 8-10, characterized in that, The content recognition model was trained in the following way: Obtain a training sample set, which includes question sample images, corresponding answer sample texts, and recognition label sequences; The answer sample text and the corresponding question sample image are predicted by a content recognition model to obtain the prediction result corresponding to the question sample image; The content recognition model is constructed based on a pre-trained text recognition model; The content recognition model is trained based on the prediction results and the corresponding recognition label sequence, and the training loss is determined to obtain the trained content recognition model.

14. The apparatus according to claim 13, characterized in that, The text recognition model includes a recognition feature extraction layer and a recognition linear layer for extracting image features from the input image. The recognition linear layer is used to perform text mapping processing on the image features output by the image extraction layer. The content recognition model includes a text feature extraction layer and an image feature extraction layer. The structural parameters of the text feature extraction layer are transposes of the structural parameters of the recognition linear layer, and the structural parameters of the image feature extraction layer are the same as those of the recognition feature extraction layer.

15. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the content recognition method as described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the content recognition method as described in any one of claims 1-7.

17. A computer program product, characterized in that, The computer program product includes at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the content recognition method as described in any one of claims 1-7.

Images