Annotating method and device of teaching material, electronic equipment and storage medium

Abstract

The present disclosure provides a teaching material annotation method and device, electronic equipment and storage medium, relates to the technical field of artificial intelligence, and particularly relates to the technical field of image processing, natural language processing, intelligent education and the like. The specific implementation scheme is: obtaining a first image of a teaching material to be annotated, and obtaining a text block sequence based on the first image; reorganizing an image region in the first image based on the text block sequence, obtaining a second image of the teaching material to be annotated, and performing word recognition on the second image to obtain words in the text block; obtaining a set of annotation information based on the text block sequence, wherein the set of annotation information includes at least one subset of annotation information of the text block; aligning and binding the set of annotation information and each text block based on the words in the text block to obtain structured annotation data; and sending the structured annotation data to a client, wherein the structured annotation data is used to render annotation information of the teaching material to be annotated.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to image processing, natural language processing, intelligent education and other technical fields, and in particular to a method, apparatus, electronic device and storage medium for annotating teaching materials. Background Technology

[0002] Currently, e-textbook products mainly rely on teaching and research personnel to manually annotate the textbook content page by page, providing auxiliary information such as word definitions, sentence analysis, and paragraph summaries. The annotated content is then bound to a pre-made textbook version and distributed to users, enabling them to interact with the e-textbook product based on the bound information. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for annotating teaching materials.

[0004] According to one aspect of this disclosure, a method for annotating a textbook is provided, comprising: acquiring a first image of the textbook to be annotated, and acquiring a text block sequence based on the first image, wherein the text blocks in the text block sequence are arranged in semantic order; reorganizing image regions in the first image based on the text block sequence to acquire a second image of the textbook to be annotated, and performing word recognition on the second image to acquire words in the text blocks; acquiring an annotation information set based on the text block sequence, wherein the annotation information set includes a subset of annotation information of at least one text block; aligning and binding the annotation information set and each text block based on the words in the text blocks to acquire structured annotation data; and sending the structured annotation data to a client, wherein the structured annotation data is used to render the annotation information of the textbook to be annotated.

[0005] According to one aspect of this disclosure, another method for annotating teaching materials is provided, comprising: acquiring a first image of the teaching material to be annotated and sending the first image to a server, the first image including multiple image regions, different image regions corresponding to different text blocks; receiving structured annotation data sent by the server, the structured annotation data including annotation information of the first image and the position information of the content associated with the annotation information in the first image, the structured annotation data being obtained by aligning and binding the annotation information set and each text block based on the words in the text blocks; and rendering the annotation information of the first image based on the structured annotation data to obtain an annotated teaching material.

[0006] According to another aspect of this disclosure, a textbook annotation device is provided, comprising: an acquisition module, configured to acquire a first image of the textbook to be annotated, and acquire a text block sequence based on the first image, wherein the text blocks in the text block sequence are arranged in semantic order; a recognition module, configured to reorganize image regions in the first image based on the text block sequence to acquire a second image of the textbook to be annotated, and perform word recognition on the second image to acquire words in the text blocks; an annotation module, configured to acquire an annotation information set based on the text block sequence, wherein the annotation information set includes a subset of annotation information of at least one text block; a binding module, configured to align and bind the annotation information set and each text block based on the words in the text blocks to acquire structured annotation data; and a sending module, configured to send the structured annotation data to a client, wherein the structured annotation data is used to render the annotation information of the textbook to be annotated.

[0007] According to another aspect of this disclosure, another textbook annotation device is provided, comprising: a sending module for acquiring a first image of the textbook to be annotated and sending the first image to a server, the first image including multiple image regions, different image regions corresponding to different text blocks; a receiving module for receiving structured annotation data sent by the server, the structured annotation data including annotation information of the first image and position information of the content associated with the annotation information in the first image, the structured annotation data being obtained by aligning and binding the annotation information set and each text block based on the words in the text blocks; and a rendering module for rendering the annotation information of the first image based on the structured annotation data to obtain an annotated textbook.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the annotation method for teaching materials described in one aspect of the above-described embodiment.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions thereon, wherein the computer instructions are used to cause the computer to execute the annotation method for teaching materials described in the above-mentioned embodiment.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the annotation method for teaching materials described in one aspect of the above-described embodiments.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 A flowchart illustrating a textbook annotation method provided in this embodiment of the disclosure; Figure 2 A flowchart illustrating another annotation method for teaching materials provided in this embodiment of the disclosure; Figure 3 A flowchart illustrating another annotation method for teaching materials provided in this embodiment of the disclosure; Figure 4 A flowchart illustrating another annotation method for teaching materials provided in this embodiment of the disclosure; Figure 5 This is a schematic diagram of the process for generating structured annotation data provided in an embodiment of the present disclosure; Figure 6 A schematic diagram of the structure of an annotation device for teaching materials provided in this embodiment of the present disclosure; Figure 7 A schematic diagram of another annotation device for teaching materials provided in this embodiment of the present disclosure; Figure 8 This is a block diagram of an electronic device used to implement the annotation method for teaching materials according to embodiments of the present disclosure. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] The annotation methods, apparatus, electronic devices, and storage media of teaching materials according to embodiments of this disclosure are described below with reference to the accompanying drawings.

[0016] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. AI hardware technologies generally include computer vision, speech recognition, natural language processing, and related technologies such as deep learning, big data processing, and knowledge graphs.

[0017] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Its main applications include machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, and speech recognition.

[0018] Image processing technology is the technique of using computers to analyze images to achieve desired results. It is also known as image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by capturing images using equipment such as industrial cameras, video cameras, and scanners. The elements of this array are called pixels, and their values ​​are called grayscale values. Image processing techniques generally include three parts: image compression, enhancement and restoration, and matching, description, and recognition.

[0019] Figure 1 This is a flowchart illustrating a textbook annotation method provided in an embodiment of this disclosure.

[0020] like Figure 1 As shown, the annotation methods in this textbook may include: S101, obtain the first image of the textbook to be annotated, and obtain a text block sequence based on the first image, wherein the text blocks in the text block sequence are arranged in semantic order.

[0021] It should be noted that the execution entity of the annotation method in the teaching materials in this embodiment can be a hardware device with data processing capabilities and / or the necessary software to drive the hardware device. Optionally, the execution entity may include a server. Optionally, the server includes, but is not limited to, a network server, an application server, or a server of a distributed system, or a server combined with blockchain, etc. This embodiment does not impose specific limitations.

[0022] In some embodiments, an image sent by the client can be received as the first image of the teaching material to be annotated. The teaching material to be annotated can be an electronic textbook. For example, a screenshot of any page of the electronic textbook taken by the client can be received as the first image.

[0023] For example, it can also receive electronic textbooks in Portable Document Format (PDF) sent by clients. By converting the PDF electronic textbooks to an image format, the first image of the textbook to be annotated can be obtained.

[0024] In some embodiments, the teaching material to be annotated can also be a printed teaching material. Users can take pictures of the printed teaching material and upload them to the client. The client can then transmit the pictures to the server, and these pictures will be the first images of the teaching material to be annotated.

[0025] In this disclosure, the teaching materials can be subject textbooks, extracurricular reading materials, test papers, etc., without specific limitations on the teaching materials.

[0026] In some embodiments, a multimodal large model can be used to perform optical character recognition (OCR) on the first image to obtain the layout information of the textbook to be annotated from the first image, thereby extracting text blocks from the first image, and sorting the extracted text blocks according to the layout information to obtain a text block sequence.

[0027] The layout information includes two-column, multi-column, and mixed text and image layouts.

[0028] In some embodiments, the text blocks in the text block sequence are arranged in a semantic order, wherein the semantic order may be a semantic order that conforms to human reading habits, such as from left to right or from top to bottom.

[0029] In some embodiments, the first image can be segmented based on layout information to divide the first image into multiple image regions that conform to semantic logic, and text can be extracted from the image regions to obtain text blocks.

[0030] For example, the image areas in the first image include: the central main title area, the right-side text area, etc.

[0031] In some embodiments, the semantic order of each logical block can be determined based on layout information, thereby sorting the text blocks according to the semantic order to obtain a text block sequence.

[0032] In some embodiments, when acquiring the text block sequence, the multimodal large model can also output the position information of each text block in the first image. This position information can be the position information of an image region in the first image.

[0033] S102, based on the text block sequence, the image region in the first image is reorganized to obtain the second image of the textbook to be annotated, and word recognition is performed on the second image to obtain the words in the text block.

[0034] In some embodiments, an image region associated with each text block in the text block sequence can be determined, and the image regions in the first image can be reorganized according to the semantic order of the text blocks. Here, the image region associated with a text block refers to the region from which the text block is extracted.

[0035] In some embodiments, reorganizing the image regions in the first image can be based on semantic order, arranging the image regions sequentially from top to bottom to obtain a reorganized second image of the textbook to be annotated.

[0036] In some embodiments, a dedicated small model can be used to perform OCR on the second image to identify each word in the second image. That is, the smallest unit in the second image, i.e., the word, can be identified to obtain the words in the text block.

[0037] In some embodiments, the OCR of a multimodal large model can output text blocks in the correct semantic order, but the positioning granularity usually stays at the block level and cannot directly provide accurate word-level recognition. By using the OCR of a dedicated small model, accurate word recognition can be achieved.

[0038] In this disclosure, two different OCR methods are used to obtain words arranged in the correct semantic order.

[0039] In some embodiments, when recognizing words in a text block, the words can also be located to obtain their positional information.

[0040] S103, Based on the text block sequence, obtain the annotation information set, wherein the annotation information set includes at least one subset of annotation information of the text block.

[0041] In some embodiments, a sequence of text blocks can be input into a large language model for semantic analysis, so that the large language model outputs the semantic analysis results for each text block in the sequence of text blocks, thereby obtaining a set of annotation information.

[0042] In some embodiments, each text block in the text block sequence can be at least one paragraph, and each paragraph consists of sentences. The large language model can perform semantic analysis on each sentence to obtain the annotation information corresponding to the sentence.

[0043] For example, for any sentence in a text block, the original text of the sentence can be analyzed to obtain its semantic understanding information, which can then be used as annotation information in its corresponding annotation information subset.

[0044] In some embodiments, the large language model can also perform semantic analysis on words in a sentence to obtain annotation information corresponding to the words.

[0045] For example, key words and phrases in a sentence, along with their definitions, pronunciations, and word combinations, can be analyzed and used as annotation information in their corresponding subset of annotation information.

[0046] In some embodiments, semantic analysis can also be performed on the paragraph containing the sentence to obtain the annotation information corresponding to the paragraph.

[0047] For example, you can analyze the main idea of ​​a paragraph and summarize the main theme of the whole text, which can then be used as annotation information in the corresponding subset of annotation information.

[0048] In some embodiments, the large language model can also identify and explain the rhetorical devices used in sentences, thereby obtaining annotation information.

[0049] It should be noted that the generation of the annotation information set can be performed after the words in the second image are obtained, that is, the generation of annotation information can be performed serially with word recognition; or it can be performed in parallel with word recognition after the text blocks in the first image are determined, so as to synchronously obtain the annotation information set and the words in the second image, thereby reducing end-to-end latency.

[0050] S104: Based on the words in the text blocks, align and bind the annotation information set and each text block to obtain structured annotation data.

[0051] It should be noted that the words in the text blocks are based on the second image recognition, while the annotation information set is obtained based on the first image. Aligning and binding the annotation information set and each text block requires aligning and binding the information in the second image with the information in the first image.

[0052] It should be noted that the structured annotation data is used to render the textbook to be annotated on the client side. The textbook to be annotated on the client side is the first image, so it is necessary to obtain the position information of words in the first image.

[0053] Since the coordinate systems of each text block and word in the first and second images are different, we can first obtain the mapping relationship between the text block and word in different images, and then map the word to the coordinate system of the first image according to the mapping relationship to achieve alignment.

[0054] Furthermore, based on the position information of words in the first image, they are bound to the position of each annotation in the annotation information set, and the bound data is processed in a structured manner to obtain structured annotation data.

[0055] In some embodiments, when identifying text blocks in the first image, the multimodal large model can output the position information of each text block in the first image. Since the second image is composed of image regions containing the text blocks, the position information of the text blocks in the second image can then be obtained.

[0056] In some embodiments, the mapping relationship between the text block and the second image can be determined based on the position information of the text block in the first image and the position information of the text block in the second image, so that the words in the text block can be mapped to the coordinate system of the first image based on the mapping relationship.

[0057] In some embodiments, each annotation can be associated with a word, sentence, or paragraph according to the annotation information set, and the annotation information associated with the word can be bound to the position information of the word in the first image.

[0058] In some embodiments, the annotation information associated with a sentence can also be bound to the region determined by the word coordinate sequence constituting the sentence. For example, the position information of each word in the first image of the sentence can be determined, and the region coordinate information can be determined based on the position information, and the annotation information associated with the sentence can be bound to the region coordinate information.

[0059] In some embodiments, the annotation information associated with the paragraph can also be bound to the position information of the text block containing the word in the first image.

[0060] In other words, structured annotation data includes location information.

[0061] S105, send structured annotation data to the client. The structured annotation data is used to render the annotation information of the textbook to be annotated.

[0062] In some embodiments, structured annotation data is sent to the client so that the client can render the teaching material to be annotated based on the structured annotation data, thereby overlaying the structured annotation data onto the first image of the teaching material to be annotated, so that the first image can carry annotation information.

[0063] In some embodiments, the client can locate the first image of the teaching material to be annotated based on the location information contained in the structured annotation data, and render the annotation information of each location information into the first image of the teaching material to be annotated.

[0064] According to the textbook annotation method provided in this disclosure, a first image of the textbook to be annotated is acquired, a text block sequence is obtained based on the first image, and the image regions in the first image are reorganized based on the text block sequence to obtain a second image of the textbook to be annotated. Word recognition is then performed on the second image to obtain the words in the text blocks. Further, an annotation information set can be obtained based on the text block sequence, and the annotation information set and each text block are aligned and bound based on the words in the text blocks to obtain structured annotation data. This structured annotation data is then sent to the client, enabling the client to render the annotation information of the textbook to be annotated based on the structured annotation data. Therefore, this solution can achieve fully automated generation of annotation information, significantly reducing manpower and time costs. By acquiring the first image of the textbook to be annotated to generate annotation information, annotations for different types of textbooks can be generated, improving the generalization ability of annotation information generation.

[0065] In addition, identifying text blocks and words in the image separately not only ensures the correctness of the semantic order of text in complex layouts, but also achieves precise word-level positioning, which helps improve the accuracy of binding annotation information with text blocks.

[0066] Figure 2 This is a flowchart illustrating another annotation method for teaching materials provided in this embodiment of the disclosure.

[0067] like Figure 2 As shown, the annotation methods in this textbook may include: S201, Obtain the first image of the textbook to be annotated, and obtain a text block sequence based on the first image, wherein the text blocks in the text block sequence are arranged in semantic order.

[0068] S202, based on the text block sequence, the image region in the first image is reorganized to obtain the second image of the textbook to be annotated, and word recognition is performed on the second image to obtain the words in the text block.

[0069] S203, Based on the text block sequence, obtain the annotation information set, wherein the annotation information set includes at least one subset of annotation information of the text block.

[0070] The details of steps S201-S203 can be found in the above embodiments and will not be repeated here.

[0071] S204, determine the first position information of the word in the first image.

[0072] In some embodiments, the words are identified based on the second image. Therefore, when performing word recognition on the second image, the second position information of the words in the text block in the second image can be determined. By mapping the second position information to the coordinate system of the first image, the first position information of the words in the first image can be determined.

[0073] It should be noted that the second image is obtained by reconstructing image regions from the first image. Each image region is associated with a text block, which can determine the position transformation relationship of the text block from the first image to the second image. Based on the position transformation relationship, the second position information is inversely transformed to determine the first position information of the word in the first image.

[0074] In some embodiments, by determining the third position information of the text block associated with the image region in the first image and the fourth position information of the text block associated with the image region in the second image, the position transformation relationship corresponding to the text block can be determined based on the third position information and the fourth position information.

[0075] In this disclosure, the position transformation relationship corresponding to the text block is determined based on the third position information of the text block in the first image and the fourth position information in the second image, thereby improving the accuracy of mapping the second position information of words to the first image based on the position transformation relationship.

[0076] In some embodiments, the position transformation relationship can be a transformation matrix between position information. The transformation matrix can then be solved based on the third and fourth position information to obtain the position transformation relationship.

[0077] In some embodiments, the inverse transformation matrix corresponding to the transformation matrix can be obtained, and the second position information can be multiplied by the inverse transformation matrix. This enables the inverse transformation of the second position information based on the position transformation relationship, thereby obtaining the full text content with the correct semantic order and the precise position of each word in the first image, realizing the structured mapping between the textbook content and the word positions.

[0078] For example, when obtaining a text block sequence based on the first image, the third position information of each text block in the first image can be obtained, denoted as A. When recombining the image regions in the first image based on the text block sequence to obtain the second image, the fourth position information of the text blocks in the second image can be determined, denoted as A'. Furthermore, based on A and A', the mapping relationship from the first image to the second image, i.e., the position transformation relationship, can be determined, denoted as T.

[0079] Furthermore, by performing word recognition on the second image, the words in the text block and their second position information in the second image are obtained, denoted as B'. Then, the inverse transform T of T is obtained. -1 and using T -1 Map B' to the coordinate system of the first image to obtain the first position information, denoted as B.

[0080] S205: Based on the first position information of words in the first image, align and bind the annotation information set and each text block to obtain structured annotation data.

[0081] In some embodiments, the words corresponding to each annotation in the annotation information set can be determined, and the first position information of the words in the first image can be bound to the annotation information to obtain structured annotation data.

[0082] It should be noted that the annotation information set includes a subset of annotation information for each text block, and the subset of annotation information includes annotation information at different levels of the text block.

[0083] In some embodiments, different levels may include word, sentence, paragraph, and other levels. By generating annotation information for multiple levels, multi-granularity analysis of text blocks can be achieved to obtain comprehensive annotation information.

[0084] In other words, by aligning and binding the annotation information set with each text block, we can bind annotation information for words, sentences, and paragraphs, thereby achieving comprehensive binding of annotation information to text blocks and improving the comprehensiveness of structured annotation data.

[0085] In some embodiments, the binding of annotation information can be implemented based on the positional information of different levels. The fifth positional information of sentences and the sixth positional information of paragraphs in a text block can be determined based on the first positional information of words, thereby determining the annotation information associated with the text block and its included words, sentences, and paragraphs, and binding the associated annotation information with their respective positional information to obtain the substructured annotation data of the text block.

[0086] In other words, for any given text block, any text block can consist of at least one paragraph, each paragraph can consist of at least one sentence, and each sentence can consist of at least one word.

[0087] For words with annotation information, the first position information can be bound to the annotation information associated with the word; for sentences with annotation information, the fifth position information can be bound to the annotation information associated with the sentence; for paragraphs with annotation information, the sixth position information can be bound to the annotation information associated with the paragraph.

[0088] In some embodiments, after obtaining the sub-structured annotation data of a text block, structured annotation data can be obtained based on the sub-structured annotation data of the text block. Optionally, the sub-structured annotation data can be arranged and combined according to the semantic order of the text block to obtain structured annotation data.

[0089] In some embodiments, the fifth position information of a sentence in a text block can be determined based on the first position information of the words contained in the sentence. Optionally, the position information of the region where the sentence is located is determined based on the first position information of each word contained in the sentence, and is used as the fifth position information.

[0090] For example, if there is a sentence consisting of 3 words, and the first position information corresponding to each word is: position 1, position 2 and position 3, then the fifth position information of the sentence can be formed based on position 1, position 2 and position 3.

[0091] In some embodiments, in order to save computing resources and quickly locate sentences, the first initial word and the first final word of the sentence can be determined, and the fifth position information of the sentence can be determined based on the first position information of the first initial word and the first position information of the first final word.

[0092] For example, based on the first position information of the first word and the first position information of the last word, the minimum bounding rectangle of the sentence can be determined, and the position information of the minimum bounding rectangle can be used as the fifth position information.

[0093] Similarly, the sixth position information of a paragraph within a text block can be determined based on the first position information of the words contained in the paragraph. Optionally, the position information of the area where the paragraph is located can be determined based on the first position information of each word contained in the paragraph, and used as the sixth position information.

[0094] In some embodiments, the second first word and the second last word of the paragraph can also be determined, and the fifth position information of the paragraph can be determined based on the first position information of the second first word and the first position information of the second last word, so as to save computing resources for calculating the sixth position information and realize rapid positioning of the paragraph.

[0095] In other words, the minimum bounding rectangle of a paragraph can be determined based on the first position information of the second word and the first position information of the second word, and the position information of the minimum bounding rectangle of the paragraph can be used as the sixth position information.

[0096] S206 sends structured annotation data to the client. The structured annotation data is used to render the annotation information of the textbook to be annotated.

[0097] The details of step S206 can be found in the above embodiments and will not be repeated here.

[0098] According to the textbook annotation method provided in this disclosure, by determining the first position information of words in the first image, and based on the first position information of words in the first image, aligning and binding the annotation information set and each text block to obtain structured annotation data, it is possible to obtain structured annotation data carrying precise position information. The structured annotation data can establish a precise correspondence between annotation information and positional information in the first image, allowing the structured annotation data to be directly consumed by the client, thereby achieving fully automatic annotation of the textbook.

[0099] Figure 3 This is a flowchart illustrating another annotation method for teaching materials provided in this embodiment of the disclosure.

[0100] like Figure 3 As shown, the annotation methods in this textbook may include: S301, Obtain the first image of the textbook to be annotated, and obtain a text block sequence based on the first image, wherein the text blocks in the text block sequence are arranged in semantic order.

[0101] The relevant content for obtaining the first image of the teaching material to be annotated in step S301 can be found in the above embodiments, and will not be repeated here.

[0102] In some embodiments, to obtain text blocks with the correct semantic order, the semantic order of the text blocks can be determined by performing semantic understanding on the first image. For example, the first image can be divided into multiple image regions by performing semantic understanding on the first image, thereby determining the semantic order of the image regions. Here, an image region is a region from which text blocks can be extracted; that is, text blocks of the first image can be extracted from the image region.

[0103] In some embodiments, semantic understanding is performed on the first image, and one or more text blocks are obtained based on the semantic understanding results. That is, the semantic understanding results may be image regions of the first image, thereby extracting text blocks from the image regions.

[0104] In some embodiments, semantic understanding of the first image may involve determining the geometric structure of the first image, such as headers, footers, column lines, and image borders, to determine the image region in the first image based on the geometric structure, which serves as the semantic understanding result.

[0105] In some embodiments, after determining the text blocks, the semantic order of one or more text blocks can be determined based on their semantics. For example, for any two text blocks, their semantic similarity is calculated, and the two text blocks with high semantic similarity are taken as adjacent text blocks, thereby determining the semantic order of one or more text blocks.

[0106] In some embodiments, one or more text blocks can be combined based on semantic order to obtain a text block sequence.

[0107] In some embodiments, a first image can be input into a multimodal first model, which outputs a text block sequence and the third position information of each text block in the first image. That is, the first image can be input into the first model, which performs semantic understanding and text block extraction on the first image to determine the text block sequence and the third position information of each text block in the first image.

[0108] In this disclosure, since the first multimodal model has semantic understanding capabilities, the order of the output text blocks is the correct semantic order, thereby avoiding the problem of disordered order in complex layouts in traditional OCR, and providing full-text content with the correct semantic order for the generation of subsequent annotation information.

[0109] S302, Determine the image area of ​​the text block in the first image.

[0110] S303, Reorganize the image regions in the first image based on semantic order to obtain the second image.

[0111] In some embodiments, the text block is extracted from an image region in the first image, and the image region associated with the text block can be determined as the image region of the text block in the first image.

[0112] It should be noted that the image regions in the first image are arranged according to the original layout order of the first image. In order to eliminate the column segmentation and layout interference of the original layout order, and to provide input with a definite order and a single layout for subsequent word-level recognition, the image regions in the first image can be reorganized based on the semantic order to obtain the second image.

[0113] For example, based on semantic order, the image regions in the first image can be arranged from top to bottom to be stitched together to form the second image.

[0114] In some embodiments, during the process of reconstructing to obtain the second image, the positional transformation relationship of each text block from the first image to the second image can be recorded, that is, the positional transformation relationship from the third positional information of the text block in the first image to the fourth positional information of the text block in the second image.

[0115] In some embodiments, after determining the second image, the second image can be input into the second model, which outputs the words in the text block and their second position information in the second image. That is, the second image can be input into the second model, which performs word-level OCR on the second image to determine the words in the text block and their second position information in the second image.

[0116] It should be noted that the first model can be a multimodal large model, while the second model can be a dedicated small model specifically for OCR of images.

[0117] In this disclosure, the semantic order of the first image is identified by the first model, and the second image is obtained by recombining the semantic order. The second model is then used to identify the words in the second image. This can achieve the fusion of two different types of OCR to obtain the full text content with the correct semantic order and the precise position of each word in the image, thus realizing a structured mapping between the textbook content and the position of words.

[0118] In some embodiments, the first model has already output a sequence of text blocks, and the first image can be cropped based on the sequence of text blocks to obtain the image region of each text block, so that the text blocks are fed into the second model in semantic order to obtain the words in the text blocks.

[0119] S304, perform word recognition on the second image to obtain words in the text block.

[0120] S305, Based on the text block sequence, obtain a set of annotation information, wherein the set of annotation information includes a subset of annotation information of at least one text block.

[0121] S306, based on the words in the text blocks, aligns and binds the annotation information set and each text block to obtain structured annotation data.

[0122] S307 sends structured annotation data to the client. The structured annotation data is used to render the annotation information of the textbook to be annotated.

[0123] The relevant content of steps S304-S307 can be found in the above embodiments, and will not be repeated here.

[0124] According to the annotation method of the teaching materials provided in the embodiments of this disclosure, by determining the image region of the text block in the first image and reorganizing the image region in the first image based on the semantic order, a second image is obtained, so as to obtain a second image with the correct semantic order, thereby recognizing words from the second image, which not only ensures the correctness of the semantic order, but also obtains precise word-level positioning.

[0125] Figure 4 This is a flowchart illustrating another annotation method for teaching materials provided in this embodiment of the disclosure.

[0126] like Figure 4 As shown, the annotation methods in this textbook may include: S401: Collect the first image of the textbook to be annotated and send the first image to the server. The first image includes multiple image regions, and different image regions correspond to different text blocks.

[0127] It should be noted that the execution entity of the annotation method in the teaching materials in this embodiment can be a hardware device with data processing capabilities and / or the necessary software to drive the hardware device. Optionally, the execution entity may include a user terminal and other smart devices. Optionally, the user terminal includes, but is not limited to, mobile phones, computers, smart voice interaction devices, etc. This embodiment does not impose specific limitations.

[0128] In some embodiments, the teaching material to be annotated can be an electronic textbook. For example, a screenshot of any page of the electronic textbook can be used as the first image and sent to the server.

[0129] In some embodiments, the teaching materials to be annotated can also be paper teaching materials. Users can take pictures of the paper teaching materials and upload them to the client. The client then sends the picture to the server as the first image.

[0130] In this disclosure, the teaching materials can be subject textbooks, extracurricular reading materials, test papers, etc., without specific limitations on the teaching materials.

[0131] In some embodiments, the first image has a layout order, that is, the first image can be divided into multiple image regions based on the layout order, and text blocks can be extracted from each image region.

[0132] S402 receives structured annotation data sent by the server.

[0133] In some embodiments, after the first image is sent to the server, the server can perform content recognition and annotation information generation based on the first image, so as to bind the annotation information with the recognized content and receive the bound structured annotation data.

[0134] In other words, the structured annotation data includes the annotation information of the first image and the location information of the content associated with the annotation information in the first image.

[0135] In some embodiments, the content associated with the annotation information can be words, sentences, paragraphs, etc. in the text block. The binding of words, sentences, paragraphs and annotation information all need to be based on the first position information of the words. That is to say, the structured annotation data is obtained by aligning and binding the annotation information set and each text block with the words in the text block.

[0136] It should be noted that the process of generating structured annotation data can be found in any of the above embodiments, and will not be repeated here.

[0137] S403, based on structured annotation data, renders the annotation information of the first image to obtain the annotated teaching material.

[0138] In some embodiments, the content in the first image can be located based on structured annotation data, such as locating words, sentences, paragraphs, etc. in the first image according to the content associated with the annotation information, thereby rendering the annotation information at the position of the words, sentences, paragraphs in the first image, thus obtaining the annotated teaching material.

[0139] In some embodiments, a first image can be used as a base image, multiple layers can be overlaid on the first image, and annotation information can be rendered on the drawn image layers to obtain an annotated teaching material. For example, the multiple images may include a highlight layer and a text layer.

[0140] By drawing transparent color blocks in the highlight layer based on the positional information of words, sentences, and paragraphs in the first image, and determining the position of the annotation information in the text layer based on the positional information of words, sentences, and paragraphs in the first image, the annotation information is drawn at that position. By compositing all layers, the annotated teaching material can be obtained.

[0141] In some embodiments, after obtaining the annotated teaching materials, the client can interact with the user, display the annotated teaching materials on the client's display interface, and interact with the user based on the display interface.

[0142] In some embodiments, by receiving user interaction operations and determining the target location information hit by the user interaction operations in the annotated teaching materials, the target annotation information can be determined and displayed based on the target location information.

[0143] In other words, by clicking on any word, sentence, or paragraph on the display interface, the client instantly displays the corresponding annotation information based on the target location information corresponding to the click operation. For example, clicking on a word displays its definition and pronunciation; clicking on a sentence displays its meaning and rhetorical explanation; clicking on a paragraph displays its main idea, etc.

[0144] In this publication, users can click on any uploaded teaching material to be annotated, or any words, sentences, or paragraphs at any location and instantly receive multi-level explanations, thus expanding the scope of interaction and improving immediacy.

[0145] In some embodiments, if any word or phrase has word-level annotation information, sentence-level annotation information, or paragraph-level annotation information, the display priority of different levels of annotation information can be set, and the annotation information can be displayed according to the set display priority.

[0146] According to the textbook annotation method provided in this disclosure, a first image of the textbook to be annotated is acquired, sent to a server, and structured annotation data based on the first image is received from the server. Based on the structured annotation data, the annotation information of the first image can be rendered to obtain the annotated textbook. Therefore, this solution can achieve fully automated generation of annotation information, significantly reducing manpower and time costs, and enabling annotation of any textbook.

[0147] Figure 5 This is a schematic diagram of the process for generating structured annotation data provided in an embodiment of this disclosure. The process involves receiving a first image of the teaching material to be annotated from a client, and then performing OCR on the first image using a first model to obtain a sequence of text blocks in the first image and the third position information of each text block in the first image. A second model is then used to perform OCR to obtain words in the text blocks and the second position information of the words in the second image.

[0148] In this context, OCR using the second model refers to recombining the image regions in the first image based on semantic order to obtain the second image, and then performing OCR on the second image using the second model.

[0149] Semantic analysis of text block sequences is performed using a large language model to determine annotation information at different levels within the text blocks, thus obtaining an annotation information set. Based on the second position information of words in the second image and the positional transformation relationship of text blocks mapped from the first image to the second image, the first position information of words in the first image is determined. Furthermore, based on the first position information, the annotation information set and each text block can be aligned and bound to obtain structured annotation data.

[0150] Corresponding to the annotation methods of teaching materials provided in the above embodiments, an embodiment of this disclosure also provides an annotation device for teaching materials. Since the annotation device for teaching materials provided in this disclosure corresponds to the annotation methods of teaching materials provided in the above embodiments, the implementation methods of the above annotation methods for teaching materials are also applicable to the annotation device for teaching materials provided in this disclosure, and will not be described in detail in the following embodiments.

[0151] Figure 6 This is a schematic diagram of the structure of a textbook annotation device provided in an embodiment of this disclosure.

[0152] like Figure 6 As shown, the textbook annotation device 600 of this embodiment includes an acquisition module 601, an identification module 602, an annotation module 603, a binding module 604, and a sending module 605.

[0153] The acquisition module 601 is used to acquire a first image of the teaching material to be annotated, and to acquire a text block sequence based on the first image, wherein the text blocks in the text block sequence are arranged in semantic order; The recognition module 602 is used to reconstruct the image region in the first image based on the text block sequence to obtain the second image of the textbook to be annotated, and to perform word recognition on the second image to obtain the words in the text block; Annotation module 603 is used to obtain an annotation information set based on the text block sequence, wherein the annotation information set includes at least one subset of annotation information for the text block; The binding module 604 is used to align and bind the annotation information set and each text block based on the words in the text block to obtain structured annotation data; The sending module 605 is used to send the structured annotation data to the client, and the structured annotation data is used to render the annotation information of the textbook to be annotated.

[0154] In one embodiment of this disclosure, the binding module 604 is further configured to: determine the first position information of the word in the first image; and based on the first position information of the word in the first image, align and bind the annotation information set and each text block to obtain the structured annotation data.

[0155] In one embodiment of this disclosure, the binding module 604 is further configured to: determine the position transformation relationship of the text block from the first image to the second image; determine the second position information of the words in the text block in the second image; and perform an inverse transformation on the second position information based on the position transformation relationship to determine the first position information of the words in the first image.

[0156] In one embodiment of this disclosure, the binding module 604 is further configured to: determine the third position information of the text block associated with the image region in the first image; determine the fourth position information of the text block associated with the image region in the second image; and determine the position transformation relationship corresponding to the text block based on the third position information and the fourth position information.

[0157] In one embodiment of this disclosure, the subset of annotation information includes annotation information at different levels of the text block. The binding module 604 is further configured to: determine the fifth position information of the sentence and the sixth position information of the paragraph in the text block based on the first position information of the words; determine the annotation information associated with the text block and its included words, sentences, and paragraphs, and bind the associated annotation information with their respective position information to obtain the sub-structured annotation data of the text block; and obtain the structured annotation data based on the sub-structured annotation data of the text block.

[0158] In one embodiment of this disclosure, the binding module 604 is further configured to: determine the first initial word and the first final word included in the sentence; and determine the fifth position information of the sentence based on the first position information of the first initial word and the first position information of the first final word.

[0159] In one embodiment of this disclosure, the binding module 604 is further configured to: determine the second first word and the second last word included in the paragraph; and determine the fifth position information of the paragraph based on the first position information of the second first word and the first position information of the second last word.

[0160] In one embodiment of this disclosure, the identification module 602 is further configured to: Determine the image region of the text block in the first image; The image regions in the first image are reorganized based on the semantic order to obtain the second image.

[0161] In one embodiment of this disclosure, the acquisition module 601 is further configured to: perform semantic understanding on the first image and acquire one or more text blocks based on the semantic understanding result; determine the semantic order of the one or more text blocks based on the semantics of the text blocks; and combine the one or more text blocks based on the semantic order to obtain the text block sequence.

[0162] In one embodiment of this disclosure, the acquisition module 601 is configured to: invoke a multimodal first model, input the first image into the first model, and output the text block sequence and the third position information of each text block in the text block sequence in the first image.

[0163] In one embodiment of this disclosure, the recognition module 602 is further configured to: invoke a second model, input the second image into the second model, and output the words in the text block and the second position information of the words in the second image.

[0164] According to the textbook annotation device provided in this embodiment, a first image of the textbook to be annotated is acquired, a text block sequence is obtained based on the first image, and the image regions in the first image are recombined based on the text block sequence to obtain a second image of the textbook to be annotated. Word recognition is then performed on the second image to obtain the words in the text blocks. Further, an annotation information set can be obtained based on the text block sequence, and the annotation information set and each text block are aligned and bound based on the words in the text blocks to obtain structured annotation data. This structured annotation data is then sent to the client, enabling the client to render the annotation information of the textbook to be annotated based on the structured annotation data. Therefore, this solution can achieve fully automated generation of annotation information, significantly reducing manpower and time costs. By acquiring the first image of the textbook to be annotated to generate annotation information, annotations for different types of textbooks can be generated, improving the generalization ability of annotation information generation.

[0165] In addition, identifying text blocks and words in the image separately not only ensures the correctness of the semantic order of text under complex layouts, but also achieves precise word-level positioning, which helps to improve the accuracy of binding annotation information with text blocks.

[0166] Figure 7 A schematic diagram of another annotation device for teaching materials provided in this embodiment of the present disclosure.

[0167] like Figure 7 As shown, the textbook annotation device 700 of this embodiment includes a sending module 701, a receiving module 702 and a rendering module 703.

[0168] The sending module 701 is used to acquire the first image of the teaching material to be annotated and send the first image to the server. The first image includes multiple image regions, and different image regions correspond to different text blocks. The receiving module 702 is used to receive structured annotation data sent by the server. The structured annotation data includes annotation information of the first image and the position information of the content associated with the annotation information in the first image. The structured annotation data is obtained by aligning and binding the annotation information set and each text block based on the words in the text block. The rendering module 703 is used to render the annotation information of the first image based on the structured annotation data to obtain the annotated teaching material.

[0169] In one embodiment of this disclosure, the rendering module 703 is further configured to: receive user interaction operations; determine the target location information hit by the user interaction operations in the annotated teaching materials; determine the hit target annotation information based on the target location information, and display the target annotation information.

[0170] The textbook annotation device provided in this embodiment of the present disclosure acquires a first image of the textbook to be annotated, sends the first image to a server, and receives structured annotation data sent by the server based on the first image. Based on the structured annotation data, the annotation information of the first image can be rendered to obtain the annotated textbook. Therefore, this solution can achieve fully automated generation of annotation information, significantly reducing manpower and time costs, and enabling annotation of any textbook.

[0171] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0172] The examples in this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and provisions. In the examples of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario, and the scope of the solution is not limited in this regard.

[0173] In this manual and the sample solutions, any processing of personal information will be conducted only on a legal basis (such as with the consent of the data subject or as necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0174] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0175] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0176] like Figure 8As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on computer programs / instructions stored in read-only memory (ROM) 802 or loaded into random access memory (RAM) 803 from storage unit 806. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0177] Multiple components in device 800 are connected to I / O interface 805, including: input units 806 such as keyboard, mouse, etc.; output units 807 such as various types of displays, speakers, etc.; storage units 808 such as disks, optical disks, etc.; and communication units 809 such as network cards, modems, wireless transceivers, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0178] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the annotation method for teaching materials. For example, in some embodiments, the annotation method for teaching materials can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as storage unit 806. In some embodiments, part or all of the computer program / instructions can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program / instructions are loaded into RAM 803 and executed by the computing unit 801, one or more steps of the annotation method for teaching materials described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the annotation method for teaching materials by any other suitable means (e.g., by means of firmware).

[0179] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs / instructions that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0180] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0181] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0182] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0183] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0184] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. The client-server relationship is created by computer programs / instructions running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0185] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in the disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this document does not impose any restrictions.

[0186] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for annotating teaching materials, wherein, The method includes: Obtain a first image of the teaching material to be annotated, and obtain a text block sequence based on the first image, wherein the text blocks in the text block sequence are arranged in semantic order; Based on the text block sequence, the image regions in the first image are reconstructed to obtain the second image of the textbook to be annotated, and word recognition is performed on the second image to obtain the words in the text block; Based on the text block sequence, an annotation information set is obtained, wherein the annotation information set includes at least one subset of annotation information for the text block; Based on the words in the text blocks, the annotation information set and each text block are aligned and bound to obtain structured annotation data; The structured annotation data is sent to the client, and the structured annotation data is used to render the annotation information of the textbook to be annotated.

2. The method according to claim 1, wherein, Before aligning and binding the annotation information set and each text block based on the words in the text block to obtain structured annotation data, the method further includes: Determine the first position information of the word in the first image; Based on the first position information of the words in the first image, the annotation information set and each text block are aligned and bound to obtain the structured annotation data.

3. The method according to claim 2, wherein, Determining the first position information of the word in the first image includes: Determine the positional transformation relationship of the text block from the first image to the second image; Determine the second position information of the words in the text block in the second image; Based on the position transformation relationship, the second position information is inversely transformed to determine the first position information of the word in the first image.

4. The method according to claim 3, wherein, Determining the positional transformation relationship of the text block from the first image to the second image includes: Determine the third position information of the text block associated with the image region in the first image; Determine the fourth position information of the text block associated with the image region in the second image; Based on the third and fourth position information, the position transformation relationship corresponding to the text block is determined.

5. The method according to any one of claims 2-4, wherein, The subset of annotation information includes annotation information at different levels of the text blocks. The step of aligning and binding the annotation information set and each text block based on the first position information of the words in the first image to obtain the structured annotation data includes: Based on the first position information of the words, the fifth position information of the sentence and the sixth position information of the paragraph in the text block are determined; Determine the annotation information associated with each of the text blocks and their included words, sentences, and paragraphs, and bind the associated annotation information with their respective position information to obtain the substructured annotation data of the text block; The structured annotation data is obtained based on the sub-structured annotation data of the text block.

6. The method according to claim 5, wherein, The process of determining the fifth position information of a sentence in the text block includes: Determine the first and last words of the sentence; Based on the first position information of the first word and the first position information of the first word, the fifth position information of the sentence is determined.

7. The method according to claim 5, wherein, The process of determining the sixth position information of a paragraph in the text block includes: Determine the second first word and the second last word included in the paragraph; Based on the first position information of the second first word and the first position information of the second last word, the fifth position information of the paragraph is determined.

8. The method according to any one of claims 1-4, wherein, The step of reconstructing the image regions in the first image based on the text block sequence to obtain the second image of the teaching material to be annotated includes: Determine the image region of the text block in the first image; The image regions in the first image are reorganized based on the semantic order to obtain the second image.

9. The method according to claim 8, wherein, The step of obtaining the text block sequence based on the first image includes: Perform semantic understanding on the first image, and obtain one or more text blocks based on the semantic understanding results; Based on the semantics of the text blocks, determine the semantic order of the one or more text blocks; Based on the semantic order, the one or more text blocks are combined to obtain the text block sequence.

10. The method according to claim 9, wherein, The method further includes: The first multimodal model is invoked, the first image is input into the first model, and the text block sequence and the third position information of each text block in the text block sequence in the first image are output.

11. The method according to claim 8, wherein, The method further includes: The second model is invoked, the second image is input into the second model, and the words in the text block and the second position information of the words in the second image are output.

12. A method for annotating teaching materials, wherein, The method includes: The first image of the textbook to be annotated is acquired and sent to the server. The first image includes multiple image regions, and different image regions correspond to different text blocks. The system receives structured annotation data sent by the server. The structured annotation data includes annotation information of the first image and the position information of the content associated with the annotation information in the first image. The structured annotation data is obtained by aligning and binding the annotation information set and each text block based on the words in the text block. Based on the structured annotation data, the annotation information of the first image is rendered to obtain an annotated teaching material.

13. The method according to claim 12, wherein, After rendering the annotation information of the first image based on the structured annotation data to obtain the annotated teaching material, the process further includes: Receive user interaction operations; Determine the target location information of the user interaction operation in the annotated teaching materials; Based on the target location information, the hit target annotation information is determined and displayed.

14. An annotation device for teaching materials, wherein, The device includes: The acquisition module is used to acquire a first image of the teaching material to be annotated, and to acquire a text block sequence based on the first image, wherein the text blocks in the text block sequence are arranged in semantic order; The recognition module is used to reconstruct the image region in the first image based on the text block sequence to obtain the second image of the textbook to be annotated, and to perform word recognition on the second image to obtain the words in the text block; An annotation module is used to obtain an annotation information set based on the text block sequence, wherein the annotation information set includes at least one subset of annotation information of the text block; The binding module is used to align and bind the annotation information set and each text block based on the words in the text block to obtain structured annotation data; The sending module is used to send the structured annotation data to the client, and the structured annotation data is used to render the annotation information of the textbook to be annotated.

15. An annotation device for teaching materials, wherein, The device includes: The sending module is used to acquire the first image of the teaching material to be annotated and send the first image to the server. The first image includes multiple image regions, and different image regions correspond to different text blocks. The receiving module is used to receive structured annotation data sent by the server. The structured annotation data includes annotation information of the first image and the position information of the content associated with the annotation information in the first image. The structured annotation data is obtained by aligning and binding the annotation information set and each text block based on the words in the text block. The rendering module is used to render the annotation information of the first image based on the structured annotation data to obtain the annotated teaching materials.

16. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-13.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-13.

18. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the method of any one of claims 1-13.

Images