A child book content semantic analysis method and system based on image recognition

Abstract

The application discloses a kind of children's book content semantic analysis method and system based on image recognition, including: graphic processing module, image recognition module, graphic matching module, semantic analysis module, intelligent answering module, to provide accurate, safe and meet cognitive level's book content semantic analysis service for children.The present application fuses multimodal big model image recognition technology, based on mass children's book graphic data training, can accurately identify the illustration, scene and character in book, and in-depth understanding of the semantic information of book content is combined with natural language processing technology.Accurately guide is provided for children by the semantic analysis of graphic and text in full combination through the image recognition mode of multiple links, accurately grasp the interest point and understanding ability of children in reading process, and dynamically according to the feedback of children, answer and content optimization, truly realize the intelligent, accurate children's book content semantic analysis.

Description

Technical Field

[0001] This invention relates to the field of computer science and technology, and is applied to semantic analysis of book content. In particular, it relates to a method and system for semantic analysis of children's book content based on image recognition. Background Technology

[0002] The core objective of semantic analysis in children's books is to provide children with book content that is appropriate for their cognitive level and safe to read, ideally answering questions during the reading process to aid comprehension. Traditional semantic analysis methods can only perform text recognition, offering limited assistance to children through simple summaries of book titles, introductions, and chapter content. However, this method has significant limitations. Children's books differ from ordinary adult books, often containing numerous images, and semantic analysis relying solely on text recognition cannot fully comprehend the content. With the development of image recognition technology, some methods attempt to convert images into text for semantic analysis, but the integration of text and image information is insufficient, failing to achieve true multimodal fusion. This limits the intelligence of semantic analysis in children's books to some extent, making it impossible to accurately grasp children's interests and comprehension abilities during reading, or to dynamically optimize content based on children's real-time feedback.

[0003] To better integrate textual and image information and overcome the challenges of semantic analysis in children's books, this invention integrates multimodal large-scale image recognition technology. Trained on massive amounts of textual and image data from children's books, it designs a novel semantic analysis system for children's book content. This system possesses strong image recognition and semantic understanding capabilities, accurately identifying illustrations, scenes, and characters in books, and combining natural language processing technology to deeply understand the semantic information of the book content. This invention also proposes an innovative method for semantic analysis of children's book content. Based on multi-link image recognition, it fully integrates textual and image information for semantic analysis, thereby providing precise guidance for children, accurately grasping their interests and comprehension abilities during reading, and dynamically providing answers and optimizing content based on children's feedback. This truly achieves intelligent and precise semantic analysis of children's book content. Summary of the Invention

[0004] To address the shortcomings of existing detection methods, a semantic analysis system for children's book content based on image recognition is proposed. The system includes: an image and text processing module, an image recognition module, an image and text matching module, a semantic analysis module, and an intelligent question-answering module.

[0005] The image and text processing module performs image and text segmentation, text processing (extraction, error correction, segmentation), image segmentation, sorting, and content security monitoring on the book pages, providing basic data for subsequent analysis;

[0006] The image recognition module identifies and analyzes image content through multiple links (text, scene, role, object), extracts key information and generates text descriptions, providing semantic information from the image side for image-text fusion;

[0007] The image-text matching module combines the semantically segmented text and image content lists, and reorders them using a multimodal large model to optimize the image-text layout order and ensure content coherence.

[0008] The semantic analysis module performs semantic analysis on the book content, extracts key information, generates chapter and full book summaries, performs sentiment analysis, and extracts knowledge into the database to provide data support for subsequent applications.

[0009] The intelligent Q&A module matches the database content with the child's questions and generates accurate answers. At the same time, it analyzes the learning progress in real time and provides personalized tutoring suggestions to help children learn.

[0010] The multimodal large model is a deep learning model capable of processing and understanding multimodal data. It combines the natural language processing capabilities of a large language model with the visual perception capabilities of a large visual model, enabling more comprehensive and intelligent information processing and text-image interaction. The multimodal large model is a large model fine-tuned and trained on specific tasks related to children's books, making it more adaptable to the system. The specific fine-tuning and training process involves designing a dedicated dataset for a specific task and using the dedicated dataset to train the multimodal large model. Compared to general multimodal large models, the multimodal large model used in this invention can better understand the semantic and visual features of children's books, thus exhibiting higher accuracy and efficiency in children's book content analysis and semantic understanding tasks.

[0011] Furthermore, this invention also relates to a semantic analysis method for children's book content based on image recognition. This method, applied to the aforementioned image recognition-based semantic analysis system for children's book content, includes the following steps:

[0012] Step S1: Use object detection technology to identify the content of each page of the book, analyze the image and text positions, segment the image and text, and obtain the initial image box and the initial text box;

[0013] Step S2: Recognize, correct, and segment the initial text box;

[0014] Step S3: Segment, sort, and perform content security monitoring on the initial image frame;

[0015] Step S4: Use a multimodal large model to understand and identify the list of secondary images from multiple links; identify all secondary images in the same initial image frame, and the identified information is in text format, which is convenient for subsequent semantic analysis;

[0016] Step S5: Reorder the semantically segmented text from Step S2 in conjunction with the secondary image content list from Step S4.

[0017] Step S6: Utilize the language understanding capabilities of a multimodal large model to perform semantic analysis on the content of children's books, extract key information, and record the data into the database;

[0018] Step S7: When a child asks a question about a certain part of the book, the child's question is matched with the content of the database. Using the intelligent question-and-answer capabilities of the multimodal big data model, an accurate answer is generated to help the child understand the book content. The child's learning progress is analyzed in real time, and personalized tutoring suggestions are provided.

[0019] This invention utilizes multimodal large-scale modeling capabilities for textual and graphical comprehension and in-depth analysis to ensure accurate interpretation and efficient learning of children's book content, thereby supporting children's all-round development. Beneficial effects

[0020] To better integrate textual and image information and overcome the challenges of semantic analysis in children's books, this invention integrates multimodal large-scale image recognition technology. Trained on massive amounts of textual and image data from children's books, it designs a novel semantic analysis system for children's book content. This system possesses strong image recognition and semantic understanding capabilities, accurately identifying illustrations, scenes, and characters in books, and combining natural language processing technology to deeply understand the semantic information of the book content. This invention also proposes an innovative method for semantic analysis of children's book content. Based on multi-link image recognition, it fully integrates textual and image information for semantic analysis, thereby providing precise guidance for children, accurately grasping their interests and comprehension abilities during reading, and dynamically providing answers and optimizing content based on children's feedback. This truly achieves intelligent and precise semantic analysis of children's book content. Attached Figure Description

[0021] Figure 1 This is an architecture diagram of a children's book content semantic analysis system based on image recognition, according to the present invention. Detailed Implementation

[0022] Embodiment 1 of the present invention relates to a semantic analysis system for children's book content based on image recognition, as shown in the attached figure. Figure 1 As shown, the system includes: a text and image processing module, an image recognition module, a text and image matching module, a semantic analysis module, and an intelligent Q&A module;

[0023] The image and text processing module performs image and text segmentation, text processing (extraction, error correction, segmentation), image segmentation, sorting, and content security monitoring on the book pages, providing basic data for subsequent analysis;

[0024] The image recognition module identifies and analyzes image content through multiple links (text, scene, role, object), extracts key information and generates text descriptions, providing semantic information from the image side for image-text fusion;

[0025] The image-text matching module combines the semantically segmented text and image content lists, and reorders them using a multimodal large model to optimize the image-text layout order and ensure content coherence.

[0026] The semantic analysis module performs semantic analysis on the book content, extracts key information, generates chapter and full book summaries, performs sentiment analysis, and extracts knowledge into the database to provide data support for subsequent applications.

[0027] The intelligent Q&A module matches the database content with the child's questions and generates accurate answers. At the same time, it analyzes the learning progress in real time and provides personalized tutoring suggestions to help children learn.

[0028] The multimodal large model is a deep learning model capable of processing and understanding multimodal data. It combines the natural language processing capabilities of a large language model with the visual perception capabilities of a large visual model, enabling more comprehensive and intelligent information processing and graphic interaction.

[0029] Embodiment 2 of the present invention relates to a semantic analysis method for children's book content based on image recognition. The method is implemented by the system of Embodiment 1 and includes the following steps:

[0030] Step S1: Use object detection technology to identify the content of each page of the book, analyze the image and text positions, segment the image and text, and obtain the initial image box and the initial text box;

[0031] Step S2: Recognize, correct, and segment the initial text box;

[0032] Step S21: Use OCR image text recognition technology to extract text from the initial text box to obtain all the text in the text box. Since the accuracy of OCR technology is limited, the text needs to be further processed.

[0033] Step S22: Correct the text obtained in step S21 by replacing the wrong words in the text with the correct words based on the obfuscation dictionary, and removing whitespace and garbled characters.

[0034] Step S23: Segment the text semantically to adapt to the image-text ordering, specifically as follows:

[0035] Step S231: Divide the text into sentences according to the end-of-sentence marker;

[0036] Step S232: Perform intent recognition on a single sentence. Intent recognition refers to the process of understanding the user's true intent by analyzing the user's language input (text or speech) and mapping it to a predefined intent category.

[0037] Step S233: Vectorize the single sentence to obtain the single sentence text vector, vectorize the intent identified in the single sentence to obtain the single sentence intent vector, and concatenate the single sentence text vector and the single sentence intent vector to obtain the single sentence vector;

[0038] Step S234: Calculate the cosine similarity of continuous single-sentence vectors. If the similarity between the upper and lower sentences is greater than the threshold, the two sentences are considered to be in the same paragraph. Thus, the semantic text segmentation is completed.

[0039] Vectorization refers to converting text into computer-computable vectors to facilitate similarity calculation.

[0040] Step S3: Segment, sort, and perform content security monitoring on the initial image frame; multiple images may exist in the initial image frame, requiring further processing;

[0041] Step S31: Based on the object detection technology, the self-trained model performs fine recognition in the initial image box, selects multiple possible images, and records them as secondary images; This invention is based on its own image data for annotation, uses YOLO technology to train an object detection model specifically for secondary image segmentation, applies it to this step, and obtains a list of secondary image boxes;

[0042] Step S32: Sort the secondary image box list to obtain the correct image order; input the coordinates of the initial image box and the secondary image box list into the multimodal large model, and input the task to make the multimodal large model give the correct image order; the multimodal large model is a deep learning model that can process and understand multimodal data. It combines the natural language processing capabilities of a large language model and the visual perception capabilities of a large vision model, and can achieve more comprehensive and intelligent information processing and graphic interaction; the multimodal large model used in this invention is a large model that has been fine-tuned and trained on specific tasks related to children's books, and can be more adapted to the system; the specific fine-tuning training process is to design a dedicated dataset for specific tasks and use the dedicated dataset to train the multimodal large model; compared with the general multimodal large model, the multimodal large model used in this invention can better understand the semantic and visual features of children's books, thus showing higher accuracy and efficiency in children's book content analysis and semantic understanding tasks.

[0043] Step S34: Conduct content security monitoring on the secondary image frame to detect whether there is any illegal content, ensure that the graphic information complies with legal and regulatory requirements, and guide children's correct values;

[0044] Step S4: Utilize a multimodal large model to understand and identify the secondary image list from multiple links; identify all secondary images within the same initial image frame, with the identified information in text format for easy subsequent semantic analysis; details are as follows:

[0045] Step S41, Text Link: Use a multimodal large model to identify the text in the secondary image to obtain the secondary image text;

[0046] Step S42, Scene Link: Use a multimodal large model to identify the scene in the secondary image and obtain the scene description of the secondary image;

[0047] Step S43, Role Link: Use a multimodal large model to identify the character appearance, character expression, character action and relationship between characters in the secondary image, collectively referred to as the secondary image role description;

[0048] Step S44, Object Linkage: Using a multimodal large model to identify objects, object states, and relationships between roles and objects in the secondary image, collectively referred to as the secondary image object description;

[0049] Step S45, Initial Image Link: Use a multimodal large model to identify the initial image from a holistic perspective, perform image summarization, and summarize it into an initial image summary;

[0050] Step S46, Multi-link Fusion: Input the initial image, the initial image information in step S45, the secondary image, and the secondary image information from steps S41 to S44 into the multimodal large model. Utilize the image and text understanding capabilities of the multimodal large model to more fully summarize the content of the secondary image.

[0051] Step S47: Perform steps S41-S46 on all secondary images to obtain a list of secondary image contents in plain text format;

[0052] This step breaks down the image elements step by step, and then combines them from multiple links to fully understand the image and accurately analyze the representative meaning of the secondary images, which is beneficial to the semantic understanding of children's books.

[0053] Step S5: Reorder the semantically segmented text from Step S2 with the secondary image content list from Step S4. Considering that the image position and the corresponding text position may not be on the same page during the typesetting process, it is necessary to reorder the text and images. The specific steps are as follows: Input the semantically segmented text and secondary image content list of three consecutive pages in the children's book into the multimodal big data model. Input the task to make the multimodal big data model give the optimal text and image typesetting order, and obtain the reordered book information. At this time, the image information is converted into text format after image recognition in Step S4. Therefore, the reordered book information is all in text format, which is convenient for content analysis. Through the intelligent decision-making ability of the multimodal big data model, the coherence of the text and image content is ensured, which is beneficial to the subsequent semantic analysis of children's book content.

[0054] Step S6: Utilize the language understanding capabilities of a multimodal large-scale model to perform semantic analysis on the content of children's books, extract key information, and record the data into a database; the specific steps are as follows:

[0055] Step S61: Divide the book information after reordering in step S5 into chapters;

[0056] Step S62, Key Sentence Extraction: Perform in-depth analysis of the text of each chapter, identify and extract key sentences, form a summary of the core content of the chapter, and ensure that key information is recorded accurately;

[0057] Step S63, Chapter Summary: Integrate the core content of each chapter to form a chapter summary;

[0058] Step S64, Chapter Sentiment Analysis: Identify the sentiment tendency of each chapter's text and determine the emotional tone;

[0059] Step S65: Overall Book Summary: Integrate the summaries of the core content of each chapter to form a summary of the entire book;

[0060] Step S66: Knowledge Extraction: Extract knowledge points from each chapter to form a knowledge base for easy application in children's education;

[0061] Step S67, Input into database: Record the content after semantic analysis in steps S61-S66 into the database for easy retrieval and application;

[0062] Step S7: When a child asks a question about a certain part of the book, the child's question is matched with the content of the database. Using the intelligent question-and-answer capabilities of the multimodal big data model, an accurate answer is generated to help the child understand the book content. The child's learning progress is analyzed in real time, and personalized tutoring suggestions are provided.

[0063] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A semantic analysis system for children's book content based on image recognition, characterized by: Image and text processing module: performs image and text segmentation, text processing, image segmentation, sorting, and content security monitoring on book pages; Image recognition module: Semantic analysis module: performs semantic analysis on book content, extracts key information, generates chapter and full book summaries, performs sentiment analysis, and extracts knowledge into the database; The image recognition module identifies and analyzes image content through multiple links, extracts key information, and generates text descriptions, providing semantic information from the image side for image-text fusion; Intelligent Q&A module: matches database content with children's questions and generates accurate answers, while analyzing learning progress in real time and providing personalized tutoring suggestions; Image and text matching module: combines the semantically segmented text and image content lists, and re-sorts them using a multimodal large model; The system executes the following methods: S1. Use object detection technology to identify the content of each page of the book, analyze the image and text positions, segment the image and text, and obtain the initial image box and the initial text box. S2. Recognize, correct, and segment the initial text box; use OCR image text recognition technology to extract the text from the initial text box to obtain all the text in the text box; S3. Segment, sort, and perform content security monitoring on the initial image frame; Based on the self-trained model, fine recognition is performed in the initial image bounding box to select multiple possible images, which are recorded as secondary images; based on the self-trained image data annotation, a target detection model specifically for secondary image segmentation is trained using YOLO to obtain a list of secondary image bounding boxes. S4. Utilize a multimodal large model to understand and identify a secondary image list from multiple links; All secondary images within the same initial image frame are identified, and the identified information is all in text format. The text in the secondary images is obtained by recognizing the text in the secondary images using a multimodal large model. S5. Reorder the semantically segmented text from S2 in conjunction with the secondary image content list from S4. S6. Utilize the language understanding capabilities of a multimodal large model to perform semantic analysis on the content of children's books, extract key information, and record the data into a database; S7. When a child asks a question about a part of the book, the system matches the child's question with the content in the database, uses the intelligent question-and-answer capabilities of the multimodal big data model to generate an answer, analyzes the child's learning progress in real time, and provides personalized tutoring suggestions.

2. The image recognition-based semantic analysis system for children's book content according to claim 1, characterized in that, Step S2 further includes: Step S22: Correct the obtained text by replacing the wrong words in the text with the correct words based on the obfuscation dictionary, and removing whitespace and garbled characters. Step S23: Segment the text from a semantic perspective to adapt to the image and text sorting.

3. The image recognition-based semantic analysis system for children's book content according to claim 2, characterized in that, The specific operation of step S23 is as follows: Step S23: Segment the text from a semantic perspective to adapt to the image and text sorting; Step S231: Divide the text into sentences according to the end-of-sentence marker; Step S232: Perform intent recognition on a single sentence. Intent recognition refers to the process of understanding the user's true intent by analyzing the user's language input and mapping it to a predefined intent category. Step S233: Vectorize the single sentence to obtain the single sentence text vector, vectorize the intent identified in the single sentence to obtain the single sentence intent vector, and concatenate the single sentence text vector and the single sentence intent vector to obtain the single sentence vector; Step S234: Calculate the cosine similarity of continuous single-sentence vectors. If the similarity between the upper and lower sentences is greater than the threshold, the two sentences are considered to be in the same paragraph. Thus, the semantic text segmentation is completed. Vectorization refers to converting text into vectors that can be computed by a computer.

4. The image recognition-based semantic analysis system for children's books according to claim 2, characterized in that, Step S3 includes: Step S32: Sort the secondary image box list to obtain the correct image order; input the coordinates of the initial image box and the secondary image box list into the multimodal large model, input the task to make the multimodal large model give the correct image order; the multimodal large model combines the natural language processing capability of the large language model and the visual perception capability of the large vision model, and can realize more comprehensive and intelligent information processing and graphic interaction; Step S34: Perform content security monitoring on the secondary image frame to detect whether there is any illegal content and ensure that the text and image information complies with legal and regulatory requirements.

5. A semantic analysis system for children's book content based on image recognition according to claim 1, characterized in that, Step S4 includes: Step S42, Scene Link: Use a multimodal large model to identify the scene in the secondary image and obtain the scene description of the secondary image; Step S43, Role Link: Use a multimodal large model to identify the character appearance, character expression, character action and relationship between characters in the secondary image, collectively referred to as the secondary image role description; Step S44, Object Linkage: Using a multimodal large model to identify objects, object states, and relationships between roles and objects in the secondary image, collectively referred to as the secondary image object description; Step S45, Initial Image Link: Use a multimodal large model to identify the initial image from a holistic perspective, perform image summarization, and summarize it into an initial image summary; Step S46, Multi-link Fusion: Input the initial image, the initial image information in step S45, the secondary image, and the secondary image information from steps S41 to S44 into the multimodal large model, and summarize the content of the secondary image; Step S47: Perform the operations of steps S41-S46 on all secondary images to obtain a list of secondary image contents in plain text format.

6. The image recognition-based semantic analysis system for children's book content according to claim 4, characterized in that, Step S5 includes: considering that the image position and the corresponding text position may not be on the same page during the typesetting process, it is necessary to reorder the text and images; the specific steps are: inputting the semantically segmented text and secondary image content list of three consecutive pages in the children's book into the multimodal big data model, inputting the task to make the multimodal big data model give the optimal text and image typesetting order, and obtaining the reordered book information. At this time, the image information is converted into text format after image recognition in step S4. Therefore, the reordered book information is all in text format, which is convenient for content analysis; through the intelligent decision-making capability of the multimodal big data model, the coherence of the text and image content is ensured, which is beneficial to the subsequent semantic analysis of children's book content.

7. The image recognition-based semantic analysis system for children's book content according to claim 4, characterized in that, Step S6 includes: Step S61: Divide the book information after reordering in step S5 into chapters; Step S62, Key Sentence Extraction: Perform in-depth analysis of the text of each chapter, identify and extract key sentences, form a summary of the core content of the chapter, and ensure that key information is recorded accurately; Step S63, Chapter Summary: Integrate the core content of each chapter to form a chapter summary; Step S64, Chapter Sentiment Analysis: Identify the sentiment tendency of each chapter's text and determine the emotional tone; Step S65: Overall Book Summary: Integrate the summaries of the core content of each chapter to form a summary of the entire book; Step S66: Knowledge Extraction: Extract knowledge points from each chapter to form a knowledge base for easy application in children's education; Step S67, Input: Record the content after semantic analysis in steps S61-S66 into the database.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein the program, when running, controls the device containing the non-volatile storage medium to perform the method described in claim 2 for the semantic analysis system of children's book content based on image recognition.

9. An electronic device, characterized in that, It includes a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions, when executed, perform the method executed by the image recognition-based children's book content semantic analysis system of claim 2.

Images