Children book content semantic analysis method and system based on image recognition

Abstract

The invention discloses a child book content semantic analysis method and system based on image recognition, and the system comprises an image-text processing module, an image recognition module, an image-text matching module, a semantic analysis module, and an intelligent question answering module, and aims to provide accurate and safe book content semantic analysis service meeting the cognitive level for children. According to the method, a multi-modal large model image recognition technology is fused, training is performed based on massive children book image-text data, illustrations, scenes and roles in books can be accurately recognized, and semantic information of book contents can be deeply understood in combination with a natural language processing technology. Through a multi-link image recognition mode, semantic analysis is carried out by fully combining images and texts, accurate guidance is provided for children, points of interest and understanding ability of the children in the reading process are accurately grasped, answering and content optimization are dynamically carried out according to feedback of the children, and intelligent and accurate semantic analysis of the children's book content is truly achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer science and technology, and is applied to semantic analysis of book content, and in particular to a method and system for semantic analysis of children's book content based on image recognition. Background Art

[0002] The core goal of semantic analysis of children's books is to provide children with book content that is in line with their cognitive level and safe, and preferably answer questions during the reading process to assist children in understanding knowledge. Traditional semantic analysis methods can only perform text recognition, and provide limited help to children by simply combing through the title, introduction, and chapter content of the book. However, this method has obvious limitations. Children's books are different from ordinary adult books and often contain a large number of pictures. Semantic analysis that relies solely on text recognition cannot fully understand the content of children's books. With the development of image recognition technology, some methods attempt to convert images into text for semantic analysis, but the combination of image and text information is not sufficient, and true multimodal fusion cannot be achieved. This limits the intelligence of semantic analysis of children's books to a certain extent, and it is impossible to accurately grasp children's interests and comprehension abilities during the reading process, nor can it dynamically optimize content based on children's real-time feedback.

[0003] In order to more fully combine graphic information and overcome the difficulties in the field of semantic analysis of children's books, the present invention integrates multimodal large-model image recognition technology, and designs a new children's book content semantic analysis system based on massive children's book graphic data for training. The system has strong image recognition and semantic understanding capabilities, and can accurately identify illustrations, scenes and characters in books, and deeply understand the semantic information of book content by combining natural language processing technology. The present invention also proposes an innovative children's book content semantic analysis method, which is based on a multi-link image recognition method and fully combines graphics and text for semantic analysis, thereby providing children with precise guidance, accurately grasping children's interests and comprehension abilities during reading, and dynamically providing answers and optimizing content based on children's feedback, truly realizing intelligent and precise children's book content semantic analysis. Summary of the invention

[0004] Aiming at the shortcomings of the existing detection methods, a children's book content semantic analysis system based on image recognition is proposed. The system includes: image and text processing module, image recognition module, image and text matching module, semantic analysis module, and intelligent question answering module; The image and text processing module performs image and text segmentation, text processing (extraction, error correction, segmentation), image segmentation, sorting, and content security monitoring on book pages, providing basic data for subsequent analysis; The image recognition module recognizes and analyzes the image content through multiple links (text, scene, role, object), extracts key information and generates text description, providing semantic information on the image side for image-text fusion; The image-text matching module combines the text and image content lists after semantic segmentation, reorders them through a multimodal large model, optimizes the image-text layout order, and ensures content coherence; The semantic analysis module performs semantic analysis on the book content, extracts key information, generates chapter and book summaries, performs sentiment analysis, and extracts knowledge into the database to provide data support for subsequent applications; The intelligent question-answering module matches the database content and generates accurate answers based on the children's questions, while also analyzing the learning progress in real time and providing personalized tutoring suggestions to help children learn; The multimodal big model is a deep learning model that can process and understand data of multiple modalities. It combines the natural language processing capabilities of a large language model and the visual perception capabilities of a large visual model, and can achieve more comprehensive and intelligent information processing and image-text interaction. The multimodal big model is a big model that has been fine-tuned and trained on specific tasks related to children's books, and can be more adaptable to the system. The specific fine-tuning training process is to design a proprietary data set for a specific task and use the proprietary data set to train the multimodal big model. Compared with the general multimodal big model, the multimodal big model used in the present invention can better understand the semantic and visual features of children's books, thereby showing higher accuracy and efficiency in children's book content analysis and semantic understanding tasks.

[0005] In addition, the present invention also relates to a children's book content semantic analysis method based on image recognition, which is applied to the above-mentioned children's book content semantic analysis system based on image recognition, and comprises the following steps: Step S1, using target detection technology to identify the content of each page of the book, analyzing the image and text positions, segmenting the image and text, and obtaining an initial image frame and an initial text frame; Step S2, identifying, correcting and segmenting the initial text frame; Step S3: segmenting, sorting, and content security monitoring the initial image frame; Step S4: Using the multimodal large model to understand and identify the secondary image list from multiple links; identifying all secondary images in the same initial image frame, and the identified information is in text format, which is convenient for subsequent semantic analysis; Step S5, reordering the semantically segmented text in step S2 in combination with the secondary image content list in step S4; Step S6: Use the language understanding ability of the multimodal large model to perform content semantic analysis on the children's books, extract key information, and form data records into the database; Step S7: When a child asks about a certain part of the book, the child's question is matched with the content of the database, and the intelligent question-answering capability of the multimodal large model is used to accurately generate answers to help the child understand the content of the book, and the child's learning progress is analyzed in real time to provide personalized tutoring suggestions; The present invention ensures accurate interpretation and efficient learning of children's book content through the graphic comprehension and in-depth analysis capabilities of a multimodal large model, thus facilitating the all-round development of children. Beneficial Effects

[0006] In order to more fully combine graphic information and overcome the difficulties in the field of semantic analysis of children's books, the present invention integrates multimodal large-model image recognition technology, and designs a new children's book content semantic analysis system based on massive children's book graphic data for training. The system has strong image recognition and semantic understanding capabilities, and can accurately identify illustrations, scenes and characters in books, and deeply understand the semantic information of book content by combining natural language processing technology. The present invention also proposes an innovative children's book content semantic analysis method, which is based on a multi-link image recognition method and fully combines graphics and text for semantic analysis, thereby providing children with precise guidance, accurately grasping children's interests and comprehension abilities during reading, and dynamically providing answers and optimizing content based on children's feedback, truly realizing intelligent and precise children's book content semantic analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is an architecture diagram of a children's book content semantic analysis system based on image recognition according to the present invention. DETAILED DESCRIPTION

[0008] Embodiment 1 of the present invention relates to a children's book content semantic analysis system based on image recognition, as shown in the attached Figure 1 As shown, the system includes: an image and text processing module, an image recognition module, an image and text matching module, a semantic analysis module, and an intelligent question answering module; The image and text processing module performs image and text segmentation, text processing (extraction, error correction, segmentation), image segmentation, sorting, and content security monitoring on book pages, providing basic data for subsequent analysis; The image recognition module recognizes and analyzes the image content through multiple links (text, scene, role, object), extracts key information and generates text description, providing semantic information on the image side for image-text fusion; The image-text matching module combines the text and image content lists after semantic segmentation, reorders them through a multimodal large model, optimizes the image-text layout order, and ensures content coherence; The semantic analysis module performs semantic analysis on the book content, extracts key information, generates chapter and book summaries, performs sentiment analysis, and extracts knowledge into the database to provide data support for subsequent applications; The intelligent question-answering module matches the database content and generates accurate answers based on the children's questions, while also analyzing the learning progress in real time and providing personalized tutoring suggestions to help children learn; The multimodal large model is a deep learning model that can process and understand data in multiple modalities. It combines the natural language processing capabilities of a large language model with the visual perception capabilities of a large visual model, and can achieve more comprehensive and intelligent information processing and image-text interaction.

[0009] Embodiment 2 of the present invention relates to a method for semantic analysis of children's book content based on image recognition, which is implemented by the system of embodiment 1 and includes the following steps: Step S1, using target detection technology to identify the content of each page of the book, analyzing the image and text positions, segmenting the image and text, and obtaining an initial image frame and an initial text frame; Step S2, identifying, correcting errors and segmenting the initial text frame; Step S21, using OCR image text recognition technology to extract text from the initial text box to obtain all the text in the text box. Due to the limited accuracy of OCR technology, the text needs to be further processed; Step S22, correcting the text obtained in step S21, replacing the wrong words hit in the text with correct words based on the obfuscation word library, and removing blanks and garbled characters; Step S23: segment the text from a semantic perspective to adapt to the sorting of pictures and texts, specifically: Step S231, dividing the text into single sentences according to the end symbol; Step S232: performing intent recognition on the single sentence, wherein intent recognition refers to a process of analyzing the user's language input (text or voice), understanding the user's true intent, and mapping it to a predefined intent category; Step S233: vectorize the single sentence to obtain a single sentence text vector, vectorize the intent recognized by the single sentence to obtain a single sentence intent vector, and concatenate the single sentence text vector and the single sentence intent vector to obtain a single sentence vector; Step S234, performing cosine similarity calculation on the continuous single sentence vectors, when the similarity between the upper and lower sentences is greater than a threshold, the two sentences are considered to be in the same paragraph, and thus the text segmentation from a semantic perspective is completed; Vectorization refers to converting text into a computer-calculated vector to facilitate the calculation of similarity; Step S3, segmenting, sorting, and content security monitoring the initial image frame; there may be multiple images in the initial image frame, which need further processing; Step S31: Based on the target detection technology, the self-trained model performs fine recognition in the initial image frame, and selects multiple images that may exist, which are recorded as secondary images. The present invention is based on the self-owned image data for annotation, and uses the yolo technology to train the target detection model dedicated to the secondary image segmentation, which is applied to this step to obtain the secondary image frame list; Step S32, sort the secondary image frame list to obtain the correct image order; input the coordinates of the initial image frame and the secondary image frame list into the multimodal large model, and input the task to make the multimodal large model give the correct image order; the multimodal large model is a deep learning model that can process and understand multimodal data, which combines the natural language processing capabilities of the large language model and the visual perception capabilities of the large visual model, and can achieve more comprehensive and intelligent information processing and image-text interaction; the multimodal large model used in the present invention is a large model that has been fine-tuned and trained on specific tasks related to children's books, and can be more adaptable to the system; the specific fine-tuning training process is to design a proprietary data set for a specific task and use the proprietary data set to train the multimodal large model; compared with the general multimodal large model, the multimodal large model used in the present invention can better understand the semantics and visual features of children's books, thereby showing higher accuracy and efficiency in children's book content analysis and semantic understanding tasks.

[0010] Step S34: monitor the content security of the secondary image frame to see if there is any illegal content, ensure that the graphic information complies with the requirements of laws and regulations, and guide children to correct values; Step S4: Using the multimodal large model to understand and identify the secondary image list from multiple links; identifying all secondary images in the same initial image frame, and the identified information is in text format, which is convenient for subsequent semantic analysis; the details are as follows: Step S41, text link: using the multimodal large model to recognize the text in the secondary image to obtain the secondary image text; Step S42, scene link: using the multimodal large model to identify the scene in the secondary image, and obtain the secondary image scene description; Step S43, character link: using the multimodal large model to identify the character appearance, character expression, character action and the relationship between characters in the secondary image, collectively referred to as the secondary image character description; Step S44, object link: using the multimodal large model to identify objects, object states, and role-object relationships in the secondary image, collectively referred to as secondary image object description; Step S45, initial image link: using the multimodal large model to identify the initial image from an overall perspective, perform image summary, and summarize it into an initial image summary; Step S46, multi-link fusion: input the initial image, the initial image information in step S45, the secondary image, and the secondary image information in steps S41 to S44 into the multimodal large model, and use the image and text understanding ability of the multimodal large model to more fully summarize the content of the secondary image; Step S47, perform the operations of steps S41 to S46 on all secondary images to obtain a secondary image content list in plain text format; This step splits the image elements step by step, and then combines them from multiple links to fully understand the image and accurately analyze the representative meaning of the secondary image, which is conducive to the semantic understanding of children's books; Step S5, reordering the semantically segmented text of step S2 in combination with the secondary image content list of step S4; considering that the image position and the corresponding text position may not be on the same page during the typesetting process, the text and image need to be reordered; the specific steps are: inputting the semantically segmented text and the secondary image content list of three consecutive pages in the children's book into the multimodal large model, inputting the task to make the multimodal large model give the optimal text and image typesetting order, and obtaining the reordered book information. At this time, the image information is converted into text format after image recognition in step S4, so the reordered book information is all in text format, which is convenient for content analysis; through the intelligent decision-making ability of the multimodal large model, the coherence of the text and image content is ensured, which is conducive to the semantic analysis of the subsequent children's book content; Step S6: Use the language understanding ability of the multimodal large model to perform content semantic analysis on the children's books, extract key information, and form data records into the database; the specific steps are as follows: Step S61, dividing the book information reordered in step S5 into chapters; Step S62, extracting key sentences from each chapter: performing in-depth analysis on the text of each chapter, identifying and extracting key sentences, forming a summary of the core content of the chapter, and ensuring that key information is recorded accurately; Step S63, chapter summary: integrating the core content of the chapter to form a chapter summary; Step S64, chapter sentiment analysis: identifying the sentiment tendency of each chapter text and determining the sentiment tone; Step S65, overall summary of the book: integrating the core content summaries of each chapter to form a summary of the entire book; Step S66, knowledge extraction: extracting knowledge points from each chapter to form a knowledge base for children's education application; Step S67, storage: recording the contents after semantic analysis in steps S61-S66 into the database for easy retrieval and application; Step S7: When a child asks about a certain part of the book, the child's question is matched with the content of the database, and the intelligent question-answering capability of the multimodal large model is used to accurately generate answers to help the child understand the content of the book, and the child's learning progress is analyzed in real time to provide personalized tutoring suggestions; The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A children's book content semantic analysis system based on image recognition, characterized in that: The system comprises: Image and text processing module: segment the image and text of the book page, process the text, and segment, sort and monitor the content security of the image, providing basic data for subsequent analysis; Image recognition module: recognizes and analyzes image content through multiple links, extracts key information and generates text descriptions, providing semantic information on the image side for image and text fusion; Image-text matching module: Combines the text and image content lists after semantic segmentation, reorders them through a multimodal large model, optimizes the image and text layout order, and ensures content coherence; Semantic analysis module: performs semantic analysis on book content, extracts key information, generates chapter and book summaries, performs sentiment analysis, and extracts knowledge into the database to provide data support for subsequent applications; Intelligent Q&A module: Based on children’s questions, it matches database content and generates accurate answers. It also analyzes learning progress in real time and provides personalized tutoring suggestions.

2. A method for semantic analysis of children's book content based on image recognition, the method being applied to the system for semantic analysis of children's book content based on image recognition as claimed in claim 1, characterized in that: The following steps are involved: Step S1, using target detection technology to identify the content of each page of the book, analyzing the image and text positions, segmenting the image and text, and obtaining an initial image frame and an initial text frame; Step S2, identifying, correcting errors and segmenting the initial text frame; Step S3: segmenting, sorting, and content security monitoring the initial image frame; Step S4, using a multimodal large model to understand and identify a secondary image list from multiple links; All secondary images in the same initial image frame are recognized, and the recognized information is in text format, which is convenient for subsequent semantic analysis; Step S5, reordering the semantically segmented text in step S2 in combination with the secondary image content list in step S4; Step S6: Use the language understanding ability of the multimodal large model to perform content semantic analysis on the children's books, extract key information, and form data records into the database; Step S7: When a child asks about a certain part of the book, the child’s question is matched with the content of the database, and the intelligent question-answering capability of the multimodal large model is used to accurately generate answers to help children understand the content of the book. The child’s learning progress is analyzed in real time to provide personalized tutoring suggestions.

3. The method for semantic analysis of children's book content based on image recognition according to claim 2, characterized in that: The step S2 comprises: Step S21, using OCR image text recognition technology to extract text from the initial text box to obtain all the text in the text box. Due to the limited accuracy of OCR technology, the text needs to be further processed; Step S22, correcting the text obtained in step S21, replacing the wrong words hit in the text with correct words based on the obfuscation word library, and removing blanks and garbled characters; Step S23: segment the text from a semantic perspective to adapt to the sorting of images and texts.

4. The method for semantic analysis of children's book content based on image recognition according to claim 3, characterized in that: The specific operation of step S23 is: Step S23, segmenting the text from a semantic perspective to adapt to the sorting of pictures and texts; Step S231, dividing the text into single sentences according to the end symbol; Step S232: performing intent recognition on the single sentence, wherein intent recognition refers to a process of analyzing the user's language input, understanding the user's true intent, and mapping it to a predefined intent category; Step S233: vectorize the single sentence to obtain a single sentence text vector, vectorize the intent recognized by the single sentence to obtain a single sentence intent vector, and concatenate the single sentence text vector and the single sentence intent vector to obtain a single sentence vector; Step S234, performing cosine similarity calculation on the continuous single sentence vectors, when the similarity between the upper and lower sentences is greater than a threshold, the two sentences are considered to be in the same paragraph, and thus the text segmentation from a semantic perspective is completed; The vectorization refers to converting text into a vector computable by a computer.

5. The method for semantic analysis of children's book content based on image recognition according to claim 2, characterized in that: The step S3 comprises: Step S31: Based on the target detection technology, the self-trained model performs fine recognition in the initial image frame, selects multiple images that may exist, and records them as secondary images; based on the self-owned image data, the target detection model dedicated to secondary image segmentation is trained using the yolo technology to obtain a list of secondary image frames; Step S32, sorting the secondary image frame list to obtain the correct image order; inputting the coordinates of the initial image frame and the secondary image frame list into the multimodal large model, and inputting the task to make the multimodal large model give the correct image order; the multimodal large model combines the natural language processing capability of the large language model and the visual perception capability of the large visual model, and can realize more comprehensive and intelligent information processing and image-text interaction; Step S34: Perform content security monitoring on the secondary image frame to monitor whether there is any illegal content to ensure that the graphic information complies with the requirements of laws and regulations.

6. The method for semantic analysis of children's book content based on image recognition according to claim 5, characterized in that: The step S4 comprises: Step S41, text link: using the multimodal large model to recognize the text in the secondary image to obtain the secondary image text; Step S42, scene link: using the multimodal large model to identify the scene in the secondary image, and obtain the secondary image scene description; Step S43, character link: using the multimodal large model to identify the character appearance, character expression, character action and the relationship between characters in the secondary image, collectively referred to as the secondary image character description; Step S44, object link: using the multimodal large model to identify objects, object states, and role-object relationships in the secondary image, collectively referred to as secondary image object description; Step S45, initial image link: using the multimodal large model to identify the initial image from an overall perspective, perform image summary, and summarize it into an initial image summary; Step S46, multi-link fusion: input the initial image, the initial image information in step S45, the secondary image, and the secondary image information in steps S41 to S44 into the multimodal large model to summarize the content of the secondary image; Step S47: Perform the operations of steps S41 to S46 on all secondary images to obtain a secondary image content list in plain text format.

7. The method for semantic analysis of children's book content based on image recognition according to claim 5, characterized in that: The step S5 includes: considering that the image position and the corresponding text position may not be on the same page during the typesetting process, the text and image need to be reordered; the specific steps are: inputting the semantically segmented text and secondary image content list of three consecutive pages in the children's book into the multimodal large model, inputting the task to make the multimodal large model give the optimal text and image typesetting order, and obtaining the reordered book information. At this time, the image information is converted into text format after image recognition in step S4, so the reordered book information is all in text format, which is convenient for content analysis; through the intelligent decision-making ability of the multimodal large model, the coherence of the text and image content is ensured, which is conducive to the subsequent semantic analysis of the children's book content.

8. The method for semantic analysis of children's book content based on image recognition according to claim 5, characterized in that: The step S6 comprises: Step S61, dividing the book information reordered in step S5 into chapters; Step S62, extracting key sentences from each chapter: performing in-depth analysis on the text of each chapter, identifying and extracting key sentences, forming a summary of the core content of the chapter, and ensuring that key information is recorded accurately; Step S63, chapter summary: integrating the core content of the chapter to form a chapter summary; Step S64, chapter sentiment analysis: identifying the sentiment tendency of each chapter text and determining the sentiment tone; Step S65, overall summary of the book: integrating the core content summaries of each chapter to form a summary of the entire book; Step S66, knowledge extraction: extracting knowledge points from each chapter to form a knowledge base for children's education application; Step S67, storage: record the contents after semantic analysis in steps S61-S66 into the database.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the method according to claim 2.

10. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the method according to claim 2 when executed.

Images