Intelligent accompanying reading method and device and all-in-one machine

Abstract

The invention provides an intelligent accompanying reading method, an intelligent accompanying reading device and an all-in-one machine. The intelligent accompanying reading method comprises the steps of determining an accompanying reading mark position of a user from a text page picture; interpreting a accompanying reading instruction of a user, and determining an accompanying reading starting indicator and an accompanying reading ending indicator; based on the accompanying reading mark position, the accompanying reading starting indicator and the accompanying reading ending indicator, character recognition is conducted on the text page picture, and an accompanying reading text is obtained; and performing intelligent accompanying reading based on the accompanying reading text. According to the method, the accompanying reading mark position of the user is obtained through the text page picture, and the accompanying reading instruction of the user is combined, so that intelligent accompanying reading can be realized. In other words, under the condition that the user cannot completely recognize characters or does not recognize the characters, text reading can be autonomously achieved, the text content can be accurately understood, guiding and accompanying of parents are not needed, the reading efficiency and the reading effect can be improved, and the reading interest of the user can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an intelligent reading companion method, device and all-in-one machine. Background Art

[0002] In recent years, preschool education has been regarded as the foundation of children's physical and mental health, habit formation and intellectual development. Picture book reading, as an important means to promote children's language development and cognitive ability, has received more and more attention.

[0003] However, the existing picture book reading method mainly relies on the company and guidance of parents. Many young children have incomplete literacy or no literacy in the preschool stage, which causes them to face greater difficulties in reading picture books and affects their interest and enthusiasm in reading. Summary of the invention

[0004] The present invention provides an intelligent reading companion method, device and all-in-one machine, which are used to solve the defects in the prior art.

[0005] The present invention provides an intelligent reading companion method, comprising the following steps: Determine the user's reading companion mark position from the text page image; Interpreting the accompanying reading instruction of the user, and determining an accompanying reading start indicator and an accompanying reading end indicator; Based on the companion reading mark position, the companion reading start indicator and the companion reading end indicator, character recognition is performed on the text page image to obtain a companion reading text; Based on the accompanying reading text, intelligent accompanying reading is performed.

[0006] According to an intelligent reading companion method provided by the present invention, character recognition is performed on the text page image based on the reading companion mark position, the reading companion start indicator and the reading companion end indicator to obtain the reading companion text, including: Starting from the position of the companion reading mark, the text page image is recognized character by character until the companion reading start indicator and the companion reading end indicator are recognized, and then the companion reading text is determined based on all recognized characters.

[0007] According to an intelligent reading companion method provided by the present invention, the method takes the reading companion mark position as a starting point and performs character-by-character recognition on the text page image, including: Taking the position of the companion reading mark as the starting point, the text page image is recognized character by character, and after each character is recognized, it is determined whether the corresponding character is the companion reading start indicator or the companion reading end indicator; If so, the character recognition in the target recognition direction is stopped; if not, the next character recognition is continued in the target recognition direction, where the target recognition direction is a straight line direction from the position of the companion reading mark to the position of the corresponding character.

[0008] According to an intelligent reading companion method provided by the present invention, character recognition is performed on the text page image based on the reading companion mark position, the reading companion start position character and the reading companion end position character to obtain the reading companion text, including: Performing character recognition on the text page image to determine a plurality of candidate companion reading start indicators and a plurality of candidate companion reading end indicators; Using the candidate companion reading start indicator closest to the companion reading mark position as the companion reading start indicator, and using the candidate companion reading end indicator closest to the companion reading mark position as the companion reading end indicator; The companion reading text is determined based on all characters between the companion reading start indicator and the companion reading end indicator.

[0009] According to an intelligent reading companion method provided by the present invention, the intelligent reading companion method based on the reading companion text includes: Receiving a language partner switching instruction from the user; Under the language partner role indicated by the language partner switching instruction, intelligent reading companion is performed based on the reading companion text.

[0010] According to an intelligent reading companion method provided by the present invention, the intelligent reading companion is performed based on the reading companion text, and then further comprises: Based on the question text of the user regarding the companion reading text, generating an answer text within the cognitive scope of the user; Output the audio file corresponding to the answer text.

[0011] According to an intelligent reading companion method provided by the present invention, the step of determining the user's reading companion mark position from a text page image includes: Performing target detection on the text page image to extract an image of the area where a target object is located, wherein the target object refers to an object used to indicate the accompanying reading position; Performing edge detection on the regional image to determine the contour position of the target object; The position of the companion mark is determined based on the contour position of the target object.

[0012] The present invention also provides an intelligent reading companion device, comprising the following modules: A determination unit, used to determine the position of the user's reading companion mark from the text page image; An interpretation unit, used for interpreting the accompanying reading instruction of the user, and determining an accompanying reading start indicator and an accompanying reading end indicator; A recognition unit, configured to perform character recognition on the text page image based on the companion reading mark position, the companion reading start indicator and the companion reading end indicator to obtain a companion reading text; The accompanying reading unit is used for performing intelligent accompanying reading based on the accompanying reading text.

[0013] The present invention also provides an all-in-one machine, comprising a memory, a processor, and a large model stored in the memory and a computer program stored in the memory and running on the processor; When the processor executes the computer program, the large model is called to implement the intelligent companion reading method.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the intelligent companion reading method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the intelligent reading companion method described above is implemented.

[0016] The intelligent reading companion method, device and all-in-one machine provided by the present invention can realize intelligent reading companion by obtaining the user's reading companion mark position through the text page image and combining the user's reading companion instruction. That is to say, even if the user cannot recognize all characters or cannot recognize any characters, the user can read the text independently and accurately understand the text content without relying on the guidance and accompaniment of parents, which can not only improve the reading efficiency and reading effect, but also improve the user's reading interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the intelligent reading companion method provided by the present invention.

[0019] Figure 2 It is a flowchart of the implementation of step 140 in the intelligent reading companion method provided by the present invention.

[0020] Figure 3 It is a flow chart of the interactive question-answering method provided by the present invention.

[0021] Figure 4 It is a flowchart of the implementation of step 110 in the intelligent reading companion method provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the intelligent reading companion device provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The traditional way of reading picture books mainly relies on the company and guidance of parents. Many young children have incomplete literacy or are illiterate in the preschool stage, which causes them to face great difficulties in reading picture books and affects their interest and enthusiasm in reading.

[0026] Traditional picture book reading is dominated by parents and lacks sufficient interactivity. Young children may be curious about the storyline, characters or pictures during reading, but due to the lack of timely feedback and guidance, their understanding and interest in the reading content may be weakened.

[0027] In addition, in modern families, many parents are busy with work and find it difficult to spare enough time to accompany their children in reading picture books, which also limits the reading experience of young children.

[0028] In view of the above problems, in order to cultivate children's interest in picture book reading from preschool, stimulate children's curiosity in the enlightenment stage, and solve the problem that parents cannot accompany their children in the reading process due to lack of time. The present invention aims to provide children with a better picture book reading experience by simulating the process of real parents accompanying children in the early childhood stage to read picture books. Especially for children with incomplete literacy or illiteracy, through picture book finger reading, free question interaction, and personalized language partner customization, children's reading interest is cultivated from the preschool enlightenment stage, and their curiosity is stimulated, so that children can experience the warm company of parents at any time. The present invention supports page turning and finger reading of all picture book contents, allowing children to actively ask questions to large language models. The large language model combines the picture book content to quickly answer in a language mode that is more in line with the child's understandable. From the outside, the present invention allows children or parents to replicate sounds, generate personalized language partners, and use replicated sounds to read picture books and discuss interactions with children at any time.

[0029] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings. Figure 1 It is a flow chart of the intelligent reading companion method provided by the present invention. The execution subject of each step in the method can be an intelligent reading companion device, which can be implemented by software and / or hardware. The device can be integrated in an electronic device. The electronic device can be an all-in-one machine, a smart phone, a personal computer, a learning machine, a local server or a cloud server, and a server cluster can also be a hardware device with various operating systems.

[0030] It should be noted that the execution subject of the embodiment of the present invention may be, for example, a central processing unit (CPU) in an electronic device in terms of hardware, or may be, for example, a large model-related service in an electronic device in terms of software, without limitation. Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0031] Step 110: determine the user's reading companion mark position from the text page image.

[0032] Specifically, the text page image refers to a text content image used for intelligent reading companion, which can be an image of a page in a picture book or an image of a page in other books, and the embodiment of the present invention does not specifically limit this. Among them, the text page image can be an image directly input by the user, or an image acquired by an image acquisition device such as a scanner, a mobile phone, or a camera, and the embodiment of the present invention does not specifically limit this.

[0033] The user's reading companion mark position can be understood as the reading position pointed to by the user in the text page image. For example, the user can point to the position to be read with a finger, and then the position of the fingertip is the reading companion mark position; for another example, the user can point to the position to be read with a reading pen, and then the position of the pen tip is the reading companion mark position. The user can be a person of the appropriate age matching the text content, such as a person of the appropriate age matching the picture book can be a toddler, that is, the user here can be a toddler.

[0034] Step 120: interpret the user's companion reading instruction and determine the companion reading start indicator and the companion reading end indicator.

[0035] Specifically, the user's accompanying reading instruction can be understood as an instruction to read the text, and the user's accompanying reading instruction can be a voice instruction sent by the user. For example, the user's accompanying reading instruction can be "help me read this paragraph".

[0036] The companion reading start indicator is used to indicate the start position of reading in the companion reading instruction, and the companion reading end indicator is used to indicate the end position of reading in the companion reading instruction. The companion reading start indicator and the companion reading end indicator can be determined by interpreting the user's companion reading instruction.

[0037] For example, if the user's accompanying reading instruction is "Help me read this paragraph", it can be determined that the user needs to read a paragraph, and at this time, the starting position and the ending position of the paragraph need to be determined. Usually, there is a "first line indent space character" at the starting position of a paragraph, and there is a "line break character" at the ending position of a paragraph. Therefore, the "first line indent space character" can be used as the accompanying reading start indicator, and the "line break character" can be used as the accompanying reading end indicator.

[0038] For another example, if the user's companion reading instruction is "Help me read this page content", it can be determined that the user needs to read the entire page content, and at this time, it is necessary to determine the starting position and the ending position of the entire page text. Usually, the starting position of the entire page text has a "first line indent space character" and no characters exist in front of it, and the ending position of the entire page text has a "line break character" and no characters exist after it. Therefore, "the first line indent space character and no characters in front of it" can be used as the companion reading start indicator, and "the line break character and no characters after it" can be used as the companion reading end indicator.

[0039] Step 130: Based on the reading companion mark position, the reading companion start indicator and the reading companion end indicator, character recognition is performed on the text page image to obtain the reading companion text.

[0040] Specifically, the accompanying text can be understood as the text between the accompanying reading start indicator and the accompanying reading end indicator, and the accompanying text includes the text pointed to by the accompanying reading mark position. In other words, the accompanying reading mark position may be located between the accompanying reading start indicator and the accompanying reading end indicator.

[0041] On this basis, the target recognition area where the accompanying reading mark is located can be determined based on the accompanying reading start indicator and the accompanying reading end indicator, and character recognition is performed on the target recognition area to obtain the accompanying reading text, which is the text that the user needs to read.

[0042] Step 140: Perform intelligent reading companionship based on the reading companion text.

[0043] Specifically, after the accompanying reading text is determined, speech synthesis can be performed on the accompanying reading text to convert the accompanying reading text into an audio file output for intelligent accompanying reading.

[0044] The intelligent reading companion method provided by the embodiment of the present invention can realize intelligent reading companion by obtaining the user's reading companion mark position through the text page image and combining the user's reading companion instruction. That is to say, even if the user cannot recognize all characters or cannot recognize any characters, the user can read the text independently and accurately understand the text content without relying on the guidance and accompaniment of parents, which can not only improve the reading efficiency and reading effect, but also improve the user's reading interest.

[0045] As an optional embodiment, the above steps 110 to 140 can all be implemented through a large language model, that is, the text page image and the user's reading instructions are input into the large language model, and the large language model executes the above steps 110 to 140 to achieve intelligent reading.

[0046] The above-mentioned large language model (LLM) is referred to as a large model, which refers to a natural language processing (NLP) model with a large parameter scale. The number of model parameters and / or the complexity of the model structure exceeds the preset threshold. The model processes large-scale text data during training and has the ability to understand and generate natural language. For example, large language models can include the Spark large model.

[0047] Based on the above embodiment, step 130 performs character recognition on the text page image based on the accompanying reading mark position, the accompanying reading start indicator and the accompanying reading end indicator to obtain the accompanying reading text, including: Starting from the position of the accompanying reading mark, the text page image is recognized character by character until the accompanying reading start indicator and the accompanying reading end indicator are recognized, and the accompanying reading text is determined based on all the recognized characters.

[0048] Specifically, the position of the companion reading mark is usually between the companion reading start indicator and the companion reading end indicator, and the text between the companion reading start indicator and the companion reading end indicator is the companion reading text. Based on this, the embodiment of the present invention needs to identify all characters between the companion reading start indicator and the companion reading end indicator, and the text composed of all the characters is the companion reading text.

[0049] Therefore, after the companion reading start indicator is recognized, it indicates that the characters before the companion reading start indicator are not characters corresponding to the companion reading text, so the characters before the companion reading start indicator can no longer be recognized. Similarly, after the companion reading end indicator is recognized, it indicates that the characters after the companion reading end indicator are not characters corresponding to the companion reading text, so the characters after the companion reading end indicator can no longer be recognized.

[0050] After all characters between the accompanying reading start indicator and the accompanying reading end indicator are identified, all characters may be concatenated to form an accompanying reading text, which is the text that the user needs to read.

[0051] Based on any of the above embodiments, taking the reading companion mark position as the starting point, character by character recognition is performed on the text page image, including: Taking the position of the companion reading mark as the starting point, the text page image is recognized character by character, and after each character is recognized, it is determined whether the corresponding character is a companion reading start indicator or a companion reading end indicator; If yes, the character recognition in the target recognition direction is stopped; if no, the next character recognition is continued in the target recognition direction. The target recognition direction is the straight line direction from the position of the companion reading mark to the position of the corresponding character.

[0052] Specifically, when the text page image is recognized character by character with the companion reading mark position as the starting point, and after each character is recognized, it is first determined whether the character is a companion reading start indicator or a companion reading end indicator. If so, it indicates that all character recognitions of the companion reading text have been completed in the target recognition direction, and the character recognition in the target recognition direction can be stopped at this time. If not, it indicates that there are still unrecognized characters in the target recognition direction, and the next character recognition is continued in the target recognition direction.

[0053] Exemplarily, the companion reading mark position is taken as the starting point, and character-by-character recognition is performed in two target recognition directions, that is, the direction in which the companion reading mark position points to the companion reading start indicator is taken as the first target recognition direction, and the companion reading mark position is taken as the starting point, and character-by-character recognition is performed in the first target recognition direction until the companion reading start indicator is recognized, and then character recognition in the first target recognition direction is stopped.

[0054] The direction from the companion reading mark position to the companion reading termination indicator is used as the second target recognition direction, and the companion reading mark position is used as the starting point to perform character recognition in the second target recognition direction until the companion reading termination indicator is recognized, and then stop character recognition in the second target recognition direction.

[0055] Next, all characters recognized in the first target recognition direction are concatenated with all characters recognized in the second target recognition direction to obtain a companion text.

[0056] For example, the text content in the text page image is as follows: {In a distant place, there is a mysterious volcano hidden. Beneath the calm surface, there is infinite power.

[0057] But today, the volcano seemed a little uneasy, with dark clouds covering it and the wind becoming muffled.

[0058] Boom! The volcano erupted! The earth began to shake violently...

[0059] The flames are like dancing dancers, and the magma is flowing, as if painting a magnificent picture on the earth.}

[0060] If the user's accompanying reading instruction is "Please help me read this paragraph", and the accompanying reading mark position is the position of "dong", then "dong" is used as the starting point, and character recognition is performed on the left and right sides respectively. When recognizing on the left, it stops when the accompanying reading start indicator (first line indentation space character) is recognized, and when recognizing on the right, it stops when the accompanying reading end indicator (line break character) is recognized. All recognized characters are spliced ​​together to obtain the accompanying reading text "Boom! The volcano is erupting! The earth is shaking... ".

[0061] Based on any of the above embodiments, character recognition is performed on the text page image based on the accompanying reading mark position, the accompanying reading start position character and the accompanying reading end position character to obtain the accompanying reading text, including: Performing character recognition on the text page image to determine a plurality of candidate companion reading start indicators and a plurality of candidate companion reading end indicators; The candidate companion reading start indicator closest to the companion reading mark position is used as the companion reading start indicator, and the candidate companion reading end indicator closest to the companion reading mark position is used as the companion reading end indicator; The companion reading text is determined based on all characters between the companion reading start indicator and the companion reading end indicator.

[0062] Specifically, the position of the companion reading mark is between the companion reading start indicator and the companion reading end indicator, and there is only one companion reading start indicator and one companion reading end indicator in the companion reading text. If there are other companion reading start indicators and / or companion reading end indicators in the text page image, they must be far away from the position of the companion reading mark.

[0063] For example, if the companion reading instruction is "Please read this paragraph for me", there will be only one companion reading start indicator (first line indent space character) and one companion reading end indicator (line break character) in this paragraph. If there are other companion reading start indicators and / or companion reading end indicators, they must be located in other paragraphs, and thus are far away from the position of the companion reading mark.

[0064] Based on this, the embodiment of the present invention performs character recognition on the text page image, and multiple candidate companion reading start indicators and multiple candidate companion reading end indicators can be obtained. Considering that there is only one companion reading start indicator and one companion reading end indicator in the companion reading text, it is necessary to filter out the companion reading start indicator from multiple candidate companion reading start indicators, and filter out the companion reading end indicator from multiple candidate companion reading end indicators. The screening strategy is the principle of the closest distance, that is, the candidate companion reading start indicator closest to the companion reading mark position is used as the companion reading start indicator, the candidate companion reading end indicator closest to the companion reading mark position is used as the companion reading end indicator, and the text composed of all characters between the companion reading start indicator and the companion reading end indicator is used as the companion reading text. Among them, the distance here can be the character spacing.

[0065] Based on any of the above embodiments, Figure 2 is a flow chart of an implementation of step 140 in the intelligent companion reading method provided by the present invention, such as Figure 2 As shown, step 140 performs intelligent reading companion based on the reading companion text, including: Step 141: receiving a language partner switching instruction from a user; Step 142: Under the language partner role indicated by the language partner switching instruction, intelligent reading companion is performed based on the reading companion text.

[0066] Specifically, the speech partner switching instruction refers to an instruction for the user to select different roles or voice styles for reading companion. For example, the user can select different roles such as "Dad", "Mom", "Grandpa", and "Grandma".

[0067] For example, a simple user interface (UI) can be created to allow users to select different speech partner roles. The speech partner roles can be predefined, such as "Dad", "Mom", "Grandpa", "Grandma", etc. The user's selection is received through a button, a drop-down menu, or voice input. After the user selects, a speech partner switching instruction is generated, and the selected speech partner role information is fed back, such as "You have selected the dad role."

[0068] Under the language partner role indicated by the language partner switching instruction, speech synthesis can be performed on the reading companion text, an audio file matching the language partner role can be generated, and the audio file can be played. Therefore, even if the parents are unable to accompany the reading, the language partner role switching can be used to simulate the scenario of parents accompanying the reading.

[0069] Among them, a text can be recorded in advance for different language partner roles (for example, the text can be "In my opinion, every new friend comes with his or her own story and experience, which is particularly interesting"), and the sound can be reproduced based on the recording, so as to obtain the speech synthesis parameters of different language partner roles (such as speaking speed, intonation, emotion, etc.), and then the accompanying text can be speech synthesized based on the speech synthesis parameters to obtain audio files corresponding to different language partner roles.

[0070] Based on any of the above embodiments, Figure 3 is a flow chart of the interactive question-answering method provided by the present invention, such as Figure 3 As shown, based on the accompanying text, intelligent accompanying reading is performed, and then it also includes: Step 310: Based on the question text of the user regarding the companion reading text, generate an answer text within the user's cognitive scope; Step 320: output the audio file corresponding to the answer text.

[0071] Specifically, the question text can be understood as the question text of the user for the accompanying reading text after the audio file corresponding to the accompanying reading text is output. The question text can be obtained by performing voice recognition on the user's question audio.

[0072] In addition, the user's cognitive scope can be understood as the scope of information and knowledge that the user can understand, process and use. If the user is a young child, the information and knowledge involved in the generated answer text needs to be within the cognitive scope of the young child. For example, when generating answer texts for young children, simple vocabulary and sentence structures can be used, combined with vivid metaphors and examples from life to help them understand and remember more easily. In this way, the cognitive scope of young children can be effectively expanded, making them more interested in the world around them.

[0073] For example, after obtaining the question text, the question text and the prompt text can be input into the big model, and the big model generates the answer text within the user's cognitive scope based on the question text, and outputs the audio file corresponding to the answer text. The prompt text can be used to prompt the big model user's cognitive scope and the requirements for generating the answer text (such as word requirements, format requirements, sentence structure requirements, etc.).

[0074] It can be seen that the embodiment of the present invention can not only realize autonomous reading by users, but also conduct interactive Q&A with users, and the knowledge involved in the generated answer text is within the user's cognitive scope, thereby stimulating the user's curiosity. For young children, it can satisfy their curiosity in the enlightenment stage and further improve the reading companion effect.

[0075] Based on any of the above embodiments, Figure 4is a flow chart of an implementation of step 110 in the intelligent reading companion method provided by the present invention, such as Figure 4 As shown, step 110 determines the user's reading companion mark position from the text page image, including: Step 111, performing target detection on the text page image, and extracting an image of the area where the target object is located, where the target object refers to an object used to indicate the position of the accompanying reading; Step 112: perform edge detection on the regional image to determine the contour position of the target object; Step 113: Determine the position of the companion mark based on the contour position of the target object.

[0076] Specifically, the target object can be understood as an object used by the user to indicate the location of the companion reading mark. For example, if the user points to the location of the companion reading mark with a finger, the target object is the fingertip; if the user points to the location of the companion reading mark with a reading pen, the target object is the pen tip.

[0077] After acquiring the text page image, target detection is performed on the text page image, and the image of the area where the target object is located is cropped according to the detected bounding box coordinates, thereby avoiding interference of other objects in the text page image with the subsequent extraction of the accompanying reading mark position.

[0078] After acquiring the regional image, edge detection is performed on the regional image to extract the edge information of the target object, and then the contour position of the target object is obtained based on the edge information. After the contour position is obtained, the center position can be calculated based on the contour position, and the center position is used as the companion mark position.

[0079] The intelligent reading companion device provided by the present invention is described below. The intelligent reading companion device described below and the intelligent reading companion method described above can be referred to each other.

[0080] Based on any of the above embodiments, Figure 5 : is a schematic diagram of the structure of the intelligent reading companion device provided by the present invention, such as Figure 5 As shown, the device comprises: A determination unit 510, used to determine the user's reading companion mark position from the text page image; The interpretation unit 520 is used to interpret the user's companion reading instruction and determine the companion reading start indicator and the companion reading end indicator; The recognition unit 530 is used to perform character recognition on the text page image based on the accompanying reading mark position, the accompanying reading start indicator and the accompanying reading end indicator to obtain the accompanying reading text; The accompanying reading unit 540 is used to perform intelligent accompanying reading based on the accompanying reading text.

[0081] Based on any of the above embodiments, character recognition is performed on the text page image based on the accompanying reading mark position, the accompanying reading start indicator and the accompanying reading end indicator to obtain the accompanying reading text, including: Starting from the position of the accompanying reading mark, the text page image is recognized character by character until the accompanying reading start indicator and the accompanying reading end indicator are recognized, and the accompanying reading text is determined based on all the recognized characters.

[0082] Based on any of the above embodiments, taking the reading companion mark position as the starting point, character by character recognition is performed on the text page image, including: Taking the position of the companion reading mark as the starting point, the text page image is recognized character by character, and after each character is recognized, it is determined whether the corresponding character is a companion reading start indicator or a companion reading end indicator; If yes, the character recognition in the target recognition direction is stopped; if no, the next character recognition is continued in the target recognition direction. The target recognition direction is the straight line direction from the position of the companion reading mark to the position of the corresponding character.

[0083] Based on any of the above embodiments, character recognition is performed on the text page image based on the accompanying reading mark position, the accompanying reading start position character and the accompanying reading end position character to obtain the accompanying reading text, including: Performing character recognition on the text page image to determine a plurality of candidate companion reading start indicators and a plurality of candidate companion reading end indicators; The candidate companion reading start indicator closest to the companion reading mark position is used as the companion reading start indicator, and the candidate companion reading end indicator closest to the companion reading mark position is used as the companion reading end indicator; The companion reading text is determined based on all characters between the companion reading start indicator and the companion reading end indicator.

[0084] Based on any of the above embodiments, intelligent reading companion is performed based on the reading companion text, including: Receive a language partner switching instruction from the user; Under the language partner role indicated by the language partner switching instruction, intelligent reading is performed based on the reading companion text.

[0085] Based on any of the above embodiments, intelligent reading companion is performed based on the reading companion text, and then further includes: Based on the user's question text regarding the companion reading text, generate an answer text within the user's cognitive scope; Output the audio file corresponding to the answer text.

[0086] Based on any of the above embodiments, determining the user's reading companion mark position from the text page image includes: Performing target detection on the text page image and extracting the image of the area where the target object is located, the target object refers to the object used to indicate the position of the accompanying reading; Perform edge detection on the regional image to determine the contour position of the target object; Based on the contour position of the target object, the position of the companion mark is determined.

[0087] Figure 6 Schematic diagram of the structure of the integrated machine provided by the present invention. Figure 6 As shown, the integrated machine may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The memory 630 stores a large model and a computer program that can be loaded on the processor 610. The processor 610 can call the large model and the computer program in the memory 630 to execute the intelligent reading companion method, which includes: determining the user's reading companion mark position from the text page image; interpreting the user's reading companion instruction to determine the reading companion start indicator and the reading companion end indicator; based on the reading companion mark position, the reading companion start indicator and the reading companion end indicator, performing character recognition on the text page image to obtain the reading companion text; and performing intelligent reading companion based on the reading companion text.

[0088] Exemplarily, the all-in-one machine may be a learning machine, which is provided with a camera for collecting text page images. Taking the user as a child and the text page image as an image corresponding to any page of a picture book as an example, the process of the learning machine performing intelligent accompanying reading is as follows: When a child opens a picture book to read and encounters a word he or she does not recognize, or does not recognize any words at all, he or she can point to the relevant content with his or her finger and send a reading instruction "Xiao Fei, help me read this paragraph." At this time, the learning machine controls the camera to collect the text page image of the current picture book, and calls the large model to determine the reading text based on the text page image and the reading instruction, generates an audio file of the reading text, and outputs the audio file "Boom! The volcano is erupting! The earth is shaking..."

[0089] In addition, children can also ask interactive questions to the learning machine. For example, a child can ask "Xiao Fei, why does a volcano erupt?" The learning machine calls the large model to convert it into a question text, generates an answer text within the child's cognitive scope, and converts the answer text into a corresponding audio file output "The volcano erupts because there is a lot of hot magma in the earth. They want to come to the ground to breathe, so they erupt with a 'bang' sound. Just like when there is too much air in a balloon, it will explode. This is very interesting, but it is very dangerous when a volcano erupts, so we can't get close!"

[0090] Referring to the above interactive process, the child can continue to ask the question "Why is it dangerous?" The learning machine calls the large model to answer "Because the magma and smoke are very, very hot, they will burn us. Moreover, the rocks may hit us, and the ash may choke us and make it difficult for us to breathe. So we have to stay away from them so that we won't get hurt."

[0091] In addition, users can also select the language partner roles that need to be customized on the learning machine, such as the preset language partner roles of father, mother, grandfather, grandmother, etc. Among them, the preset language partner roles need to record an audio in advance, and the corresponding text of the audio can be "In my opinion, every new friend comes with his own story and experience, which is very interesting." For the audio recorded for each language partner role, the speech synthesis parameters of each language partner role are determined, and then the accompanying text can be speech synthesized later, and the audio file of the corresponding language partner role can be output, so as to realize the free switching of language partner roles.

[0092] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0093] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the intelligent reading companion method provided by the above-mentioned methods, which includes: determining the user's reading companion mark position from the text page image; interpreting the user's reading companion instruction to determine the reading companion start indicator and the reading companion end indicator; based on the reading companion mark position, the reading companion start indicator and the reading companion end indicator, performing character recognition on the text page image to obtain the reading companion text; and performing intelligent reading companion based on the reading companion text.

[0094] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the intelligent reading companion method provided by the above-mentioned methods, the method comprising: determining the user's reading companion mark position from a text page image; interpreting the user's reading companion instruction to determine a reading companion start indicator and a reading companion end indicator; based on the reading companion mark position, the reading companion start indicator and the reading companion end indicator, performing character recognition on the text page image to obtain a reading companion text; and performing intelligent reading companion based on the reading companion text.

[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent reading companion method, characterized in that: include: Determine the user's reading companion mark position from the text page image; Interpreting the accompanying reading instruction of the user, and determining an accompanying reading start indicator and an accompanying reading end indicator; Based on the companion reading mark position, the companion reading start indicator and the companion reading end indicator, character recognition is performed on the text page image to obtain a companion reading text; Based on the accompanying reading text, intelligent accompanying reading is performed.

2. The intelligent reading companion method according to claim 1, characterized in that: The method of performing character recognition on the text page image based on the companion reading mark position, the companion reading start indicator and the companion reading end indicator to obtain the companion reading text includes: Starting from the position of the companion reading mark, the text page image is recognized character by character until the companion reading start indicator and the companion reading end indicator are recognized, and then the companion reading text is determined based on all recognized characters.

3. The intelligent reading companion method according to claim 2, characterized in that: The step of taking the position of the companion reading mark as a starting point and performing character-by-character recognition on the text page image includes: Taking the position of the companion reading mark as the starting point, the text page image is recognized character by character, and after each character is recognized, it is determined whether the corresponding character is the companion reading start indicator or the companion reading end indicator; If so, the character recognition in the target recognition direction is stopped; if not, the next character recognition is continued in the target recognition direction, where the target recognition direction is a straight line direction from the position of the companion reading mark to the position of the corresponding character.

4. The intelligent reading companion method according to claim 1, characterized in that: The method of performing character recognition on the text page image based on the accompanying reading mark position, the accompanying reading start position character and the accompanying reading end position character to obtain the accompanying reading text includes: Performing character recognition on the text page image to determine a plurality of candidate companion reading start indicators and a plurality of candidate companion reading end indicators; Using the candidate companion reading start indicator closest to the companion reading mark position as the companion reading start indicator, and using the candidate companion reading end indicator closest to the companion reading mark position as the companion reading end indicator; The companion reading text is determined based on all characters between the companion reading start indicator and the companion reading end indicator.

5. The intelligent reading companion method according to any one of claims 1 to 4, characterized in that: The intelligent accompanying reading based on the accompanying reading text includes: Receiving a language partner switching instruction from the user; Under the language partner role indicated by the language partner switching instruction, intelligent reading companion is performed based on the reading companion text.

6. The intelligent reading companion method according to any one of claims 1 to 4, characterized in that: The intelligent accompanying reading is performed based on the accompanying reading text, and then further comprises: Based on the question text of the user regarding the companion reading text, generating an answer text within the cognitive scope of the user; Output the audio file corresponding to the answer text.

7. The intelligent reading companion method according to any one of claims 1 to 4, characterized in that: Determining the position of the user's accompanying reading mark from the text page image includes: Performing target detection on the text page image to extract an image of the area where a target object is located, wherein the target object refers to an object used to indicate the accompanying reading position; Performing edge detection on the regional image to determine the contour position of the target object; The position of the companion mark is determined based on the contour position of the target object.

8. An intelligent reading companion device, characterized in that: include: A determination unit, used to determine the position of the user's reading companion mark from the text page image; An interpretation unit, used for interpreting the accompanying reading instruction of the user, and determining an accompanying reading start indicator and an accompanying reading end indicator; A recognition unit, configured to perform character recognition on the text page image based on the companion reading mark position, the companion reading start indicator and the companion reading end indicator to obtain a companion reading text; The accompanying reading unit is used for performing intelligent accompanying reading based on the accompanying reading text.

9. An all-in-one machine, comprising a memory, a processor, a large model stored in the memory, and a computer program stored in the memory and running on the processor; when the processor executes the computer program, the large model is called to implement the intelligent reading companion method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the intelligent reading companion method as claimed in any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the intelligent reading companion method as claimed in any one of claims 1 to 7 is implemented.

Images