Method and device for realizing auxiliary learning in vehicle, equipment and storage medium

Abstract

The invention relates to the technical field of automobile intelligence, in particular to a method, device and equipment for realizing auxiliary learning in a vehicle and a storage medium, and the method comprises the steps: obtaining an internal space image of a target vehicle, the internal space image comprises a target page region, and the target page region is a region range currently read by a learning object carried in the target vehicle; determining page contents correspondingly displayed in the target page area, and obtaining area coordinates of sub-page contents in the page contents; obtaining a reading interaction key point in the internal space image; and based on position mapping between the position coordinates of the reading interaction key points and the area coordinates of the sub-page contents, obtaining the finger-reading contents of the learning object in the target page area, and carrying out auxiliary learning on the finger-reading contents. According to the invention, accurate positioning and identification of the finger-read content of the learning object in the vehicle can be realized, and the hardware cost is reduced while the learning interestingness of the learning object is improved.

Description

Technical Field

[0001] This application relates to the field of automotive intelligent technology, specifically to methods, devices, equipment, and storage media for enabling assisted learning within vehicles. Background Technology

[0002] Many car models on the market, such as MPVs and large SUVs, are now equipped with large rear entertainment screens for passenger entertainment, and OMS (Occupant Monitoring System) cameras on the roof to monitor the rear seats. Parents and children often use these screens for leisure and entertainment, while the OMS cameras monitor children's behavior in the back seat for safety. However, with the development of intelligent technology, people's needs for in-car functions are becoming increasingly diverse and personalized, especially in family travel scenarios. Providing a good learning environment for children has become a focus for many parents.

[0003] Currently, there is no technical record on how to accurately and effectively recognize content pointed to by children inside a vehicle, thereby helping children to learn effectively while in the car. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, device, and storage medium for assisted learning in a vehicle, to solve the problem that there is currently no relevant technology to record how to accurately and effectively recognize the content pointed to and read by children in a vehicle, thereby helping children to learn effectively in the car.

[0005] In a first aspect, this application provides a method for implementing assisted learning in a vehicle, the method comprising: Acquire an image of the interior space of the target vehicle, wherein the interior space image includes the target page region, which is the area currently being read by the learning object accommodating the target vehicle. Determine the page content to be displayed corresponding to the target page area, and obtain the area coordinates of each subpage content within the page content; The key points for reading interaction in the internal space image are obtained. The key points for reading interaction are the point information selected from the key points based on the reading method after determining the reading method based on multiple key points. Based on the positional mapping between the location coordinates of key points in the reading interaction and the area coordinates of the subpage content, the reading content of the learning object in the target page area is obtained, and the reading content is used for auxiliary learning.

[0006] Furthermore, determine the page content to be displayed corresponding to the target page area, including: The target page area is identified by its book category and page number, thus obtaining the target book category and page number corresponding to the target page area; Based on the target book category and target page number, determine the page content corresponding to the target page area.

[0007] Furthermore, the target page area is identified by its book category and page number to obtain the target book category and page number corresponding to the target page area, including: Input the target page area into the classification model, and the output will be the initial book category to which the target page area belongs and the initial page number to which the page is displayed; The initial book category and initial page number are verified, and the target page area is identified based on the verification results. The initial book category and initial page number are then adjusted to obtain the target book category and target page number.

[0008] Furthermore, based on the positional mapping between the location coordinates of key points in the reading interaction and the area coordinates of the subpage content, the reading content pointed to by the learning object in the target page area is obtained, including: The position coordinates of the key points of the reading interaction in the image coordinate system are transformed into the area coordinates of the subpage content in the target page area coordinate system, so as to obtain the target coordinates of the page content pointed to by the learning object in the reading state within the target page area. The reading content is obtained based on the target coordinates.

[0009] Furthermore, supplementary learning methods are provided for the content to be read aloud, including: It recognizes the content being read aloud and plays the content aloud. or, After receiving a question request from the learner, determine the learner's cognitive level based on the content they are pointing to and reading. Based on the cognitive level and the question request, a comprehensive answer is obtained, and voice feedback is provided on the comprehensive answer.

[0010] Furthermore, after obtaining the interior space map of the target vehicle, the method also includes: identifying the target page region contained in the interior space image; Identify target page regions contained within the interior space image, including: The internal spatial image is segmented to obtain the initial page region by segmenting the page outline; The image features within the initial page area are corrected to obtain the repaired target page area.

[0011] Furthermore, the internal spatial image is segmented into page contours to obtain the initial page region, including: Input the interior space image into the instance segmentation model, and output the page bounding box and segmentation mask; The initial page region is obtained by post-processing the page bounding box based on the page bounding box and the segmentation mask.

[0012] Furthermore, the image features within the initial page area are corrected to obtain the repaired target page area, including: Input the initial page area into the generator to obtain the corrected page image; The corrected page image is input into the discriminator, which compares the differences between the original page image and the page image until the discriminator outputs the target value, thus obtaining the target page region. The original page image is the reference image used when correcting the image features within the initial page region.

[0013] Secondly, this application provides an in-vehicle device for assisted learning, the device comprising: The first acquisition module is used to acquire an image of the interior space of the target vehicle, wherein the image of the interior space includes a target page area, which is the area currently being read by the learning object riding in the target vehicle. The determination module is used to determine the page content displayed corresponding to the target page area and obtain the area coordinates of each subpage content in the page content; The second acquisition module is used to acquire reading interaction key points in the internal space image. The reading interaction key points are point information selected from the key points based on the reading method after determining the reading method based on multiple key points. The module is used to obtain the reading content of the learning object in the target page area based on the positional mapping between the positional coordinates of the reading interaction key points and the area coordinates of the subpage content, and to assist in the learning of the reading content.

[0014] Thirdly, this application provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the in-vehicle assisted learning method described in the first aspect or any corresponding embodiment thereof.

[0015] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the in-vehicle assisted learning method of the first aspect or any corresponding embodiment described above.

[0016] In this embodiment, an internal space image of the target vehicle is acquired, including a target page area, which is the area currently being read by the learner in the vehicle. The page content corresponding to the target page area is then determined, and the area coordinates of each subpage within the page content are obtained. Next, key reading interaction points in the internal space image are acquired. Based on the positional mapping between the coordinates of these key points and the area coordinates of the subpage content, the content the learner is pointing to in the target page area is obtained, and then assisted learning is performed on this content. This embodiment utilizes an OMS camera in the target vehicle to acquire an internal space image and then obtains key reading interaction points within that image. Since these key points are selected from multiple key points based on the learner's reading style, mapping the coordinates of these key points to the area coordinates of each subpage content allows for accurate location and recognition of the learner's pointing content, increasing the learner's interest while reducing hardware costs. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a method for implementing assisted learning in a vehicle according to an embodiment of this application; Figure 2 This is a flowchart illustrating another method for implementing assisted learning in a vehicle according to an embodiment of this application; Figure 3 This is a schematic diagram of page repair using a generative adversarial network according to an embodiment of this application; Figure 4 This is a schematic diagram of the overall process of a method for implementing assisted learning in a vehicle according to an embodiment of this application; Figure 5 This is a structural block diagram of a vehicle-mounted assisted learning device according to an embodiment of this application. Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application; Figure 7 This is a schematic structural block diagram of a vehicle according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] Currently, many autonomous vehicles are equipped with roof-mounted OMS cameras to take overhead shots of the rear seats, using these cameras to predict and protect children's behavior. However, with the advancement of intelligent technology, people's needs for in-car functions are becoming increasingly diverse and personalized, especially in family travel scenarios. Providing a good learning environment for children has become a major concern for many parents. Currently, there is no documented technology demonstrating how existing OMS cameras in vehicles can effectively facilitate children's learning inside the car.

[0023] To address the aforementioned problems, according to an embodiment of this application, a method for implementing assisted learning in a vehicle is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0024] This embodiment provides a method for implementing assisted learning within a vehicle. Figure 1 This is a flowchart illustrating a method for implementing assisted learning in a vehicle according to an embodiment of this application, as shown below. Figure 1 As shown, this method can be applied to the vehicle's in-vehicle infotainment system, and the method process includes the following steps: Step S101: Obtain an image of the interior space of the target vehicle, wherein the interior space image includes a target page area, which is the area currently being read by the learning object riding in the target vehicle.

[0025] Specifically, in this embodiment, the OMS camera is generally installed on the roof of the vehicle. It can capture images of the interior space of the target vehicle by taking downward-facing shots. The target vehicle can refer to any vehicle traveling on the road. The current goal is to assist the learning object (e.g., a child, or an adult) inside the target vehicle. This assisted learning refers to recognizing the paper book or electronic book displayed on an electronic device (e.g., an in-vehicle entertainment screen, the learning object's terminal electronic device) that the learning object is currently reading, and obtaining the target page area contained within the interior space image.

[0026] Understandably, the interior space images captured by the OMS camera usually contain some background information, such as the chair fabric, the pattern of the small table, and other interfering elements. The current task is to identify the area that the learning object is currently reading from the interior space image. This area refers to some page areas, which are called target page areas.

[0027] Step S102: Determine the page content to be displayed corresponding to the target page area, and obtain the area coordinates of each subpage content in the page content.

[0028] Optionally, since the target page area is a page outline, it is necessary to further determine the page content displayed within the page outline based on the target page area. It is understood that the page content contains multiple subpage contents (such as a piece of text, an image, etc.), and each subpage content corresponds to a region coordinate within the page content. These region coordinates are the coordinate information of each subpage content within the page content.

[0029] Step S103: Obtain reading interaction key points in the internal space image. The reading interaction key points are point information selected from the key points based on the reading method after determining the reading method based on multiple key points.

[0030] Optionally, the application scenario of this application is to identify the learning situation of the learner reading a book inside the target vehicle, so it is currently necessary to detect the key points of the learner's hand and the key points of the pen.

[0031] Currently, 21 key points of the hand and 2 key points of the pen are obtained from the internal spatial image. Based on the detection of the 21 key points of the hand and the 2 key points of the pen, the reading method adopted by the learning object when reading is determined.

[0032] This application uses Yolo11-pose as an example, specifically designed for real-time keypoint detection tasks. Specifically, an internal spatial image is input into Yolo11-pose, and 21 keypoints on the hand are detected: 0-wrist, 1-thumb metacarpophalangeal joint, 2-thumb proximal interphalangeal joint, 3-thumb distal interphalangeal joint, 4-thumb fingertip, 5-index finger metacarpophalangeal joint, 6-index finger proximal interphalangeal joint, 7-index finger distal interphalangeal joint, 8-index fingertip, 9-middle finger metacarpophalangeal joint, 10-middle finger proximal interphalangeal joint, 11-middle finger distal interphalangeal joint, 12-middle fingertip, 13-ring finger metacarpophalangeal joint, 14-ring finger proximal interphalangeal joint, 15-ring finger distal interphalangeal joint, 16-ring fingertip, 17-little finger metacarpophalangeal joint, 18-little finger proximal interphalangeal joint, 19-little finger distal interphalangeal joint, 20-little fingertip, for a total of 21 keypoints. The key points of a pen include 1-the nib and 2-the end of the pen.

[0033] Having key information about the child's hand and pen is sufficient, but it's impossible to determine whether the child is reading or using their finger or pen tip to read. Therefore, a trained neural network classification model is needed to identify the child's current reading style. This model can categorize children into seven types: index finger reading, middle finger reading, ring finger reading, little finger reading, pen tip reading, pen tail reading, and no finger reading. Inputting the obtained key information into the trained neural network classification model will output the classification result, revealing the child's reading style.

[0034] After determining the reading method of the learning target, it is necessary to select reading interaction key points corresponding to that reading method from the 21 key points of the hand and the 2 key points of the pen. For example, if the current learning target's reading method is index finger pointing, it is assumed that the current learning target is reading by pointing at the page content with their index finger, and the reading interaction key point 8 - index fingertip is selected from the 21 key points of the hand. Step S104: Based on the positional mapping between the position coordinates of the reading interaction key points and the area coordinates of the subpage content, the reading content pointed at by the learning target in the target page area is obtained, and the reading content pointed at is used for assisted learning.

[0035] Optionally, after obtaining the reading interaction key points in step S103 above, the position coordinates of the reading interaction key points in the internal spatial image and the area coordinates of the subpage content learned by the learning object in the page content (which can also be understood as the area coordinates in the target page area) are obtained. The position coordinates of the reading interaction key points and the area coordinates of the subpage content are combined to realize position mapping, thereby obtaining the reading content of the learning object in the target page area.

[0036] The content being read aloud here refers to what the learner is pointing to with their finger or pen tip. Finally, the vehicle's infotainment system analyzes this content and uses one or more learning aids to help the learner master it.

[0037] In this embodiment, an internal space image of the target vehicle is acquired, including a target page area, which is the area currently being read by the learner in the vehicle. The page content corresponding to the target page area is then determined, and the area coordinates of each subpage within the page content are obtained. Next, key reading interaction points in the internal space image are acquired. Based on the positional mapping between the coordinates of these key points and the area coordinates of the subpage content, the content the learner is pointing to in the target page area is obtained, and then assisted learning is performed on this content. This embodiment utilizes an OMS camera in the target vehicle to acquire an internal space image and then obtains key reading interaction points within that image. Since these key points are selected from multiple key points based on the learner's reading style, mapping the coordinates of these key points to the area coordinates of each subpage content allows for accurate location and recognition of the learner's pointing content, increasing the learner's interest while reducing hardware costs.

[0038] This embodiment provides a method for implementing assisted learning within a vehicle, such as... Figure 2 As shown, Figure 2 This is a flowchart of another method for implementing assisted learning in a vehicle according to an embodiment of this application, the process including the following steps: Step S201: Obtain an interior space image of the target vehicle, wherein the interior space image includes a target page region, which is the area currently being read by the learning object accommodating the target vehicle. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0039] Step S202: Determine the page content to be displayed corresponding to the target page area and obtain the area coordinates of the page content.

[0040] Specifically, step S202 includes: Step S2021: Identify the book category and page number of the target page area to obtain the target book category and target page number corresponding to the target page area.

[0041] Optionally, a Transformer-based attention mechanism model can be used to encode the page image in blocks, and capture global features such as the cover logo and book title (for book classification) and local features of the page numbers (for page number recognition) through a self-attention mechanism.

[0042] Alternatively, a traditional template-matching-based recognition method can be used: pre-store standard templates for various book covers (such as the main color scheme and SIFT features of key patterns), calculate the feature matching degree (such as Euclidean distance) between the page to be identified and the template, and the page with the highest matching degree belongs to the category; locate the page number area in the corner of the page, extract the digit outline features, compare them with the 0-9 digit template, and splice them into a complete page number.

[0043] As a preferred approach, step S2021 also includes: Step a1: Input the target page area into the classification model, and output the initial book category to which the target page area belongs and the initial page number displayed on the page; Step a2: Verify the initial book category and initial page number, and identify the target page area based on the verification result. Adjust the initial book category and initial page number to obtain the target book category and target page number.

[0044] Optionally, in this embodiment of the application, a classification model can be trained to specifically identify the book category and page number of the acquired image.

[0045] Specifically, the first step is the establishment of a book database. To meet the recognition needs of different children's books and various versions of children's textbooks, a database needs to be built covering the rich variety of children's books and various versions of children's textbooks available on the market. During this process, electronic versions of the books will be saved, and this data will serve as an important basis for subsequent algorithm queries. By building a comprehensive and rich book database, it can be ensured that the in-vehicle infotainment system can accurately identify and process various types of children's books.

[0046] Then, a shared feature extraction network is used, with two branches in the output layer to identify the book category and page number respectively. Taking MobileNetV4 as an example of a pre-trained classification model, two classification branches are added after feature extraction to output the book category and page number. The implementation process is as follows: MobileNetV4's backbone network employs "depthiable separable convolutions," which reduces the number of parameters while preserving key features. For the needs of book and page number recognition, its feature extraction process is divided into three stages: 1. Shallow feature extraction (convolution layers 1-5): Through 3×3 depthwise separable convolution, shallow features such as page edges and color blocks are extracted, corresponding to basic information such as the outline of the book cover pattern (such as the cartoon character in the picture book) and the number lines of the page numbers. 2. Mid-level feature extraction (convolutional layers 6-12): Introducing "inverse residual structure" and "linear bottleneck" to enhance the expressive power of features and extract mid-level features such as local texture of the page (e.g., font style of textbook cover, font size of page number) and color distribution (e.g., red title of Chinese textbook, blue formula of math textbook); 3. Deep Feature Extraction (Layer 13-18 Convolution): By using convolutional kernels with a larger receptive field (e.g., 5×5), global features are fused to extract deep features such as the overall style of the book (e.g., the hand-drawn style of picture books, the printing style of textbooks) and the positional pattern of page numbers on the page (e.g., page numbers are usually located in the corner of the page). Finally, a 7×7×1024 feature map is output (7×7 is the spatial dimension, 1024 is the channel dimension, representing 1024 feature vectors).

[0047] After the MobileNetV4 feature extraction layer, two independent classification branches are added to implement "book category recognition" and "page number recognition" respectively. The branch structure and functions are as follows: (1) Book category recognition branch: Branch structure: It consists of three parts: "global average pooling - fully connected layer - normalized exponential function activation". Global average pooling: Converts the 7×7×1024 feature map into a 1×1×1024 feature vector (averaging the 7×7 pixels of each channel), reducing the number of parameters and preserving global features; Fully connected layer: Set up 2 fully connected layers. The first layer compresses the 1024-dimensional features to 256 dimensions. The second layer maps the 256-dimensional features to the "number of book categories" dimension (e.g., if it supports 500 common children's books, the output dimension is 500). Normalized exponential function activation: Converts the numerical values ​​output by the fully connected layer into a probability distribution (the sum of the probabilities of all categories is 1), and the category with the highest probability is the identified "book category" (e.g., "XX edition of primary school Chinese, third grade, first semester").

[0048] Core function: By using global features such as the patterns, colors, and fonts on book covers, it distinguishes different types of children's books and provides an index for subsequent access to electronic data in the book database.

[0049] (2) Page number recognition branch: Branching structure: Consistent with the book category branching structure, but the output dimensions are adjusted according to the page number range: Global average pooling: For the same book category branch, the 7×7×1024 feature map is converted into a 1×1×1024 feature vector; Fully connected layer: The first layer is also compressed to 256 dimensions, and the output dimension of the second layer is the "maximum page number" (e.g., children's books usually have no more than 200 pages, so the output dimension is 200). Softmax activation: Outputs the probability of each page number, and the page number with the highest probability is the recognized "page number" (e.g., "page 25").

[0050] In addition, considering the potential for textbook cover wear and illegible page numbers in a vehicle environment, a result verification mechanism needs to be added: (1) Book library matching verification: The identified “book category” is matched with the preset “in-vehicle book library” (which stores the cover features and page number range of 500 children’s books). If the book library does not contain the category, or the identified page number exceeds the actual page number range of the book (e.g., the maximum page number of the book is 150, but 160 is identified), it is judged as “identification abnormal”, triggering the re-identification of the target page area, and finally obtaining the target book category and target page number that are correctly identified.

[0051] (2) Multi-frame fusion error correction: Five consecutive frames of images captured by the OMS camera are used to identify books and page numbers respectively. The result with the most occurrences in the five frames is taken as the final output (e.g., if three frames are identified as "Class A books" and two frames are identified as "Class B books", then it is finally determined to be "Class A books". If three frames are identified as "page 25" and two frames are identified as "page 26", then it is finally determined to be "page 25"), thus reducing misidentification caused by blurry single-frame images.

[0052] Step S2022: Based on the target book category and target page number, determine the page content corresponding to the target page area.

[0053] Optionally, after determining the target book category and target page number, the page content currently being read by the learner can be determined. It's understood that the current page content could be the entire page the learner is currently reading; it's a broad category representing the printed or electronic content the learner is reading. To understand the detailed content the learner is pointing to, it's necessary to obtain the mapping results of the learner's reading interaction key point location coordinates to the area coordinates of each subpage of the page content in order to determine the content the learner is pointing to.

[0054] Step S203: Obtain reading interaction key points in the internal spatial image, whereby the reading interaction key points are used to represent the reading methods that the learning object can adopt when reading. For details, please refer to... Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0055] Step S204: Based on the positional mapping between the position coordinates of the reading interaction key points and the area coordinates of the subpage content, the reading content of the learning object in the target page area is obtained, and the reading content is used for assisted learning.

[0056] Specifically, step S204 includes: Step S2041: The position coordinates of the reading interaction key points in the image coordinate system are transformed into the area coordinates of the subpage content in the target page area coordinate system, so as to obtain the target coordinates of the page content pointed to by the learning object in the reading state within the target page area.

[0057] Step S2042: Obtain the reading content based on the target coordinates.

[0058] Optionally, after obtaining the reading mode, it is determined that the learner is currently reading. Taking pen-tip reading as an example, the position coordinates of the pen tip key points in the image coordinate system are first obtained: Use a lightweight detection model (such as YOLO11) to locate the pen in the image and output the bounding box coordinates of the pen; Within the bounding box of the pen, a key point detection algorithm (such as YOLO11-pose or HRNet) is used to locate the pen tip and obtain its position coordinates.

[0059] Next, obtain the region coordinates of the subpage content within the region coordinate system of the target page area: Targeted boundary detection is used to determine the type of subpage content (such as text paragraphs, images, tables, formulas, etc.) and the coordinates of the subpage boundaries.

[0060] The detected sub-content boundary coordinates are transformed from the original image coordinate system to the target page region coordinate system to obtain the region coordinates.

[0061] The coordinates of the detected pen tip key points in the image coordinate system are converted into the regional coordinates of the subpage content in the target page area coordinate system through a perspective transformation matrix. Then, the relative position of the pen tip in the page is matched to obtain the target coordinates of the page content pointed to by the learning object within the target page area, thereby locating the reading content pointed to by the pen tip.

[0062] Specifically, step S204 further includes: Step S2043: Identify the content being read and play it aloud.

[0063] or, Step S2044: After receiving a question request from the learner, determine the learner's cognitive level based on the content being read aloud; Step S2045: Based on the cognitive level and the question request, obtain a comprehensive answer and provide voice feedback on the comprehensive answer.

[0064] Optionally, when children are engaged in learning activities in the car, they can use the vehicle's infotainment system to achieve multi-dimensional intelligent interaction, with the core interaction methods divided into two main scenarios.

[0065] One feature is interactive reading with finger pointing: When children browse electronic picture books, textbooks, and other learning content on the in-vehicle screen and touch specific text, paragraphs, or illustrations on the screen with their fingers, the in-vehicle system will capture the finger's position in real time through screen touch positioning technology and then trigger the reading with finger pointing function. This function reads aloud the content of the finger-pointing area in a soft child's voice or clear speech speed that is adapted to the child's auditory habits. At the same time, simple annotations can be added according to the content type (such as descriptions of the characters' emotions in picture books or pronunciation prompts for new words in textbooks) to enhance the immersive learning experience.

[0066] Secondly, there's the voice-based question-and-answer interaction: children don't need to operate it manually. They simply activate the question-and-answer mode using a preset child-friendly wake word (such as "xx, I want to ask"), and then ask learning-related questions in natural language (such as "Why do leaves turn yellow in autumn?" or "How do you spell this English word?"). The in-vehicle system first uses a noise-reducing speech recognition model to accurately capture and translate the child's voice questions, generating structured question text. Then, it automatically captures the learning content images currently displayed on the screen (such as illustrations and knowledge point charts for the corresponding chapter), and integrates the image information with the question text in a multimodal manner, inputting them into a customized educational model. The model combines the learning content images with the core of the question to generate a concise answer that matches the child's cognitive level (avoiding complex terminology and using everyday examples when necessary). Finally, the in-vehicle system outputs the answer through a human-like voice (adjustable speed and tone), and simultaneously displays the key points of the answer in text and images on the screen, achieving dual "auditory + visual" feedback to ensure accurate understanding by the child. The entire interaction process does not require the child to leave their seat or manually operate complex buttons, adapting to in-vehicle safety scenarios, while stimulating children's learning interest through diverse interactive forms, balancing practicality and fun.

[0067] In this embodiment, based on key points of reading interaction, accurate recommendations for voice reading or learning extension content are achieved, providing learners with a more convenient and intelligent in-car learning companion service.

[0068] As an optional embodiment, after step S101, the method further includes: identifying a target page region contained in the internal space image; The identification of target page regions contained in the interior space image includes: Step b1: Segment the internal space image into page outlines to obtain the initial page region.

[0069] Optionally, the MaskR-CNN model is used to apply a two-stage framework to the input internal spatial image. First, the RPN network generates page candidate boxes, and then the pixels in the candidate boxes are classified and masked for prediction, outputting an accurate page outline.

[0070] Alternatively, the U-Net model can be used to perform end-to-end page region pixel segmentation by fusing high and low layer features through an encoder-decoder structure and skip connections on the input internal spatial image.

[0071] Step b1 above includes: Step b11: Input the interior space image into the instance segmentation model and output the page bounding box and segmentation mask; Step b12: Post-process the page bounding box based on the page bounding box and the segmentation mask to obtain the initial page region.

[0072] Optionally, the instance segmentation model YOLO11-seg (YouOnlyLookOnceVersion11-Segmentation, the 11th generation of single-stage object detection and segmentation) can be used to perform instance segmentation of the page contours in the internal spatial images captured by OMS. The implementation process is as follows: (1) Preprocessing the acquired internal space images YOLO11-seg is an algorithm specifically designed for real-time instance segmentation. However, in page contour segmentation using an in-vehicle OMS camera, the overhead view of the OMS may result in trapezoidal or irregular quadrilateral distortions. Furthermore, the lighting conditions during OMS capture are unstable, with varying intensity of light. Additionally, the background may contain interfering elements such as seat fabric and tray table patterns. Therefore, preprocessing is necessary. Distortion correction: By using the intrinsic parameters (such as focal length and pixel size) and extrinsic parameters (installation angle and height) of the OMS camera, a perspective transformation algorithm is used to initially correct the tilted image taken from above to an image that is close to the normal viewing angle, thereby reducing the stretching and distortion of the page edges; Light normalization: An adaptive histogram equalization algorithm is used to suppress overexposed areas of the page caused by strong light inside the car, enhance page details in low light environments (such as textbook edges and text outlines), and ensure that the grayscale difference between the page area and the background is more obvious. Noise filtering: Gaussian filtering (5×5 convolution kernel) is used to remove image noise that may exist in the vehicle environment (such as blur caused by camera shake, noise caused by interference from electronic devices in the vehicle), providing a clear image for subsequent segmentation.

[0073] (2) Dataset construction: annotation of page scenes in the textbook The performance of YOLO11-seg depends on the training dataset for scene matching. A "vehicle-mounted overhead camera textbook dataset" needs to be constructed, specifically including: Data collection: Different types of reading materials (such as children's books, picture books, primary school Chinese textbooks, and math workbooks) were placed on the small tables of the second-row seats in different vehicle models (MPV, large SUV). Different lighting conditions (natural light during the day and interior lights at night) and different placement angles (textbook tilted from 0° to 30°) were simulated. More than 10,000 images were captured using an OMS camera, covering common textbook sizes (16mo and 32mo) and cover colors (color and monochrome). Labeling rules: Use the LabelMe labeling tool to label the textbook pages in each image in two ways: first, a "page area mask" to accurately delineate the pixel range of the textbook page and distinguish the page from the background (such as a small table or finger); second, a "page category label" labeled as "children's textbook page" to ensure that the algorithm only focuses on the target area. Data partitioning: The dataset is divided into a training set (70%, used for model parameter learning), a validation set (20%, used for adjusting model hyperparameters such as learning rate), and a test set (10%, used for evaluating the final segmentation effect) in a 7:2:1 ratio. Data augmentation techniques such as image flipping, rotation (±5°), and brightness adjustment are used to improve the model's adaptability to changes in the vehicle scene.

[0074] (3) Model training: Adapting to the real-time requirements of in-vehicle systems The YOLO11-seg network structure consists of three parts: backbone (feature extraction), neck (feature fusion), and head (detection and segmentation head). The training set is input into the YOLO11-seg model, which outputs initial segmentation results. Model parameter tuning is then performed. The test set is then input into the YOLO11-seg model, and parameter tuning is repeated. Finally, the optimized and trained YOLO11-seg is obtained. The optimized training strategy for automotive scenarios aims to ensure a balance between segmentation accuracy and real-time performance. Optimization directions include: Backbone optimization: The CSPDarknet-53 backbone network of YOLO11-seg is used, but the number of channels in the last two convolutional layers is reduced (from 1024 to 512), and the number of model parameters is reduced (from about 25M to 18M) to adapt to the computing power of the vehicle system. Loss function design: A combined loss function of "bounding box loss + mask loss" is adopted: the bounding box loss is used to optimize the positioning accuracy of the rectangular box of the page area to ensure that the selection range covers the entire page; the mask loss is used to optimize the accuracy of the page pixel-level segmentation and reduce the confusion between the page edges and the background.

[0075] (4) Model inference: Real-time output of page segmentation results After the trained YOLO11-seg model is deployed to the vehicle system, it performs inference on the interior space images captured in real time by the OMS camera, and outputs the page bounding box and mask. Then, through morphological closing operations to fill the segmentation holes and adjusting the bounding box to remove the background, the initial segmented page region is finally obtained.

[0076] Step b2: Correct the image features within the initial page area to obtain the repaired target page area.

[0077] Optionally, since the image information contained in the initial page area may be inaccurate, the image features in the initial page area can be corrected based on geometric correction of perspective transformation to obtain the repaired target page area.

[0078] Alternatively, a panoramic correction method based on image stitching can be used: continuously capture multiple frames of page images from different perspectives, extract feature points from each frame; match feature points of adjacent frames, calculate the transformation matrix, and stitch the multiple frames into a complete page; apply perspective transformation to the stitched complete page to correct it to a forward view, thus obtaining the repaired target page area.

[0079] Step b2 above includes: Step b21: Input the initial page area into the generator to obtain the corrected page image; Step b23: Input the corrected page image into the discriminator, and use the discriminator to compare the difference between the original page image and the page image until the discriminator outputs the target value to obtain the target page region. The original page image is the image that needs to be referenced when correcting the image features in the initial page region.

[0080] Optionally, since the segmented area is not necessarily oriented, and OMS is a top-down view with a certain tilt angle, it will affect subsequent recognition.

[0081] Therefore, this application proposes a page repair method based on generative adversarial networks. For example... Figure 3 As shown, the GAN network, based on the Generative Adversarial Network (GAN) feature inpainting network structure, consists of a generator and a discriminator. The generator corrects the page image, generating a corrected page image. The discriminator compares the feature-inpainted page image with the original page image. The generator and discriminator iterate until the discriminator can no longer determine whether the newly generated image features are genuine identifier data features.

[0082] It should be noted that the discriminator cannot determine whether the generated new image features are genuine identifier data features until the discriminator outputs a target value. This target value can be 1, which represents that there is no difference between the original page image and the decoded page image.

[0083] a. Generator like Figure 3 As shown, the generator mainly consists of two parts: an encoding network and a decoding network for the image. First, the content of the initial page region is cropped out as the original page input to the generator. The generator then learns image features through the encoding network and generates the corrected page.

[0084] The encoding network chosen is MobileNetV4. After encoding, two fully connected layers are added to pass the encoded features to the decoding network. To ensure image accuracy, the decoding network decodes the generated new features into an image and sends it to the discriminator for comparison with the original page image. The structure of the decoding network is the same as that of the encoding network, except that the pooling layers in the decoding network are replaced with unpooling layers. The original page image is the standard sample in the training set and is also the reference image used to correct the image features within the initial page region.

[0085] b. Discriminator The discriminator uses a ResNet-18 architecture and adds a sigmoid binary classifier at the end of the network. This part discriminates the corrected images generated by the generator.

[0086] As an alternative embodiment, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the overall process of implementing assisted learning in a vehicle according to an embodiment of this application. The overall logical flow is as follows: Children study in the second row of the car, with their books placed on small tables. The OMS camera takes a downward-facing shot to obtain an image of the interior space. The captured internal space image is used to detect page regions, and then the page orientation of the detected page regions is corrected to obtain the page content in the corrected page regions. The algorithm identifies specific books and page numbers based on the page content. Simultaneously, it recognizes key points of the child's hand and pen, determining whether the child is reading with a finger or a pen. Then, by clearly matching the position of the fingertip or pen tip with the detected page content, it pinpoints the specific location, locates the book in the database, and obtains the specific paragraph the child is pointing to, thus achieving page content localization.

[0087] Afterwards, you can ask the vehicle's infotainment system questions via voice to provide accompanying reading or related knowledge recommendations for the located page content, and the system can respond using a large model.

[0088] This embodiment also provides an in-vehicle device for assisted learning, which implements the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0089] This embodiment provides a device for assisted learning within a vehicle, such as... Figure 5 As shown, it includes: The first acquisition module 501 is used to acquire an image of the interior space of the target vehicle, wherein the image of the interior space includes a target page area, which is the area currently being read by the learning object riding in the target vehicle. The determination module 502 is used to determine the page content displayed corresponding to the target page area and obtain the area coordinates of each subpage content in the page content; The second acquisition module 503 is used to acquire reading interaction key points in the internal space image. The reading interaction key points are point information selected from the key points based on the reading method after determining the reading method based on multiple key points. Module 504 is used to obtain the reading content of the learning object in the target page area based on the positional mapping between the positional coordinates of the reading interaction key points and the area coordinates of the subpage content, and to assist in the learning of the reading content.

[0090] In this embodiment, an internal space image of the target vehicle is acquired, including a target page area, which is the area currently being read by the learner in the vehicle. The page content corresponding to the target page area is then determined, and the area coordinates of each subpage within the page content are obtained. Next, key reading interaction points in the internal space image are acquired. Based on the positional mapping between the coordinates of these key points and the area coordinates of the subpage content, the content the learner is pointing to in the target page area is obtained, and then assisted learning is performed on this content. This embodiment utilizes an OMS camera in the target vehicle to acquire an internal space image and then obtains key reading interaction points within that image. Since these key points are selected from multiple key points based on the learner's reading style, mapping the coordinates of these key points to the area coordinates of each subpage content allows for accurate location and recognition of the learner's pointing content, increasing the learner's interest while reducing hardware costs.

[0091] In this embodiment of the application, the determining module 502 is used to identify the book category and page number of the target page area to obtain the target book category and target page number corresponding to the target page area; and to determine the page content corresponding to the target page area based on the target book category and target page number.

[0092] In this embodiment of the application, the determining module 502 is used to input the target page area into the classification model, output the initial book category to which the target page area belongs and the initial page number displayed on the page; verify the initial book category and the initial page number, and identify the target page area according to the verification result, adjust the initial book category and the initial page number, and obtain the target book category and the target page number.

[0093] In this embodiment, module 504 is used to convert the position coordinates of the reading interaction key points in the image coordinate system to the area coordinates of the subpage content in the target page area coordinate system, so as to obtain the target coordinates of the page content pointed to by the learning object in the finger reading state within the target page area; and obtain the finger reading content based on the target coordinates.

[0094] In this embodiment of the application, module 504 is used to identify the content being read and play it aloud; or, after receiving a question request initiated by the learner, to determine the learner's cognitive level based on the content being read; and based on the cognitive level and the question request, to obtain a comprehensive answer and provide voice feedback on the comprehensive answer.

[0095] In this embodiment of the application, the device further includes: The recognition module is used to identify target page regions contained in the interior space image after acquiring the interior space image of the target vehicle.

[0096] In this embodiment of the application, the recognition module is used to segment the page outline of the internal space image to obtain an initial page region; and to correct the image features within the initial page region to obtain a repaired target page region.

[0097] In this embodiment, the recognition module is used to input the internal spatial image into the instance segmentation model and output the page bounding box and segmentation mask; and to perform post-processing on the page bounding box and segmentation mask to obtain the initial page region.

[0098] In this embodiment, the identification module is used to input the initial page region into the generator to obtain the corrected page image; input the corrected page image into the discriminator, and use the discriminator to compare the difference between the original page image and the page image until the discriminator outputs the target value to obtain the target page region. The original page image is the image that needs to be referenced when correcting the image features within the initial page region.

[0099] In this embodiment, the in-vehicle device for assisted learning is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0100] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0101] The following is a detailed reference. Figure 6 This diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present disclosure. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0102] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0103] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the in-vehicle assisted learning method of embodiments of this disclosure.

[0104] Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0105] This application also provides a vehicle, such as... Figure 7 As shown, the vehicle may include: The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0106] When the processor 702 executes the program, it implements the in-vehicle assisted learning method provided in the above embodiments.

[0107] Furthermore, the vehicle also includes: Communication interface 703 is used for communication between memory 701 and processor 702.

[0108] The memory 701 is used to store computer programs that can run on the processor 702.

[0109] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0110] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0111] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0112] Processor 702 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. This application also provides a computer-readable storage medium in which the methods described in the embodiments of this application can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium may be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the in-vehicle assisted learning method shown in the above embodiments is implemented.

[0113] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0114] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method of enabling assisted learning within a vehicle, characterized by, The method includes: Acquire an image of the interior space of the target vehicle, wherein the image includes a target page region, which is the area currently being read by the learning object accommodating the target vehicle. Determine the page content displayed corresponding to the target page area, and obtain the area coordinates of each subpage content in the page content; The key points for reading interaction in the internal space image are obtained, wherein the key points for reading interaction are point information selected from the key points based on the reading method after determining the reading method based on multiple key points; Based on the positional mapping between the location coordinates of the key points of the reading interaction and the area coordinates of the subpage content, the reading content pointed to by the learning object in the target page area is obtained, and the reading content is used for assisted learning.

2. The method of claim 1, wherein, Determining the page content displayed corresponding to the target page area includes: The target page area is identified by its book category and page number to obtain the target book category and target page number corresponding to the target page area; Based on the target book category and target page number, determine the page content corresponding to the target page area.

3. The method of claim 2, wherein, The step of identifying the book category and page number of the target page area to obtain the target book category and target page number corresponding to the target page area includes: Input the target page area into the classification model, and output the initial book category to which the target page area belongs and the initial page number displayed on the page; The initial book category and the initial page number are verified, and the target page area is identified based on the verification results. The initial book category and the initial page number are then adjusted to obtain the target book category and the target page number.

4. The method of claim 1, wherein, The location mapping between the position coordinates of the reading interaction key points and the area coordinates of the subpage content, to obtain the reading content pointed to by the learning object in the target page area, includes: The position coordinates of the reading interaction key points in the image coordinate system are transformed into the area coordinates of the subpage content in the target page area coordinate system to obtain the target coordinates of the page content pointed to by the learning object in the finger reading state within the target page area. The reading content is obtained based on the target coordinates.

5. The method of claim 1, wherein, The method of assisting learning the pointed-to-read content includes: The system identifies the content being read aloud and plays it aloud via voice. or, Upon receiving a question request from the learning subject, the cognitive level of the learning subject is determined based on the content being read aloud. Based on the cognitive level and the question request, a comprehensive answer is obtained, and voice feedback is provided on the comprehensive answer.

6. The method of claim 1, wherein, After obtaining the interior space image of the target vehicle, the method further includes: identifying target page regions contained in the interior space image; Identifying the target page region contained in the interior space image includes: The internal space image is segmented into page outlines to obtain an initial page region; The image features within the initial page area are corrected to obtain the repaired target page area.

7. The method of claim 6, wherein, The step of segmenting the internal space image into page contours to obtain an initial page region includes: The internal space image is input into the instance segmentation model, and the page bounding box and segmentation mask are output. The initial page region is obtained by post-processing the page bounding box based on the page bounding box and the segmentation mask.

8. The method of claim 6, wherein, The step of correcting the image features within the initial page region to obtain the repaired target page region includes: The initial page area is input into the generator to obtain the corrected page image; The corrected page image is input into a discriminator, which compares the differences between the original page image and the page image until the discriminator outputs a target value, thus obtaining the target page region. The original page image is the image that needs to be referenced when correcting the image features within the initial page region.

9. An apparatus for enabling assisted learning in a vehicle, the apparatus comprising: The device includes: The first acquisition module is used to acquire an image of the interior space of the target vehicle, wherein the image of the interior space includes a target page area, which is the area currently being read by the learning object riding in the target vehicle. The determination module is used to determine the page content displayed corresponding to the target page area and obtain the area coordinates of each subpage content in the page content; The second acquisition module is used to acquire reading interaction key points in the internal space image, wherein the reading interaction key points are point information selected from the key points based on the reading method after determining the reading method based on multiple key points; The module is used to obtain the reading content pointed to by the learning object in the target page area based on the positional mapping between the positional coordinates corresponding to the reading interaction key points and the area coordinates of the subpage content, and to assist in learning the reading content pointed to.

10. A computer device, comprising: include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the in-vehicle assisted learning method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the method for assisted learning in a vehicle as described in any one of claims 1 to 8.

Images