A multi-modal interaction data processing method and system for children's picture book reading

Abstract

The application provides a kind of multi-modal interaction data processing method and system of children's picture book reading, it is related to data processing technical field, the method includes: to multi-modal original interaction data set is processed in parallel, extracts and outputs visual semantic feature, interactive speech semantic feature and interactive behavior semantic feature;In visual semantic feature and interactive speech semantic feature, dynamically determine at least three key semantic anchor points, i.e., visual key semantic anchor point corresponds to the core role space coordinate of current picture book page and key text area center point, interactive speech key semantic anchor point corresponds to the core emotion frame timestamp in speech spectrum and question word position;Based on the key semantic anchor point, construct dynamic semantic geometric relationship graph in cross-modal joint feature space.The application can realize the accurate adaptive fusion of cross-modal feature, effectively improve the accuracy of children's picture book reading interaction data processing and the adaptability of interactive response.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal interactive data processing method and system for children's picture book reading. Background Technology

[0002] With the deep integration of artificial intelligence technology and early childhood education, multimodal interaction technology is gradually becoming the core supporting technology for children's smart picture book reading products.

[0003] Current multimodal interactive data processing solutions for children's picture book reading suffer from the following technical shortcomings: the fusion process of cross-modal semantic features lacks dynamic geometric association constraints based on precise semantic anchors, relying solely on static weighting or simple splicing methods. This fails to establish a precise geometric mapping and dynamic association mechanism based on core semantics across visual, speech, and behavioral multimodalities. Consequently, the accuracy of cross-modal semantic fusion is insufficient, making it difficult to adapt to the dynamic changes in semantic associations within children's picture book reading scenarios. For example, in a specific scenario where a child points to a key character on a picture book page and asks a question with a questioning tone, existing technologies cannot accurately match the spatial semantic information of the key character at the visual level and the temporal semantic information of the question word and emotion at the vocal level. Static fusion logic loses crucial cross-modal semantic association information, resulting in the generated multimodal joint semantic representation failing to accurately match the core semantics of the current picture book page and the child's actual interaction state. Moreover, due to the lack of constraints from dynamic semantic geometric association, the binding effect between the fused multimodal semantic representation and the picture book plot and the child's interactive intention is poor, resulting in low accuracy in judging interactive intention and assessing cognitive state. The output interactive response is difficult to match with the evolution rhythm of the picture book plot and the child's real-time reading needs, and cannot achieve personalized and coherent picture book reading interaction guidance. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a multimodal interactive data processing method and system for children's picture book reading, which can realize accurate adaptive fusion of cross-modal features and effectively improve the accuracy of interactive data processing and the adaptability of interactive response in children's picture book reading.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] Firstly, a multimodal interactive data processing method for children's picture book reading, the method comprising:

[0007] Parallel processing is performed on the original multimodal interaction dataset to extract and output visual semantic features, interactive speech semantic features, and interactive behavior semantic features;

[0008] In visual semantic features and interactive speech semantic features, key semantic anchors are dynamically determined. These key semantic anchors include visual key semantic anchors and interactive speech key semantic anchors. Visual key semantic anchors include candidate visual spatial anchors and candidate text region anchors. Interactive speech key semantic anchors include candidate emotion temporal anchors and candidate question semantic anchors. Candidate visual spatial anchors correspond to the spatial coordinates of the core character on the current picture book page. Candidate text region anchors correspond to the center point of the key text region on the current picture book page. Candidate emotion temporal anchors correspond to the timestamp of the core emotion frame in the speech spectrum. Candidate question semantic anchors correspond to the position of the question word in the speech spectrum.

[0009] Based on the key semantic anchors, a dynamic semantic geometric relationship graph is constructed in the cross-modal joint feature space; wherein, a semantic ellipse is generated with the visual key semantic anchors as the focus; wherein, the center of the circle is obtained by weighted calculation of the core emotion frame timestamp anchor and the question word position anchor in the key semantic anchors of interactive speech, so as to generate a semantic perception circle.

[0010] Calculate the geometric inclusion degree and area overlap rate between the semantic ellipse and the semantically perceived circle, and generate a cross-modal semantic fusion adjustment coefficient;

[0011] A cross-modal semantic fusion adjustment coefficient is used to weight and fuse visual semantic features, interactive speech semantic features, and interactive behavior semantic features to generate a multimodal joint semantic representation.

[0012] Based on multimodal joint semantic representation, joint reasoning is performed to parse and output the results of children's interaction intention judgment and cognitive state assessment. Based on this, the multimodal joint semantic representation is iteratively updated to generate a coherent reading conversation context representation.

[0013] Based on the contextual representation of the coherent reading conversation and the results of the interaction intent determination, personalized interactive response instructions that adapt to the evolution of the picture book plot are generated and output.

[0014] Furthermore, the original multimodal interaction dataset is processed in parallel to extract and output visual semantic features, interactive speech semantic features, and interactive behavior semantic features, including:

[0015] From the acquired multimodal raw interaction dataset, picture book image data, children's voice data, touch pressure data, and 3D pose data were separated.

[0016] Edge preprocessing is performed on the picture book image data to obtain the preprocessed page image; the preprocessed page image is segmented by picture book page features and object recognition is performed to locate the text area and illustration element area; text semantic information is extracted from the text area and visual semantic information of characters and scenes is extracted from the illustration element area; text semantic information and visual semantic information are integrated to form and output visual semantic features.

[0017] Noise reduction and endpoint detection are performed on children's speech data to obtain clean speech segments; acoustic feature analysis and speech recognition are performed on the clean speech segments to extract acoustic features from the speech spectrum and transcribe them into text information; combined with the acoustic features and text information, interactive speech semantic features containing reading content, question content and tone of voice are analyzed and output.

[0018] The touch pressure data and 3D posture data are synchronized and time-aligned to obtain a synchronized behavior sequence. By parsing the page turning pressure change pattern, page flipping angle sequence and gaze focus change trajectory from the synchronized behavior sequence, interactive behavior semantic features that represent reading rhythm and concentration state are generated and output.

[0019] Furthermore, in the visual semantic features and interactive speech semantic features, key semantic anchors are dynamically determined. These key semantic anchors include visual key semantic anchors and interactive speech key semantic anchors. Visual key semantic anchors include candidate visual spatial anchors and candidate text region anchors. Interactive speech key semantic anchors include candidate emotion temporal anchors and candidate interrogative semantic anchors. Candidate visual spatial anchors correspond to the spatial coordinates of the core character on the current picture book page; candidate text region anchors correspond to the center point of the key text region on the current picture book page; candidate emotion temporal anchors correspond to the timestamp of the core emotion frame in the speech spectrum; and candidate interrogative semantic anchors correspond to the position of the interrogative word in the speech spectrum. These include:

[0020] Based on visual semantic features, the core character objects and key text regions in the current picture book page are identified. The center coordinates of the bounding box of the core character objects in the page image coordinate system are extracted as candidate visual space anchor points; the geometric center coordinates of the key text regions are extracted as candidate text region anchor points.

[0021] Based on interactive speech semantic features, we can identify emotional fluctuation frames and interrogative sentence patterns in children's speech segments. At the same time, we can locate the center timestamp of the emotional fluctuation frame on the speech timeline as a candidate emotional time anchor point; and locate the position index of the interrogative word in the text sequence as a candidate interrogative semantic anchor point.

[0022] Furthermore, based on the key semantic anchors, a dynamic semantic geometric relationship graph is constructed in the cross-modal joint feature space; wherein, a semantic ellipse is generated with the visual key semantic anchors as the focus; wherein, by performing a weighted calculation on the core emotion frame timestamp anchors and question word position anchors in the interactive voice key semantic anchors, the center of the circle is obtained to generate a semantic perception circle, including:

[0023] Using the spatial coordinates of the core character and the center point of the key text region in the visual key semantic anchor points as the focus, the focal length of the ellipse is calculated based on the preset semantic association distance threshold; based on the calculated ellipse focal length and the semantic saliency weight, a semantic ellipse is generated in the visual modal subspace of the cross-modal joint feature space.

[0024] By weighting the core emotion frame timestamp anchor and question word position anchor in the key semantic anchors of interactive speech, and using the calculation result as the center, the perception radius is calculated by the acoustic feature intensity and emotional confidence in the semantic features of interactive speech; based on the center and perception radius, a semantic perception circle is generated in the speech modal subspace of the cross-modal joint feature space.

[0025] The semantic ellipse and the semantic perception circle are mapped to the same cross-modal joint feature space coordinate system to form a dynamic semantic geometric relationship graph. The semantic ellipse represents the distribution range of the visual semantics of the picture book, and the semantic perception circle represents the perception range of the semantics of children's voice interaction. The semantic ellipse and the semantic perception circle together construct the geometric constraint framework of cross-modal semantic association.

[0026] Furthermore, the geometrical inclusion degree and area overlap rate between the semantic ellipse and the semantically perceived circle are calculated to generate a cross-modal semantic fusion adjustment coefficient, including:

[0027] Under the same coordinate system mapped by the dynamic semantic geometric relationship graph, the focal position, major axis and minor axis length of the semantic ellipse, as well as the center coordinates and perception radius of the semantic perception circle are obtained;

[0028] Based on the obtained center coordinates and focus positions, the relative positional relationship between the semantically perceptual circle and the semantic ellipse is calculated; according to the relative positional relationship, it is determined whether the center of the semantically perceptual circle is located inside or on the boundary of the semantic ellipse, and the corresponding first geometrical inclusion value is calculated.

[0029] Calculate the area of ​​the intersection region between the semantically perceived circle and the semantic ellipse, and divide the area of ​​the intersection region by the area of ​​the semantically perceived circle to obtain the second area overlap rate value;

[0030] The first geometric inclusion value and the second area overlap rate value are weighted and synthesized, and combined with the visual semantic distribution confidence represented by the semantic ellipse, to generate the cross-modal semantic fusion adjustment coefficient.

[0031] Furthermore, cross-modal semantic fusion adjustment coefficients are used to weight and fuse visual semantic features, interactive speech semantic features, and interactive behavior semantic features to generate a multimodal joint semantic representation, including:

[0032] Obtain cross-modal semantic fusion adjustment coefficients and simultaneously read visual semantic features, interactive speech semantic features, and interactive behavior semantic features;

[0033] The cross-modal semantic fusion adjustment coefficients are input into the set weight mapping function to calculate the visual weights of visual semantic features, the speech weights of interactive speech semantic features, and the behavioral weights of interactive behavior semantic features.

[0034] Based on the visual weight, voice weight, and behavior weight, the corresponding visual semantic features, interactive voice semantic features, and interactive behavior semantic features are weighted and summed to obtain a preliminary fusion feature vector.

[0035] The initial fusion feature vectors are normalized and dimensionality reduced to output a multimodal joint semantic representation that is precisely bound to the semantic content of the current picture book page.

[0036] Furthermore, based on multimodal joint semantic representation, joint reasoning is performed to parse and output the child's interaction intention determination results and cognitive state assessment results. The multimodal joint semantic representation is then iteratively updated to generate a coherent reading conversation context representation, including:

[0037] Obtain the multimodal joint semantic representation corresponding to the current picture book page;

[0038] The multimodal joint semantic representation is input into the pre-trained intention and cognition joint reasoning model to extract the multimodal semantic association patterns in the multimodal joint semantic representation, and to perform pattern matching and reasoning based on the knowledge graph of the picture book plot, outputting a multimodal semantic matching and reasoning result.

[0039] Based on the results of multimodal semantic matching and reasoning, the interaction intent determination results and cognitive state assessment results are parsed and output.

[0040] The multimodal joint semantic representation, interaction intent determination results, and cognitive state evaluation results are fused with the real-time stored historical reading session context representation in a temporal sequence; the temporal fusion is used to update and generate a coherent reading session context representation for the current moment.

[0041] Furthermore, based on the coherent reading conversation context representation and the interaction intent determination result, personalized interactive response instructions adapted to the evolution of the picture book plot are generated and output, including:

[0042] Obtain the current reading session context representation and interaction intent determination result;

[0043] Based on the current reading session context representation and combined with the picture book plot knowledge graph, the current reading process is predicted to be located in the story and the subsequent plot development, and a plot adaptation strategy is generated.

[0044] Based on the interaction intent determination result and the plot adaptation strategy, the most suitable response template is matched and retrieved from the preset multimodal response template library;

[0045] The key semantic information in the reading session context representation is filled into the most suitable response template to generate specific personalized interactive response instructions, which are then output to the interactive execution terminal.

[0046] Secondly, a multimodal interactive data processing system for children's picture book reading includes:

[0047] The preprocessing and extraction module is used to process the multimodal raw interaction dataset in parallel, extract and output visual semantic features, interactive speech semantic features and interactive behavior semantic features;

[0048] The key semantic anchor point localization module is used to dynamically determine key semantic anchor points from visual semantic features and interactive speech semantic features. The key semantic anchor points include visual key semantic anchor points and interactive speech key semantic anchor points. The visual key semantic anchor points include candidate visual spatial anchor points and candidate text region anchor points. The interactive speech key semantic anchor points include candidate emotion temporal anchor points and candidate question semantic anchor points. The candidate visual spatial anchor points correspond to the spatial coordinates of the core characters on the current picture book page. The candidate text region anchor points correspond to the center point of the key text region on the current picture book page. The candidate emotion temporal anchor points correspond to the timestamp of the core emotion frame in the speech spectrum. The candidate question semantic anchor points correspond to the position of the question word in the speech spectrum.

[0049] The geometric relationship construction module is used to construct a dynamic semantic geometric relationship graph in the cross-modal joint feature space based on the key semantic anchor points; wherein, a semantic ellipse is generated with the visual key semantic anchor points as the focus; wherein, the center of the circle is obtained by weighted calculation of the core emotion frame timestamp anchor points and question word position anchor points in the interactive voice key semantic anchor points to generate a semantic perception circle.

[0050] The fusion coefficient calculation module is used to calculate the geometric inclusion degree and area overlap rate between the semantic ellipse and the semantically perceived circle, and generate a cross-modal semantic fusion adjustment coefficient.

[0051] The multimodal feature fusion module is used to adjust the coefficients through cross-modal semantic fusion, and to perform weighted fusion of visual semantic features, interactive speech semantic features, and interactive behavior semantic features to generate a multimodal joint semantic representation.

[0052] The Intent and Cognitive Reasoning module is used to perform joint reasoning based on multimodal joint semantic representation, parse and output the results of children's interaction intent judgment and cognitive state assessment, and iteratively update the multimodal joint semantic representation accordingly to generate a coherent reading conversation context representation.

[0053] The personalized response generation module is used to generate and output personalized interactive response instructions that adapt to the evolution of the picture book plot based on the contextual representation of the coherent reading conversation and the results of the interaction intent determination.

[0054] The above-described solution of the present invention has at least the following beneficial effects:

[0055] By employing techniques that dynamically determine key semantic anchors, construct dynamic semantic geometric relationship graphs, calculate cross-modal semantic fusion adjustment coefficients, and perform multimodal feature weighted fusion accordingly, this approach overcomes the technical problems of existing technologies, such as the lack of precise semantic anchors in dynamic geometric association constraints, reliance on static weighting or simple splicing leading to insufficient fusion accuracy, and low accuracy in determining interactive intent and assessing cognitive state. This improves the accuracy of multimodal joint semantic representation, enhances the accuracy of children's interactive intent determination and cognitive state assessment, achieves personalized and coherent interactive responses adapted to the evolution of the picture book plot, and optimizes the intelligent interactive experience of children's picture book reading. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating a multimodal interactive data processing method for children's picture book reading provided by an embodiment of the present invention.

[0057] Figure 2 This is a schematic diagram of a multimodal interactive data processing system for children's picture book reading provided by an embodiment of the present invention. Detailed Implementation

[0058] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0059] like Figure 1 As shown, an embodiment of the present invention proposes a multimodal interactive data processing method for children's picture book reading, the method comprising the following steps:

[0060] Step 1: Perform parallel processing on the original multimodal interaction dataset to extract and output visual semantic features, interactive speech semantic features, and interactive behavior semantic features;

[0061] Step 2: Dynamically determine key semantic anchors from visual semantic features and interactive speech semantic features. Key semantic anchors include visual key semantic anchors and interactive speech key semantic anchors. Visual key semantic anchors include candidate visual spatial anchors and candidate text region anchors. Interactive speech key semantic anchors include candidate emotion temporal anchors and candidate question semantic anchors. Candidate visual spatial anchors correspond to the spatial coordinates of the core character on the current picture book page. Candidate text region anchors correspond to the center point of the key text region on the current picture book page. Candidate emotion temporal anchors correspond to the timestamp of the core emotion frame in the speech spectrum. Candidate question semantic anchors correspond to the position of the question word in the speech spectrum.

[0062] Step 3: Based on the key semantic anchors, construct a dynamic semantic geometric relationship graph in the cross-modal joint feature space; wherein, a semantic ellipse is generated with the visual key semantic anchors as the focus; wherein, by performing weighted calculation on the core emotion frame timestamp anchors and question word position anchors in the interactive voice key semantic anchors, the center of the circle is obtained to generate a semantic perception circle.

[0063] Step 4: Calculate the geometric inclusion degree and area overlap rate between the semantic ellipse and the semantic perception circle to generate a cross-modal semantic fusion adjustment coefficient;

[0064] Step 5: Using cross-modal semantic fusion adjustment coefficients, visual semantic features, interactive speech semantic features, and interactive behavior semantic features are weighted and fused to generate a multimodal joint semantic representation.

[0065] Step 6: Perform joint reasoning based on multimodal joint semantic representation, parse and output the child's interaction intention judgment results and cognitive state assessment results, and iteratively update the multimodal joint semantic representation accordingly to generate a coherent reading conversation context representation.

[0066] Step 7: Based on the coherent reading conversation context representation and the interaction intent determination result, generate and output personalized interactive response instructions that adapt to the evolution of the picture book plot.

[0067] In this embodiment of the invention, parallel processing refers to employing a multi-threaded or distributed processing architecture to simultaneously extract features from picture book image data, children's voice data, touch pressure data, and 3D posture data. Each modality of data undergoes preprocessing, feature segmentation, and semantic parsing synchronously in an independent processing unit, ultimately outputting three types of semantic features in parallel: visual, speech, and behavioral, thereby improving overall processing efficiency and real-time interaction. By extracting multimodal semantic features in parallel, the core information of picture book visuals, children's speech, and interactive behavior is accurately captured. Dynamically determined key semantic anchors provide precise positioning for cross-modal associations, and combined with the construction of a dynamic semantic geometric relationship graph, effective association constraints between visual and speech modalities are established. Fusion adjustment coefficients generated based on geometric inclusion degree and area overlap rate achieve adaptive and precise fusion of multimodal features, improving the reliability of multimodal joint semantic representation. Subsequent joint reasoning and context iteration updates ensure the accuracy and coherence of interactive intent determination and cognitive state assessment. The final generated personalized interactive response commands accurately adapt to the evolution of the picture book plot and children's real-time reading needs, effectively optimizing the intelligent interactive experience of children's picture book reading.

[0068] In a preferred embodiment of the present invention, step 1 above may include:

[0069] Step 1.1: From the acquired multimodal raw interactive dataset, separate the picture book image data, children's voice data, touch pressure data, and 3D posture data. Specifically, this includes: synchronously collecting data through the multimodal acquisition components mounted on the children's picture book reading interactive terminal. The image acquisition sensor is responsible for capturing the complete visual image of the picture book page, the sound acquisition microphone synchronously records the children's voice information such as reading aloud and asking questions during the reading process, the touch pressure sensor senses the pressure signals generated when the children touch the picture book and turn the pages in real time, and the 3D posture capture camera continuously collects the children's spatial posture data such as head rotation and limb swing. All acquisition devices maintain synchronous acquisition time, and finally, a multimodal raw interactive dataset covering four types of information: picture book images, children's voice, touch pressure, and 3D posture is formed.

[0070] To address the differences in characteristics among different modalities in the dataset, we first briefly analyze the signal carriers, encoding formats, and physical attributes of each type of data. For example, picture book image data uses light signals as carriers and employs image encoding formats, while children's speech data uses sound waves as carriers and exhibits waveform variation characteristics. Based on the analysis results, we then formulate splitting rules that are layered according to signal carriers and precisely matched according to attribute features. Next, we separate the data category by category according to the rules: by identifying image encoding identifiers and pixel matrix features, we filter and separate picture book image data from the dataset; by detecting audio waveform patterns and audio encoding features, we separate the corresponding children's speech data; by extracting continuous pressure value fluctuation segments and combining them with the set pressure signal value range, we separate touch pressure data; and by locating dynamic sequences containing three-dimensional spatial coordinates X, Y, and Z axes, we separate three-dimensional posture data. Finally, we archive the four types of separated modal data into the terminal's dedicated storage area, naming them according to the rules of collection timestamps and data types, while verifying the integrity of each type of data to ensure no data confusion or redundancy. Ultimately, we complete the independent splitting, classified storage, and orderly output of the four types of modal data.

[0071] Step 1.2: Perform edge preprocessing on the picture book image data to obtain the preprocessed page image; perform picture book page feature segmentation and object recognition on the preprocessed page image to locate the text area and illustration element area; extract text semantic information from the text area and visual semantic information of characters and scenes from the illustration element area; integrate the text semantic information and visual semantic information to form and output visual semantic features, specifically including: first, transmitting the separated picture book image data to the interactive terminal for processing; starting the edge-side standardized image preprocessing process; normalizing the grayscale values ​​of the picture book image data to unify the brightness and contrast of the image; then removing noise and blur interference in the image through filtering operations; next, correcting the distortion caused by the angle problem during image acquisition; finally, adjusting the image to a uniform size specification to obtain a preprocessed page image with clear image quality and consistent specifications; then, based on the text area and illustration area... Differences in brightness and color distribution were analyzed region by region in the preprocessed page image to accurately distinguish and locate text layout areas and illustration element areas. Then, the line outlines and shape features of illustration element areas were carefully analyzed to identify story characters and scene objects, confirming the position and basic attributes of these visual elements. Next, the character content of the text area was identified line by line, and the semantic information of the story description, character lines, etc., was analyzed by combining the connection between the text. At the same time, the color characteristics, outline shape, and spatial position relationship between different elements of the illustration elements were observed to extract visual semantic information such as the image features of the characters and the environmental features of the scene. Finally, the text semantic information and visual semantic information were aligned in dimensions, and the interrelated content was supplemented and improved to form a logically coherent information set. In the end, the visual semantic features that can comprehensively and accurately represent the core content of the current picture book page were obtained and a stable output was completed.

[0072] Step 1.3: Denoising and endpoint detection are performed on the children's speech data to obtain clean speech segments; acoustic feature analysis and speech recognition are performed on the clean speech segments to extract acoustic features from the speech spectrum and transcribe them into text information; combining the acoustic features and text information, interactive speech semantic features containing reading content, question content, and tone and emotion are analyzed and output, specifically including: environmental noise processing of the separated children's speech data, carefully removing external background noise, spatial reverberation, and subtle background noise generated by the acquisition device in the reading scene; then effective content identification is performed on the processed speech signal to distinguish the effective part containing actual speech from meaningless blank and redundant noise parts; the speech content containing only effective interactive behaviors such as children reading picture books and actively asking questions is accurately extracted to obtain clean and complete speech segments; then a comprehensive analysis of the clean speech segments is performed. Sound characteristic analysis meticulously captures core sound features such as changes in energy intensity, pitch fluctuations, and rhythmic patterns of speech at different times. Simultaneously, pure speech segments are converted into corresponding text content, and the converted text undergoes sentence organization and deviation correction to form accurate and fluent text information. Then, combining the extracted sound features with the organized text information, the original text of the picture book read by the child and the questions raised regarding the plot and characters are clearly distinguished. Furthermore, based on changes in pitch, energy intensity, and rhythm, the emotional types of tone, such as curiosity, doubt, and pleasure, are determined in the speech. Finally, the three types of information—reading content, questions, and emotional tone—are systematically integrated to form interactive speech semantic features that comprehensively reflect the child's interactive intentions and real-time emotional state during picture book reading, and this is then output in a standardized manner.

[0073] Step 1.4 involves synchronizing and aligning the touch pressure data and 3D posture data to obtain a synchronized behavior sequence. This sequence is then used to extract the page turning pressure change pattern, page flipping angle sequence, and gaze focus change trajectory. This generates and outputs semantic features representing reading rhythm and focus state. Specifically, this includes: first, performing unified timestamp calibration on the separated touch pressure data and 3D posture data. Using the initial data acquisition time as a unified benchmark, timing interpolation and timeline mapping are used to match and align the acquisition time points of both types of data one by one, eliminating discrepancies. The timing deviation caused by the inconsistent acquisition rate of the same sensor is addressed by aligning the data and then performing a simple verification of the synchronization results. This confirms that each set of touch pressure data precisely corresponds to the three-dimensional posture data at the same moment, forming a synchronized behavior sequence with complete synchronization in the time dimension and one-to-one correspondence of data frames. The synchronized behavior sequence is then decomposed frame by frame and its features are analyzed. By analyzing the trend of continuous changes in touch pressure values, different operation behaviors such as light touch to flip pages and pressing to fix the page are distinguished. The corresponding pressure peak, pressure change rate and pressure duration are extracted to obtain a flipping pressure change pattern that can fully represent the operation force and operation habits.

[0074] By calculating the spatial angle parameters in the 3D posture data frame by frame, different page operations such as forward page turning and back page review are distinguished. The tilt angle change, rotation speed and final termination angle of the entire page flipping process are recorded to obtain a coherent and complete page flipping angle sequence. By continuously tracking and connecting the spatial coordinates of the gaze points, the different dwelling states of children's gaze in the text area and the illustration character area are distinguished. The movement path of the gaze, the dwell time in each area and the core focus area are depicted to obtain a clear and detailed trajectory of gaze focus change. Finally, based on the obtained reading pressure change pattern, page flipping angle sequence and gaze focus change trajectory, the reading speed, page operation frequency and attention focus duration are comprehensively analyzed to quantitatively assess children's reading rhythm and concentration. The assessment results are integrated and standardized with behavioral characteristics to generate interactive behavioral semantic features that can objectively and comprehensively represent children's real-time reading status and complete effective output.

[0075] In a preferred embodiment of the present invention, step 2 above may include:

[0076] Step 2.1: Based on visual semantic features, identify the core characters and key text regions on the current picture book page, extract the center coordinates of the bounding box of the core characters in the page image coordinate system as candidate visual space anchor points; extract the geometric center coordinates of the key text regions as candidate text region anchor points. Specifically, this includes: accurately retrieving the output visual semantic features, which integrate the text semantic information and illustration visual semantic information of the current picture book page, and extracting the cumulative frequency of all characters in the entire picture book from the global character statistics library of the picture book. Combined with cross-validation of the text description and illustration layout of the current page, the characters with the highest cumulative frequency of occurrence and those that play a core narrative role in the current plot are selected, such as those that are highlighted in the text, are located at the visual center in the illustration, or are related to the core action, so as to accurately identify the core characters that play a decisive role in the plot; at the same time, the key text regions that carry the core plot description, key lines of the characters, or thematic sublimation content are selected, and text blocks containing core narrative words such as verbs and adjectives are given priority.

[0077] Then, a standardized pixel coordinate system is established on the image of the current picture book page. First, the blank areas at the page edges are removed, and the coordinate system is set only within the effective content area. The first pixel at the top left corner of the effective content area is used as the origin, with the X-axis pointing horizontally to the right and the Y-axis pointing vertically downwards. The coordinate unit is a single pixel, ensuring that picture book pages of different sizes can calculate coordinates using a uniform pixel scale. For the core character object, if there is partial occlusion, the largest bounding rectangle of the visible outline is used as the bounding box. Then, the coordinates of the four vertices of the rectangle are adjusted to make it fit as closely as possible to the visible outline edge of the character, and the pixel coordinates of the top left corner of the rectangle are calculated. , ) and bottom right pixel coordinates ( , The average value of the bounding box center coordinates is used to obtain the coordinates of the bounding box. And serve as candidate visual space anchor points.

[0078] The specific process for calculating the geometric center of the key text region is as follows: First, extract the pixel coordinates of all characters within the key text region, and find the leftmost, rightmost, topmost, and bottommost pixels among all characters. Using these four points as boundaries, draw the smallest enclosing rectangle that exactly contains all the characters. If the text is horizontal, first calculate the vertical center coordinates and take the average of the pixel coordinates of the upper and lower boundaries of the rectangle. Then, calculate the total horizontal spacing between characters and combine it with the pixel difference between the left and right boundaries of the rectangle to obtain the total horizontal length of the text region. Take the midpoint of the total length as the horizontal center coordinate, so that the center matches the visual center of gravity of the horizontal text. If the text is vertical, first calculate the horizontal center coordinates and take the average of the pixel coordinates of the left and right boundaries of the rectangle. Then, calculate the total vertical spacing between characters and combine it with the pixel difference between the upper and lower boundaries of the rectangle to obtain the total vertical length of the text region. Take the midpoint of the total length as the vertical center coordinate, ensuring that the center accurately corresponds to the visual center of gravity of the vertical text. Finally, determine the calculated center coordinates as the anchor points of the candidate text region, ensuring that the coordinates accurately correspond to the semantic core position of the key text.

[0079] Step 2.2: Based on interactive speech semantic features, identify emotional fluctuation frames and interrogative sentences in children's speech segments. Simultaneously, locate the center timestamp of the emotional fluctuation frame on the speech timeline as a candidate emotional time anchor point; locate the position index of the interrogative word in the text sequence as a candidate interrogative semantic anchor point. Specifically, this includes: calling the generated interactive speech semantic features, which contain complete information such as the child's reading content, question content, and tone and emotion. Based on this information, analyze the changing patterns of the child's speech signal segment by segment. Combined with tone and emotion information and context, when the short-term energy of the speech increases by more than 30% from the stable range, or the fundamental frequency fluctuation amplitude exceeds 20Hz, the segment is marked as an emotional fluctuation frame. At the same time, combined with the text content transcribed from the speech, if the segment corresponds to a question or exclamation, it is further confirmed as an emotional fluctuation frame.

[0080] On the audio timeline starting from the moment the child begins reading the current page, the start and end times of each emotional fluctuation frame are precisely recorded. The arithmetic mean of the two times is calculated to obtain the center timestamp, which is set as the candidate emotional time anchor. The timeline unit is milliseconds to ensure that the timestamp is synchronized with the page reading progress. At the same time, from the complete text information transcribed from speech, sentences containing interrogative words such as "what" and "why," and with a pause of more than 500ms after the interrogative words, are judged as valid interrogative sentences based on the intonation characteristics of the speech and identified as interrogative sentences. Then, the text sequence of each interrogative sentence is scanned character by character. Starting from the first character of the text sequence, each character is assigned a unique sequence number. The sequence number corresponding to the first character of the interrogative word is used as the position index. This index is associated with the corresponding audio timestamp to ensure that the anchor can accurately associate with the core interactive information of the speech semantics. At the same time, the interrogative sentence is semantically matched with the visual anchors of the current page. If the question content points to a core character or key text area, the anchor is retained first.

[0081] In a preferred embodiment of the present invention, step 3 above may include:

[0082] Step 3.1: Using the core character spatial coordinates and the center point of the key text region in the visual key semantic anchor points as focal points, calculate the ellipse focal length based on a preset semantic association distance threshold; based on the calculated ellipse focal length and the semantic saliency weights, generate a semantic ellipse in the visual modality subspace of the cross-modal joint feature space. Specifically, this includes: first, accurately extracting two core focal points from the visual key semantic anchor points; one is the determined core character bounding box center coordinates, i.e., the candidate visual spatial anchor point; the other is the geometric center coordinates of the key text region, i.e., the candidate text region anchor point; simultaneously, retrieving and recording the semantic saliency weight values ​​corresponding to the two anchor points from step 2.3; then... The complete boundary parameters of the effective content area of ​​the current picture book page are retrieved. After removing the invalid white space at the edge of the page, the actual carrying range of visual semantics is determined. Then, based on the actual size of the current picture book page, the distribution density of visual elements, and the coverage requirements of the core semantics, a semantic association distance threshold is preset: the basic threshold is dynamically adjusted according to the density of visual elements, with the diagonal length of the effective content area of ​​the page as the benchmark. If the page elements are dense, the density coefficient of the basic threshold T0 is 0.3 to 0.4; if the elements are sparse, the density coefficient is 0.4 to 0.5. The basic threshold T0 is calculated as the diagonal length of the effective content area of ​​the page × the density coefficient of the visual elements.

[0083] Based on this fundamental threshold, the threshold is slightly fine-tuned by incorporating the semantic salience weights of the two anchor points. The specific fine-tuning process is as follows: First, the definitions of the relevant parameters for fine-tuning are clarified, including the semantic salience weight scores of the core role anchor points. The semantic saliency weight score of anchor points in key text regions (All scores are taken from the scoring results in step 2.3, ranging from 4 to 10 points). The absolute difference between the weight scores of the two anchor points is the weight difference. A preset small-scale adjustment coefficient k is used, ranging from 0.05 to 0.1, to ensure that the adjustment range is controllable. A preset maximum weight value MaxW = 10 is used, corresponding to the highest score related to the core plot in the scoring system of step 2.3, to normalize the weight difference. The adjusted semantic association distance threshold T = the base threshold. +Base threshold × fine-tuning coefficient k × (weight difference) ÷Maximum weight value MaxW); The formula logic is: normalize the result by the ratio of the weight difference ΔW to the maximum weight value MaxW, with the result ranging from (0, 1), and then combine it with the basic threshold. The fine-tuning factor k is used to calculate a small adjustment amount. Finally, the base threshold and the fine-tuning amount are added together to obtain the final threshold T. This ensures that the threshold corresponding to the anchor point with the higher weight score is only slightly increased, with an increase of 1 / 2. The percentage is 0.5% to 10%, which not only meets the coverage requirements of the core semantics, but also avoids interference from irrelevant visual elements.

[0084] Next, the elliptical focal length is calculated based on the semantic association distance threshold T determined after fine-tuning and the semantic saliency weights of the two focal points. First, the straight-line distance between the two focal points is calculated as the basic spacing. It is then checked whether the basic spacing is within the effective range of the threshold T. If it exceeds the range, the basic spacing is redefined according to the upper limit of the threshold T. Then, the total focal length is allocated according to the weight scores of the two focal points. If the anchor point of the core role has a higher weight, 60% of the total focal length is allocated to its corresponding focal length segment. If the anchor point of the key text area has a higher weight, the allocation ratio is adjusted accordingly. If the weights are equal, they are allocated in an equal proportion to ensure that the focal length allocation is completely matched with the semantic importance of the anchor points.

[0085] Finally, a semantic ellipse is generated in the visual modality subspace of the cross-modal joint feature space. The two determined foci are used as the bifoci of the ellipse. The basic shape of the ellipse is determined by combining the calculated focal length. Then, by statistically analyzing the distribution density of visual semantic features around the two foci, the lengths of the major and minor axes of the ellipse are adjusted. The axis length is appropriately extended in the direction where the core semantic elements are more densely distributed, and appropriately shortened in the direction where the distribution is sparse. At the same time, the interference of irrelevant elements such as non-core decorative patterns and corner patterns on the page is eliminated. The final semantic ellipse can completely wrap the core characters, key texts and related visual elements around them. All areas inside the ellipse are the effective association range of the visual semantics of the picture book.

[0086] Step 3.2 involves weighting the core emotion frame timestamp anchors and question word position anchors in the key semantic anchors of the interactive speech, and using the calculation result as the center. The perceptual radius is then calculated using the acoustic feature intensity and emotional confidence in the semantic features of the interactive speech. Based on the center and perceptual radius, a semantic perceptual circle is generated in the speech modality subspace of the cross-modal joint feature space. Specifically, this includes: first, extracting the core emotion frame timestamps and question word positions from the key semantic anchors of the interactive speech; second, verifying whether the core emotion frame timestamps and question word positions belong to the same continuous valid interactive speech segment, and removing invalid numbers with temporal mismatches. Then, coordinate transformation and fusion are performed on the two to determine the center of the semantic perception circle. The core emotion frame timestamp is converted into the temporal coordinates of the speech modality subspace, marked as G. The time when the child starts reading the current page is taken as the origin of the coordinates, and the time unit is set to milliseconds. The value of the timestamp is directly mapped to the corresponding coordinate value of the temporal axis. The question word position index is combined with the character temporal distribution of the speech text and converted into the semantic axis coordinates, marked as H. The first character of the speech-to-text corresponds to the temporal origin. Each character is assigned a unique temporal interval according to the actual speech playback duration. The midpoint of the temporal interval corresponding to the first character of the question word is the coordinate value of the semantic axis.

[0087] Then, fusion weights are assigned based on the child's current voice interaction state, and the final precise center coordinates are obtained through weighted fusion calculation. The specific process is as follows: First, the relevant parameters for weighted fusion are defined, and the fusion weights are denoted as follows: (Weight of the temporal coordinate G of the emotion frame) (Weight of the semantic axis coordinate H of the interrogative word), the final center coordinate is C; the weighted fusion formula is: C=G× +H× Formula application logic explanation: If the child is in a clear questioning interaction state, that is, simultaneously including standard question sentence structure and obvious emotional fluctuation characteristics, then =40% =60%, substituting into the formula, we get C=G×40%+B×60%; if the interaction is dominated by emotional fluctuations, that is, there are no explicit interrogative sentences but the intensity of emotional fluctuations is high, then =60% =40%, substituting into the formula, we get C=G×60%+H×40%.

[0088] Next, the perceptual radius of the semantic perception circle is calculated. First, the acoustic feature intensity in the interactive speech semantic features is quantized and graded in three levels. At the same time, the emotional confidence value output during the speech semantic recognition process is extracted. Then, the radius is finely adjusted by combining the actual speech rate characteristics. If the speech rate is fast, the radius is slightly reduced; if the speech rate is slow, the radius is slightly expanded. When the acoustic feature is at high intensity and the emotional confidence is higher than the preset standard value, the radius is adjusted upwards based on the set basic time sequence length. When the acoustic feature is at medium intensity and the emotional confidence is in the medium range, the basic radius is used directly. When the acoustic feature is at low intensity or the emotional confidence is lower than the preset value, the radius is adjusted upwards. When using the standard value, the base radius is appropriately lowered to ensure that the size of the perception radius is fully adapted to the directional strength of the semantics of children's speech interaction. Finally, in the speech modality subspace of the cross-modal joint feature space, a semantic perception circle is drawn with the determined center and the calculated perception radius. During the drawing process, invalid areas corresponding to blank silent segments and irrelevant noise segments in the speech are removed. At the same time, the covered speech temporal segments and corresponding core semantic content are marked inside the circle. The entire area covered by the perception circle is the effective perception range of children's speech interaction semantics, which can accurately define the speech semantic interval directly related to children's real-time interaction intentions.

[0089] Step 3.3: Map the semantic ellipse and semantic perception circle to the same cross-modal joint feature space coordinate system to form a dynamic semantic geometric relationship diagram. The semantic ellipse represents the distribution range of visual semantics in the picture book, and the semantic perception circle represents the perceptual range of children's speech interaction semantics. The semantic ellipse and semantic perception circle jointly construct the geometric constraint framework for cross-modal semantic association. Specifically, this includes: first, constructing a unified cross-modal joint feature space coordinate system. This coordinate system adopts a three-dimensional design, where the X and Y axes correspond to the spatial dimension of the visual modality, maintaining the same proportional relationship as the page pixel coordinate system established in Step 2.1; the Z axis corresponds to the temporal dimension of the speech modality, with the time unit set to milliseconds, fully compatible with the speech temporal coordinates in Step 3.2; and using the coordinates of the upper left corner vertex of the current picture book page's effective content area in visual space as the origin of the entire coordinate system. Simultaneously, preset the calibration coefficients for the coordinate mapping. The specific refinement process is as follows: First, clarify the classification and definition of the calibration coefficients, which are divided into visual dimension calibration coefficients. (Adapted visual spatial coordinate mapping for X and Y axes) and voice dimension calibration coefficients (Adapted to the Z-axis audio time-series coordinate mapping), both types of coefficients are initially set to 1.0, representing the unbiased basic state; the adjustment is based on two core influencing factors: one is the size ratio of the picture book page, that is, the width-to-height ratio of the effective content area of ​​the page, and the other is the average audio playback rate of the child reading the current picture book.

[0090] Calibration coefficients for visual dimensions If the aspect ratio of the picture book page deviates from the standard ratio (the default standard ratio is 3:4, which is suitable for common picture book sizes), the page will be adjusted according to the degree of deviation. For example, when the aspect ratio is greater than 3:4 (the page is too wide), Fine-tune to 0.95 to 0.98 to reduce the X-axis mapping scale; when the aspect ratio is less than 3:4 (page is too tall), Fine-tuning was performed from 1.02 to 1.05, increasing the mapping scale along the X-axis to ensure a balanced proportion of visual content of different sizes in the joint coordinate system; calibration coefficients were applied to the speech dimension. By statistically analyzing the average playback rate of the audio on the child's current page in real time and comparing it with the preset standard speaking speed (set to 3 to 4 characters per second), if the actual speaking speed is higher than the standard speaking speed, then... Fine-tuning to 0.9 to 0.95 compresses the mapping range of the Z-axis time coordinates; if the actual speech rate is lower than the standard speech rate, then... The value was fine-tuned to 1.05 to 1.1 to expand the mapping range of the Z-axis temporal coordinates and achieve scale adaptation between speech temporal sequence and visual space. Through the above targeted adjustments, it is ensured that the two types of calibration coefficients can accurately compensate for the mapping deviation caused by different picture book sizes and different reading speeds, thus ensuring the accuracy of subsequent coordinate transformation.

[0091] Subsequently, coordinate transformation and bidirectional mapping operations are performed. When mapping the generated semantic ellipse from the visual modality subspace to the joint coordinate system, its spatial coordinates on the X and Y axes are fully preserved. The Z-axis coordinate is set to the average time series value of children's reading on the current page, which is obtained by statistically analyzing the reading time of similar picture book pages in history, and combined with the visual dimension calibration coefficient. The coordinate values ​​are corrected; when mapping the generated semantically aware circle from the speech modality subspace to the joint coordinate system, its temporal coordinates of the Z-axis are fully preserved. The X-axis and Y-axis coordinates are determined according to the visual content association relationship corresponding to the speech semantics. If the question content points to the core role, the spatial coordinates of that role are matched; if the question content points to key text, the center coordinates of that text region are matched. The speech dimension calibration coefficient is also combined. The coordinate values ​​are corrected, and slight coordinate deviations are corrected through linear interpolation during the transformation process. At the same time, the semantic tags of the two geometric figures are bound synchronously to ensure that the geometric positions correspond one-to-one with the corresponding semantic cores and that no semantic association attributes are lost.

[0092] Next, based on the spatial relationship between the mapped semantic ellipse and the semantic perception circle, a dynamic semantic geometric relationship graph is constructed. The graph accurately marks the center coordinates of the two figures, the lengths of the major and minor axes of the ellipse, and the radius of the perception circle. Simultaneously, it calculates and marks the center distance, the area of ​​the overlapping region, and the specific semantic content corresponding to the overlapping region. It also records the positional change parameters of the figures in real time, reserving a data interface for subsequent dynamic updates of semantic association. The semantic ellipse fully and accurately represents the core distribution range of the picture book's visual semantics, while the semantic perception circle fully and accurately represents the effective perception range of children's speech interaction semantics. Together, they construct a geometric constraint framework for cross-modal semantic association within a unified coordinate system. Within this framework, the strength of the association between visual and speech semantics is clearly defined by the proportion of the overlapping area. The center distance assists in judging the consistency of semantic direction, and further clarifies the subsequent processing logic guidance corresponding to different association strengths. This provides clear geometric constraints and quantitative judgment standards for subsequent accurate cross-modal semantic matching and the recognition of interaction intentions.

[0093] In a preferred embodiment of the present invention, step 4 above may include:

[0094] Step 4.1: Under the same coordinate system mapped by the dynamic semantic geometric relationship graph, obtain the focal position, major and minor axis lengths of the semantic ellipse, and the center coordinates and perceptual radius of the semantic perception circle. Specifically, this includes: reconfirming and locking the unique cross-modal joint feature space coordinate system corresponding to the constructed dynamic semantic geometric relationship graph, and checking the origin position of the coordinate system (the top left corner vertex of the effective content area of ​​the current picture book page), the XY axis visual space scale ratio (consistent with the pixel coordinate system in Step 2.1), the Z axis speech temporal scale ratio (milliseconds / unit scale), and the previously set visual dimension calibration coefficient. Voice dimension calibration coefficient To ensure all coordinate system parameters are completely consistent with the previous mapping and transformation standards, with no parameter drift, the full set of core geometric parameters for the semantic ellipse and semantic-aware circle are then retrieved. For the semantic ellipse, the precise coordinates of the two calibrated foci are extracted sequentially, denoted as... , The Z-axis represents the average reading time of the current page, the actual length of the major axis L, and the actual length of the minor axis S. At the same time, the azimuth angle α between the major axis and the X-axis of the coordinate system is extracted, with the accuracy retained to 0.1 degrees, and is used to determine the orientation of the elliptical space.

[0095] For semantically perceptual circles, extract the passing through circles sequentially. The final coordinates of the calibrated center C The perceptual radius R, after dual calibration of speech rate and emotional confidence, is calculated. Simultaneously, the acoustic feature intensity grading records (high, medium, and low) from the radius calculation process, the original emotional confidence values ​​(range 0 to 1), and the basic temporal length parameters are retrieved. After parameter retrieval, a three-level integrity and validity check is performed: the first level is format verification, checking whether the numerical type of all parameters (integer or floating-point) and the three-dimensional coordinate dimensions conform to preset standards; the second level is range verification, comparing each parameter with a reasonable range, such as ensuring the radius R is greater than 0 and the major axis L is greater than the minor axis S, removing abnormal parameters that deviate from the range by more than ±10%; the third level... Level 3 is the association verification, which verifies the mapping relationship between the focal coordinates of the semantic ellipse and the original anchor point coordinates in step 2.1, and the consistency between the center coordinates of the semantic perception circle and the weighted fusion result in step 3.2, ensuring that the source of the parameters is traceable. If a single parameter is missing, it is supplemented by reverse deduction through the associated geometric relationship. For example, if one focus, the length of the major axis and the azimuth angle of the ellipse are known, the coordinates of the other focus can be deduced. If there are more than two abnormal parameters, the geometric figure generation process in steps 3.1 to 3.3 is re-triggered, ultimately ensuring that all parameters participating in the subsequent calculations are real, valid and accurately match the current dynamic semantic geometric relationship graph.

[0096] Step 4.2: Based on the obtained center coordinates and focal positions, calculate the relative positional relationship between the semantically perceived circle and the semantic ellipse; according to the relative positional relationship, determine whether the center of the semantically perceived circle is located inside or on the boundary of the semantic ellipse, and calculate the corresponding first geometric inclusion value, specifically including: based on the verified center coordinates C of the semantically perceived circle... Coordinates of the two foci of the semantic ellipse , First, calculate the straight-line distance from the center of the circle to the two foci using the formula. (C to) ), d2 (C to During the calculation, six decimal places are retained to ensure precision, and intermediate values ​​such as coordinate differences and sums of squares are recorded for later reference; subsequently... and The sum of the distances is D = + , considering the tiny errors existing in the coordinate calculation process, a determination tolerance Δ of ±0.5 pixel visual dimension and ±1 millisecond timing dimension is preset; the total distance D is accurately compared with the length L of the major axis of the semantic ellipse, and the relative position relationship is determined according to the following rules: if D < L - Δ, it is determined that the center of the circle is in the internal area of the semantic ellipse; if L - Δ ≤ D ≤ L + Δ, it is determined that the center of the circle is in the boundary area of the semantic ellipse, and it is considered that the boundary fits within the tolerance range; if D > L + Δ, it is determined that the center of the circle is in the external area of the semantic ellipse; after the position determination is completed, a first geometric inclusion degree value is generated according to the preset grading assignment standard, and the value range is from 0 to 1, with 2 decimal places reserved: when the center of the circle is inside the ellipse, the assignment is 1.0, representing that the core semantics of speech and the core semantics of vision achieve a deep position fit; when it is in the boundary area, the assignment is 0.8, representing that the two achieve a critical position fit; when it is in the external area, the assignment is 0.0, representing that there is no position association between the two; after the assignment is completed, a determination report is generated synchronously, recording 、 、the specific values of D, L, Δ and the final determination result, ensuring that the generation process of the first geometric inclusion degree value is traceable and verifiable throughout.

[0097] Step 4.3, calculate the intersection area between the semantic perception circle and the semantic ellipse, and divide the intersection area by the area of the semantic perception circle to obtain the second area overlap rate value, specifically including: first, based on the obtained full set of geometric parameters, perform high-precision contour fitting in the unified cross-modal joint feature space coordinate system: for the semantic ellipse, according to the foci 、 、major axis L, minor axis S and azimuth angle α, use the piecewise fitting method to generate a smooth ellipse contour, taking a sampling point every 1°, ensuring the contour accuracy; for the semantic perception circle, generate a circular contour according to the center C and radius R, and the sampling point density is the same as that of the ellipse to ensure the contour matching degree; then use the contour intersection point-by-point detection algorithm to locate the intersection area between the two: based on the sampling points of the ellipse contour, judge point by point whether the point is inside the circular contour by judging whether the distance from the point to the center C is less than or equal to R, and at the same time, based on the sampling points of the circular contour, verify in the reverse direction whether it is inside the ellipse by judging whether the sum of the distances from the point to F1 and F2 is less than or equal to L, and finally screen out all the intersection points and connect them in sequence to form a closed intersection area.

[0098] If the intersection area is a regular figure, such as a symmetric figure enclosed by an elliptical arc and a circular arc, directly use the corresponding combined geometric area formula to calculate the area; if it is an irregular figure, use the minimum bounding rectangle segmentation method to construct a bounding rectangle with the maximum X, Y, Z range of the intersection area, and divide the irregular area into several triangular and rectangular sub-regions along the contour tangent direction, calculate the area of each sub-region respectively, the area of the triangle is calculated as base × height ÷ 2, and the area of the rectangle is calculated as length × width, and the total area of the intersection area is obtained after accumulation. The precision is retained to two decimal places; then, the full area is calculated based on the semantically aware circle radius R. =πR 2 π is taken as 3.1416, and... and The ratio is calculated to obtain the second area overlap rate value. = / Round to two decimal places; if the two figures have no intersection points, i.e. =0, then assign a value directly. =0; if the circle is completely inside the ellipse, that is... = Then assign a value =1.0, this value directly reflects the degree of overlap and fit between the semantic perception range of children's voice interaction and the distribution range of visual semantics in picture books.

[0099] Step 4.4 involves weighting and synthesizing the first geometric inclusion value and the second area overlap rate value, and combining this with the visual semantic distribution reliability represented by the semantic ellipse to generate the cross-modal semantic fusion adjustment coefficient. Specifically, this includes: determining a weighting scheme based on 1000 sets of historical experimental data from three typical scenarios of cross-modal interaction in children's picture books (question-based interaction, emotional feedback interaction, and flat reading interaction): since geometric position fit directly determines whether the semantic core is related, it has a higher priority than range overlap, therefore, the weighting of the first geometric inclusion value is set. =0.6, the weighted average of the second area overlap rate values. =0.4, this weight allocation has been experimentally verified to improve the accuracy of semantic association determination by 15% to 20%; subsequently, the basic fusion coefficient = first geometric inclusion value × +Second area overlap rate value × The rules are used to complete the weighted synthesis, and four decimal places are retained during the calculation to avoid loss of precision. Then, the confidence score Conf generated for the current picture book page is retrieved. This confidence score is a weighted synthesis of three core indicators: the accuracy of core character recognition (weight 0.5), the completeness of key text region extraction (weight 0.3), and the effectiveness of visual semantic segmentation (weight 0.2). Each indicator has been verified and calibrated multiple times. Finally, the value of Conf is stable between 0 and 1, that is, Conf = accuracy of core character recognition × 0.5 + completeness of key text region extraction × 0.3 + effectiveness of visual semantic segmentation × 0.2.

[0100] Next, the basic fusion coefficient is multiplied by Conf to obtain the initial fusion adjustment coefficient. Then, the min-max normalization method is used to calibrate the initial coefficient to ensure that the final cross-modal semantic fusion adjustment coefficient K is always within a reasonable range of 0 to 1, i.e., K = min(max(basic fusion coefficient × Conf, 0), 1). Specifically, the calibration operation rule is as follows: First, calculate the larger value between the initial fusion coefficient and the number 0. This step is used to remove abnormally negative values ​​less than 0 in the initial fusion coefficient, ensuring that the minimum value of the coefficient is not lower than 0. Then, calculate the smaller value between the result obtained in the previous step and the number 1. This step is used to remove abnormally large values ​​greater than 1 in the initial fusion coefficient, ensuring that the maximum value of the coefficient is not higher than 1. The above two layers of constraint operations are integrated into a clear operation formula, that is, the final cross-modal semantic fusion adjustment coefficient K is equal to the maximum value of the initial fusion coefficient and 0, and then the minimum value of the maximum value and 1.

[0101] After the coefficients are generated, an additional layer of validity verification is added: if K ≥ 0.8, it is determined to be a strong semantic association; if 0.3 ≤ K < 0.8, it is determined to be a medium semantic association; if K < 0.3, it is determined to be a weak semantic association. The association level is output synchronously with the coefficient K. The higher the final K value, the stronger the integration and adaptation between children's voice interaction semantics and picture book visual semantics, and the higher the credibility of the association. This coefficient will directly serve as the core adjustment parameter for subsequent cross-modal semantic accurate alignment and intelligent analysis of children's reading interaction intentions, providing quantitative support for the semantic processing accuracy of the entire picture book interaction system.

[0102] In a preferred embodiment of the present invention, step 5 above may include:

[0103] Step 5.1: Obtain the cross-modal semantic fusion adjustment coefficients and simultaneously read visual semantic features, interactive voice semantic features, and interactive behavior semantic features. Specifically, this includes: first, retrieving the output cross-modal semantic fusion adjustment coefficients K and their corresponding semantic association levels (strong, medium, weak association), and simultaneously verifying whether the value of coefficient K is within a reasonable range of 0 to 1. If any anomalies are found, return to step 4.4 to regenerate. Then, selectively read the three types of core semantic features: for visual semantic features, using the current picture book page as the scope, extract the previously analyzed core character outline features, key text semantic codes, and spatial distribution relationships of visual elements, ensuring that the extraction scope is strictly limited to... For the current page, avoid cross-page semantic interference; for the semantic features of interactive voice, associate with the reading time period of the current picture book page, and extract the acoustic feature sequence, emotion type label, and speech-to-text semantic vector corresponding to the child's interactive voice during that time period; for the semantic features of interactive behavior, first summarize the raw data of children's behavior collected in real time during the reading process on the current page, and then transform the raw data into semantic features including gaze focus coordinates, page dwell time, and hand page-turning action sequence; after completing the reading of all features, conduct feature integrity verification to ensure that the core parameters of each type of feature are not missing and the data format is consistent, providing a reliable data foundation for subsequent weighted fusion.

[0104] Step 5.2: Input the cross-modal semantic fusion adjustment coefficient into the set weight mapping function to calculate the visual weight of visual semantic features, the speech weight of interactive speech semantic features, and the behavioral weight of interactive behavior semantic features. Specifically, this includes: combining the semantic fusion rules and association level adaptation requirements of multimodal interaction in children's picture books, pre-setting a set of continuously differentiable weight mapping functions. This function takes the cross-modal semantic fusion adjustment coefficient K as the only input variable and outputs the visual weights corresponding to the visual semantic features. Speech weights corresponding to interactive speech semantic features Behavior weights corresponding to semantic features of interactive behaviors And always satisfy + + The normalization constraint of =1, the specific weight mapping function expression is divided into two parts: the basic mapping formula and the hierarchical correction formula. The basic mapping formula is: =0.5×K+0.2, =0.4×K+0.2, =1- - The graded correction formula is as follows: when K ≥ 0.8, = +0.05, = +0.05, = -0.1; when 0.3≤K<0.8, the calculation result of the basic mapping formula takes effect directly without additional correction; when K<0.3, = -0.05, = -0.05, = +0.1, where K takes a fixed value between 0 and 1; Used to characterize the proportion of visual semantics in the multimodal fusion process; Used to characterize the proportion of contribution of speech semantics in the multimodal fusion process; Used to characterize the proportion of contribution of behavioral semantics in the multimodal fusion process.

[0105] Substituting the verified cross-modal semantic fusion adjustment coefficient K into the aforementioned weight mapping function, the initial values ​​of the three types of weights are first calculated using the basic mapping formula. Then, based on the semantic association level matching corresponding to K, the corresponding hierarchical correction operation is performed to obtain the preliminary visual weight, speech weight, and behavioral weight. Subsequently, the sum of the three types of weights is precisely calculated. If the sum of the values ​​deviates slightly from 1, normalization correction is performed by scaling proportionally to ensure that the sum of the three types of weights is strictly equal to 1. This ensures that the weight allocation conforms to the normalization specification of multimodal feature fusion and can adapt to the feature contribution allocation logic under different semantic association strengths.

[0106] Step 5.3: Based on the visual weight, speech weight, and behavior weight, perform weighted summation on the corresponding visual semantic features, interactive speech semantic features, and interactive behavior semantic features to obtain a preliminary fusion feature vector. Specifically, this includes: performing dimension alignment preprocessing on the visual semantic features, interactive speech semantic features, and interactive behavior semantic features; mapping the three types of features to a high-dimensional semantic space with N dimensions (a preset fixed positive integer) using a unified feature encoding conversion rule, resulting in dimension-aligned visual semantic feature vector V, dimension-aligned interactive speech semantic feature vector A, and dimension-aligned interactive behavior semantic feature vector B, avoiding fusion bias caused by dimensional differences; subsequently, feature fusion is completed through weighted summation. The core calculation expression is: Preliminary fusion feature vector F = V × +A× +B× The meanings of each character are as follows: F is the initial fused feature vector, and V, A, and B are the visual, speech, and behavioral semantic feature vectors after dimension alignment, respectively. , , The corresponding weights are obtained by multiplying V, A, and B by their respective weights to obtain weighted feature vectors V', A', and B'. Then, the values ​​of the three corresponding dimensions are summed element by element, i.e., the value of the i-th dimension of F = the value of the i-th dimension of V' + the value of the i-th dimension of A' + the value of the i-th dimension of B', where i = 1, 2, ..., N. The resulting F fully integrates the three types of semantic core information and accurately reflects the contribution of different semantics through the weights.

[0107] Step 5.4 involves normalizing and reducing the dimensionality of the initial fused feature vector to output a multimodal joint semantic representation that is precisely bound to the semantic content of the current picture book page. Specifically, this includes: normalizing the initial fused feature vector by using a min-max normalization method to compress the values ​​of each dimension of the vector to the range of 0 to 1, eliminating dimensional differences between different feature dimensions, and ensuring a balanced contribution of each dimension to the final representation; subsequently, dimensionality reduction is performed on the normalized fused feature vector by using principal component analysis (PCA) to extract the core principal components in the vector, retaining those that can explain more than 90% of the data. The principal component dimension is reduced to eliminate redundant feature information and reduce the computational complexity of subsequent semantic processing. After dimensionality reduction, the final multimodal joint semantic representation is generated. This representation is accurately associated with the core semantic content of the current picture book page through the semantic binding module. The binding information includes page number, core characters, key plot points, etc. Finally, the validity of the output joint semantic representation is verified by comparing the matching degree between the representation and the core semantics of the page. If the matching degree is ≥85%, it is directly output; otherwise, it returns to step 5.3 for weighted fusion again to ensure that the output representation can accurately reflect the semantic content of the current picture book page and the child's interactive intention.

[0108] In a preferred embodiment of the present invention, step 6 above may include:

[0109] Step 6.1: Obtain the multimodal joint semantic representation corresponding to the current picture book page. This specifically includes: first, locking the picture book page currently being read and interacted with, obtaining the unique identifier information of the page, such as the page number and core plot tags; then, retrieving the multimodal joint semantic representation precisely bound to the current page based on the identifier information; immediately after retrieval, verifying the validity of the representation, checking whether the dimensions of the representation meet the set standards, whether it carries the core semantic binding information of the current page, such as core characters and key plots, and comparing the preliminary matching degree between the representation and the semantic content of the current page. The preset matching degree threshold is 0.8, which is calibrated based on historical multimodal representation matching data and used to define the core fit between the representation and the semantic core of the page; if the matching degree is lower than 0.8 or the representation has missing dimensions, return to step 5.4 to regenerate, ensuring that the finally obtained multimodal joint semantic representation can accurately correspond to the semantic core and children's interaction information of the current picture book page.

[0110] Step 6.2: Input the multimodal joint semantic representation into the pre-trained intention and cognition joint reasoning model to extract multimodal semantic association patterns from the multimodal joint semantic representation, and perform pattern matching and reasoning based on the knowledge graph of the picture book plot to output a multimodal semantic matching and reasoning result. Specifically, this includes: first, confirming the core architecture and construction logic of the pre-trained intention and cognition joint reasoning model. This model is derived from the Transformer architecture and improved based on the special characteristics of cross-modal interaction scenarios in children's picture books. It adds a multimodal semantic alignment layer and a picture book knowledge graph interaction layer. Its core improvement lies in strengthening the association mining ability of visual, speech, and behavioral semantics, while improving the adaptability with picture book plot knowledge. The specific model construction process is as follows: First, the input layer is designed with a three-dimensional input interface, specifically adapted to the multimodal joint semantic representation output in step 5.4. It supports receiving fused feature vectors of fixed dimensions and reserves an interface for subsequent access to structured data from the picture book plot knowledge graph. Second, the multimodal semantic alignment layer is constructed by calculating the association weights of each dimension within the representation through a self-attention mechanism. This accurately extracts cross-modal semantic association patterns between vision and speech, speech and behavior, and vision and behavior. For example, it identifies the directional association between speech question semantics and the coordinates of the core visual character, and the focusing association between behavioral dwell time and key text regions. Third, the picture book knowledge graph interaction layer is constructed by transforming the pre-built picture book plot knowledge graph, including all plot nodes, character relationships, key knowledge points, and semantic association rules, into a structured tensor. Through a cross-attention mechanism, it achieves semantic matching between multimodal semantic association patterns and the knowledge graph, compensating for the semantic bias of pure data-driven reasoning. Fourth, the reasoning output layer is constructed using a dual-branch structure. One branch outputs the multimodal semantic matching degree and association pattern type, while the other outputs the reasoning basis, ensuring the traceability of the results.

[0111] The model training process closely aligns with the children's picture book reading interaction scenario of this invention: The first step involves dataset construction, collecting real reading interaction data from 500 children's picture books on different themes, including fairy tales, science books, and fables, covering the reading behavior of children aged 3 to 8. The data content includes multimodal joint semantic representations of each page, manually labeled multimodal semantic association patterns (e.g., association between questioning voices and character visuals, association between pause behavior and key plot points), and node labels corresponding to the picture book plot knowledge graph. Simultaneously, interactive intentions such as plot questions, character cognition, and emotional feedback, as well as cognitive states such as focused or distracted attention, and sufficient or insufficient plot comprehension are labeled as supervisory tags. The second step involves data preprocessing, cleaning the collected dataset, removing abnormal data generated by unintentional child interactions such as accidental touches or invalid noise, normalizing the multimodal joint semantic representations to ensure a consistent data format, and then dividing it into training, validation, and test sets in a 7:2:1 ratio.

[0112] The third step, the pre-training stage, employs a strategy of pre-training with general picture book data and fine-tuning with specific scenarios. First, large-scale general children's picture book semantic data is used for pre-training, allowing the model to initially grasp the basic association rules between visual elements and textual semantics in picture books. Then, a constructed specific interactive dataset is used for fine-tuning, focusing on optimizing cross-modal semantic association extraction and plot knowledge matching capabilities. The fourth step, training optimization, uses the cross-entropy loss function to calculate the error between the model's output and manually labeled results. The Adam optimizer iteratively updates the model parameters. After each round of training, a validation set is used to evaluate model performance. The core evaluation metrics are semantic association pattern recognition accuracy and inference result matching degree. When the validation set metrics show no improvement for five consecutive rounds, an early stopping strategy is used to terminate training. Simultaneously, to address performance shortcomings in different interactive scenarios, such as questions from young children and distracted reading, the attention weights of the picture book knowledge graph interaction layer are specifically adjusted. Finally, a pre-trained model that meets the standards for generalization ability and scenario adaptability is obtained.

[0113] After inputting the validated multimodal joint semantic representation into the pre-trained model, the model first mines core association patterns through a multimodal semantic alignment layer. For example, in the application scenario where a child asks "Where is the bear?", the model can extract a strong association between the semantics of the voice question and the visual core character, i.e., the outline features of the bear, from the representation. At the same time, it captures the behavioral association of the gaze focusing coordinate pointing to the bear area, integrating them to form a composite association pattern of voice question, visual character, and behavioral focus. Then, it calls the picture book plot knowledge graph and uses the knowledge graph interaction layer to accurately compare the extracted composite association pattern with the plot nodes corresponding to the current page, such as the character position nodes in the bear's adventure plot. It combines the semantic association rules of characters, plot, and spatial position in the graph to complete the reasoning logic and eliminate invalid association patterns of voice question and non-core characters, i.e., grass. Finally, it outputs a multimodal semantic matching and reasoning result containing three core parts: first, the semantic matching degree, i.e., the degree of fit between the representation association pattern and the knowledge graph node; second, the association pattern type, such as character-oriented association and plot question association; and third, the reasoning basis, i.e., the specific source of the matched knowledge graph node and association features.

[0114] The advantages and benefits of adopting this improved Transformer model are reflected in three aspects: First, the self-attention mechanism of the Transformer architecture can efficiently capture long-distance associations of multimodal semantics, adapting to temporal association scenarios in children's reading, such as lingering voice questions behind visual observations and synchronizing behavioral pauses with plot comprehension; Second, the added multimodal semantic alignment layer solves the problem of misalignment of different modal semantics in traditional models, improving the accuracy of cross-modal association extraction, such as avoiding misassociating emotional voices with non-corresponding visual elements; Third, the picture book knowledge graph interaction layer introduces prior picture book plot knowledge, significantly reducing the model's dependence on sparse interaction data, especially suitable for scenarios with less interaction data for young children, while improving the rationality and interpretability of inference results and avoiding erroneous inferences unrelated to the picture book plot.

[0115] Step 6.3: Based on the multimodal semantic matching and reasoning results, analyze and output the interaction intent determination results and cognitive state assessment results. Specifically, this includes: analyzing the output multimodal semantic matching and reasoning results, extracting the core association information and matching degree data, and determining the interaction intent type in conjunction with typical interactive scenarios of children's picture book reading. Common types include plot questioning intent, character cognition intent, emotional feedback intent, and bland reading intent. If there is a clear question semantic association pattern in the reasoning results and the matching degree is high, it is determined to be a questioning intent; if there is a strong semantic association of the core character and an accompanying observation behavior association, it is determined to be a character cognition intent. At the same time, based on the semantic understanding depth and feature association completeness data in the reasoning results, assess the child's cognitive state from three dimensions: attention concentration, plot comprehension, and knowledge point cognition. For example, if the semantic matching degree of key plots is high and the behavior is focused, it is assessed as focused attention and clear plot comprehension; if the semantic association is loose and there are multiple matching deviations, it is assessed as scattered attention and insufficient plot comprehension. Finally, output the structured interaction intent determination results and cognitive state assessment results, along with the core basis for the determination and assessment.

[0116] Step 6.4 involves temporally fusing the multimodal joint semantic representation, interaction intent determination results, and cognitive state assessment results with the real-time stored historical reading session context representation. This temporal fusion updates and generates a coherent reading session context representation for the current moment. Specifically, this includes: first, retrieving the real-time stored historical reading session context representation, which integrates the multimodal joint semantic representation of past pages within the current reading session, historical interaction intent determination results, historical cognitive state assessment results, and complete interaction temporal information; then, performing temporal weighted fusion: first, establishing temporal weight allocation rules, using the order of interaction as the core basis, dividing the page into three temporal levels: current page, recent historical pages, and earlier historical pages, assigning higher weights to recent information, and ensuring all weights satisfy the normalization constraint, i.e., the sum of the weights is 1; defining core parameters: the weight of the current page information is... The top 3 most recent historical pages, ranked from newest to oldest, have the following weights: , , The total weight of the remaining early historical pages is The current page information set is denoted as The recent three pages of information are denoted as follows: , , The early page summary information collection is denoted as The new contextual representation after fusion is denoted as U.

[0117] The time-weighted fusion formula, expressed in words and mathematical symbols, is: U= × + × + × + × + × The optimal weight value verified by experiments is: =0.4, =0.25, =0.15, =0.1, =0.1, which highlights the dominant role of the latest interactive information while retaining the reference value of key historical semantics; during fusion, each information set is first multiplied by its corresponding weight, and then accumulated and integrated according to semantic dimensions to complete the temporal fusion; after that, the fusion result is checked for coherence, and the semantic evolution logic between the current information and the historical information is checked to see if it is smooth. If there is a logical conflict, it is corrected by backtracking with historical interactive data. Finally, a coherent reading conversation context representation at the current moment is generated and stored back to the conversation storage module in real time, providing complete context support for the interactive reasoning of subsequent pages.

[0118] In a preferred embodiment of the present invention, step 7 above may include:

[0119] Step 7.1: Obtain the current reading conversation context representation and interaction intent determination result. Specifically, this includes: locating the currently ongoing children's reading conversation and retrieving the updated current reading conversation context representation, while simultaneously extracting the associated interaction intent determination result from the representation; immediately after retrieval, perform information integrity verification to check whether the context representation contains a complete historical interaction sequence, multimodal semantic association information, and cognitive state evolution trajectory, and confirm whether the interaction intent determination result clearly marks the intent type and core basis. If there is missing or ambiguous information, return to step 6.4 to regenerate the context representation, ensuring that the two types of information obtained are accurate, complete, and logically coherent, providing a reliable foundation for subsequent plot prediction and response matching.

[0120] Step 7.2, based on the current reading session context representation and combined with the picture book plot knowledge graph, predict the current reading process's position in the story's narrative and the subsequent plot development, generating a plot adaptation strategy. Specifically, this includes: semantically matching the current reading session context representation with the pre-constructed picture book plot knowledge graph; locating the current reading page's specific position within the entire picture book's story narrative through plot nodes, character relationships, and narrative association rules in the graph; confirming whether the current stage is the beginning, development, climax, or ending of the story; simultaneously, combining the historical reading trajectory and children's interactive intentions in the context representation to predict possible subsequent plot developments. For example, if the current stage is development and the child repeatedly focuses on the core character's adventurous behavior, it can be predicted that subsequent plot developments are likely to revolve around that character's adventurous challenges; based on the positioning... The strategy for generating plot adaptation based on the predicted outcome and direction includes three core components: First, the extraction of core knowledge points, which extracts knowledge points such as character relationships, plot causality, and key concepts from the current plot unit that are compatible with children's cognition; second, the direction of subsequent plot guidance, which determines whether to guide children to focus on character growth, plot logic, or knowledge point application, while pre-setting 2 to 3 flexible guidance nodes to adapt to children's subsequent interactions; and third, the grading of plot interpretation depth, which divides the interpretation depth into basic cognitive level, deep understanding level, and extended level based on the cognitive state assessment results in the context representation. The basic level only explains the surface content of the plot, the deep level analyzes the plot logic, and the extended level connects with similar picture book plots or real-life scenarios, ensuring that the strategy strictly adheres to the original logic of the picture book story and accurately matches children's reading progress, interaction needs, and comprehension abilities.

[0121] Step 7.3: Based on the interaction intent determination result and the plot adaptation strategy, match and retrieve the most suitable response template from the preset multimodal response template library. Specifically, this includes: first, determining the classification logic of the preset multimodal response template library. The preset multimodal response template library is classified in two layers according to the interaction intent type and the plot stage of the picture book, covering multiple major template categories such as plot question and answer, character cognition guidance, emotional feedback and empathy, and plot transition and connection. Each major category is further subdivided into sub-templates for different plot stages, and the template language style is adapted to the comprehension level of children aged 3 to 8. Combining the obtained interaction intent determination result and the generated plot adaptation strategy, first filter the corresponding major template category according to the intent type, and then further refine the matching according to the current plot stage and the guidance direction in the adaptation strategy. For example, if the intent is a plot question and the current stage is the story development stage, the plot answer template exclusive to the development stage can be selected. During the matching process, the child's cognitive state in the context representation is also taken into account, and the template complexity that matches the cognitive level is prioritized. Finally, the most suitable response template is determined and retrieved.

[0122] Step 7.4 involves filling the key semantic information from the reading session context representation into the most suitable response template, generating specific personalized interactive response instructions, and outputting them to the interactive execution terminal. This includes: extracting key semantic information from the current reading session context representation, including the core character names on the current page, key plot details, the core focus of the child's interactive behavior, and key assessment conclusions of the historical cognitive state; then accurately filling the extracted key semantic information into the corresponding placeholders of the retrieved suitable response template, such as filling the core character names into the character placeholders and the key plot details into the plot description placeholders, ensuring semantic coherence and logical flow of the filled content; and generating specific personalized interactive response instructions after filling. If the terminal is a speech recognition terminal... For audio players, the instructions include voice text and tone parameters adapted to children's auditory habits; for example, an upward tone corresponds to a questioning intent, and a soft tone corresponds to an empathetic intent. If the terminal is a display screen, the instructions include a combination of text and images; for example, images should match the current scene, and text should use clear and easy-to-read children's fonts. If it is a multimodal terminal, voice and text instructions are integrated. After generation, the response instructions undergo comprehensive format verification to check whether the encoding format and parameter settings of the instructions conform to the terminal execution specifications, whether the voice text is unambiguous, and whether the text and image resources are complete. After verification, the instructions are output to the corresponding interactive execution terminal. At the same time, the generated response instructions are synchronously associated with the reading session context representation to provide historical response references for subsequent page interactions, ensuring the continuity and consistency of the interaction.

[0123] like Figure 2 As shown, embodiments of the present invention also provide a multimodal interactive data processing system for children's picture book reading, comprising:

[0124] The preprocessing and extraction module is used to process the multimodal raw interaction dataset in parallel, extract and output visual semantic features, interactive speech semantic features and interactive behavior semantic features;

[0125] The key semantic anchor point localization module is used to dynamically determine key semantic anchor points from visual semantic features and interactive speech semantic features. The key semantic anchor points include visual key semantic anchor points and interactive speech key semantic anchor points. The visual key semantic anchor points include candidate visual spatial anchor points and candidate text region anchor points. The interactive speech key semantic anchor points include candidate emotion temporal anchor points and candidate question semantic anchor points. The candidate visual spatial anchor points correspond to the spatial coordinates of the core characters on the current picture book page. The candidate text region anchor points correspond to the center point of the key text region on the current picture book page. The candidate emotion temporal anchor points correspond to the timestamp of the core emotion frame in the speech spectrum. The candidate question semantic anchor points correspond to the position of the question word in the speech spectrum.

[0126] The geometric relationship construction module is used to construct a dynamic semantic geometric relationship graph in the cross-modal joint feature space based on the key semantic anchor points; wherein, a semantic ellipse is generated with the visual key semantic anchor points as the focus; wherein, the center of the circle is obtained by weighted calculation of the core emotion frame timestamp anchor points and question word position anchor points in the interactive voice key semantic anchor points to generate a semantic perception circle.

[0127] The fusion coefficient calculation module is used to calculate the geometric inclusion degree and area overlap rate between the semantic ellipse and the semantically perceived circle, and generate a cross-modal semantic fusion adjustment coefficient.

[0128] The multimodal feature fusion module is used to adjust the coefficients through cross-modal semantic fusion, and to perform weighted fusion of visual semantic features, interactive speech semantic features, and interactive behavior semantic features to generate a multimodal joint semantic representation.

[0129] The Intent and Cognitive Reasoning module is used to perform joint reasoning based on multimodal joint semantic representation, parse and output the results of children's interaction intent judgment and cognitive state assessment, and iteratively update the multimodal joint semantic representation accordingly to generate a coherent reading conversation context representation.

[0130] The personalized response generation module is used to generate and output personalized interactive response instructions that adapt to the evolution of the picture book plot based on the contextual representation of the coherent reading conversation and the results of the interaction intent determination.

[0131] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal interactive data processing method for children's picture book reading, characterized in that, The method includes: Parallel processing is performed on the original multimodal interaction dataset to extract and output visual semantic features, interactive speech semantic features, and interactive behavior semantic features; In visual semantic features and interactive speech semantic features, key semantic anchors are dynamically determined. These key semantic anchors include visual key semantic anchors and interactive speech key semantic anchors. Visual key semantic anchors include candidate visual spatial anchors and candidate text region anchors. Interactive speech key semantic anchors include candidate emotion temporal anchors and candidate question semantic anchors. Candidate visual spatial anchors correspond to the spatial coordinates of the core character on the current picture book page. Candidate text region anchors correspond to the center point of the key text region on the current picture book page. Candidate emotion temporal anchors correspond to the timestamp of the core emotion frame in the speech spectrum. Candidate question semantic anchors correspond to the position of the question word in the speech spectrum. Based on the key semantic anchors, a dynamic semantic geometric relationship graph is constructed in the cross-modal joint feature space; wherein, a semantic ellipse is generated with the visual key semantic anchors as the focus; wherein, the center of the circle is obtained by weighted calculation of the core emotion frame timestamp anchor and the question word position anchor in the key semantic anchors of interactive speech, so as to generate a semantic perception circle. Calculate the geometric inclusion degree and area overlap rate between the semantic ellipse and the semantically perceived circle, and generate a cross-modal semantic fusion adjustment coefficient; A cross-modal semantic fusion adjustment coefficient is used to weight and fuse visual semantic features, interactive speech semantic features, and interactive behavior semantic features to generate a multimodal joint semantic representation. Based on multimodal joint semantic representation, joint reasoning is performed to parse and output the results of children's interaction intention judgment and cognitive state assessment. Based on this, the multimodal joint semantic representation is iteratively updated to generate a coherent reading conversation context representation. Based on the contextual representation of the coherent reading conversation and the results of the interaction intent determination, personalized interactive response instructions that adapt to the evolution of the picture book plot are generated and output.

2. The multimodal interactive data processing method for children's picture book reading according to claim 1, characterized in that, Parallel processing is performed on the original multimodal interaction dataset to extract and output visual semantic features, interactive speech semantic features, and interactive behavior semantic features, including: From the acquired multimodal raw interaction dataset, picture book image data, children's voice data, touch pressure data, and 3D pose data were separated. Edge preprocessing is performed on the picture book image data to obtain the preprocessed page image; the preprocessed page image is segmented by picture book page features and object recognition is performed to locate the text area and illustration element area; text semantic information is extracted from the text area and visual semantic information of characters and scenes is extracted from the illustration element area; text semantic information and visual semantic information are integrated to form and output visual semantic features. Noise reduction and endpoint detection are performed on children's speech data to obtain clean speech segments; acoustic feature analysis and speech recognition are performed on the clean speech segments to extract acoustic features from the speech spectrum and transcribe them into text information; combined with the acoustic features and text information, interactive speech semantic features containing reading content, question content and tone of voice are analyzed and output. The touch pressure data and 3D posture data are synchronized and time-aligned to obtain a synchronized behavior sequence. By parsing the page turning pressure change pattern, page flipping angle sequence and gaze focus change trajectory from the synchronized behavior sequence, interactive behavior semantic features that represent reading rhythm and concentration state are generated and output.

3. The multimodal interactive data processing method for children's picture book reading according to claim 2, characterized in that, In visual semantic features and interactive speech semantic features, key semantic anchors are dynamically determined. These key semantic anchors include visual key semantic anchors and interactive speech key semantic anchors. Visual key semantic anchors include candidate visual spatial anchors and candidate text region anchors. Interactive speech key semantic anchors include candidate emotion temporal anchors and candidate interrogative semantic anchors. Candidate visual spatial anchors correspond to the spatial coordinates of the core character on the current picture book page; candidate text region anchors correspond to the center point of the key text region on the current picture book page; candidate emotion temporal anchors correspond to the timestamp of the core emotion frame in the speech spectrum; and candidate interrogative semantic anchors correspond to the position of the interrogative word in the speech spectrum. These include: Based on visual semantic features, the core character objects and key text regions in the current picture book page are identified. The center coordinates of the bounding box of the core character objects in the page image coordinate system are extracted as candidate visual space anchor points; the geometric center coordinates of the key text regions are extracted as candidate text region anchor points. Based on interactive speech semantic features, we can identify emotional fluctuation frames and interrogative sentence patterns in children's speech segments. At the same time, we can locate the center timestamp of the emotional fluctuation frame on the speech timeline as a candidate emotional time anchor point; and locate the position index of the interrogative word in the text sequence as a candidate interrogative semantic anchor point.

4. The multimodal interactive data processing method for children's picture book reading according to claim 3, characterized in that, Based on the key semantic anchors, a dynamic semantic geometric relationship graph is constructed in the cross-modal joint feature space; wherein, a semantic ellipse is generated with the visual key semantic anchors as the focus; wherein, by weighting the core emotion frame timestamp anchors and question word position anchors in the interactive voice key semantic anchors, the center of the circle is obtained to generate a semantic perception circle, including: Using the spatial coordinates of the core character and the center point of the key text region in the visual key semantic anchor points as the focus, the focal length of the ellipse is calculated based on the preset semantic association distance threshold; based on the calculated focal length of the ellipse and the semantic saliency weight, a semantic ellipse is generated in the visual modal subspace of the cross-modal joint feature space. By weighting the core emotion frame timestamp anchor and question word position anchor in the key semantic anchors of interactive speech, and using the calculation result as the center, the perception radius is calculated by the acoustic feature intensity and emotional confidence in the semantic features of interactive speech; based on the center and perception radius, a semantic perception circle is generated in the speech modal subspace of the cross-modal joint feature space. The semantic ellipse and the semantic perception circle are mapped to the same cross-modal joint feature space coordinate system to form a dynamic semantic geometric relationship graph. The semantic ellipse represents the distribution range of the visual semantics of the picture book, and the semantic perception circle represents the perception range of the semantics of children's voice interaction. The semantic ellipse and the semantic perception circle together construct the geometric constraint framework of cross-modal semantic association.

5. The multimodal interactive data processing method for children's picture book reading according to claim 4, characterized in that, Calculate the geometric inclusion degree and area overlap rate between the semantic ellipse and the semantically perceived circle to generate a cross-modal semantic fusion adjustment coefficient, including: Under the same coordinate system mapped by the dynamic semantic geometric relationship graph, the focal position, major axis and minor axis length of the semantic ellipse, as well as the center coordinates and perception radius of the semantic perception circle are obtained; Based on the obtained center coordinates and focus positions, the relative positional relationship between the semantically perceptual circle and the semantic ellipse is calculated; according to the relative positional relationship, it is determined whether the center of the semantically perceptual circle is located inside or on the boundary of the semantic ellipse, and the corresponding first geometrical inclusion value is calculated. Calculate the area of ​​the intersection region between the semantically perceived circle and the semantic ellipse, and divide the area of ​​the intersection region by the area of ​​the semantically perceived circle to obtain the second area overlap rate value; The first geometric inclusion value and the second area overlap rate value are weighted and synthesized, and combined with the visual semantic distribution confidence represented by the semantic ellipse, to generate the cross-modal semantic fusion adjustment coefficient.

6. The multimodal interactive data processing method for children's picture book reading according to claim 5, characterized in that, A cross-modal semantic fusion adjustment coefficient is used to weight and fuse visual semantic features, interactive speech semantic features, and interactive behavior semantic features to generate a multimodal joint semantic representation, including: Obtain cross-modal semantic fusion adjustment coefficients and simultaneously read visual semantic features, interactive speech semantic features, and interactive behavior semantic features; The cross-modal semantic fusion adjustment coefficients are input into the set weight mapping function to calculate the visual weights of visual semantic features, the speech weights of interactive speech semantic features, and the behavioral weights of interactive behavior semantic features. Based on the visual weight, voice weight, and behavior weight, the corresponding visual semantic features, interactive voice semantic features, and interactive behavior semantic features are weighted and summed to obtain a preliminary fusion feature vector. The initial fusion feature vectors are normalized and dimensionality reduced to output a multimodal joint semantic representation that is precisely bound to the semantic content of the current picture book page.

7. The multimodal interactive data processing method for children's picture book reading according to claim 6, characterized in that, Joint reasoning is performed based on multimodal joint semantic representation. The results of determining children's interactive intentions and assessing their cognitive states are parsed and output. Based on this, the multimodal joint semantic representation is iteratively updated to generate a coherent reading conversation context representation, including: Obtain the multimodal joint semantic representation corresponding to the current picture book page; The multimodal joint semantic representation is input into the pre-trained intention and cognition joint reasoning model to extract the multimodal semantic association patterns in the multimodal joint semantic representation, and to perform pattern matching and reasoning based on the knowledge graph of the picture book plot, outputting a multimodal semantic matching and reasoning result. Based on the results of multimodal semantic matching and reasoning, the interaction intent determination results and cognitive state assessment results are parsed and output. The multimodal joint semantic representation, interaction intent determination results, and cognitive state evaluation results are fused with the real-time stored historical reading session context representation in a temporal sequence; the temporal fusion is used to update and generate a coherent reading session context representation for the current moment.

8. The multimodal interactive data processing method for children's picture book reading according to claim 7, characterized in that, Based on the coherent reading conversation context representation and the interaction intent determination result, personalized interactive response instructions adapted to the evolution of the picture book plot are generated and output, including: Obtain the current reading session context representation and interaction intent determination result; Based on the current reading session context representation and combined with the picture book plot knowledge graph, the current reading process is predicted to be located in the story and the subsequent plot development, and a plot adaptation strategy is generated. Based on the interaction intent determination result and the plot adaptation strategy, the most suitable response template is matched and retrieved from the preset multimodal response template library; The key semantic information in the reading session context representation is filled into the most suitable response template to generate specific personalized interactive response instructions, which are then output to the interactive execution terminal.

9. A multimodal interactive data processing system for children's picture book reading, the system implementing the method as described in any one of claims 1 to 8, characterized in that, include: The preprocessing and extraction module is used to process the multimodal raw interaction dataset in parallel, extract and output visual semantic features, interactive speech semantic features and interactive behavior semantic features; The key semantic anchor point localization module is used to dynamically determine key semantic anchor points from visual semantic features and interactive speech semantic features. The key semantic anchor points include visual key semantic anchor points and interactive speech key semantic anchor points. The visual key semantic anchor points include candidate visual spatial anchor points and candidate text region anchor points. The interactive speech key semantic anchor points include candidate emotion temporal anchor points and candidate question semantic anchor points. The candidate visual spatial anchor points correspond to the spatial coordinates of the core characters on the current picture book page. The candidate text region anchor points correspond to the center point of the key text region on the current picture book page. The candidate emotion temporal anchor points correspond to the timestamp of the core emotion frame in the speech spectrum. The candidate question semantic anchor points correspond to the position of the question word in the speech spectrum. The geometric relationship construction module is used to construct a dynamic semantic geometric relationship graph in the cross-modal joint feature space based on the key semantic anchor points; wherein, a semantic ellipse is generated with the visual key semantic anchor points as the focus; wherein, the center of the circle is obtained by weighted calculation of the core emotion frame timestamp anchor points and question word position anchor points in the interactive voice key semantic anchor points to generate a semantic perception circle. The fusion coefficient calculation module is used to calculate the geometric inclusion degree and area overlap rate between the semantic ellipse and the semantically perceived circle, and generate a cross-modal semantic fusion adjustment coefficient. The multimodal feature fusion module is used to adjust the coefficients through cross-modal semantic fusion, and to perform weighted fusion of visual semantic features, interactive speech semantic features, and interactive behavior semantic features to generate a multimodal joint semantic representation. The Intent and Cognitive Reasoning module is used to perform joint reasoning based on multimodal joint semantic representation, parse and output the results of children's interaction intent judgment and cognitive state assessment, and iteratively update the multimodal joint semantic representation accordingly to generate a coherent reading conversation context representation. The personalized response generation module is used to generate and output personalized interactive response instructions that adapt to the evolution of the picture book plot based on the contextual representation of the coherent reading conversation and the results of the interaction intent determination.

Images