A kind of method and system for evaluating language ability of young children based on multi-modal interaction and dynamic task allocation

Abstract

The application provides a kind of based on multimodal interaction and dynamic task allocation's infant language ability evaluation method and system, it is related to language test technical field.Method includes: determine the current language target of the infant to be evaluated and corresponding expression type evaluation task and understanding type evaluation task;Based on expression type evaluation task, collect language response data and participation behavior data, determine task participation state and expression response state;When effective participation and expression do not pass, keep the current language target and assign understanding type evaluation task, determine understanding response state and language ability state according to understanding behavior data;According to language ability state, dynamically determine subsequent language target and evaluation task, and determine language ability evaluation result according to the language ability state of multiple language targets.The application can distinguish the expression mastery, understanding mastery and unmastered state of infant, improve the pertinence of language ability evaluation.

Description

Technical Field

[0001] This invention relates to the field of language assessment technology, and in particular to a method and system for assessing early childhood language abilities based on multimodal interaction and dynamic task allocation. Background Technology

[0002] Early childhood language ability assessments typically include assessments of language comprehension and language expression, with different stimulus materials and response methods used for different assessment contents. In their study, "A Reliability and Validity Study of a Vocabulary Assessment Scale for Children's Language Development" (Chinese Journal of Hearing and Speech Rehabilitation Science, 2024, Vol. 22, No. 1, pp. 23-26), Zhang Xianda et al. used the "Children's Chinese Language Development Assessment Scale—Vocabulary Section" to assess preschool children. This vocabulary subscale includes separate vocabulary comprehension and vocabulary expression questions to evaluate children's vocabulary comprehension and expression abilities. The vocabulary subscale in this study included 20 vocabulary comprehension questions and 25 vocabulary expression questions, indicating that language comprehension and language expression can be used as different assessment contents in children's vocabulary ability assessment.

[0003] In the area of ​​automatic assessment of children's language abilities, Chinese invention patent application CN116595152A discloses a method for testing children's language abilities based on a large language model superimposed with an adapter. This method trains a children's language dialogue model using the child's voice input, determines the language ability test result based on the child's voice and the model's predicted voice dialogue, and discloses adaptive prompts in the form of text and image instructions. Building upon this, Chinese invention patent application CN119028594A discloses a method for assessing the language abilities of children with autism. This method collects children's language, visual, and behavioral data during interaction. It extracts acoustic, lexical, sentence structure, and semantic information from the language data, and visual attention points, eye movements, facial expressions, body movements, head movements, and fixation behaviors from the visual and behavioral data. Based on the resulting multimodal response features, it generates interactive tasks that match the child's current language ability, and then uses the language data obtained from multiple rounds of interaction to assess language ability.

[0004] Therefore, the assessment of children's language ability has been expanded from a single speech test to an assessment method that combines language response, visual response, and behavioral response with an adjusted interactive task.

[0005] However, existing multimodal interactive language assessment methods mainly determine subsequent interactive tasks based on children's comprehensive response characteristics, and the task adjustments disclosed in CN119028594A also mainly change the interactive content based on children's current language ability, emotional state, and attention state.

[0006] For the assessment of early childhood language abilities, when children do not develop the required language expression responses in expressive assessment tasks, current technologies lack an assessment method that, while maintaining consistency with the language goals being assessed, allows for a further switch to comprehension-based assessment tasks based on the children's task participation status, and determines the corresponding language goal mastery status by combining the response results under different response modes. Summary of the Invention

[0007] In order to overcome the shortcomings of the prior art, the purpose of this invention is to provide a method and system for assessing early childhood language ability based on multimodal interaction and dynamic task allocation. This invention solves the problem in the prior art that it is difficult to further distinguish between language comprehension and language expression states under the same language goal when young children have not completed language expression.

[0008] To achieve the above objectives, the present invention provides the following solution: A method for assessing early childhood language abilities based on multimodal interaction and dynamic task allocation, comprising: Determine the current language goals for the children to be assessed and the corresponding expressive and comprehension assessment tasks; Based on the expressive assessment task, language response data and participation behavior data of the children to be assessed are collected. The task participation status is determined based on the participation behavior data, and the expressive response status is determined based on the language response data. When the task participation status is invalid, the language proficiency status of the current language target is uncertain, and the assessment task corresponding to the current language target is reassigned; when the task participation status is valid and the expression response status is expression pass, the language proficiency status of the current language target is determined to be expression mastery; when the task participation status is valid and the expression response status is expression fail, the current language target remains unchanged and the comprehension assessment task is assigned. Based on the comprehension-based assessment task, comprehension behavior data of the children to be assessed are collected, comprehension response status is determined based on the comprehension behavior data, and the language ability status of the current language goal is determined as either comprehension mastery or non-mastery based on the comprehension response status. Based on the current language proficiency status of the language objective, determine subsequent language objectives and corresponding assessment tasks; The subsequent language goals are evaluated as new current language goals, and the language proficiency status of multiple language goals is determined in sequence. Based on the language proficiency status of the multiple language goals, the language proficiency assessment result of the child to be evaluated is determined.

[0009] A language proficiency assessment system for young children based on multimodal interaction and dynamic task allocation includes: The task determination module is used to determine the current language goals of the children to be assessed and the corresponding expressive assessment tasks and comprehension assessment tasks. The expressive response processing module is used to collect language response data and participation behavior data of the child to be assessed based on the expressive assessment task, determine the task participation status based on the participation behavior data, and determine the expressive response status based on the language response data. The task allocation module is used to: determine the language proficiency status of the current language target when the task participation status is invalid participation, and reallocate the assessment task corresponding to the current language target; determine the language proficiency status of the current language target as expression mastery when the task participation status is valid participation and the expression response status is expression failure; and keep the current language target unchanged and allocate the comprehension assessment task when the task participation status is valid participation and the expression response status is expression failure. The comprehension response processing module is used to collect comprehension behavior data of the child to be assessed based on the comprehension assessment task, determine the comprehension response status based on the comprehension behavior data, and determine the language ability status of the current language target as comprehension mastery or non-mastery based on the comprehension response status. The subsequent task determination module is used to determine the subsequent language goals and corresponding assessment tasks based on the language proficiency status of the current language goal. The language proficiency assessment module is used to assess the subsequent language goals as new current language goals, sequentially determine the language proficiency status of multiple language goals, and determine the language proficiency assessment result of the child to be assessed based on the language proficiency status of the multiple language goals.

[0010] The present invention discloses the following technical effects: This invention provides a method and system for assessing early childhood language abilities based on multimodal interaction and dynamic task allocation. The invention acquires language response data and participation behavior data of the children to be assessed based on expressive assessment tasks, and determines the task participation state and expressive response state respectively. In cases of ineffective participation, the assessment task corresponding to the current language goal is reassigned. In cases of effective participation but failure to pass the expression assessment, the current language goal remains unchanged, and a comprehension-based assessment task is assigned. The comprehension response state is then determined based on the comprehension behavior data, enabling the language ability state of the current language goal to be distinguished as expressive mastery, comprehension mastery, or lack of mastery. Further, subsequent language goals and corresponding assessment tasks are determined based on the language ability states of these goals, and the language ability assessment result is determined based on the language ability states of multiple language goals. This allows for dynamic adjustment of subsequent assessment tasks based on the actual response states of children in different assessment tasks, improving the differentiation of different language ability states during the assessment process. Attached Figure Description

[0011] Figure 1 A flowchart of a method for assessing early childhood language ability based on multimodal interaction and dynamic task allocation, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating language ability status determination and dynamic task allocation provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a preschool language ability assessment system provided in an embodiment of the present invention.

[0012] Figure label: 1-Task Determination Module, 2-Expression Response Processing Module, 3-Task Allocation Module, 4-Understanding Response Processing Module, 5-Subsequent Task Determination Module, 6-Language Proficiency Assessment Module. Detailed Implementation

[0013] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] like Figure 1 As shown, this invention provides a method for assessing early childhood language abilities based on multimodal interaction and dynamic task allocation, including: Step 100: Determine the current language goals for the child to be assessed and the corresponding expressive and comprehension assessment tasks; Step 200: Based on the expressive assessment task, collect the language response data and participation behavior data of the child to be assessed, determine the task participation status based on the participation behavior data, and determine the expressive response status based on the language response data; Step 300: When the task participation state is invalid participation state, the language ability state of the current language target is uncertain, and the assessment task corresponding to the current language target is reassigned; when the task participation state is valid participation state and the expression response state is expression pass state, the language ability state of the current language target is determined to be expression mastery state; when the task participation state is valid participation state and the expression response state is expression fail state, the current language target remains unchanged and the comprehension assessment task is assigned. Step 400: Based on the comprehension assessment task, collect the comprehension behavior data of the child to be assessed, determine the comprehension response status according to the comprehension behavior data, and determine the language ability status of the current language goal as comprehension mastery or non-mastery according to the comprehension response status; Step 500: Determine subsequent language goals and corresponding assessment tasks based on the current language proficiency status; Step 600: The subsequent language goals are evaluated as new current language goals, the language proficiency status of multiple language goals is determined in sequence, and the language proficiency assessment result of the child to be evaluated is determined based on the language proficiency status of the multiple language goals.

[0015] In this embodiment, the preschool language ability assessment can be performed on a smart terminal with audio acquisition, image acquisition, and interactive operation functions. The smart terminal can be a smartphone or tablet running a preschool language ability assessment mini-program. The smart terminal uses a microphone to collect the speech produced by the child being assessed, a camera to capture the child's behavior during the task presentation, and records the child's interactive operations on the assessment interface. The collected data can be processed locally on the smart terminal or sent to a server for processing.

[0016] In this embodiment, the current language goal represents the language content that needs to be assessed in the current evaluation to determine the child's mastery. The current language goal can be vocabulary, action meaning, simple semantic relationships, or language instructions. The expressive evaluation task is used to obtain the language expressions actively generated by the child in response to the current language goal; the comprehension evaluation task is used to determine the child's comprehension level based on their behavior in response to the current language goal, while keeping the current language goal unchanged.

[0017] The language response data, participation behavior data, and comprehension behavior data each serve different processing functions. The language response data originates from children's speech collected during the expressive assessment task and is used to determine the expressive response status; the participation behavior data is used to determine whether the child to be assessed has actually entered the expressive assessment task; the comprehension behavior data is generated from the comprehension assessment task and is used to determine whether the child to be assessed can complete the selection, identification, or instruction execution behaviors according to the requirements corresponding to the current language goal.

[0018] For step 100, determine the current language goals of the child to be assessed and the corresponding expressive assessment tasks and comprehension assessment tasks.

[0019] Once the assessment begins, the age information of the child to be assessed is obtained, and the corresponding language development stage is determined based on a pre-established relationship between language development stages. Multiple language goals are configured for each language development stage, and each language goal records its language goal identifier, language ability dimensions, expressive assessment tasks, and comprehension assessment tasks. For implementation scenarios using other language development standards, the mapping relationship between language development stages and language goals can be adjusted according to the corresponding standard without altering the subsequent status determination and task allocation process.

[0020] For example, when the current language target is the vocabulary word "apple," the expressive assessment task can present an apple picture on the assessment interface and prompt the child being assessed to say what the picture represents; the comprehension assessment task can simultaneously present an apple picture and several non-target pictures and prompt the child being assessed to point out the apple. The current language target for both assessment tasks is "apple," and the language content being assessed remains consistent when switching from the expressive assessment task to the comprehension assessment task.

[0021] For step 200, based on the expressive assessment task, language response data and participation behavior data of the child to be assessed are collected, task participation status is determined based on the participation behavior data, and expressive response status is determined based on the language response data.

[0022] When the expressive assessment task begins, audio and video data are simultaneously collected during the task response. The audio data is used to form the language response data, and the video data is used to form the participation behavior data. To ensure that the task participation status has a repeatable basis for judgment, this embodiment statistically analyzes the orientation of the child to be assessed towards the task presentation area based on video frames.

[0023] Assume that a total of [number] data points are obtained during the task response period. The first valid video frame, the... The task attention markers corresponding to each valid video frame are: When the child's face is within the valid orientation range corresponding to the terminal task presentation area, ;otherwise, Task participation rate Determined according to the following formula: ; In the formula, This refers to the percentage of participants in the task. The number of valid video frames obtained during the task response period; For the first Task attention markers corresponding to each valid video frame.

[0024] The facial orientation can be determined by calculating the pose using facial key points in the video frames. The terminal only needs to obtain the determination result of whether the child to be evaluated is facing the task presentation area in each valid video frame, and does not require the use of a specific face detection model. When the task participation ratio reaches the participation determination threshold, the task participation state is determined to be a valid participation state; when the task participation ratio does not reach the participation determination threshold, the task participation state is determined to be an invalid participation state.

[0025] The participation determination threshold is determined using calibration data labeled with valid or invalid participation. For candidate thresholds... The Youden index is calculated using the following formula. : ; In the formula, To correctly determine the number of samples that are considered valid participants; This represents the number of valid participation samples that were incorrectly identified as invalid participation. To determine the number of samples correctly identified as invalid participants; This represents the number of invalid participation samples incorrectly classified as valid participation. After iterating through the candidate thresholds, the candidate threshold with the highest Youden index is used as the corresponding decision threshold. The acoustic decision threshold, semantic decision threshold, and understanding decision threshold, which will be discussed later, can also be determined using the same calibration method to avoid directly combining different data items into a single score using manual weights.

[0026] Table 1 provides a set of exemplary results used to illustrate threshold calibration and formula verification. The data in the table are for illustrating calculation relationships and do not represent statistical conclusions for specific clinical samples.

[0027] Table 1 Example of threshold calibration

[0028] For example, the Youden index corresponding to the task participation rate in Table 1 is: ; The Youden index corresponding to the correct proportion is: ; When processing the language response data, the audio data can be divided into continuous speech frames, and MFCC acoustic features can be extracted from each speech frame. To avoid the influence of differences in the value range of different MFCC coefficients on distance calculation, the MFCC coefficients of each dimension are standardized using the mean and standard deviation corresponding to the calibration samples. ; In the formula, For the first The first audio frame MFCC coefficients; and The first one in the calibration sample Mean and standard deviation of the MFCC coefficients; These are the standardized MFCC coefficients. The target acoustic template is standardized using the same mean and standard deviation to ensure that the language responses of the children to be assessed are on the same characteristic scale as the target acoustic template.

[0029] The standardized audio frames were arranged according to their acquisition time to form an acoustic feature sequence corresponding to the child to be evaluated: ; In the formula, The acoustic feature sequence corresponding to the child to be evaluated; For the first The standardized MFCC feature vector corresponding to each speech frame; The number of speech frames contained in the language response data.

[0030] For each current language target, multiple target acoustic templates that have been manually verified as correct expressions are pre-saved, forming a target acoustic template set: ; In the formula, For the target acoustic template set; For the first One target acoustic template; This represents the number of target acoustic templates corresponding to the current language target. When different pronunciation habits exist, multiple correct expression templates can be saved under the same current language target without changing the subsequent calculation method.

[0031] Because young children speak at different rates when uttering the same language content, the acoustic feature sequence and the target acoustic template typically have different lengths. This embodiment utilizes dynamic time warping to align the two sequences in time. For the first... Given a target acoustic template, let the alignment path formed by dynamic time warping be... Normalized path distance Determined according to the following formula: ; In the formula, To assess the acoustic feature sequence of the preschool children and the first Normalized path distance between target acoustic templates; The number of matching points contained in the alignment path; For the first In the target acoustic template Aligned MFCC feature vectors; This represents the Euclidean distance.

[0032] The minimum normalized path distance among multiple target acoustic templates is taken as the normalized acoustic distance of the language response data. : ; The smaller the normalized acoustic distance, the closer the language response produced by the child to be evaluated is to the correct expression template corresponding to the current language target in terms of acoustic features.

[0033] The language response data is subjected to speech recognition to obtain language response text, and the language response text and the current language target are converted into Word2Vec word vectors. When the language response text contains multiple valid words, the arithmetic mean vector of the word vectors corresponding to each valid word can be used as the word vector corresponding to the language response text. Let the word vector corresponding to the language response text be... The word vector corresponding to the current language target is semantic similarity Determined according to the following formula: ; In the formula, For semantic similarity; It is the inner product of two word vectors; and These are the L2 norms of the two word vectors. When there is no content in the language response text that can form a valid word vector, the expression response state is determined to be an expression failure state, and the semantic similarity calculation is not performed.

[0034] Under the parameters of the embodiments shown in Table 1, when the normalized acoustic distance satisfies: ; And the semantic similarity satisfies: ; If the expression response state is successful, it is determined as an expression pass state; otherwise, it is determined as an expression fail state. The thresholds in Table 1 are only used to illustrate one implementation method, and the actual thresholds are determined by the corresponding calibration samples according to the aforementioned threshold calibration process.

[0035] In an evaluation where "ball" is the current language target, the dynamic time warped path corresponding to the selected target acoustic template contains four matching positions. The Euclidean distances of these matching positions are 0.25, 0.33, 0.29, and 0.37, respectively. Therefore, the normalized acoustic distance is: ; The inner product of the word vectors corresponding to the same language response is 4.10, and the L2 norms of the two word vectors are 2.10 and 2.14 respectively. Therefore, the semantic similarity is: ; Therefore, the normalized acoustic distance meets the acoustic determination requirements, and the semantic similarity meets the semantic determination requirements, thus determining the expression response state corresponding to "ball" as the expression pass state. This process utilizes both acoustic and semantic information, but does not employ artificially set acoustic and semantic weights, avoiding different weight combinations from altering the judgment meaning of the same language response.

[0036] As another implementation method for language response analysis, MFCC feature sequences can be input into a trained LSTM model to obtain language content recognition results, and then the expressive response state can be determined based on these results. The training samples consist of pre-annotated children's speech samples and can be divided into training, validation, and test sets. The model includes an input layer, an LSTM layer, a fully connected layer, and an output layer, trained using a cross-entropy loss function and the Adam optimizer. The LSTM model is used for language response analysis but does not participate in determining the task participation state, nor does it alter the subsequent processing relationship of maintaining the current language objective and switching to a comprehension-based assessment task.

[0037] For step 300, the subsequent processing corresponding to the current language target is determined based on the task participation status and the expression response status.

[0038] When the task participation status is invalid, the language proficiency status of the current language target is not determined based on the current language response. The current language target remains unchanged, and an assessment task corresponding to the current language target is reassigned. During reassignment, the presentation format of the expressive assessment task can be adjusted. For example, the initial presentation uses a static image; after reassignment, local dynamic prompts or sound prompts are added to the static image; after a second presentation, the language response data and participation behavior data are re-collected.

[0039] During a certain task response, a total of 100 valid video frames were obtained, of which 38 valid video frames had a task attention flag of 1. Then: ; Since the task participation rate is lower than the participation judgment threshold of 0.60 in Table 1, the task participation status is determined to be invalid participation status, and the language ability status of the current language target is uncertain for this task.

[0040] After adjusting the task presentation format and reassigning expressive assessment tasks corresponding to the same current language target, a total of 82 valid video frames have a task attention marker of 1. Therefore: ; When the task participation rate reaches the participation determination threshold, the task participation status is determined to be a valid participation status. After using an independent task participation status determination, the first instance of no actual participation in the evaluation remains at the point of branch reallocation, and will not directly result in an unmastered status due to a single failure to produce the expected language output.

[0041] When the task participation status is effective participation and the expression response status is expression pass, the language ability status of the current language target is determined as expression mastery status.

[0042] When the task participation status is valid participation and the expression response status is expression failure, the current language goal remains unchanged, and a comprehension-based assessment task corresponding to the current language goal is assigned.

[0043] For example, when "apple" is the current language target, after re-presenting the expressive assessment task, the task participation rate is 0.82. The four local alignment distances between the language response and the selected target acoustic template are 0.50, 0.56, 0.52, and 0.54, respectively. Then: ; The inner product of the word vector corresponding to the language response text and the word vector corresponding to the current language target is 3.48, and the L2 norms of the two word vectors are 2.05 and 2.08, respectively. Therefore: ; Although the semantic similarity meets the semantic judgment requirements, the normalized acoustic distance does not meet the acoustic judgment requirements. Therefore, the expression response state is determined to be an expression failure state. At this time, the current language target "apple" is evaluated again without being replaced with other words, and the comprehension-based assessment task corresponding to "apple" is assigned.

[0044] Keeping the current language goals unchanged, the two assessments correspond to the same language content. The previous task requires the child to express themselves actively, while the next task requires the child to perform comprehension actions based on language instructions. This allows for the continued differentiation between the states of mastery in expression, mastery in comprehension, and lack of mastery without altering the language goals.

[0045] like Figure 2 As shown, when the task participation status is valid participation and the expression response status is expression failure, the current language goal remains unchanged and the comprehension assessment task is assigned; when the expression response status is expression success, the language ability status of the current language goal is determined to be expression mastery; for the current language goal entering the comprehension assessment task, the language ability status is determined to be comprehension mastery or non-mastery based on the comprehension response status, and subsequent language goals and corresponding assessment tasks are determined based on the language ability status.

[0046] For step 400, based on the comprehension-based assessment task, the comprehension behavior data of the child to be assessed is collected, the comprehension response status is determined according to the comprehension behavior data, and the language ability status of the current language target is determined as either a comprehension mastery status or a non-mastery status according to the comprehension response status.

[0047] This embodiment uses an image identification task to illustrate the formation process of the comprehension behavior data. The evaluation interface simultaneously displays a target image corresponding to the current language target and multiple non-target images, and plays the language instruction corresponding to the current language target. To reduce the impact of fixed positions on the behavior results, three image identification sub-tasks are presented consecutively for the same current language target. Each sub-task only changes the position of the target image and non-target images on the interface, while the current language target and language instruction remain unchanged.

[0048] Let the first The target image area corresponding to each sub-task is The coordinates of the first effective touch response generated by the child after receiving the language instruction were to be evaluated. The first valid touch refers to the first touch that occurs within any candidate image area and during the current subtask response. (Single understanding result) Determined according to the following formula: ; In the formula, Indicates the first In each sub-task, correctly identify the target image. Indicates the first The target image was not correctly identified in the sub-task.

[0049] After completing the three sub-tasks, understand the correct proportions. Determined according to the following formula: ; When the correct comprehension rate reaches the comprehension determination threshold, the comprehension response state is determined to be a comprehension pass state; otherwise, the comprehension response state is determined to be a comprehension fail state. The comprehension determination threshold used in the embodiment shown in Table 1 is 0.667, corresponding to at least 2 out of 3 sub-tasks being correct.

[0050] For example, if the child to be assessed has single-test comprehension results of 1, 0, and 1 in the subtask of identifying the three pictures corresponding to "apple," then: ; The understanding response state is the understanding passed state, and the language ability state corresponding to "apple" is determined to be the understanding mastery state.

[0051] If the results of the three subtasks corresponding to the other language goal are 1, 0, and 0 respectively, then: ; The comprehension response state is a "comprehension failure" state, which determines the language ability state corresponding to the current language target as "unmastered". Utilizing multiple sub-tasks involving changes in the target image's position for comprehension assessment reduces the impact of a single accidental touch directly determining the language ability state. Furthermore, the three sub-tasks always revolve around the same current language target, without altering the language content being evaluated.

[0052] To clarify the state transition relationships between steps 200 and 400, state variables are defined. : ; In the formula, The threshold for participation in the judgment; This indicates that the expression response state is an expression pass state. This indicates that the expression response status is an expression failure status; This indicates that the understood response status is "understanding passed". This indicates that the understanding response status is a "understanding failed" status. When When the language proficiency status of the current language target is uncertain, the assessment task corresponding to the current language target is reassigned; when When the current language goal is to express mastery, then... When the current language goal is to understand and master the language; when At that time, the current language target is in an unmastered state.

[0053] The state variables are used only to represent different processing branches, and do not assign weights to the three language ability states, nor are they used for weighted scoring. These state variables enable the task participation state, expressive response state, and comprehension response state to form clear discrete processing results, allowing subsequent task allocation to directly read the corresponding language ability state.

[0054] Table 2 lists a set of example data processed consecutively from step 200 to step 400. Each value is consistent with the aforementioned judgment criteria and can be used to verify different state branches.

[0055] Table 2 Examples of Key Assessment Processes

[0056] In Table 2, the normalized acoustic distance for "drinking water" is 0.47, which is higher than the acoustic judgment threshold of 0.45. Although the semantic similarity reaches 0.836, the expression response state is still considered "expression failed." Subsequently, the correct comprehension rate reaches 0.667, and the language ability state is determined to be "understanding and mastery." The normalized acoustic distance and semantic similarity for "cup" do not meet the expression passing condition, and the subsequent correct comprehension rate is only 0.333, ultimately determining it as "not mastered." Therefore, the expression failed state is not directly equivalent to the "not mastered" state.

[0057] For step 500, subsequent language goals and corresponding assessment tasks are determined based on the language proficiency status of the current language goal.

[0058] To facilitate the direct execution of subsequent language goal determination processes, a state-branching relationship is established for each language goal. Each language goal records the subsequent language goal identifiers corresponding to the states of mastery (expression), mastery (comprehension), and lack of mastery. This state-branching relationship is established during the task allocation phase based on the sequence of language development stages and language goals.

[0059] Let the current language target be... The expression mastery of the subsequent goals, understanding and mastery of the subsequent goals, and lack of mastery of the subsequent goals corresponding to the current language goal are respectively... , and Then the subsequent language goals Determined according to the following formula: ; when If this happens, no further language target selection is performed; the current language target remains unchanged, and the evaluation task corresponding to the current language target is reassigned.

[0060] Table 3 presents one state branching configuration method. The language goals and transition relationships in the table are used to illustrate the task configuration and state reading process, and do not indicate that each word has a unique and fixed linguistic developmental order.

[0061] Table 3 Examples of Subsequent Language Target State Branches

[0062] Based on Tables 2 and 3, G01 "apple" is presented again and is understood and mastered. Therefore, the understanding and mastery branch of G01 is read, and G02 "ball" is identified as the new current language target. G02 "ball" is presented and is expressed and mastered. Therefore, the expression and mastery branch of G02 is read, and G04 "drinking water" is identified as the new current language target. G04 "drinking water" is presented and is understood and mastered. Therefore, the understanding and mastery branch of G04 is read, and G03 "cup" is identified as the new current language target. G03 "cup" is in an unmastered state, and the corresponding branch is the end of this stage. Therefore, the task sequence corresponding to the current language development stage ends.

[0063] The above process only reads the corresponding state branch based on the language ability state that has already been determined in the preceding language objectives. It does not require training a reinforcement learning model for selecting the next task, nor does it require manually weighting the acoustic distance, semantic similarity, and correct comprehension ratio again. In this way, the state results formed in steps 200 to 400 can be directly passed to step 500, reducing the influence of intermediate scoring variables on the subsequent task selection relationship.

[0064] For step 600, the subsequent language goals are evaluated as new current language goals, the language proficiency status of multiple language goals is determined in sequence, and the language proficiency assessment result of the child to be evaluated is determined based on the language proficiency status of the multiple language goals.

[0065] After taking the subsequent language goals determined in step 500 as the new current language goals, repeat steps 200 to 500 until the task sequence of the current language development stage ends, or the pre-configured language ability dimensions have all completed the assessment of the corresponding language goals.

[0066] After assessment of multiple language objectives, language competence is categorized according to the dimensions of phonetics, semantics, grammar, and pragmatics. For any given language competence dimension... Let the number of language objectives that have been assessed be... The number of language targets corresponding to the states of mastery, comprehension, and lack of mastery are respectively... , and ,but: ; To preserve the distinction between expressive mastery and comprehension mastery, no artificial weights are assigned to the three language proficiency levels. This embodiment determines the language target mastery ratio. : ; And determine the proportion of expressions achieved in the language learning objectives that have been mastered. : ; when season .

[0067] Unknown proportions Determined according to the following formula: ; From the above definition, we can obtain: ; In the formula, The proportion of language goals that have been achieved in the corresponding language proficiency dimension, either in terms of comprehension or expression, reflects the percentage of language goals that have been mastered. It reflects the proportion of language objectives that have been mastered and that individuals are able to actively express themselves. It reflects the proportion of language goals not mastered in the corresponding language proficiency dimension.

[0068] For example, if an assessment is completed across four language objectives in the semantic dimension, with one objective being mastery of expression, two objectives being mastery of comprehension, and one objective being a state of non-mastery, then: ; ; ; This allows us to retain two layers of information simultaneously: "whether the corresponding language goal has been mastered" and "whether the mastered language goal can be actively expressed," without compressing the expression mastery status and comprehension mastery status into a single score with artificial weights.

[0069] Table 4 shows the summary data of different language proficiency dimensions after one assessment.

[0070] Table 4 Summary of Language Proficiency Dimensions (Example)

[0071] Taking grammar as an example, the percentage of language target mastery is: ; The proportion of the expression is: ; The percentage of items not known is: ; The calculation results above are consistent with those in Table 4. The language ability assessment results for the children to be assessed can retain the number of expressions mastered, the number of comprehensions mastered, the number of expressions not mastered, the proportion of language goals mastered, the proportion of expressions achieved, and the proportion of expressions not mastered for each language ability dimension. The assessment terminal can also generate language development radar charts, growth curves, or text reports based on the language ability assessment results; the display method does not participate in the determination of the aforementioned language ability status and subsequent language goals.

[0072] The above implementation process forms a continuous data transmission relationship. The expressive assessment task first generates the language response data and the participation behavior data; the participation behavior data is used to form the task participation state, and the language response data is used to form the expressive response state; the invalid participation state enters the task reassignment branch of the current language goal, the valid participation state and the expression passed state form the expression mastery state, and the valid participation state and the expression failed state maintain the current language goal and enter the comprehension assessment task; the comprehension behavior data is used to form the comprehension response state, and further form the comprehension mastery state or the non-mastery state; the language ability state is used to determine the subsequent language goals; after multiple language goals are assessed, the language ability assessment result is formed according to the language ability state corresponding to each language goal.

[0073] In the above process, failing to produce the expected language expression once does not directly equate to not mastering the current language goal. The task participation status is used to first distinguish cases where the user has not actually entered the current assessment task; maintaining the current language goal when the user has effectively participated but the expression is not satisfactory allows the comprehension-based assessment task to continue judging the same language content; forming a comprehension response status through multiple image identification sub-tasks with varying positions can reduce the impact of a single accidental interaction directly determining the language ability status; subsequent language goals are determined by the already formed language ability status, eliminating the need to introduce manual weights for task selection again. Thus, the expression mastery status, comprehension mastery status, and non-mastery status can be retained separately during the assessment process and continuously used for subsequent task allocation and final language ability assessment.

[0074] After obtaining the language ability assessment results of the children to be assessed, age-matched status judgment benchmarks were established according to the children's age information, focusing on the dimensions of phonetics, semantics, grammar, and pragmatic language ability. These benchmarks were then used by age group. and language proficiency dimension For example, each reference sample in the peer reference sample has a normal reference label or a language development delay risk label confirmed by a professional standardized language development assessment, and obtains the corresponding language target mastery ratio according to the aforementioned step 600. For candidate risk thresholds Binary classification calibration was performed one by one, using the Youden index. The discriminative power of the candidate thresholds is evaluated, and the candidate threshold with the highest Youden index is determined as the risk threshold for slowing down corresponding to the age group and language ability dimension. Wherein, To accurately determine the number of samples at risk of language development delay, The number of samples at risk of language development delay that were incorrectly classified as normal reference samples. To correctly determine the number of normal reference samples, This represents the number of normal reference samples that were incorrectly identified as samples at risk of language development delay. Further, the normal reference samples in the same age group are arranged from smallest to largest according to the proportion of language target mastery. ,in This is the normal reference sample size; it will be no less than The smallest integer is denoted as It will be no less than The smallest integer is denoted as and respectively according to and Determine the normal reference median and the good reference threshold. Where, and These are the sorting positions corresponding to the 50th and 75th percentiles, respectively. This is the normal reference median value. The good reference threshold is determined by the calibration results corresponding to the professional assessment label; the normal reference median and the good reference threshold are determined by the data distribution of normal reference samples of the same age.

[0075] Taking the semantic dimension of a 36-month-old child as an example, to illustrate the acquisition of the above thresholds and the state judgment process, 10 calibration samples with language development delay risk labels and 10 normal reference samples were taken. The language target mastery ratios of the 10 risk samples were 0.35, 0.40, 0.44, 0.47, 0.50, 0.52, 0.54, 0.56, 0.59, and 0.63, respectively, while those of the 10 normal reference samples were 0.54, 0.57, 0.59, 0.71, 0.73, 0.76, 0.79, 0.83, 0.86, and 0.92, respectively. When the language target mastery ratio is not higher than the candidate threshold as the risk judgment condition, the candidate threshold of 0.52 corresponds to... A candidate threshold of 0.56 corresponds to A candidate threshold of 0.60 corresponds to Therefore, among the candidate thresholds, 0.56 has the largest Youden index, and 0.56 is determined as the lag risk threshold for the semantic dimension of 36 months of age. When ranking the above 10 normal reference samples, ,therefore Take 5, which corresponds to the 5th sorted value of 0.73; The smallest integer not less than 7.5 is 8, therefore Taking 8, corresponding to the 8th sorting value of 0.83, the normal reference median value for the semantic dimension is 0.73, and the good reference threshold is 0.83. Using the same method, the "lagging risk threshold, normal reference median value, and good reference threshold" for the speech, semantic, grammatical, and pragmatic dimensions of 36-month-old infants were obtained as "0.55, 0.72, 0.84", "0.56, 0.73, 0.83", "0.50, 0.68, 0.80", and "0.57, 0.74, 0.85". For the first 36-month-old child awaiting assessment, the percentages of language goals mastered across the four language ability dimensions were 0.86, 0.78, 0.82, and 0.88, respectively. All four dimensions were above the corresponding delay risk threshold. Phonetics, grammar, and pragmatics reached the corresponding good reference thresholds. While the semantic dimension did not reach 0.83, it was above the normal reference median of 0.73. Therefore, the condition of "at least three dimensions reaching the good reference threshold and the remaining dimensions not lower than the normal reference median" was met, and the assessment status was determined to be in a good language development state. For the second 36-month-old child awaiting assessment, the four dimensions were 0.76, 0.69, 0.72, and 0.80, none of which met the corresponding delay risk criteria. However, only some dimensions reached the normal reference median, and the aforementioned good language development state condition was not met. Therefore, the child was determined to be in a normal language development state. For the third 36-month-old child awaiting assessment, the scores for the four dimensions were 0.74, 0.51, 0.70, and 0.77, respectively. The semantic dimension score of 0.51 was not higher than the corresponding delay risk threshold of 0.56. Even though the other three dimensions did not meet the risk criteria, the assessment status was still determined as a language development delay risk state, and the semantic dimension was recorded as a key focus dimension. The above values ​​are used to illustrate the specific execution process of threshold formation, comparison order, and status branching. The actual thresholds were determined by the same method using reference samples of the same age group.

[0076] After determining the assessment status, language development suggestions are generated based on the deviation of each language ability dimension from the corresponding benchmark and the language ability status of each language goal within that dimension. For language development delay risk status, suggestions that meet the criteria are prioritized. The language proficiency dimension is the primary focus dimension. Within this dimension, language goals in the unmastered state are addressed first, followed by those in the understood state but not yet in the expressive state. Taking the aforementioned 36-month-old child as an example, if the semantic dimension score of 0.51 is below the delay risk threshold of 0.56, and if "cup" and "drinking water" are in the unmastered state while "apple" is in the understood state, then basic comprehension assessment tasks and corresponding basic expressive assessment tasks are first configured for "cup" and "drinking water," followed by picture naming or interactive expression tasks for "apple," and simultaneously, prompts for further professional language development assessment are generated. For children with normal language development, according to... Calculate the difference between each language proficiency dimension and the median value of the normal reference, where, For the first The median difference among the language proficiency dimensions was used to prioritize the development of the language proficiency dimension with the smallest median difference. Taking the second-ranked 36-month-old child as an example, the median differences in the phonetics, semantics, grammar, and pragmatics dimensions were respectively... , , and Therefore, the semantic dimension is identified as the priority development dimension, and language goals for comprehension and mastery are selected first from the semantic dimension to configure active expression tasks. For those with good language development, training content is determined from higher-level language goals that have not yet been completed at the current language development stage or from language goals corresponding to adjacent subsequent language development stages, configuring tasks such as sentence expansion, contextual expression, or continuous interaction. The language development suggestions are generated only based on the existing language ability assessment results and language ability status, and do not change the aforementioned results of determining the expression mastery status, comprehension mastery status, lack of mastery status, and subsequent language goals.

[0077] like Figure 3As shown, in some embodiments, the present invention also provides a language ability assessment system for young children based on multimodal interaction and dynamic task allocation. The system includes a task determination module 1, an expression response processing module 2, a task allocation module 3, a comprehension response processing module 4, a subsequent task determination module 5, and a language ability assessment module 6. The task determination module 1 is used to determine the current language goal of the child to be assessed and the corresponding expression-based assessment task and comprehension-based assessment task; the expression response processing module 2 is used to collect language response data and participation behavior data based on the expression-based assessment task, and determine the task participation state and the expression response state; the task allocation module 3 is used to determine, based on the task participation state and the expression response state, to reassign assessment tasks, determine the expression mastery state, or assign comprehension-based assessment tasks; the comprehension response processing module 4 is used to collect comprehension behavior data based on the comprehension-based assessment task, and determine the comprehension mastery state or the lack thereof; the subsequent task determination module 5 is used to determine subsequent language goals and corresponding assessment tasks based on the language ability state of the current language goal; the language ability assessment module 6 is used to sequentially determine the language ability states of multiple language goals, and determine the language ability assessment result of the child to be assessed based on the language ability states of the multiple language goals.

Claims

1. A method for assessing early childhood language abilities based on multimodal interaction and dynamic task allocation, characterized in that, include: Determine the current language goals for the children to be assessed and the corresponding expressive and comprehension assessment tasks; Based on the expressive assessment task, language response data and participation behavior data of the children to be assessed are collected. The task participation status is determined based on the participation behavior data, and the expressive response status is determined based on the language response data. When the task participation status is invalid, the language proficiency status of the current language target is uncertain, and the assessment task corresponding to the current language target is reassigned; when the task participation status is valid and the expression response status is expression pass, the language proficiency status of the current language target is determined to be expression mastery; when the task participation status is valid and the expression response status is expression fail, the current language target remains unchanged and the comprehension assessment task is assigned. Based on the comprehension-based assessment task, comprehension behavior data of the children to be assessed are collected, comprehension response status is determined based on the comprehension behavior data, and the language ability status of the current language goal is determined as either comprehension mastery or non-mastery based on the comprehension response status. Based on the current language proficiency status of the language objective, determine subsequent language objectives and corresponding assessment tasks; The subsequent language goals are evaluated as new current language goals, and the language proficiency status of multiple language goals is determined in sequence. Based on the language proficiency status of the multiple language goals, the language proficiency assessment result of the child to be evaluated is determined.

2. The method for assessing early childhood language ability based on multimodal interaction and dynamic task allocation according to claim 1, characterized in that, The process of determining the current language goals of the child to be assessed and the corresponding expressive and comprehension assessment tasks includes: Obtain the age information of the child to be assessed; The language development stage of the child to be assessed is determined based on the age information. The current language goal is determined based on the language development stage, and the expressive assessment task and the comprehension assessment task corresponding to the current language goal are determined respectively.

3. The method for assessing early childhood language ability based on multimodal interaction and dynamic task allocation according to claim 1, characterized in that, The process of collecting language response data and participation behavior data of the children to be assessed based on the expressive assessment task, determining task participation status based on the participation behavior data, and determining expressive response status based on the language response data includes: The expressive assessment task is presented to the child to be assessed, and audio and video data are collected during the task presentation. Extract the language response data generated by the child to be assessed from the audio data, and extract the participation behavior data generated by the child to be assessed during the task presentation from the video data; The task participation status is determined based on the participation behavior data, and the expression response status is determined based on the language response data.

4. The method for assessing early childhood language ability based on multimodal interaction and dynamic task allocation according to claim 1, characterized in that, Determining the task participation status based on the participation behavior data includes: Based on the participation behavior data, determine whether the child to be assessed exhibited a behavioral response corresponding to the expressive assessment task during the presentation of the expressive assessment task; When the child to be assessed produces a behavioral response corresponding to the expressive assessment task, the task participation state is determined as the effective participation state; When the child to be assessed does not produce a behavioral response corresponding to the expressive assessment task, the task participation status is determined as the invalid participation status.

5. The method for assessing early childhood language ability based on multimodal interaction and dynamic task allocation according to claim 1, characterized in that, Determining the expression response state based on the language response data includes: Extract acoustic and semantic features from the language response data; The pronunciation clarity of the child to be evaluated is determined based on the acoustic features, and the semantic conformity of the language response data with respect to the current language target is determined based on the semantic features. The expression response state is determined as either the expression passed state or the expression failed state based on the pronunciation clarity and the semantic conformity.

6. The method for assessing early childhood language ability based on multimodal interaction and dynamic task allocation according to claim 1, characterized in that, The expressive evaluation task is an image naming task for the current language target, and the comprehension evaluation task is an instruction following task for the current language target.

7. The method for assessing early childhood language ability based on multimodal interaction and dynamic task allocation according to claim 6, characterized in that, The process of collecting comprehension behavior data of the children to be assessed based on the comprehension-based assessment task, determining the comprehension response status based on the comprehension behavior data, and determining the language ability status of the current language goal as either a comprehension mastery state or a non-mastery state based on the comprehension response status includes: Based on the instructions, follow the task to determine the target behaviors that the child to be assessed needs to complete; Collect the comprehension behavior data generated by the child to be assessed in performing the instruction-following task, and determine the actual behavior completed by the child to be assessed based on the comprehension behavior data; The actual behavior is matched with the target behavior. When the actual behavior matches the target behavior, the understanding response state is determined to be the comprehension passed state, and the language ability state of the current language target is determined to be the comprehension mastered state. When the actual behavior does not match the target behavior, the understanding response state is determined to be the comprehension failed state, and the language ability state of the current language target is determined to be the unmastered state.

8. The method for assessing early childhood language ability based on multimodal interaction and dynamic task allocation according to claim 1, characterized in that, When the task participation status is invalid, the language proficiency status of the current language target is uncertain, and the assessment task corresponding to the current language target is reassigned, including: The language proficiency status of the current language target remains undetermined; Adjust the presentation format of the expressive assessment task corresponding to the current language goal, and reassign the adjusted expressive assessment task; Based on the adjusted expressive assessment task, the language response data and the participation behavior data were re-collected, and the task participation status and the expressive response status were re-determined.

9. The method for assessing early childhood language ability based on multimodal interaction and dynamic task allocation according to claim 1, characterized in that, The process of evaluating subsequent language goals as new current language goals, sequentially determining the language proficiency status of multiple language goals, and determining the language proficiency assessment result of the child to be evaluated based on the language proficiency status of multiple language goals includes: The multiple language objectives are classified according to the language competence dimensions of phonetics, semantics, grammar, and pragmatics; Based on the language proficiency status of multiple language objectives corresponding to each of the aforementioned language proficiency dimensions, the assessment result of the corresponding language proficiency dimension is determined; The language ability assessment results of the child to be assessed are determined based on the assessment results of each of the aforementioned language ability dimensions.

10. A language ability assessment system for young children based on multimodal interaction and dynamic task allocation, characterized in that, include: The task determination module is used to determine the current language goals of the children to be assessed and the corresponding expressive assessment tasks and comprehension assessment tasks. The expressive response processing module is used to collect language response data and participation behavior data of the child to be assessed based on the expressive assessment task, determine the task participation status based on the participation behavior data, and determine the expressive response status based on the language response data. The task allocation module is used to: determine the language proficiency status of the current language target when the task participation status is invalid participation, and reallocate the assessment task corresponding to the current language target; determine the language proficiency status of the current language target as expression mastery when the task participation status is valid participation and the expression response status is expression failure; and keep the current language target unchanged and allocate the comprehension assessment task when the task participation status is valid participation and the expression response status is expression failure. The comprehension response processing module is used to collect comprehension behavior data of the child to be assessed based on the comprehension assessment task, determine the comprehension response status based on the comprehension behavior data, and determine the language ability status of the current language target as comprehension mastery or non-mastery based on the comprehension response status. The subsequent task determination module is used to determine the subsequent language goals and corresponding assessment tasks based on the language proficiency status of the current language goal. The language proficiency assessment module is used to assess the subsequent language goals as new current language goals, sequentially determine the language proficiency status of multiple language goals, and determine the language proficiency assessment result of the child to be assessed based on the language proficiency status of the multiple language goals.

Images