An ASD child language ability evaluation method and system based on distillation LLMs and application thereof

Abstract

The application discloses an ASD child language ability evaluation method based on distillation LLMs, and the method comprises the following steps: step one, collecting ASD child natural language samples and generating dialogue text; step two, designing a prompt for guiding the standardized evaluation of a large language model; step three, constructing model training data based on "hard constraints and soft guidance", training and distilling the large language model to obtain a distilled large language model; and step four, processing input language samples to be evaluated by using the distilled large language model to obtain a child language ability evaluation result. The application also discloses an evaluation system for implementing the above evaluation method, and has wide application value.

Description

Technical Field

[0001] This invention belongs to the field of language ability assessment technology, and relates to a method, system and application for assessing the language ability of children with ASD based on distilled LLMs. Background Technology

[0002] Traditional language proficiency assessments typically rely on standardized measurement tools and parent reports. Commonly used assessment tools include the Early Language Milestone Scale (ELMS). [1] ,Peabody Picture Vocabulary Test(PPVT) [2] Mullen Early Learning Scale (MSEL) [3] Clinical Language Foundation Assessment (CELF) [4] Language Development Survey (LDS) [5] These tools provide systematic language assessments for children of different ages, covering comprehension, expression, vocabulary, and communication. ELMS is suitable for infants aged 0-36 months, assessing auditory reception, visual language, and auditory expression abilities; PPVT assesses vocabulary comprehension in individuals aged 2.5 years and older; MSEL is suitable for infants aged 0-68 months, assessing language comprehension and expression; CELF assesses language abilities in children, adolescents, and young adults; and LDS assesses vocabulary and early language development in toddlers through parent reports. However, these traditional tools rely on professional analysis, resulting in high subjectivity and low efficiency. To improve the objectivity and efficiency of language ability assessment, many researchers have begun exploring innovative methods combining machine learning technology in recent years, leading to the development of various automated assessment methods. [6-10] .

[0003] Traditional assessment tools (such as clinical assessment scales, questionnaires, and structured interviews) are often subjective and inefficient. Although existing computer-aided methods have improved efficiency, they typically have limited assessment dimensions and fail to fully utilize children's natural language samples.

[0004] Language impairment is a common comorbidity in children with ASD. Language ability assessment, as the primary method for identifying language impairment, mainly relies on standardized assessment scales or reports in clinical practice, such as the Early Language Milestone Scale (ELMS), Mullen Scales of Early Learning, and the Language Development Survey (LDS). While these methods provide direct assessments of language ability, their applicable age ranges and assessment content vary, and the entire process depends on professional analysis, resulting in issues such as high subjectivity and time costs.

[0005] To improve evaluation efficiency and reduce subjectivity, researchers have begun to explore intelligent methods, such as the LENA (Language Environment Analysis) system.

[10] It automates and quantifies language data by recording children's vocal environment and analyzing metrics such as the number of vocalizations and turns. While the LENA system overcomes the limitations of traditional methods, it primarily focuses on quantitative metrics and remains insufficient in understanding dialogue content and assessing higher-level language skills. To address this, Gale R. et al. [7] Researchers have proposed an automated method based on sentence repetition task (SRT) and speech recognition to assess children’s expressive language abilities through regression models. This method improves assessment efficiency and objectivity, but still has limitations such as a single assessment dimension and a lack of authenticity in natural communication. Summary of the Invention

[0006] Traditional assessment scales or reports suffer from high subjectivity and low efficiency, while existing intelligent assessments are limited by their singular assessment dimensions and lack of authenticity in reflecting natural communication. To address these issues, this invention proposes a method, system, and application for assessing the language abilities of children with ASD based on a distilled large language model. This invention leverages the powerful understanding and generation capabilities of a large language model to analyze natural language samples from children with ASD, enabling a multi-dimensional assessment of their language abilities. Specifically, this invention designs a training data generation and model distillation method under "hard constraints and soft guidance." This involves optimizing the data generation strategy and using a token-based model training method to distill the large language model. The distilled large language model is then used to process and evaluate input language samples, achieving a low-cost, high-performance assessment of the language abilities of children with ASD.

[0007] This invention provides a method for assessing the language abilities of children with ASD based on distilled LLMs, the method comprising the following steps:

[0008] Step 1: Collect natural language samples from children with ASD and generate dialogue text;

[0009] Step 2: Design a prompt to guide the standardized evaluation of the large language model;

[0010] Step 3: Construct model training data based on "hard constraints and soft guidance", perform training distillation on the large language model, and obtain the distilled large language model;

[0011] Step 4: Process the input language samples to be evaluated using the distilled large language model to obtain the assessment results of children's language abilities.

[0012] In step one, the natural language samples of children with ASD include one or more sets of dialogue audio, which contain the voices of parents and children talking.

[0013] Using the publicly available children's speech dataset CHILDES, we fine-tuned the wav2vec 2.0 speech recognition model and the ERes2Net speaker recognition model. Based on the fine-tuned model, we completed speech recognition and speaker recognition of the collected natural language sample audio of ASD children and generated dialogue text.

[0014] Sensitive information such as names, addresses, and phone numbers in the dialogue text is desensitized.

[0015] The sensitive information is processed and anonymized into a general identifier related to the information category.

[0016] In step two, the structure of the prompt includes: role definition, assessment dimensions, child information, dialogue content, and standardized workflow;

[0017] The aforementioned role definition refers to defining the large language model that performs standardized evaluations of children's natural language samples as an expert in the field of children's language development;

[0018] The assessment dimensions include language comprehension, vocabulary mastery, logical expression, and communication skills.

[0019] The child information includes the child's gender and age;

[0020] The dialogue content includes the dialogue text generated after processing the natural language samples of ASD children collected in step one;

[0021] The standardized workflow includes understanding the assessment dimensions, providing assessment criteria, conducting language ability analysis and assessment, and summarizing the assessment results.

[0022] The language comprehension assessment evaluates a child's ability to understand word meanings, semantics, syntax, or pragmatics.

[0023] The vocabulary mastery assessment evaluates a child's ability to understand, remember, and use vocabulary.

[0024] The logical expression ability assessment evaluates a child's ability to organize language and state ideas;

[0025] The communication skills assessment evaluates a child’s comprehensive ability to effectively express ideas and understand others, including communication initiative, dialogue turn-taking, and pragmatic functions.

[0026] The evaluation results for each of the evaluation dimensions are categorized as very poor, poor, average, good, and excellent. The highest total score is then evenly distributed across the four dimensions, and within each dimension, the evaluation result is assigned a score according to an arithmetic progression.

[0027] And / or,

[0028] The evaluation results are scored as follows: Excellent 37.5 points, Good 30 points, Average 22.5 points, Poor 15 points, and Very Poor 7.5 points.

[0029] In step three, model training data based on "hard constraints and soft guidance" is constructed, and the large language model is trained and distilled to obtain the distilled large language model.

[0030] In one specific embodiment of the present invention, training distillation is performed based on the Qwen2-7B-Instruct large language model;

[0031] In practice, further, by using a "hard constraint, soft guidance" strategy on the large language model to generate training data for distillation training, and performing model distillation on the large language model, the requirements for computing power and other aspects are reduced, making it easier to deploy.

[0032] Specifically, a "hard constraint, soft guidance" strategy is used for data generation of large language models. By constructing multi-dimensional conditional constraint templates, the publicly available large language models are guided to generate dialogue data that conforms to standards. The Guidance library is used to force the generated data to meet key conditions through constrained decoding. By embedding comparative examples in the prompt, the model is guided to learn the mapping relationship between MSEL scores and language features. Through positive and negative example comparisons, the model's understanding of conditional boundaries is strengthened, thus achieving soft guidance; and simulated dialogues generated by children with ASD are obtained.

[0033] For the generated dialogues, the children's language abilities are independently assessed using multiple publicly available large language models (including but not limited to one or more of Qwen2-72B-Instruct, GPT4o, Doubao-Pro-32k, etc.). The consistency score between the assessment results and the hard constraints is calculated, and dialogues and assessment results with a consistency of ≥80% are retained as training data for distilling large language models.

[0034] Using the training data obtained above, including dialogue text and evaluation results, Qwen2-7B-Instruct is subjected to supervised fine-tuning. By using a dynamic decay weight allocation strategy, the weights of key tokens in the evaluation level are gradually decayed while the weights of non-key tokens are maintained. This optimizes the loss and yields a distilled large language model.

[0035] The language abilities of children with ASD were assessed using the distilled and retrained Qwen2-7B-Instruct, and the results of the language ability assessment were obtained.

[0036] To make the evaluation results of the large language model distilled by this invention more robust, a multi-expert voting mechanism can be used to evaluate the same language sample of the ASD child to be evaluated multiple times using the same distillation model, and the weighted calculation can be performed to obtain evaluation results with higher credibility and stronger stability.

[0037] The evaluation result of the multi-expert voting mechanism is obtained by weighting the evaluation results of the same distillation model multiple times, as shown in the following formula:

[0038]

[0039] Among them, y i It is the output of the i-th expert, w i This is the weight of the i-th expert; in practice, the weights can be set to the same for the same distillation model.

[0040] In practice, the distillation model and other existing public large language models can be used to independently evaluate the same language sample of children with ASD. After obtaining their respective evaluation results, a weighted calculation can be performed on the multiple independent evaluation results through a multi-expert voting mechanism to obtain a more reliable evaluation result. The weights of different large language models can be set according to actual needs.

[0041] Alternatively, a specific large language model (including but not limited to the following publicly available LLMs capable of performing Chinese tasks: Qwen2-7B-Instruct, Qwen2-72B-Instruct, GPT4o, Doubao-Pro-32k, etc.) can be selected individually to perform multiple assessments on the same language sample of a child with ASD to be evaluated. The final assessment result for the corresponding specific large language model is obtained after weighted calculation. The assessment results generated by different specific large language models can be used for further comparison of the results.

[0042] And / or,

[0043] The evaluation results of the distilled large language model were compared and quantified with the MSEL language development quotient results using Pearson correlation coefficient and p-value. The scores output by the distilled large language model were compared with the scores of the language development quotient results to verify the effectiveness of the distilled large language model.

[0044] The present invention also provides an evaluation system for implementing the above evaluation method, the evaluation system comprising: a data acquisition module, a language processing module, a data construction and model training module, and an evaluation and verification module;

[0045] The data acquisition module is used to collect language data from children with ASD.

[0046] The language processing module is used to design a prompt to guide the large language model in language ability assessment, in order to guide the large language model in standardized assessment.

[0047] The data construction and model training module is used to construct and generate new high-quality training data for distillation training, and to train a small-scale, easily deployable large language model (in one specific embodiment of the invention, Qwen2-7B-Instruct is trained).

[0048] The assessment and verification module is used to assess children's language abilities using the distilled large language model, and / or to test the assessment effect of the distilled large language model using a multi-expert voting mechanism, and / or to compare and analyze the assessment performance with a variety of publicly available large language models, and / or to verify the effectiveness of the distilled large language model using standardized scales.

[0049] The present invention also provides the application of the above-mentioned assessment method or assessment system in assessing children's language abilities and improving the accuracy of assessments.

[0050] The beneficial effects of this invention include: significantly improving assessment efficiency and effectively reducing subjectivity in the assessment process by utilizing large language models to analyze natural language samples to evaluate children's language abilities. The method of this invention designs four assessment dimensions, effectively solving the problem of single-dimensionality in existing assessment methods and providing a more comprehensive perspective on the assessment results. This invention fully utilizes children's natural language samples, and through intelligent assessment, it compensates for the authenticity problem ignored by traditional assessment methods that rely on structured data, making the assessment results closer to actual language development levels. This invention designs a training data generation and model distillation method under "hard constraints and soft guidance," including a hard constraint + soft guidance data generation strategy, a multi-model validation and screening mechanism, and model training based on token importance, reducing costs while ensuring assessment stability and effectiveness. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart of the method of the present invention.

[0053] Figure 2 This is a diagram of the prompt design components of the present invention.

[0054] Figure 3 This is a flowchart of the evaluation process for Large Language Models (LLMs) of this invention.

[0055] Figure 4 This is a complete prompt example diagram of the present invention.

[0056] Figure 5 This is an example diagram of the JSON format conditional input template of this invention.

[0057] Figure 6 This is an example diagram illustrating the implementation of hard constraints using the Guidance library in this invention.

[0058] Figure 7 This is a comparative example diagram in the Prompt of this invention. Detailed Implementation

[0059] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0060] The following are explanations of some of the terms used in this invention:

[0061] LLMs: Large Language Models, a type of generative artificial intelligence model based on the Transformer Decoder architecture.

[0062] ASD: Autism Spectrum Disorder.

[0063] Children's language ability: the development of children at different ages.

[0064] Role definition: The purpose is to assign a real-world profession or character category to a large model.

[0065] Prompt: Prompt is the entry point for the model to receive information. It provides the model with context or instructions, telling the model what kind of results need to be generated.

[0066] Multi-expert voting mechanism: It uses a voting method to arrive at the final decision by combining the judgments of multiple independent "experts" (in this invention, this refers to multiple or more large language models, or multiple evaluations of the same large language model).

[0067] This invention provides a method for assessing the language abilities of children with ASD based on distilled LLMs, such as Figure 1 As shown, it includes the following steps:

[0068] Step 1: Collect natural language samples from children with ASD and generate dialogue text.

[0069] This invention collected audio recordings of parent-child conversations from 50 children aged 2 to 6 years old with autism spectrum disorder (ASD) whose native language is Mandarin. Each group consisted of one parent and one child. Each audio recording was approximately 15 minutes long, totaling 750 minutes. All children underwent the MSEL (Mullen Scales of Early Learning) test, which focused on assessing their language comprehension and expression abilities. The wav2vec 2.0 speech recognition model and the ERes2Net speaker recognition model were used to process the children's speech. However, existing technologies are primarily trained and fine-tuned based on adult datasets, and there is still room for improvement in recognizing children's speech, especially that of children with ASD. To improve the model's performance on children's speech, this invention first used the publicly available children's dataset CHILDES to further fine-tune the speech recognition and speaker recognition models, making them better suited to children's speech characteristics. Subsequently, based on the fine-tuned model, speech recognition and speaker recognition were performed on parent-child conversation audio samples from ASD children's natural language samples, generating the corresponding parent-child conversation text.

[0070] To protect user privacy, sensitive privacy information (including names, addresses, and phone numbers) in the conversation text has been anonymized. Specifically, this information is anonymized into generic identifiers related to its category; for example, names are replaced with "XXX," addresses with "location," and phone numbers with "number." This effectively protects participants' privacy while ensuring the data can be used for subsequent research and analysis. Furthermore, privacy information appearing in the text can be anonymized based on privacy requirements.

[0071] Step 2: Design a prompt to guide the standardized evaluation of the large language model.

[0072] Figure 2This is a schematic diagram of the prompt structure designed in this invention. The prompt structure includes role definition, evaluation dimension design, text content corresponding to the child's voice, child's gender and age, and standardized workflow, etc.

[0073] Role Definition: To enhance the understanding and application of Large Language Models (LLMs) in specialized fields, this invention defines a specific role for the model. Specifically, the model is positioned as an expert in the field of child language development, specializing in assessing children's language abilities by analyzing parent-child dialogues in free play scenarios. This role definition significantly improves the model's applicability and expertise in analyzing children's dialogue content, thereby better supporting research and practice in child language development.

[0074] Assessment Dimensions Design: The assessment primarily includes four core dimensions for language proficiency: language comprehension, vocabulary mastery, logical expression, and communication skills. Specifically, based on the latest literature research, clinical assessment scales, questionnaires, and structured interviews, the following four language assessment dimensions and specific assessment content were designed, covering key aspects of children's language development.

[0075] 1) Language comprehension. This dimension assesses a child's ability to understand word meaning, semantics, syntax, or pragmatics.

[11] The Diagnostic and Statistical Manual of Mental Disorders (DSM-5) states that many individuals with ASD have speech deficits, ranging from complete speech absence to poor overall comprehension of speech. This study referenced the assessment methods of the Mullen Early Learning Scale (MSEL) and the Diagnostic Observational Scale for Autism Scale, Second Edition (ADOS-2), tools that primarily assess language comprehension by observing children's understanding and responses to verbal instructions, questions, and social interactions. Furthermore, numerous studies have shown that children with ASD have significant impairments in language comprehension. [12-14] This is mainly manifested in lagging vocabulary, semantics, and syntax learning, often only understanding the literal meaning, and having difficulty understanding and executing multi-step instructions.

[0076] 2) Vocabulary mastery. This dimension assesses a child's ability to understand, remember, and use vocabulary. Research shows that children with autism spectrum disorder (ASD) differ significantly from developmentally normal (TD) children in vocabulary mastery and use. [15,16] The Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) emphasizes that individuals with autism spectrum disorder often exhibit stereotyped or repetitive speech patterns, including imitation and specific phrases, which directly impact their vocabulary development. In clinical practice, professionals use the Peabody Picture Vocabulary Test (PPVT).

[17] Tools such as the Language Development Survey (LDS) were used to detect accepting words.

[18] This allows for the assessment of expressive vocabulary. These well-established research findings, diagnostic criteria, and clinical assessment tools collectively provide a solid foundation for designing a vocabulary competence dimension to evaluate the language skills of children with ASD.

[0077] 3) Logical expression ability. This dimension focuses on a child's ability to organize language and state ideas. In terms of expression ability, children with ASD exhibit high individual heterogeneity.

[18] Most children with ASD often experience difficulties in language organization and use, employing inappropriate language forms and content in situations, and fixating on topics that are not context-appropriate.

[19] Some researchers have compared the language abilities of children with ASD, DD, and TD through experiments and found that individuals with ASD generally exhibit syntactic disorders. [20,21] In addition, the CELF assessment also includes tests on complex syntactic structures and logical expression abilities.

[0078] 4) Communication Skills. This dimension assesses a child's overall ability to effectively express their thoughts and understand others. The DSM-5 diagnostic criteria for autism spectrum disorder state that individuals with ASD have deficits in social interaction, such as: inability to engage in normal conversation, reduced sharing of interests, or inability to respond to social interactions. (Autism Diagnostic Interview-Revised, ADI-R)

[22] Children's communication skills are assessed through structured parent interviews. Some studies have found that children with ASD are unable to proactively initiate, respond to, take turns, repair, or terminate conversations based on the context.

[23] This dimension assesses aspects such as communicative initiative, dialogue turn-taking, and pragmatic functions.

[0079] The specific details of the four evaluation dimensions are shown in Table 1 below:

[0080] Table 1. Details of the four language assessment dimensions

[0081]

[0082] Text content corresponding to children's speech: This part is the dialogue text after the ASD children's natural language sample data collection and processing in step one.

[0083] The child's gender and age. This invention targets children aged 2 to 6 with ASD, taking into account the differences in assessment standards across different age groups in practice. Although the specific assessment dimensions and scope have been defined, it is still necessary to utilize a large language model, combined with its knowledge base, to provide more detailed assessment standards for children of each age group. Therefore, the prompt must clearly display the child's age and gender information so that the model can deeply understand and generate targeted assessment content.

[0084] Standardized Workflow: 1. Understanding Assessment Dimensions: The Large Language Model (LLM) carefully analyzes and understands the four core assessment dimensions and the specific assessment content corresponding to each dimension. 2. Providing Assessment Standards: Based on the understanding in step one, the LLM, combined with the child's basic information (such as gender and age), formulates personalized assessment standards for each assessment dimension (specific examples are given later). 3. Conducting Dimension Analysis: According to the assessment standards formulated in step two, the LLM analyzes the dialogue content between the child and parents, provides a reasonable level assessment of the child's language development level in each dimension, and provides detailed assessment reasons (detailed examples are given later). Scoring Rules: This invention references the MSEL language assessment standard, dividing language ability into five levels: very poor, poor, average, good, and excellent. To quantify the assessment results of LLMs, a numerical mapping method is designed. This method, based on the previously defined four assessment dimensions and five levels, converts the level assessment results into numerical form for quantitative analysis. Considering children's language development, the total score range for language ability assessment by LLMs is set to 0-150 points, which is consistent with the observation that children's language development quotient usually does not exceed 150 (the calculation method of language development quotient is detailed in formula (1)). Based on the maximum total score of 150 points, the scores of each dimension and level are allocated as follows: First, the total score is equally distributed among the four assessment dimensions, with the highest score for each dimension being 37.5 points (150 / 4). Second, within each dimension, the score is allocated to the five levels using an arithmetic progression method, specifically: Excellent (37.5 points), Good (30 points), Average (22.5 points), Poor (15 points), and Very Poor (7.5 points). This design ensures that the score difference between adjacent levels remains constant (7.5 points), thereby guaranteeing the continuity and discrimination of the scoring. 4. Summarize the assessment results: The large language model summarizes the assessment results in JSON format, where the assessment dimension is the key and the corresponding level is the value.

[0085]

[0086] Step 3: Construct model training data based on "hard constraints and soft guidance", perform training distillation on the large language model, and obtain the distilled large language model.

[0087] The training data generation and model training method under the "hard constraints and soft guidance" includes a hard constraint + soft guidance data generation strategy, a multi-model validation and screening mechanism, and a model training based on token importance, which reduces costs while ensuring evaluation stability and effectiveness.

[0088] Due to data privacy and security issues with closed-source large language models, and the potential difficulty in deploying open-source large language models due to their massive scale, this invention, in a specific implementation, considers assessing the language abilities of children with autism spectrum disorder (ASD) by training a small-scale language ability assessment model (the model trained in this invention is Qwen2-7B-Instruct). The core process of data construction and model training is as follows:

[0089] Teacher model data generation: Through "hard constraints + soft guidance", the system controls the generation of dialogue text that conforms to the characteristics of the ASD language by using publicly available large language models such as GPT4o.

[0090] Multi-model validation screening: The capabilities of newly generated dialogues are evaluated using publicly available large language models, including Qwen2-72B-Instruct, Doubao-Pro-32k, and GPT4o. High-quality data that meets the constraints is selected as training data for training the distilled large language model.

[0091] Student model distillation training: The Qwen2-7B-Instruct model is trained using the filtered data. A model training based on token importance is introduced to make it approximate the evaluation ability of the teacher model, thus obtaining the distilled Qwen2-7B-Instruct model.

[0092] 1. Data generation strategy of hard constraints + soft guidance

[0093] To guide large language models in generating standard-compliant dialogue data, multi-dimensional conditional constraint templates need to be constructed.

[0094] Examples of key condition parameters and control logic during the data generation process are shown in Table 2 below:

[0095] Table 2. Examples of Key Condition Parameters and Control Logic

[0096]

[0097] Example of a JSON format conditional input template: Figure 5 As shown.

[0098] To ensure the quality of generated data, the following combined control strategy of "hard constraints + soft guidance" is adopted:

[0099] ① Hard Constraints

[0100] Implementation method: When generating data, key conditions are enforced by constrained decoding, such as... Figure 6 As shown, hard constraints can be implemented, for example, through the Guidance library.

[0101] ② Soft Guidance

[0102] Contrastive examples are embedded in the prompt to guide the model in learning the mapping relationship between MSEL scores and language features. By comparing positive and negative examples, the model's understanding of conditional boundaries is strengthened; the contrastive examples in the prompt are as follows: Figure 7 As shown.

[0103] 2. Multi-model validation and screening mechanism

[0104] For the generated dialogue, language capabilities were independently evaluated using multiple validated high-performance large language models (including Qwen2-72B-Instruct, Doubao-Pro-32k, GPT4o, etc.). The consistency score between the evaluation results and the hard constraints was calculated as shown in Equation (2) below:

[0105]

[0106] Retain dialogues and evaluation results with ≥80% consistency as training data.

[0107] 3. Model training based on token importance

[0108] When performing supervised fine-tuning on the Qwen2-7B-Instruct model, this invention proposes a dynamic decay weight allocation strategy. Initial reinforcement phase: Key evaluation tokens (Excellent, Good, Average, Poor, Very Poor) are initially assigned high weights (1.5), forcing the model to quickly establish a strong correlation between evaluation criteria and language features in the early stages of training. Gradual decay phase: As the number of training steps increases, the weights of key tokens are gradually decayed from 1.5 to 1.0 according to a cosine curve, allowing the model to consider general language pattern learning in the later stages. Baseline weight maintenance: Non-key tokens always maintain a weight of 1.0, ensuring that the basic language modeling ability is not disturbed.

[0109] The loss design is as follows (3):

[0110] L(t)=-∑ i w i (t)y i log(p i (3)

[0111] Among them, w i(t) is the weight of the i-th token at time step t, and the weight changes dynamically as shown in formula (4); y i For real labels; p i This represents the probability value predicted by the model.

[0112]

[0113] Where t is the current training step; T is the total number of training steps; β is the initial weight coefficient (e.g., 1.5); initial weights: key tokens w i (0) = β; Final weight: key tokens w i (T) = 1.

[0114] After training, the distilled large language model (in one specific embodiment of the present invention, the distilled Qwen2-7B-Instruct) is used to assess the language ability of the child with ASD to obtain the language ability assessment results of the child with ASD.

[0115] In one specific implementation plan, a multi-expert voting mechanism is used for weighted evaluation to make the evaluation results of the distilled large language model more stable.

[0116] Specifically, the large language model guided by the prompt designed in step two is used to assess children's language abilities. This invention employs a multi-expert voting mechanism during the assessment process. Each large language model (in one specific implementation, the large language model is a distilled Qwen2-7B-Instruct) is considered an expert, and these experts independently assess the performance of the children's language samples. The final assessment result is calculated by weighting the scores from multiple experts.

[0117]

[0118] Among them, y i It is the output of the i-th expert, w i It is the weight of the i-th expert.

[0119] To verify the effectiveness of the language assessment method, a correlation analysis was conducted, comparing the assessment results of the Distilled Large Language Model with the language development quotient results of MSEL (Mullen Scales of Early Learning). The Pearson correlation coefficient (r) and the corresponding p-value were calculated to quantify the degree of association and statistical significance between the two sets of results.

[0120] The evaluation results of the distilled large language model constructed in this invention have a high degree of consistency with the evaluation results of existing publicly available large language models, indicating that the distilled large language model constructed in this invention can be more lightweight while ensuring good evaluation results.

[0121] Example

[0122] Taking a 6-year-old child with ASD as an example, the detailed process of assessment using the trained Qwen2-7B-Instruct is as follows:

[0123] Step 1: Collect natural language samples from children and generate dialogue text.

[0124] Basic information about the child was collected: age 6, female. Next, the child underwent the MSEL (Mullen Scales of Early Learning) test, focusing on assessing her language comprehension and expression abilities, and calculating a comprehensive language development quotient to quantify her language development level. Then, during free play with the child, natural interactive audio was recorded using a recording device to ensure the audio samples accurately reflected her language abilities. For the collected audio, ERes2Net speaker recognition technology, which had undergone further pre-training and fine-tuning, was first used to accurately extract the speaker's identity and corresponding timestamp for each segment of the audio. Next, wav2vec 2.0 speech recognition technology was used to transcribe each audio segment into text, generating a complete text record of the child's dialogue. This text not only preserved the content of the language expression but also provided high-quality input data for subsequent language ability analysis.

[0125] Step 2: Design a prompt to guide the standardized evaluation of the large language model.

[0126] A complete prompt design, such as Figure 4 As shown, it includes the following key parts: role definition, four assessment dimensions and their detailed assessment scope, basic information about the child and the dialogue content, and a standardized workflow. The role definition, assessment dimensions and scope, and basic information about the child are already included. Figure 4 The full details are shown in the text. It should be noted that, to fully protect the privacy of children with special needs, the complete dialogue will not be disclosed at this time.

[0127] Step 3: Use prompts to guide the large language model in evaluation, and provide language proficiency assessment results through multi-expert voting.

[0128] According to the standardized workflow in the prompt, the large language model will be understood, analyzed, and evaluated step by step in the following steps.

[0129] (1) The first step is for the large language model (the results shown below are based on the evaluation of Qwen2-7B-Instruct after training) to deeply understand the evaluation dimensions and output the following content based on its understanding:

[0130] Language comprehension assessment evaluates children’s performance in understanding language and dialogue, specifically examining whether children can understand the literal and contextual meaning; whether they can understand abstract concepts, metaphors and other rhetorical devices; whether their answers are accurate, whether they have deviated from the topic, and whether there are any misunderstandings; and whether they can correctly use and distinguish personal pronouns.

[0131] Vocabulary mastery assessment evaluates a child's vocabulary acquisition and usage, specifically examining whether the child possesses a rich vocabulary of common words; whether they repeatedly use the same words, phrases, or short sentences; whether they frequently repeat what their parents say; and whether they use novel or made-up words or phrases to express meaning.

[0132] Logical expression ability assessment evaluates a child's performance in organizing and expressing information, specifically including: whether the child uses words in a confused manner; whether the child has difficulty organizing information, such as lack of connection between sentences, grammatical errors, and incoherent expression.

[0133] The assessment of communication skills evaluates children's interactive performance in conversations, specifically including: whether children can initiate, respond, take turns, and terminate conversations; whether there are communication difficulties; whether they talk to themselves or frequently change topics; and whether they only focus on topics that interest them.

[0134] (2) In the second step, the large language model combines the child's basic information with the assessment dimensions and scope specified in the prompt to provide the following assessment criteria for children of this age group:

[0135] Language comprehension: Children can understand the meaning of most words and context; they can understand simple abstract concepts and metaphors; they can answer questions accurately without straying from the topic; and they can accurately use and distinguish common personal pronouns.

[0136] Vocabulary mastery: Children master and can use a rich vocabulary of common words; they rarely or never repeat the same words or phrases; they do not often repeat what their parents say; and they do not use novel or made-up words or phrases.

[0137] Logical expression ability: Children use words accurately and without confusion; their sentences are coherent and have relatively complete grammatical structures; their information is clearly organized and their expression is logical.

[0138] Communication skills: Children can initiate, respond, take turns, and terminate conversations; communication is relatively smooth with no obvious difficulties; they rarely talk to themselves; topics are relatively fixed and do not change frequently.

[0139] (3) Third step: Analyze the dialogue content and conduct an evaluation

[0140] Language comprehension ability: In the conversation, the child responded with "Mmm" more often and did not show obvious semantic comprehension difficulties; there were no obvious irrelevant answers, but no active expression of complex understanding; the child could understand the parent's instructions for specific actions (such as "draw with your right hand" or "insert in an empty place"), but did not actively express understanding of abstract content; pronouns such as "he" and "you" did not appear frequently, so the pronoun discrimination ability could not be assessed.

[0141] Level Assessment: General

[0142] Vocabulary mastery: The child repeatedly used "en" in response, and the vocabulary used was relatively simple; the child did not show any use of novel or fabricated words; there was no repetition of the parent's words.

[0143] Rating: Poor

[0144] Logical expression ability: The child reproduced language less, and most of the responses were single words "en". The coherence of sentences and information organization were not effectively assessed; however, no obvious word confusion was shown.

[0145] Level Assessment: General

[0146] Communication skills: The child mainly responds with "yes" and does not initiate or respond to conversations much; there is a certain degree of passivity and singularity in communication; there is no frequent change of topics, talking to oneself, or focusing only on one's own topic.

[0147] Rating: Poor

[0148] (4) Fourth step: Summarize the evaluation results

[0149]

[0150] The above are the evaluation results provided by a single large language model expert.

[0151] To further enhance the robustness of the evaluation results, a multi-expert voting mechanism can be introduced. This involves conducting multiple evaluations of the same large language model, or taking a weighted average of the evaluation results from multiple different large language models, to arrive at a more reliable final evaluation conclusion. The specific method is as follows:

[0152] 1. Multi-model evaluation: Multiple large language models that meet the requirements are evaluated independently. Each model generates corresponding evaluation results based on the standardized evaluation process and evaluation criteria in the prompt.

[0153] 2. Weight allocation: Assign weights to each model. If there is a lack of clear performance differences, equal weight allocation can be used initially.

[0154] 3. Weighted average of results: The specific indicators under each evaluation dimension are weighted and averaged, as shown in formula (5).

[0155] References

[0156] [1]J.Coplan,Early language milestone scale,Edisi ke-2(2005).

[0157] [2]Dunn, DM (2019). Peabody Picture Vocabulary Test (5th ed.) [Measurement instrument]. Bloomington, MN: NCS Pearson.

[0158] [3] EMMullen, et al., Mullen scales of early learning, AGS CirclePines, MN, 1995.

[0159] [4]T.Paslawski, The clinical evaluation of language fundamentals, (celf-4)areview, Canadian Journal of School Psychology 20(1-2)(2005)129134.

[0160] [5] L. Rescorla, The language development survey: A screening tool for delayed language in toddlers, Journal of Speech and Hearing disorders 54(4)(1989)587–599.

[0161] [6] K.Carson, G.Gillon, T.Boustead, Computer-administrated versus paper-based assessment of school-entry phonological awareness ability, Asia Pacific Journal of Speech, Language and Hearing 14(2)(2011)85–101.

[0162] [7]R.Gale,J.Dolata,E.Prud'hommeaux,J.Van Santen,M.Asgari,Automated assessment of language ability in children with and without typical development,in:2020,pp.6111–6

[0163] [8]A.Xu,R.Hebbar,R.Lahiri,T.Feng,L.Butler,L.Shen,H.TagerFlusberg,S.Narayanan,ArXiv preprint arXiv:2305.14117(2023).

[0164] [9]EJTenenbaum,KLCarpenter,M.Sabatos-DeVito,J.Hashemi,S.Vermeer,G.Sapiro,G.Dawson,A six-minute measure of vocalizations in toddlers with autism spectrum disorder,Autism Research 13(8)(2020)1373–1382.

[0165]

[10] J.Gilkerson,JARichards,The Lena Natural Language Study,Boulder,CO:LENA Foundation.Retrieved March 3(2009)(2008)15–1

[0166]

[11] January Journal.Orthopedic Surgery of Emergency Chemistry[J].Current Journal of Medical Research, 2019,17(03):161-165.

[0167]

[12] Zhou Mingzhu, Niu Juanli, Lu Haidan. Analysis of the characteristics of picture book reading comprehension ability of autistic children aged 4-6 years with language age [J]. Modern Special Education, 2023(02):68-72.

[0168]

[13] M.Zhou, J.Niu, H.Lu, Analysis of picture book reading comprehension characteristics in autistic children with language age of 4-6years, ModernSpecial Education(02)(2023)68–72.

[14] YESu, L.Naigles, Expressive language abilities of mandarin-exposed preschool children with autism spectrum disorder (asd), Chinese Journal of Language Policy and Planning 5(02)(2020)25–34.doi:10.19689 / j.cnki.cn10-1361 / h.20200202.

[0169]

[15] W. Bi-jun, S. Xiao-na, L. Ling, Z. Rong, Analysis of language development level in children with autism, Chinese Journal of Child Health Care (2017).

[0170]

[16] A.Shield, F.Cooley, RPMeier, Sign language echolalia in deafchildren with autism spectrum disorder, Journal of Speech, Language, and HearingResearch 60(6)(2017)1622–1634.

[0171]

[17] D.Dunn, D.Dunn, Peabody picture vocabulary test [measurement instrument], NCS Pearson (2019).

[0172]

[18] L.Rescorla,The language development survey:A screening tool fordelayed language in toddlers,Journal of Speech and Hearing disorders 54(4)(1989)587–599.

[0173]

[19] S.Yi,X.Qian-qian,and S.Lin-yan,“Cross-population comparison ofearly expressive language profiles in asd,dd and ld,”Chinese Journal ofClinical Psychology,vol.28,no.03,pp.508–512+517,2020.

[0174]

[20] H.M.Geurts and M.Embrechts,“Language profiles in asd,sli,andadhd,”Journal of autism and developmental disorders,vol.38,pp.19311943,2008.

[0175]

[20] I.-M.Eigsti,L.Bennetto,and M.B.Dadlani,“Beyond pragmatics:Morphosyntactic development in autism,”Journal of autism and developmentaldisorders,vol.37,pp.1007–1023,2007.

[0176]

[21] J.Boucher,“Research review:structural language in autisticspectrum disorder–characteristics and causes,”Journal of child psychology andpsychiatry,vol.53,no.3,pp.219–233,2012.

[0177]

[22] M.Rutter, A.Le Couteur, C.Lord et al., "Autism diagnostic interview revised," Los Angeles, CA: Western Psychological Services, vol.29, no.2003, p.30, 2003.

[0178]

[23] D.-m.MA, J.Zhi-mei, P.Wei, L.Hong, G.Lan-min, L.Xin, L.Meixuan, and N.Tian-yu, "Exploration and analysis of language functions for autism spectrum disorders," Chinese Scientific Journal of Hearing and Speech Rehabilitation, vol.17, no.03, pp.175–178, 2019.

[0179] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0180] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0181] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0182] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0183] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0184] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A method for assessing the language abilities of children with ASD based on distilled LLMs, characterized in that, The method includes the following steps: Step 1: Collect natural language samples from children with ASD and generate dialogue text; Step 2: Design a prompt to guide the standardized evaluation of the large language model; Step 3: Construct model training data based on "hard constraints and soft guidance", perform training distillation on the large language model, and obtain the distilled large language model; In step three, a "hard constraint, soft guidance" strategy is used to generate data from the large language model. A multi-dimensional conditional constraint template is constructed to guide the large language model to generate standard-compliant dialogue data. The Guidance library is used to force the generated data to meet key conditions through restricted decoding. By embedding comparative examples in the prompt, the model is guided to learn the mapping relationship between MSEL scores and language features. Through positive and negative example comparisons, the model's understanding of conditional boundaries is strengthened, thus achieving soft guidance. Simulated dialogue generated by ASD children is obtained. The conditional dimensions of the multidimensional conditional constraints include the target LDQ interval, age, language feature description, and dialogue structure requirements; The target LDQ interval is the segmented interval of the MSEL language development quotient; For the simulated generated dialogue, language ability is independently evaluated using multiple publicly available large language models. The consistency score between the evaluation results and hard constraints is calculated, and dialogues with consistency ≥80% and evaluation results are retained as training data for distilled large language models. Supervised fine-tuning of a large language model is performed using training data including dialogue text and evaluation results. A dynamic decay weight allocation strategy is employed to gradually reduce the weights of key evaluation-level tokens while maintaining the weights of non-key tokens, thereby optimizing the loss and obtaining a distilled large language model. The weight allocation strategy includes: Initial reinforcement phase: Set initial high weights for key tokens in the evaluation level, forcing the model to quickly establish a strong correlation between the evaluation criteria and language features in the early stages of training; Gradual decay phase: As the number of training steps increases, the weights of key tokens are gradually decayed according to a cosine curve, so that the model can take into account the learning of general language patterns in the later stage. Maintain baseline weights: Non-critical tokens always maintain their weights to ensure that the basic language modeling capabilities are not disturbed; Step 4: Process the input language samples to be evaluated using the distilled large language model to obtain the assessment results of children's language abilities.

2. The evaluation method as described in claim 1, characterized in that, In step one, the natural language samples of children with ASD include one or more sets of dialogue audio, which contain the voices of parents and children talking. Using the publicly available children's speech dataset CHILDES, we fine-tuned the wav2vec 2.0 speech recognition model and the ERes2Net speaker recognition model. Based on the fine-tuned model, we completed speech recognition and speaker recognition of the collected sample audio and generated dialogue text. Sensitive information in the dialogue text is anonymized; the sensitive information includes name, address, and phone number. The sensitive information is processed and anonymized into a general identifier related to the information category.

3. The evaluation method as described in claim 1, characterized in that, In step two, the structure of the prompt includes: role definition, assessment dimensions, child information, dialogue content, and standardized workflow; The aforementioned role definition refers to defining the large language model that performs standardized evaluations of children's natural language samples as an expert in the field of children's language development; The assessment dimensions include language comprehension, vocabulary mastery, logical expression, and communication skills. The child information includes the child's gender and age; The dialogue content includes the dialogue text generated after processing the natural language samples of ASD children collected in step one; The standardized workflow includes understanding the assessment dimensions, providing assessment criteria, conducting language ability analysis and assessment, and summarizing the assessment results.

4. The evaluation method as described in claim 3, characterized in that, The language comprehension assessment evaluates a child's ability to understand word meanings, semantics, syntax, or pragmatics. The vocabulary mastery assessment evaluates a child's ability to understand, remember, and use vocabulary. The logical expression ability assessment evaluates a child's ability to organize language and state ideas; The communication skills assessment evaluates a child’s comprehensive ability to effectively express ideas and understand others, including communication initiative, dialogue turn-taking, and pragmatic functions.

5. The evaluation method as described in claim 3, characterized in that, The evaluation results for each of the evaluation dimensions are categorized as very poor, poor, average, good, and excellent. The highest total score is then evenly distributed across the four dimensions, and within each dimension, the evaluation result is assigned a score according to an arithmetic progression. And / or, The evaluation results are scored as follows: Excellent 37.5 points, Good 30 points, Average 22.5 points, Poor 15 points, and Very Poor 7.5 points.

6. The evaluation method as described in claim 1, characterized in that, The method also includes: using a multi-expert voting mechanism, selecting the same model to evaluate the same ASD child's language sample multiple times, and obtaining the evaluation result by weighting; or using different large language models to evaluate the same ASD child's language sample independently, and obtaining the evaluation result by weighting. The calculation formula is as follows: ， in, It is the output of the i-th expert. It is the weight of the i-th expert; And / or, The effectiveness of the distilled large language model was verified by comparing and quantifying the evaluation results of the distilled large language model with the MSEL language development quotient results using Pearson correlation coefficient and p-value.

7. An evaluation system for implementing the evaluation method as described in any one of claims 1-6, characterized in that, The evaluation system includes: a data acquisition module, a language processing module, a training data construction and model training module, and an evaluation and verification module; The data acquisition module is used to collect language data from children with ASD. The language processing module is designed to provide a prompt for the large language model to perform language ability assessment and to guide the large language model to perform standardized assessment. The data construction and model training module is used to construct and generate training data for distillation training, and to perform training distillation on the large language model. The assessment and verification module uses the distilled large language model to assess children's language abilities, and / or uses a multi-expert voting mechanism to test the assessment effect of the distilled large language model, and / or compares and analyzes the assessment performance with a variety of publicly available large language models, and / or uses standardized scales to verify the effectiveness of the distilled large language model.

8. The application of the assessment method as described in any one of claims 1-6, or the assessment system as described in claim 7, in assessing children's language abilities and improving the accuracy of assessments.

Images