A method and device for evaluating children's abilities based on multimodal data fusion

Abstract

The present invention provides a method and device for assessing children's abilities based on multimodal data fusion, comprising the following steps: S1. Collecting voice data from parental communication and video data of the child; S2. Converting the voice data obtained in S1 into text data and preprocessing the video data; S3. Inputting the text data and preprocessed video data into a multimodal large language model; S4. Inputting a questionnaire into the multimodal large language model, which then generates a child's ability assessment result. The present invention has the beneficial effect of reducing subjective bias among assessors by collecting child video data and parental voice data and performing assessments using an artificial intelligence-based multimodal large language model. Furthermore, the format of the child's video data makes it easier for parents to collect data, reducing inaccurate assessment results due to children's nervousness.

Description

Technical Field

[0001] The present invention belongs to the technical field of children's ability assessment, and in particular relates to a children's ability assessment method and device based on multimodal data fusion. Background Art

[0002] During the medical consultation process for children with developmental and behavioral problems, evaluators usually communicate with the children and parents and observe their behavior as an important source of data for the evaluation.

[0003] This assessment method has the following problems: First, when evaluating children's abilities, parents often have different opinions. Parents' judgments are usually based on their subjective feelings in daily life. They may have high expectations for their children's performance and even ignore potential problems. For example, parents may think that their children perform well in a certain ability, but this judgment is often limited to the family or familiar environment; second, the judgment of the evaluator is also subjective to a certain extent. Different evaluators have different assessment standards, resulting in low accuracy of the assessment results; third, the current assessment method requires evaluators to observe children's behavior. First, children's behavior is sporadic. When children are in a nervous state in the physical examination center, their behavior may deviate, resulting in inaccurate assessment results. Summary of the Invention

[0004] In view of this, the present invention aims to propose a child ability assessment method based on multimodal data fusion, in order to solve at least one of the above-mentioned technical problems.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] A first aspect of the present invention provides a method for evaluating children's abilities based on multimodal data fusion, comprising the following steps:

[0007] S1. Collect voice data of parents’ communication and video data of children;

[0008] S2, converting the voice data obtained in S1 into text data and preprocessing the video data;

[0009] S3, inputting the text data and the preprocessed video data into the multimodal large language model;

[0010] S4. Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the child's ability assessment results.

[0011] Furthermore, in said S2, the voice data is converted into text data through the Whisper model;

[0012] The preprocessing of the video data includes the following steps: analyzing the video frames of the video data, extracting key frames with obvious changes, and skipping repeated or static content;

[0013] If there is audio in the video, the timestamp is used for alignment.

[0014] Furthermore, the questionnaire in S4 includes background information, task description, and evaluation criteria;

[0015] The evaluation criteria include gross motor skills, fine motor skills, adaptability, language skills, and social behavior skills for each age group;

[0016] Different weights are assigned to each ability in the evaluation criteria, and a comprehensive ability score is calculated.

[0017] Furthermore, the text data and video data in S4 generate two ability scores for the questionnaire respectively;

[0018] If the ability score of text data is greater than the ability score of video data;

[0019] Then make the following judgment:

[0020] If the difference percentage of the two ability scores is within the first threshold range, the two ability scores are given the same weight to calculate the score of each ability;

[0021] If the difference percentage between the two ability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data;

[0022] If the difference percentages between the two ability scores are within a third threshold range, an exception report is issued, prompting the child's video data to be recollected;

[0023] The first threshold value ranges from 0 to the first threshold value, and the value range of the first threshold value is 10% to 20%;

[0024] The second threshold range is from the first threshold to the second threshold, and the value range of the second threshold is 20% to 35%;

[0025] The third threshold range is from the second threshold to the third threshold, and the value range of the third threshold is 40% to 60%;

[0026] If the difference percentage between the two ability scores is greater than a third threshold, the assessment is determined to be wrong.

[0027] Furthermore, the value ranges of the first threshold, the second threshold, and the third threshold are inversely proportional to the age of the child, and the calculation formula is as follows:

[0028]

[0029] Among them, y1 ratio is the first threshold, y2 ratio is the second threshold, y3 ratio is the third threshold, and n is the age of the child.

[0030] Furthermore, if the difference percentage between the two ability scores is within a second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data;

[0031] The formula for calculating the percentage difference between two ability scores is:

[0032] C = (A2-A1)A1;

[0033] Among them, C is the percentage difference between the two ability scores, A2 is the text data ability score, and A1 is the video data ability score.

[0034] The calculation formula for the weight of video data is:

[0035] β s1 =0.5+K(C-y1)(y2-y1);

[0036] Among them, β S1 is the weight of the video data, K is the proportional coefficient proportional to the age of the child, y1 is the first threshold, and y2 is the second threshold;

[0037] The proportional coefficient is calculated as follows:

[0038]

[0039] Where n is the age of the child.

[0040] Further, if the capability score of the text data is less than the capability score of the video data;

[0041] Then increase the weight of video data. The increase in the weight of video data is inversely proportional to the age of the child. The calculation formula is as follows:

[0042]

[0043] β S1 is the weight of the video data, and n is the age of the child.

[0044] A second aspect of the present invention provides a device for evaluating children's abilities based on multimodal data fusion, comprising:

[0045] a data collection module configured to collect voice data of parental communication and video data of children;

[0046] A data preprocessing module is configured to convert the voice data obtained by the data acquisition module into text data and preprocess the video data;

[0047] a data analysis module configured to input text data and pre-processed video data into a multimodal large language model;

[0048] Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the child's ability assessment results;

[0049] The result optimization module is configured to compare the difference between the ability score of the text data and the ability score of the video data, and assign different weights to them respectively.

[0050] The third aspect of the present invention provides a server, comprising at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to enable the at least one processor to execute the method described in the first aspect.

[0051] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which implements the method described in the first aspect when executed by a processor.

[0052] Compared with the prior art, the method and device for evaluating children's abilities based on multimodal data fusion described in the present invention have the following beneficial effects:

[0053] (1) The method for evaluating children's abilities based on multimodal data fusion described in the present invention collects children's video data and parents' voice data and performs evaluation through a multimodal large language model, thereby reducing the subjective bias of the evaluator. In addition, the format of the children's video data makes it easier for parents to collect the data, thereby reducing the inaccurate evaluation results caused by children's nervousness.

[0054] (2) The method for evaluating children's abilities based on multimodal data fusion described in the present invention reduces the influence of parents' subjective judgment and improves the accuracy of the results by comparing the ability scores of children's video data with the ability scores of parents' voice data and setting different thresholds. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0056] Figure 1 Schematic diagram of the evaluation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0058] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0059] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0060] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0061] Example 1:

[0062] like Figure 1 As shown, a method for evaluating children's abilities based on multimodal data fusion includes the following steps:

[0063] S1. Collect voice data of parents’ communication and video data of children;

[0064] S2, converting the voice data obtained in S1 into text data and preprocessing the video data;

[0065] In S2, voice data is converted into text data through the Whisper model;

[0066] The preprocessing of the video data includes the following steps: analyzing the video frames of the video data, extracting key frames with obvious changes, and skipping repeated or static content;

[0067] If there is audio in the video, the timestamp is used for alignment.

[0068] S3. Input the text data and the preprocessed video data into a multimodal large language model; the multimodal large language model adopts but is not limited to the existing CogVLM2 model.

[0069] S4. Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the child's ability assessment results.

[0070] The questionnaire in S4 includes background information, task description, and evaluation criteria;

[0071] Background information:

[0072] You are an intelligent model specializing in child development assessment, processing multimodal data (video frames and text) to assess children's cognition, language, motor skills, and emotions according to scientific developmental standards. Your task is to analyze the behaviors and expressions in the videos and the language in the text, combined with historical scoring records, to generate a comprehensive assessment. During each assessment, you will need to consider the current input information, the child's age group, and previous scoring records to ensure a more accurate and comprehensive assessment.

[0073] Task Description:

[0074] Your task is to assess a child's cognitive, language, motor, and social-emotional abilities based on the input video frames and text data, using the following age-based assessment criteria. You will combine the current input with historical scoring records to generate a detailed assessment report in the output format, analyzing the behaviors and expressions in the video frames and the language expressions in the text. Each output should include the original input (text and video frame analysis), combined with historical records and the child's age group for comprehensive analysis and scoring.

[0075] Evaluation criteria:

[0076] Since the evaluation criteria for children of different ages are different, the children's age information extracted from the text data is collected and the corresponding evaluation criteria are selected.

[0077] The assessment criteria include gross motor skills, fine motor skills, adaptability, language skills, and social behavior skills for each age group;

[0078] The following are assessment criteria for children of different ages.

[0079] 0.5 years old;

[0080] Gross motor skills: Able to lift head, lift head while lying on stomach, stand with support from armpits, sit up with slight pull, and turn over.

[0081] Fine motor skills: holding toys, playing with hands, grabbing nearby toys, tearing paper.

[0082] Adaptability: Eyes follow the movement of objects, can look at nearby toys, and start looking for dropped toys.

[0083] Language: Has guttural response, can pronounce simple vowels, laughs, and responds to people and sounds.

[0084] Social behavior: Smiles at familiar people, can recognize relatives, imitates games, reacts to food, and tries to feed himself.

[0085] 1 year old;

[0086] Gross motor skills: Able to sit alone, stand with support, crawl, sit up by himself, hold hands or stand alone.

[0087] Fine motor skills: Able to grasp building blocks by themselves, pinch small pills with thumb and index finger, and learn to manipulate building blocks, cups and other objects with hands.

[0088] Adaptability: changing hands with building blocks, opening the handkerchief to find building blocks, imitating patting dolls, and putting on bottle caps.

[0089] Language: Imitate pronunciation, use gestures to express simple needs, understand "no", and consciously call mom and dad.

[0090] Social behavior: Recognize common objects and relatives, take off hats, cooperate in dressing, and learn to drink water from a cup.

[0091] 1.5 years old;

[0092] Gross motor skills: walking independently and throwing a ball at will.

[0093] Fine motor skills: Start to doodle spontaneously, learn to imitate drawing and handle small objects.

[0094] Adaptability: Ability to turn pages of books, build high blocks, and place blocks into templates.

[0095] Language: Able to recognize eyes, ears, nose and mouth, able to speak 3-5 words and understand simple instructions.

[0096] Social behavior: Able to take off socks by himself, control urination and defecation during the day, and learn to eat with a spoon.

[0097] 2 years old;

[0098] Gross movements: walking on tiptoe, hopping on both feet, and climbing stairs.

[0099] Fine motor skills: Learn to thread buttonholes, zip up, etc. with your hands.

[0100] Adaptability: building tall blocks, identifying the color red, and turning pages of a book.

[0101] Language: Able to answer simple questions, speak sentences of 3-5 words, and recite simple poems or nursery rhymes.

[0102] Social behavior: Able to express personal needs, play imaginative games, and take the initiative to say hello.

[0103] 2.5 years old;

[0104] Gross motor skills: Going up and down stairs alone, standing on one leg.

[0105] Fine motor skills: imitate drawing lines and learn to button clothes.

[0106] Adaptability: recognize size, understand the difference between the number 1 and many, and assemble building block templates.

[0107] Language: Able to speak sentences of 7-10 words, answer questions, and describe the contents of pictures.

[0108] Social behavior: Begins to develop a sense of right and wrong, undress independently, and handle simple tasks.

[0109] 3 years old;

[0110] Gross movement: standing long jump, jumping with alternating feet.

[0111] Fine motor skills: imitate drawing circles and tightening screws.

[0112] Adaptability: Build tall blocks, recognize the number 3, and understand two colors.

[0113] Language: Pronunciation is clear, and one can distinguish inside from outside and express gender.

[0114] Social behavior: Learn to put on shoes, unbutton clothes, and understand feelings such as "hungry" and "cold".

[0115] 3.5 years old;

[0116] Gross movement: Alternate going up and down stairs, and hopping down the bottom step with your feet together.

[0117] Fine motor skills: Learn to use scissors and put together shapes.

[0118] Adaptability: Recognize four colors and understand the number 5.

[0119] Language: Say antonyms and identify simple shapes.

[0120] Social behavior: Learn to put on clothes and understand the principle of washing hands before meals.

[0121] 4 years old;

[0122] Gross movement: Stand on one foot for 5 seconds and jump down the last step of the stairs.

[0123] Fine motor skills: imitate drawing a square and be able to assemble according to the picture.

[0124] Adaptability: find the differences and fill in the gaps with pictures.

[0125] Language: Imitate complex sentences and understand the uses of common objects.

[0126] Social behavior: Participates in group games and can distinguish between men's and women's restrooms.

[0127] 4.5 years old;

[0128] Gross motor skills: standing on one foot for 10 seconds, walking toe-to-heel. Fine motor skills: folding paper, picking up peanuts with chopsticks.

[0129] Adaptability: Make analogies and fill in the gaps with pictures.

[0130] Language: Can rinse mouth and recognize numbers.

[0131] Social behavior: understands the concepts of morning and afternoon, and can count on fingers.

[0132] 5 years old;

[0133] Gross exercise: hopping on one foot, riding a scooter.

[0134] Fine motor skills: Able to put together shapes and cut out circles.

[0135] Adaptability: Identify differences in pictures and fill in the blanks. Language: Can say surnames and describe two round objects.

[0136] Social behavior: Knows home address and understands simple rules.

[0137] 5.5 years old;

[0138] Gross movements: catching the ball, walking backwards.

[0139] Fine Motor Skills: Write your name and cut out smooth circles.

[0140] Adaptability: Able to stand between trees and cut apples.

[0141] Language: Knows Chinese zodiac signs and can count backwards.

[0142] Social behavior: Understand the purpose of crosswalks and know that chickens can't swim.

[0143] 6 years old;

[0144] Gross movements: continuous bouncing of the ball, jumping.

[0145] Fine motor skills: assemble rectangles and copy shapes.

[0146] Adaptability: spot the differences and understand the concepts of left and right.

[0147] Language: Able to describe pictures and understand the uses of simple objects. Social Behavior: Know the concepts of the four seasons and recognize basic signs.

[0148] 6.5 years old;

[0149] Gross exercise: kicking a ball with a rope and bouncing the ball.

[0150] Fine motor skills: copying a hexagon and tying a slipknot.

[0151] Adaptability: Understands graphic analogies and knows the uses of flour.

[0152] Language: Summarize the theme of the picture and recognize the clock.

[0153] Social behavior: Can identify days of the week and read in the rain.

[0154] 7 years old;

[0155] Big movement: kicking a ball with a rope and stepping on the pedals alternately.

[0156] Fine motor skills: Learn to twist a rope and tie a slipknot.

[0157] Adaptability: Understand numerical analogies and know which animals don't have feet.

[0158] Language: Understand the necessity of vaccinations and recognize the similarities between sweaters, pants and shoes.

[0159] Social behavior: Understand how to use the emergency phone and how owls catch mice.

[0160] Expected output:

[0161] You need to generate a detailed child development assessment report covering five areas: gross motor skills, fine motor skills, adaptive skills, language skills, and social-emotional skills. Each area should be comprehensively analyzed based on the behavioral performance in the video frames and the language expression in the text, and the score should be adjusted based on historical records. The output report must include the text and video analysis of the current input, and a comprehensive evaluation based on the new input and historical records.

[0162] Different weights are assigned to each ability in the evaluation criteria, and a comprehensive ability score is calculated.

[0163] When evaluating, we not only refer to the last score, but also combine multiple historical scores as the experience value range to determine whether the current score is within a reasonable range. For example:

[0164] Adaptability Assessment:

[0165] Historical Record: Based on multiple previous assessment records, children's adaptability scores are usually between 80 and 85 (showing strong environmental adaptability and interaction skills).

[0166] Current performance: In this input, although the child was able to quickly find the dropped ball and pick it up, his reaction was slightly slow and he showed a certain degree of hesitation.

[0167] Score Adjustment: Taking into account the score range of adaptability in previous assessments, this performance is slightly lower than its average performance. The current score is 80 points, which is still within its historical experience range, but slightly lower than the previous high score.

[0168] Athletic capacity assessment:

[0169] Historical Record: In past evaluations, children's gross motor scores typically ranged from 78 to 82, reflecting good gross motor control.

[0170] Current Performance: In this input, the child's movements in picking up the ball were quick and coordinated, demonstrating superior motor responses to previous attempts.

[0171] Score Adjustment: Based on the child's historical score range in gross motor skills, this performance exceeded the previous average performance, so the score was increased to 83 points, slightly higher than his previous performance.

[0172] The weight of each ability:

[0173] This weight distribution can balance children's performance in different developmental dimensions and ensure that the assessment results can fully reflect the child's overall development status.

[0174] Big Movement (weight: 20%):

[0175] Rationale: Gross motor skills (such as walking, jumping, and running) reflect the development of a child's overall body coordination, muscle strength, and motor skills. While these abilities are fundamental to a child's daily activities, their impact on overall development is slightly smaller than that of fine motor skills and adaptive abilities, and therefore is set at 20%.

[0176] Fine motor skills (weight: 20%):

[0177] Rationale: Fine motor skills (such as fine manipulation of the hands and grasping objects) directly impact children's cognitive and operational abilities, which have a significant impact on future learning (such as writing and drawing). While the development of fine motor skills is as important as gross motor skills, their impact on children's daily lives is slightly lower than that of adaptive skills, so they are also weighted at 20%.

[0178] Adaptability (weight: 25%):

[0179] Rationale: Adaptability (such as responding to environmental changes, understanding and exploring objects) is a key indicator of children's cognitive and social functioning. It is a critical ability for children to learn and cope with new environments. Because adaptability affects children's ability to learn, interact, and cope in new environments, and is closely related to adaptation and independence in daily life, it is weighted at 25%, the highest among all indicators.

[0180] Language proficiency (weight: 20%):

[0181] Rationale: Language skills reflect a child's ability to communicate with others, understand information, and express ideas. Language development is a core component of a child's social interaction and cognitive development. While language skills are crucial, given their complementary role with fine motor skills and gross motor abilities, they are weighted at 20% to ensure balance with other core skills.

[0182] Social-emotional skills (weighting: 15%):

[0183] Rationale: Social-emotional skills reflect a child's ability to manage emotions, socialize, and interact with peers. While important, they have a slightly lower impact on overall development than other skills directly related to a child's developmental process, such as gross motor skills, fine motor skills, and language skills. Therefore, they are weighted at 15%.

[0184] In S4, the text data and video data generate two ability scores for the questionnaire respectively;

[0185] In some embodiments, the scores of each ability of the two ability scores are compared and weighted to calculate the total score;

[0186] In other embodiments, after calculating the total score, the total score is compared and weighted to obtain the total score;

[0187] If the ability score of text data is greater than the ability score of video data;

[0188] Then make the following judgment:

[0189] If the difference percentage of the two ability scores is within the first threshold range, the two ability scores are given the same weight to calculate the score of each ability;

[0190] If the difference percentage between the two ability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data;

[0191] If the difference percentages between the two ability scores are within a third threshold range, an exception report is issued, prompting the child's video data to be recollected;

[0192] The first threshold value ranges from 0 to the first threshold value, and the value range of the first threshold value is 10% to 20%;

[0193] The second threshold range is from the first threshold to the second threshold, and the value range of the second threshold is 20% to 35%;

[0194] The third threshold range is from the second threshold to the third threshold, and the value range of the third threshold is 40% to 60%;

[0195] If the percentage difference between the two ability scores is greater than a third threshold, the assessment is considered incorrect. By setting the threshold, inaccurate assessment results caused by parents' subjective judgment are effectively reduced.

[0196] The value ranges of the first threshold, the second threshold, and the third threshold are inversely proportional to the age of the child, and the calculation formula is as follows:

[0197]

[0198] Among them, y1 ratio is the first threshold, y2 ratio is the second threshold, y3 ratio is the third threshold, and n is the age of the child.

[0199] As children grow older, the accuracy of their video data increases, and the threshold can be lowered to ensure the accuracy of the evaluation results.

[0200] If the difference percentage between the two ability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data;

[0201] The formula for calculating the percentage difference between two ability scores is:

[0202] C = (A2-A1)A1;

[0203] Among them, C is the percentage difference between the two ability scores, A2 is the text data ability score, and A1 is the video data ability score.

[0204] The calculation formula for the weight of video data is:

[0205] β s1 =0.5+K(C-y1)(y2-y1);

[0206] Among them, β S1 is the weight of the video data, K is the proportional coefficient proportional to the age of the child, y1 is the first threshold, and y2 is the second threshold;

[0207] The proportional coefficient is calculated as follows:

[0208]

[0209] Where n is the child's age. As the child grows older, the accuracy of their video data increases, and the weight is increased to ensure the accuracy of the evaluation results.

[0210] If the ability score of text data is less than that of video data;

[0211] Then increase the weight of video data. The increase in the weight of video data is inversely proportional to the age of the child. The calculation formula is as follows:

[0212]

[0213] β S1 is the weight of the video data, and n is the age of the child. As the child grows older, the accuracy of their video data increases, and the weight is increased to ensure the accuracy of the evaluation results.

[0214] By collecting children's video data and parents' voice data and evaluating them through a multimodal large language model, the subjective bias of evaluators is reduced. The format of children's video data makes it easier for parents to collect, reducing inaccurate evaluation results caused by children's nervousness. Finally, by comparing the ability scores of children's video data with the ability scores of parents' voice data and setting different thresholds, the influence of parents' subjective judgment is reduced and the accuracy of the results is improved.

[0215] A device for evaluating children's abilities based on multimodal data fusion, comprising:

[0216] a data collection module configured to collect voice data of parental communication and video data of children;

[0217] A data preprocessing module is configured to convert the voice data obtained by the data acquisition module into text data and preprocess the video data;

[0218] a data analysis module configured to input text data and pre-processed video data into a multimodal large language model;

[0219] Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the child's ability assessment results;

[0220] The result optimization module is configured to compare the difference between the ability score of the text data and the ability score of the video data, and assign different weights to them respectively.

[0221] Example 2:

[0222] A server includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to enable the at least one processor to perform the method of embodiment 1.

[0223] Example 3:

[0224] A computer-readable storage medium stores a computer program, which implements the method of embodiment 1 when executed by a processor.

[0225] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

[0226] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for evaluating children's abilities based on multimodal data fusion, characterized in that: The following steps are involved: S1. Collect voice data of parents’ communication and video data of children; S2, converting the voice data obtained in S1 into text data and preprocessing the video data; S3, inputting the text data and the preprocessed video data into the multimodal large language model; S4. Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the child's ability assessment results; The questionnaire in S4 includes background information, task description, and evaluation criteria; The evaluation criteria include gross motor skills, fine motor skills, adaptability, language skills, and social behavior skills for each age group; Assign different weights to each ability in the evaluation criteria and calculate the comprehensive ability score; The text data and video data in S4 generate two ability scores for the questionnaire respectively; If the ability score of text data is greater than the ability score of video data; Then make the following judgment: If the difference percentage of the two ability scores is within the first threshold range, the two ability scores are given the same weight to calculate the score of each ability; If the difference percentage between the two ability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data; If the difference percentages between the two ability scores are within a third threshold range, an exception report is issued, prompting the user to re-collect the child's video data.

2. The method for evaluating children's abilities based on multimodal data fusion according to claim 1, characterized in that: In S2, the voice data is converted into text data through the Whisper model; The preprocessing of the video data includes the following steps: analyzing the video frames of the video data, extracting key frames with obvious changes, and skipping repeated or static content; If there is audio in the video, the timestamp is used for alignment.

3. The method for evaluating children's abilities based on multimodal data fusion according to claim 1, characterized in that: The first threshold value ranges from 0 to the first threshold value, and the value range of the first threshold value is 10% to 20%; The second threshold range is from the first threshold to the second threshold, and the value range of the second threshold is 20% to 35%; The third threshold range is from the second threshold to the third threshold, and the value range of the third threshold is 40% to 60%; If the difference percentage between the two ability scores is greater than a third threshold, the assessment is determined to be wrong.

4. The method for evaluating children's abilities based on multimodal data fusion according to claim 3, characterized in that: The value ranges of the first threshold, the second threshold, and the third threshold are inversely proportional to the age of the child, and the calculation formula is as follows: Among them, y1 is the first threshold, y2 is the second threshold, y3 is the third threshold, and n is the age of the child.

5. The method for evaluating children's abilities based on multimodal data fusion according to claim 3, characterized in that: If the difference percentage between the two ability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data; The formula for calculating the percentage difference between two ability scores is: Where C is the percentage difference between the two ability scores, A2 is the text data ability score, and A1 is the video data ability score; The calculation formula for the weight of video data is: Among them, β S1 is the weight of the video data, K is the proportional coefficient proportional to the age of the child, y1 is the first threshold, and y2 is the second threshold; The proportional coefficient is calculated as follows: Where n is the age of the child.

6. The method for evaluating children's abilities based on multimodal data fusion according to claim 3, characterized in that: If the ability score of text data is less than that of video data; Then increase the weight of video data. The increase in the weight of video data is inversely proportional to the age of the child. The calculation formula is as follows: β S1 is the weight of the video data, and n is the age of the child.

7. A device for evaluating children's abilities based on multimodal data fusion, characterized in that: include: a data collection module configured to collect voice data of parental communication and video data of children; A data preprocessing module is configured to convert the voice data obtained by the data acquisition module into text data and preprocess the video data; a data analysis module configured to input text data and pre-processed video data into a multimodal large language model; Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the child's ability assessment results; a result optimization module configured to compare the difference between the ability score of the text data and the ability score of the video data, and assign different weights to each; The questionnaire included background information, task description, and evaluation criteria; The evaluation criteria include gross motor skills, fine motor skills, adaptability, language skills, and social behavior skills for each age group; Assign different weights to each ability in the evaluation criteria and calculate the comprehensive ability score; The text data and video data each generate two ability scores for the questionnaire; If the ability score of text data is greater than the ability score of video data; Then make the following judgment: If the difference percentage of the two ability scores is within the first threshold range, the two ability scores are given the same weight to calculate the score of each ability; If the difference percentage between the two ability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data; If the difference percentages between the two ability scores are within a third threshold range, an exception report is issued, prompting the user to re-collect the child's video data.

8. A server, characterized in that: The method comprises at least one processor and a memory in communication with the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images