Child ability assessment method and device based on multi-modal data fusion

Abstract

The invention provides a children's ability assessment method and device based on multi-modal data fusion, and the method comprises the following steps: S1, collecting the voice data of parent communication, and collecting the video data of children; s2, converting the voice data obtained in S1 into text data, and preprocessing video data; s3, inputting the text data and the preprocessed video data into a multi-modal large language model; and S4, inputting a questionnaire into the multi-modal large language model, and generating a child ability evaluation result by the multi-modal large language model. The beneficial effects of the invention are that by acquiring the child video data and the parent voice data and performing evaluation through the artificial intelligence multi-mode large language model, the subjective deviation of the evaluation personnel is reduced, and the child video data form is convenient for parents to acquire and reduces the inaccurate evaluation result caused by child tension.

Description

Technical Field

[0001] The present invention belongs to the technical field of children's ability assessment, and in particular, relates to a children's ability assessment method and device based on multimodal data fusion. Background Art

[0002] During the medical consultation of children with developmental and behavioral problems, evaluators usually communicate with the children and parents and observe their behavior as an important source of data for the evaluation.

[0003] This assessment method has the following problems: First, when assessing children's abilities, parents often have different opinions. Parents' judgments are usually based on their subjective feelings in daily life. They may have high expectations for their children's performance and even ignore potential problems. For example, parents may think that their children perform well in a certain ability, but this judgment is often limited to the family or familiar environment. Second, the judgment of the evaluator is also subjective. Different evaluators have different assessment standards, resulting in low accuracy of the assessment results. Third, the current assessment method requires evaluators to observe the behavior of children. First of all, children's behavior is sporadic. When children are in a nervous state at the physical examination center, their behavior may be biased, resulting in inaccurate assessment results. Summary of the invention

[0004] In view of this, the present invention aims to propose a child ability assessment method based on multimodal data fusion, in order to solve at least one of the above-mentioned technical problems.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] The first aspect of the present invention provides a method for evaluating children's abilities based on multimodal data fusion, comprising the following steps:

[0007] S1. Collect voice data of parents’ communication and video data of children;

[0008] S2, converting the voice data obtained in S1 into text data and preprocessing the video data;

[0009] S3, inputting the text data and the preprocessed video data into the multimodal large language model;

[0010] S4. Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the children's ability assessment results.

[0011] Furthermore, in S2, the voice data is converted into text data through the Whisper model;

[0012] The preprocessing of the video data includes the following steps: analyzing the video frames of the video data, extracting key frames with obvious changes, and skipping repeated or static content;

[0013] If there is audio in the video, the timestamp is used for alignment.

[0014] Furthermore, the questionnaire in S4 includes background information, task description, and evaluation criteria;

[0015] The evaluation criteria include gross motor skills, fine motor skills, adaptability, language skills, and social behavior skills for each age group;

[0016] Different weights are assigned to each ability in the evaluation criteria, and a comprehensive ability score is calculated.

[0017] Further, the text data and the video data in S4 generate two ability scores for the questionnaire respectively;

[0018] If the ability score of the text data is greater than the ability score of the video data;

[0019] Then make the following judgment:

[0020] If the difference percentage of the two ability scores is within the first threshold range, the two ability scores are given the same weight to calculate the score of each ability;

[0021] If the difference percentage of the two ability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data;

[0022] If the difference percentages between the two ability scores are within a third threshold range, an abnormality report is issued to prompt the child's video data to be recollected;

[0023] The first threshold value ranges from 0 to the first threshold value, and the value range of the first threshold value is 10% to 20%;

[0024] The second threshold value ranges from the first threshold value to the second threshold value, and the value range of the second threshold value is 20% to 35%;

[0025] The third threshold value ranges from the second threshold value to the third threshold value, and the value range of the third threshold value is 40% to 60%;

[0026] If the difference percentage between the two ability scores is greater than a third threshold, the assessment is determined to be wrong.

[0027] Furthermore, the value ranges of the first threshold, the second threshold, and the third threshold are inversely proportional to the age of the child, and the calculation formula is as follows:

[0028]

[0029] Among them, y1 is the first threshold, y2 is the second threshold, y3 is the third threshold, and n is the age of the child.

[0030] Furthermore, if the difference percentage between the two capability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data;

[0031] The percentage difference between two competency scores is calculated as:

[0032] C = (A2-A1)A1;

[0033] Among them, C is the difference percentage between the two ability scores, A2 is the text data ability score, and A1 is the video data ability score.

[0034] The calculation formula for the weight of video data is:

[0035] β s1 =0.5+K(C-y1)(y2-y1);

[0036] Among them, β S1 is the weight of the video data, K is the proportional coefficient proportional to the age of the child, y1 is the first threshold, and y2 is the second threshold;

[0037] The proportionality coefficient is calculated as follows:

[0038]

[0039] Where n is the age of the child.

[0040] Further, if the capability score of the text data is less than the capability score of the video data;

[0041] Then increase the weight of video data, which is inversely proportional to the age of the child. The calculation formula is as follows:

[0042]

[0043] β S1 is the weight of the video data, and n is the age of the child.

[0044] A second aspect of the present invention provides a child ability assessment device based on multimodal data fusion, comprising:

[0045] A data collection module configured to collect voice data of parental communication and video data of children;

[0046] A data preprocessing module, which is configured to convert the voice data obtained by the data acquisition module into text data and preprocess the video data;

[0047] A data analysis module configured to input text data and preprocessed video data into a multimodal large language model;

[0048] Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the children's ability assessment results;

[0049] The result optimization module is configured to compare the difference between the ability score of the text data and the ability score of the video data, and assign different weights respectively.

[0050] The third aspect of the present invention provides a server, comprising at least one processor, and a memory communicatively connected to the processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the processor so that the at least one processor executes the method described in the first aspect.

[0051] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0052] Compared with the prior art, the method and device for evaluating children's ability based on multimodal data fusion described in the present invention have the following beneficial effects:

[0053] (1) The method for evaluating children's abilities based on multimodal data fusion described in the present invention collects children's video data and parents' voice data and performs evaluation through a multimodal large language model based on artificial intelligence, thereby reducing the subjective bias of the evaluator. In addition, the format of children's video data is convenient for parents to collect, thereby reducing inaccurate evaluation results caused by children's nervousness.

[0054] (2) The child ability assessment method based on multimodal data fusion described in the present invention reduces the influence of parents' subjective judgment and improves the accuracy of the results by comparing the ability scores of children's video data and the ability scores of parents' voice data and setting different thresholds. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0056] Figure 1 The figure is a flow chart of the evaluation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0058] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0059] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0060] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0061] Embodiment 1:

[0062] like Figure 1 As shown, a method for evaluating children's abilities based on multimodal data fusion includes the following steps:

[0063] S1. Collect voice data of parents’ communication and video data of children;

[0064] S2, converting the voice data obtained in S1 into text data and preprocessing the video data;

[0065] In S2, the speech data is converted into text data through the Whisper model;

[0066] The preprocessing of the video data includes the following steps: analyzing the video frames of the video data, extracting key frames with obvious changes, and skipping repeated or static content;

[0067] If there is audio in the video, the timestamp is used for alignment.

[0068] By collecting image data of captured video frames and using artificial intelligence's multimodal large language model, the image data is analyzed and image features are extracted as an important basis for judging children's abilities.

[0069] S3. Input the text data and the preprocessed video data into a multimodal large language model; the multimodal large language model adopts but is not limited to the existing CogVLM2 model.

[0070] The CogVLM2 model is an artificial intelligence model, which is a multimodal large model launched by Zhipu AI. A multimodal large model refers to an artificial intelligence model that can process and generate multiple types of data (such as images, text, sound, etc.).

[0071] The advantages of AI include: Efficient computing: AI can quickly process large amounts of data and perform complex calculations and statistical analysis.

[0072] Reducing human errors: AI can reduce errors caused by human factors, especially in highly repetitive and rule-based tasks.

[0073] Processing large amounts of data: AI can process and analyze much larger data sets than humans, helping people discover patterns and insights in the data.

[0074] Automation and intelligence: AI can automate many tasks, reduce manpower requirements, and improve work efficiency.

[0075] Creation of new services and products: AI has given rise to new services and products such as self-driving cars, smart personal assistants, etc.

[0076] S4. Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the children's ability assessment results.

[0077] The questionnaire in S4 includes background information, task description, and evaluation criteria;

[0078] Background information:

[0079] You are an intelligent model specializing in child development assessment, capable of processing multimodal data (video frames and text) and assessing children's cognition, language, movement, and emotions according to scientific development standards. Your task is to generate a comprehensive assessment by analyzing the behaviors and expressions in the video and the language performance in the text, combined with historical scoring records. During each assessment, you need to refer to the current input information, the child's age group, and previous historical scoring records to ensure a more accurate and comprehensive assessment.

[0080] Task Description:

[0081] Your task is to evaluate the cognitive, language, motor, and social-emotional abilities of children based on the image data and text data of the input video frames, combined with the following age-group evaluation criteria. You need to make an evaluation based on the current input and historical scoring records, and generate a detailed evaluation report in the output format, analyzing the behaviors and expressions in the video frames, as well as the language expressions in the text. Each output should contain the original input (text and video frame analysis) of the time, and conduct a comprehensive analysis and scoring based on historical records and the child's age group.

[0082] Evaluation criteria:

[0083] The evaluation criteria for children of different ages are different. The children's age information extracted from the text data is collected and the corresponding evaluation criteria are selected.

[0084] The assessment criteria include gross motor skills, fine motor skills, adaptability, language skills, and social behavior skills for each age group;

[0085] The following are assessment criteria for children of different ages.

[0086] 0.5 years old;

[0087] Gross movements: Able to lift head, lift head while lying on stomach, stand with support from armpits, sit up with slight pull, and turn over.

[0088] Fine motor skills: holding toys, playing with hands, grabbing toys close by, tearing paper.

[0089] Adaptability: Eyes follow the movement of objects, can focus on nearby toys, and start looking for dropped toys.

[0090] Language: Has guttural response, can pronounce simple vowels, laughs, and responds to people and sounds.

[0091] Social behavior: Smiles at familiar people, can recognize relatives, imitates games, reacts to food, and tries to feed himself.

[0092] 1 year old;

[0093] Gross motor skills: Able to sit alone, stand with support, crawl, sit up by oneself, hold hands or stand alone.

[0094] Fine motor skills: Able to grasp building blocks by themselves, pinch small pills with thumb and index finger, and learn to manipulate building blocks, cups and other objects with hands.

[0095] Adaptability: changing hands with building blocks, opening the handkerchief to find the building blocks, imitating patting dolls, and putting on bottle caps.

[0096] Language: Imitate pronunciation, use gestures to express simple needs, understand "no", and consciously call mom and dad.

[0097] Social behavior: Recognize common objects and relatives, take off hats, cooperate in dressing, and learn to drink water from a cup.

[0098] 1.5 years old;

[0099] Gross motor skills: walking independently and throwing a ball at will.

[0100] Fine motor skills: Start to doodle spontaneously, learn to imitate drawing and handle small objects.

[0101] Adaptability: Ability to turn pages of books, build high blocks, and place blocks into templates.

[0102] Language: Able to recognize eyes, ears, nose and mouth, able to speak 3-5 words, and understand simple instructions.

[0103] Social behavior: Able to take off socks by himself, control urination and defecation during the day, and learn to eat with a spoon.

[0104] 2 years old;

[0105] Gross movements: walking on tiptoe, hopping on both feet, and going up stairs.

[0106] Fine motor skills: Learn to thread buttonholes, zip up, etc. with your hands.

[0107] Adaptability: building high blocks, identifying the color red, and turning pages of a book.

[0108] Language: Able to answer simple questions, speak sentences of 3-5 words, and recite simple poems or nursery rhymes.

[0109] Social behavior: Ability to express personal needs, play imaginative games, and take the initiative to say hello.

[0110] 2.5 years old;

[0111] Gross motor skills: Going up and down stairs alone, standing on one foot.

[0112] Fine motor skills: imitate drawing lines and learn to button clothes.

[0113] Adaptability: recognize size, understand the difference between the number 1 and many, and assemble building block templates.

[0114] Language: Able to speak sentences of 7-10 words, answer questions, and describe the contents of pictures.

[0115] Social behavior: Begins to develop a sense of right and wrong, undress independently, and handle simple tasks.

[0116] 3 years old;

[0117] Gross movement: standing long jump, jumping with alternating feet.

[0118] Fine motor skills: imitate drawing circles and tightening screws.

[0119] Adaptability: Build high blocks, recognize the number 3, and understand two colors.

[0120] Language: Pronunciation is clear, able to distinguish inside from outside, and express gender.

[0121] Social behavior: learn to put on shoes, unbutton clothes, and understand feelings such as "hungry" and "cold".

[0122] 3.5 years old;

[0123] Large movement: Alternate going up and down the stairs, and hopping down the bottom step with feet together.

[0124] Fine motor skills: Learn to use scissors and put together shapes.

[0125] Adaptability: Recognize four colors and understand the number 5.

[0126] Language: Say antonyms and recognize simple shapes.

[0127] Social behavior: learn to put on clothes and understand the principle of washing hands before meals.

[0128] 4 years old;

[0129] Large movement: Stand on one foot for 5 seconds and jump down the last step of the stairs.

[0130] Fine motor skills: imitate drawing a square and be able to assemble according to the picture.

[0131] Adaptability: find the differences and fill in the gaps with pictures.

[0132] Language: Imitate complex sentences and understand the uses of common objects.

[0133] Social behavior: Can participate in group games and distinguish between men's and women's toilets.

[0134] 4.5 years old;

[0135] Gross movement: stand on one foot for 10 seconds, walk toe to heel. Fine movement: fold paper, pick up peanuts with chopsticks.

[0136] Adaptability: Make analogies and fill in the gaps with pictures.

[0137] Language: Can rinse mouth and recognize numbers.

[0138] Social behavior: understands the concepts of morning and afternoon, and can count on fingers.

[0139] 5 years old;

[0140] Gross exercise: hopping on one foot, riding a scooter.

[0141] Fine motor skills: Able to put together shapes and cut out circles.

[0142] Adaptability: Identify differences in pictures and fill in the blanks. Language: Can say last name and describe two round objects.

[0143] Social Behavior: Knows home address and understands simple rules.

[0144] 5.5 years old;

[0145] Gross movements: catching the ball, walking backwards.

[0146] Fine Motor Skills: Write your name and cut out smooth circles.

[0147] Adaptability: Able to stand between trees and cut apples.

[0148] Language: Knows the Chinese zodiac sign and can count backwards.

[0149] Social behavior: understand the purpose of crosswalks and know that chickens can't swim.

[0150] 6 years old;

[0151] Gross movements: continuous bouncing and jumping.

[0152] Fine motor skills: assemble rectangles and copy shapes.

[0153] Adaptability: spot the differences and understand left and right concepts.

[0154] Language: Able to describe the content of pictures and understand the uses of simple daily objects. Social behavior: Know the concept of the four seasons and recognize basic signs.

[0155] 6.5 years old;

[0156] Gross exercise: kicking a ball with a string, bouncing the ball.

[0157] Fine motor skills: copying hexagons and tying slipknots.

[0158] Adaptability: Understand graphic analogies and know the uses of flour.

[0159] Language: Summarize the theme of the picture and recognize the clock.

[0160] Social behavior: Can recognize days of the week and read in the rain.

[0161] 7 years old;

[0162] Large movement: kicking a ball with a rope and stepping on the pedals alternately.

[0163] Fine motor skills: Learn to flip a rope and tie a slipknot.

[0164] Adaptability: Understand numerical analogies and know which animals have no feet.

[0165] Language: Understand the necessity of vaccinations and recognize the similarities between sweaters, pants and shoes.

[0166] Social behavior: Understand how to use the emergency phone and know how owls catch mice.

[0167] Expected Output:

[0168] You need to generate a detailed child development assessment report covering five aspects: gross motor, fine motor, adaptive, language, and social-emotional. Each part should be analyzed comprehensively based on the behavioral performance in the video frame and the language expression in the text, and the score should be adjusted based on the historical records. The output report should include the text and video analysis of the current input, and make a comprehensive evaluation based on the new input and historical records.

[0169] Different weights are assigned to each ability in the evaluation criteria, and a comprehensive ability score is calculated.

[0170] When evaluating, we not only refer to the last score, but also combine multiple historical scores as the experience value range to determine whether the current score is within a reasonable range. For example:

[0171] Adaptability Assessment:

[0172] Historical Record: Based on multiple previous assessment records, children's adaptability scores are usually between 80-85 (showing strong environmental adaptability and interaction skills).

[0173] Current performance: In this input, although the child was able to quickly find the dropped ball and pick it up, his reaction was a little slow and he showed a certain degree of hesitation.

[0174] Rating adjustment: Taking into account the score range of adaptability in previous assessments, this performance is slightly lower than its average performance. The current score is 80 points, which is still within its historical experience range, but slightly lower than the previous high score.

[0175] Athletic performance assessment:

[0176] History: In past evaluations, children's gross motor scores have typically been between 78 and 82, reflecting good gross motor control.

[0177] Current Performance: In this input, the child's movements in picking up the ball were quick and coordinated, demonstrating better motor responses than before.

[0178] Score Adjustment: Based on the child's historical score range in gross motor skills, this performance exceeded the previous average performance, so the score was increased to 83 points, slightly higher than his previous performance.

[0179] The weight of each ability:

[0180] This weight distribution can balance children's performance in different developmental dimensions and ensure that the assessment results fully reflect the child's overall development status.

[0181] Big Movement (Weight: 20%):

[0182] Rationale: Gross motor skills (such as walking, jumping, running, etc.) reflect the development of children's overall body coordination, muscle strength and motor ability. Although these abilities are the basis of children's daily activities, their impact on overall development is slightly smaller than that of fine motor and adaptive abilities, so they are set at 20%.

[0183] Fine motor skills (weight: 20%):

[0184] Reason: Fine motor skills (such as fine manipulation of hands, grasping objects, etc.) directly affect children's cognitive and operational abilities, which have an important impact on future learning (such as writing, painting and other skills). The development of fine motor skills is as important as gross motor skills, but its impact on children's daily lives is slightly lower than that of adaptive ability, so the weight is also set at 20%.

[0185] Adaptability (weight: 25%):

[0186] Rationale: Adaptability (such as response to environmental changes, understanding and exploration of objects) is an important manifestation of children's cognitive and social functions. It is a key ability for children to learn and cope with new environments. Because adaptability affects children's ability to learn, interact and cope in new environments, and this ability is closely related to adaptation and independence in daily life, the weight is set at 25%, the highest among all indicators.

[0187] Language proficiency (weight: 20%):

[0188] Rationale: Language ability reflects a child's ability to communicate with others, understand information, and express ideas. Language development is a core part of a child's social interaction and cognitive development. Although language ability is very important, given that it complements fine motor skills and gross motor skills, the weight is set at 20% to ensure that it is balanced with other core abilities.

[0189] Social-emotional skills (weight: 15%):

[0190] Rationale: Social-emotional skills reflect children's ability to manage emotions, social interactions, and peer relationships. Although this ability is very important, it has a slightly lower impact on overall development than other skills directly related to children's development process (such as gross motor, fine motor, and language skills). Therefore, the weight is set at 15%.

[0191] In S4, the text data and video data generate two ability scores for the questionnaire respectively;

[0192] In some embodiments, the scores of each ability of the two ability scores are compared and weighted to calculate the total score;

[0193] In other embodiments, after calculating the total score, the total score is compared and weighted to obtain the total score;

[0194] If the ability score of the text data is greater than the ability score of the video data;

[0195] Then make the following judgment:

[0196] If the difference percentage of the two ability scores is within the first threshold range, the two ability scores are given the same weight to calculate the score of each ability;

[0197] If the difference percentage of the two ability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data;

[0198] If the difference percentages between the two ability scores are within a third threshold range, an abnormality report is issued to prompt the child's video data to be recollected;

[0199] The first threshold value ranges from 0 to the first threshold value, and the value range of the first threshold value is 10% to 20%;

[0200] The second threshold ranges from the first threshold to the second threshold, and the value range of the second threshold is 20% to 35%;

[0201] The third threshold ranges from the second threshold to the third threshold, and the value range of the third threshold is 40% to 60%;

[0202] If the difference percentage of the two ability scores is greater than the third threshold, the assessment is judged to be wrong. By setting the threshold, the inaccuracy of the assessment results caused by the subjective judgment of parents is effectively reduced.

[0203] The value ranges of the first threshold, the second threshold, and the third threshold are inversely proportional to the age of the child, and the calculation formula is as follows:

[0204]

[0205] Among them, y1 is the first threshold, y2 is the second threshold, y3 is the third threshold, and n is the age of the child.

[0206] As children grow older, the accuracy of their video data increases, and the threshold can be lowered to ensure the accuracy of the evaluation results.

[0207] If the difference percentage of the two ability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data;

[0208] The formula for calculating the percentage difference between two ability scores is:

[0209] C = (A2-A1)A1;

[0210] Among them, C is the difference percentage between the two ability scores, A2 is the text data ability score, and A1 is the video data ability score.

[0211] The calculation formula for the weight of video data is:

[0212] β s1 =0.5+K(C-y1)(y2-y1);

[0213] Among them, β S1 is the weight of the video data, K is the proportional coefficient proportional to the age of the child, y1 is the first threshold, and y2 is the second threshold;

[0214] The proportionality coefficient is calculated as follows:

[0215]

[0216] Where n is the age of the child. As the age of the child increases, the accuracy of the video data increases, and the weight is increased to ensure the accuracy of the evaluation results.

[0217] If the ability score of the text data is less than the ability score of the video data;

[0218] Then increase the weight of video data, which is inversely proportional to the age of the child. The calculation formula is as follows:

[0219]

[0220] β S1 is the weight of the video data, and n is the age of the child. As the age of the child increases, the accuracy of the video data increases, and the weight is increased to ensure the accuracy of the evaluation results.

[0221] By collecting children's video data and parents' voice data and evaluating them through an artificial intelligence multimodal large language model, the subjective bias of evaluators is reduced. The format of children's video data makes it easier for parents to collect data, reducing inaccurate evaluation results caused by children's nervousness. Finally, by comparing the ability scores of children's video data with the ability scores of parents' voice data and setting different thresholds, the impact of parents' subjective judgment is reduced and the accuracy of the results is improved.

[0222] A child ability assessment device based on multimodal data fusion, comprising:

[0223] A data collection module configured to collect voice data of parental communication and video data of children;

[0224] A data preprocessing module, which is configured to convert the voice data obtained by the data acquisition module into text data and preprocess the video data;

[0225] A data analysis module configured to input text data and preprocessed video data into a multimodal large language model;

[0226] Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the children's ability assessment results;

[0227] The result optimization module is configured to compare the difference between the ability score of the text data and the ability score of the video data, and assign different weights respectively.

[0228] Embodiment 2:

[0229] A server includes at least one processor and a memory connected to the processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes the method of the first embodiment.

[0230] Embodiment three:

[0231] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method of embodiment 1 is implemented.

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0233] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for evaluating children's abilities based on multimodal data fusion, characterized in that: The following steps are involved: S1. Collect voice data of parents’ communication and video data of children; S2, converting the voice data obtained in S1 into text data and preprocessing the video data; S3, inputting the text data and the preprocessed video data into the multimodal large language model; S4. Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the children's ability assessment results.

2. The method for evaluating children's abilities based on multimodal data fusion according to claim 1, characterized in that: In S2, the voice data is converted into text data through the Whisper model; The preprocessing of the video data includes the following steps: analyzing the video frames of the video data, extracting key frames with obvious changes, and skipping repeated or static content; If there is audio in the video, the timestamp is used for alignment.

3. The method for evaluating children's abilities based on multimodal data fusion according to claim 1, characterized in that: The questionnaire in S4 includes background information, task description, and evaluation criteria; The evaluation criteria include gross motor skills, fine motor skills, adaptability, language skills, and social behavior skills for each age group; Different weights are assigned to each ability in the evaluation criteria, and a comprehensive ability score is calculated.

4. The method for evaluating children's abilities based on multimodal data fusion according to claim 3, characterized in that: The text data and video data in S4 generate two ability scores for the questionnaire respectively; If the ability score of the text data is greater than the ability score of the video data; Then make the following judgment: If the difference percentage of the two ability scores is within the first threshold range, the two ability scores are given the same weight to calculate the score of each ability; If the difference percentage of the two ability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data; If the difference percentages between the two ability scores are within a third threshold range, an abnormality report is issued, prompting the child's video data to be recollected; The first threshold value ranges from 0 to the first threshold value, and the value range of the first threshold value is 10% to 20%; The second threshold value ranges from the first threshold value to the second threshold value, and the value range of the second threshold value is 20% to 35%; The third threshold value ranges from the second threshold value to the third threshold value, and the value range of the third threshold value is 40% to 60%; If the difference percentage between the two ability scores is greater than a third threshold, the assessment is determined to be wrong.

5. The method for evaluating children's abilities based on multimodal data fusion according to claim 4, characterized in that: The value ranges of the first threshold, the second threshold, and the third threshold are inversely proportional to the age of the child, and the calculation formula is as follows: Among them, y1 is the first threshold, y2 is the second threshold, y3 is the third threshold, and n is the age of the child.

6. The method for evaluating children's abilities based on multimodal data fusion according to claim 4, characterized in that: If the difference percentage of the two ability scores is within the second threshold range, the weight of the video data is increased, and the greater the difference, the greater the weight of the video data; The percentage difference between two competency scores is calculated as: C = (A2-A1)A1; Among them, C is the difference percentage between the two ability scores, A2 is the text data ability score, and A1 is the video data ability score; The calculation formula for the weight of video data is: β s1 =0.5+K(C-y1)(y2-y1); Among them, β S1 is the weight of the video data, K is the proportional coefficient proportional to the age of the child, y1 is the first threshold, and y2 is the second threshold; The proportionality coefficient is calculated as follows: Where n is the age of the child.

7. The method for evaluating children's abilities based on multimodal data fusion according to claim 4, characterized in that: If the ability score of the text data is less than the ability score of the video data; Then increase the weight of video data, which is inversely proportional to the age of the child. The calculation formula is as follows: β S1 is the weight of the video data, and n is the age of the child.

8. A device for evaluating children's abilities based on multimodal data fusion, characterized in that: include: A data collection module configured to collect voice data of parental communication and video data of children; A data preprocessing module, which is configured to convert the voice data obtained by the data acquisition module into text data and preprocess the video data; A data analysis module configured to input text data and preprocessed video data into a multimodal large language model; Input the questionnaire into the multimodal large language model, and the multimodal large language model generates the children's ability assessment results; The result optimization module is configured to compare the difference between the ability score of the text data and the ability score of the video data, and assign different weights respectively.

9. A server, characterized in that: It includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images