Assessment method and system for children's voice disorders based on artificial intelligence

Abstract

The present invention provides an evaluation method and system for children's voice disorders based on artificial intelligence, and relates to the technical field of voice evaluation. The method includes: obtaining a voice signal to be tested, air velocity data and a test video of a test child; dividing the voice signal to be tested, obtaining voice signal segments and their maximum amplitude and amplitude average values; obtaining the voice spectrum and the first peak frequency of the voice signal segment; obtaining a reference sound wave signal through a sound wave generation neural network model; obtaining a child's voice acoustic evaluation score according to the first peak frequency, the voice spectrum, the voice signal segment and the reference sound wave signal; determining the child's voice aerodynamic evaluation score according to the test video, air velocity data, amplitude average value and the voice signal segment, and then determining the child's voice disorder score. According to the present invention, the severity of the child's voice disorder can be comprehensively evaluated from the two aspects of acoustics and aerodynamics, and the evaluation accuracy and objectivity can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of voice assessment, and in particular to an assessment method and system for children's voice disorders based on artificial intelligence. Background Art

[0002] In the relevant technology, the layered structure of the vocal cords in children is not fully developed, and the composition of the lamina propria is different from that of adults, which makes children less tolerant and capable of repairing vocal damage than adults. In addition, children have poor control over their own behavior and are prone to crying and shouting. The above factors may lead to the occurrence of voice disorders in children. However, the characteristics of voice disorders vary greatly from person to person, making it difficult to accurately assess the severity of voice disorders.

[0003] The information disclosed in the background technology section of this application is only intended to deepen the understanding of the general background technology of this application, and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0004] The present invention provides an artificial intelligence-based method and system for evaluating children's voice disorders, which can accurately evaluate the severity of children's voice disorders.

[0005] According to a first aspect of the present invention, there is provided an artificial intelligence-based method for assessing children's voice disorders, comprising:

[0006] Receiving the pronunciation of a preset text by the test child to obtain a voice signal to be tested;

[0007] When the test child pronounces, the air flow velocity data in front of the test child's mouth is measured in real time by an air flow sensor, and a test video of the front of the test child is taken by a camera, wherein the field of view of the camera includes the mouth of the test child;

[0008] Performing time domain division on the voice signal to be tested to obtain voice signal segments corresponding to each word in the preset text;

[0009] Sampling the voice signal segment of each word to obtain the maximum amplitude and the average amplitude of the voice signal segment of each word;

[0010] Performing a fast Fourier transform on the voice signal segment corresponding to each word to obtain the voice spectrum of the voice signal segment, and determining the first peak frequency corresponding to each word;

[0011] Input the preset text, the first peak frequency, the maximum amplitude and the average amplitude of the voice signal segment of each word into the trained sound wave generation neural network model to obtain a reference sound wave signal of the preset text;

[0012] Obtaining a child's voice acoustic assessment score according to the first peak frequency, the voice spectrum, the voice signal segment of each word, and the reference sound wave signal;

[0013] Determine a child's voice aerodynamic assessment score based on the test video, the air velocity data, the amplitude average value, and the voice signal segment;

[0014] A child's voice disorder score is determined based on the child's voice acoustic assessment score and the child's voice aerodynamic assessment score.

[0015] According to a second aspect of the present invention, there is provided an evaluation system for children's voice disorders based on artificial intelligence, comprising:

[0016] A receiving module, used for receiving the pronunciation of a preset text by a test child to obtain a voice signal to be tested;

[0017] a measuring module, used for measuring the air velocity data in front of the test child's mouth in real time through an airflow sensor when the test child pronounces, and taking a test video of the front of the test child through a camera, wherein the field of view of the camera includes the mouth of the test child;

[0018] A division module, used for performing time domain division on the voice signal to be tested to obtain voice signal segments corresponding to each word in a preset text;

[0019] A sampling module is used to sample the voice signal segment of each word to obtain the maximum amplitude and the average amplitude of the voice signal segment of each word;

[0020] A peak frequency module, used for performing a fast Fourier transform on the voice signal segment corresponding to each word, obtaining the voice spectrum of the voice signal segment, and determining a first peak frequency corresponding to each word;

[0021] A reference sound wave signal module, used to input the preset text, the first peak frequency, the maximum amplitude and the average amplitude of the voice signal segment of each word into a trained sound wave generation neural network model to obtain a reference sound wave signal of the preset text;

[0022] A child voice acoustic assessment score module, configured to obtain a child voice acoustic assessment score according to the first peak frequency, the voice spectrum, the voice signal segment of each word and the reference sound wave signal;

[0023] A child voice aerodynamic assessment score module, used to determine a child voice aerodynamic assessment score based on the test video, the air velocity data, the amplitude average value and the voice signal segment;

[0024] The child voice disorder scoring module is used to determine the child voice disorder score based on the child voice acoustic assessment score and the child voice aerodynamic assessment score.

[0025] Technical effect: According to the present invention, a reference sound wave signal can be generated by a trained sound wave generation neural network model, so as to compare and evaluate the voice signal to be tested of the test child, obtain the child's voice acoustic evaluation score, and determine the child's voice aerodynamic evaluation score according to the measured data such as the video frame and air flow rate data when the test child pronounces, and then comprehensively evaluate the severity of the child's voice disorder through the child's voice acoustic evaluation score and the child's voice aerodynamic evaluation score, thereby improving the accuracy and objectivity of the evaluation. When determining the child's voice acoustic evaluation score, the relative pronunciation efficiency of the voice signal segment relative to the reference sound wave signal segment can be determined from three aspects: the amplitude decrease rate of the noise frequency, the number of noise frequencies, and the amplitude of the peak frequency, so as to comprehensively analyze the test child's voice pronunciation efficiency and objectively and accurately evaluate the possibility of the test child having a voice disorder. When determining the child's voice aerodynamic evaluation score, the pronunciation efficiency of the test child when voicing can be determined from the perspective of aerodynamics by the amplitude of the sound wave generated by the unit air flow, so as to determine the child's voice aerodynamic evaluation score, and the possibility of the test child having a voice disorder can be accurately and objectively evaluated. When training the sound wave generation neural network model, the generative adversarial training method can be used to achieve a balance between the pronunciation efficiency and authenticity of the training sound wave signals generated by the sound wave generation neural network model, so that the model has the characteristics of high pronunciation efficiency and high authenticity, and is close to healthy human voice, providing an important basis for generating accurate reference sound wave signals.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present invention. Other features and aspects of the present invention will become more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative work.

[0028] Figure 1 A schematic diagram exemplarily illustrates a flow chart of an artificial intelligence-based method for evaluating children's voice disorders according to an embodiment of the present invention;

[0029] Figure 2The block diagram of the evaluation system for children's voice disorders based on artificial intelligence according to an embodiment of the present invention is exemplarily shown. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0032] Figure 1 A flowchart of an artificial intelligence-based method for assessing children's voice disorders according to an embodiment of the present invention is exemplarily shown, the method comprising:

[0033] Step S101, receiving the pronunciation of a preset text by a test child to obtain a voice signal to be tested;

[0034] Step S102, when the test child pronounces, the air flow velocity data in front of the test child's mouth is measured in real time by an air flow sensor, and a test video of the front of the test child is captured by a camera, wherein the field of view of the camera includes the mouth of the test child;

[0035] Step S103, dividing the voice signal to be tested in the time domain to obtain voice signal segments corresponding to each word in the preset text;

[0036] Step S104, sampling the voice signal segment of each word to obtain the maximum amplitude and the average amplitude of the voice signal segment of each word;

[0037] Step S105, performing a fast Fourier transform on the voice signal segment corresponding to each word, obtaining the voice spectrum of the voice signal segment, and determining the first peak frequency corresponding to each word;

[0038] Step S106, inputting the preset text, the first peak frequency, the maximum amplitude and the average amplitude of the voice signal segment of each word into the trained sound wave generation neural network model to obtain a reference sound wave signal of the preset text;

[0039] Step S107, obtaining a child voice acoustic assessment score according to the first peak frequency, the voice spectrum, the voice signal segment of each word, and the reference sound wave signal;

[0040] Step S108, determining a child's voice aerodynamic assessment score according to the test video, the air velocity data, the amplitude average value and the voice signal segment;

[0041] Step S109, determining the child's voice disorder score according to the child's voice acoustic evaluation score and the child's voice aerodynamic evaluation score.

[0042] According to the artificial intelligence-based child voice disorder assessment method of an embodiment of the present invention, a reference sound wave signal can be generated through a trained sound wave generation neural network model, thereby comparing and evaluating the voice signal to be tested of the test child, obtaining the child's voice acoustic assessment score, and determining the child's voice aerodynamic assessment score based on measured data such as video frames and air flow rate data when the test child pronounces, and then comprehensively assessing the severity of the child's voice disorder through the child's voice acoustic assessment score and the child's voice aerodynamic assessment score, thereby improving the accuracy and objectivity of the assessment.

[0043] According to one embodiment of the present invention, in step S101, the individual differences of the tested children are large. In order to make the evaluation more universal and objective, the same preset text can be set. When the tested children are evaluated, each word in the preset text can be read out, and the voice signal to be tested can be obtained. The voice signal to be tested is a sound wave signal, which can be used for subsequent analysis and evaluation of the child's voice.

[0044] According to one embodiment of the present invention, in step S102, during the pronunciation of the test child, an airflow sensor may be arranged in front of the mouth of the test child to obtain air flow rate data in front of the test child's mouth, that is, the flow rate of the airflow exhaled by the test child when the test child pronounces. A test video of the front of the test child may be captured by a camera, and the camera may be directly facing the mouth of the test child, so as to capture the mouth shape of the test child when pronouncing and the area of ​​the mouth opening.

[0045] According to one embodiment of the present invention, in step S103, the voice signal to be tested is a sound wave signal, and the sound wave signal can be divided, that is, the sound wave signal is divided into multiple segments according to time, and each sound wave signal segment only includes the pronunciation of one word in the preset text, so as to obtain the voice signal segment corresponding to each word. Since the parameters such as the pitch, frequency, and volume of each word are different when pronouncing each word, the pronunciation of each word can be analyzed word by word through the voice signal segment, thereby improving the accuracy of the assessment of voice disorders.

[0046] According to an embodiment of the present invention, in step S104, each voice signal segment is a sound wave signal, and therefore, the amplitude of the voice signal segment can be sampled, and the maximum amplitude value is obtained from multiple sampled amplitudes, and the multiple sampled amplitudes are averaged to obtain an amplitude average value. Each voice signal segment is processed as above to obtain the maximum amplitude value and the amplitude average value of each voice signal segment.

[0047] According to an embodiment of the present invention, in step S105, a fast Fourier transform may be performed on the voice signal segment corresponding to each word to obtain a voice spectrum of the voice signal segment, and the voice spectrum may be used to describe the frequency distribution information in the voice signal segment of each word. Further, a first peak frequency corresponding to each word may be determined in the voice signal segment of each word, for example, the frequency with the highest intensity (for example, the largest amplitude) in the sound spectrum may be determined as the first peak frequency.

[0048] According to one embodiment of the present invention, in step S106, the sound wave generation neural network model is a deep learning neural network model, such as a convolutional neural network model or a recursive neural network model, etc. The present invention does not limit the specific type of the sound wave generation neural network model. The sound wave generation neural network model can generate a sound wave signal based on the input text and the setting parameters for the pronunciation of each word in the text (for example, the peak frequency, the maximum amplitude and the average amplitude of each word for pronouncing the word), and the sound wave signal is a sound wave signal for simulating the voice of a person with healthy vocal cords to read and pronounce the input text. In this step, the preset text, the first peak frequency, the maximum amplitude and the average amplitude of the voice signal segment of each word can be input into the sound wave generation neural network model, so as to obtain a reference sound wave signal for simulating the healthy voice of the vocal cord to read and pronounce the preset text, and the generated reference sound wave signal has the same peak frequency as the voice of the test child, so the reference sound wave signal can be used as the sound wave signal of the voice when the vocal cords of the test child are healthy.

[0049] According to an embodiment of the present invention, in step S107, after obtaining the reference sound wave signal, the voice signal to be tested actually emitted by the test child may be compared and evaluated based on the reference sound wave signal.

[0050] According to one embodiment of the present invention, step S107 includes: performing time domain division on the reference sound wave signal to obtain reference sound wave signal segments corresponding to each word in a preset text; performing fast Fourier transform on the reference sound wave signal segments to obtain reference sound wave spectrum of the reference sound wave signal segments; determining a second peak frequency of the reference sound wave spectrum; determining a first amplitude corresponding to the first peak frequency, and a second amplitude corresponding to the second peak frequency; setting a preset ratio, and screening out a first noise frequency having an amplitude greater than or equal to the product of the first amplitude and the preset ratio from multiple maxima of the voice spectrum; screening out a second noise frequency having an amplitude greater than or equal to the product of the second amplitude and the preset ratio from multiple maxima of the reference sound wave spectrum; and determining a child's voice acoustic assessment score based on the first peak frequency, the first amplitude, the first noise frequency, the second peak frequency, the second amplitude, and the second noise frequency.

[0051] According to one embodiment of the present invention, the reference sound wave signal segments corresponding to each word are obtained, and the reference sound wave spectrum and the second peak frequency of each reference sound wave signal segment are obtained. This is similar to the method of obtaining the voice signal segment and obtaining the voice spectrum and the first peak frequency of the voice signal segment, and will not be repeated here.

[0052] According to an embodiment of the present invention, in the voice spectrum, the first amplitude corresponding to the first peak frequency is the maximum amplitude in the voice spectrum, and similarly, the second amplitude is the maximum amplitude in the reference sound wave spectrum.

[0053] According to one embodiment of the present invention, noise frequencies that have little impact on the voice signal segment can be screened out by a preset ratio. There are multiple maximum values ​​in the voice spectrum, which can represent the amplitude of a certain noise frequency. Noise frequencies whose amplitudes are less than the product of the first amplitude and the preset ratio can be screened out, that is, noise frequencies that have little impact on the voice signal segment are screened out, thereby improving the analysis efficiency. The remaining first noise frequency has a greater impact on the voice signal segment, and the voice disorder of the child can be evaluated by analyzing the impact of the first noise frequency. Similarly, the noise frequency that has little impact on the reference sound wave signal segment in the reference sound wave spectrum can be screened out to obtain the second noise frequency.

[0054] According to one embodiment of the present invention, determining a child voice acoustic assessment score according to the first peak frequency, the first amplitude, the first noise frequency, the second peak frequency, the second amplitude and the second noise frequency includes: determining the child voice acoustic assessment score S according to formula (1): A ,

[0055]

[0056] Among them, n is the number of words in the preset text, F f,2,i is the second peak frequency of the i-th word, F f,1,i is the first peak frequency of the i-th word, A 1,i is the first amplitude of the i-th word, A 2,i is the second amplitude of the i-th word, F no,1,i,j is the jth first noise frequency of the ith word, A no,1,i,j is the amplitude corresponding to the jth first noise frequency of the ith word, F no,2,i,k is the kth second noise frequency of the ith word, A no,2,i,k is the amplitude corresponding to the kth second noise frequency of the ith word, n 1,i is the number of the first noise frequency of the voice spectrum of the i-th word, n 2,i is the number of the second noise frequency of the reference sound wave spectrum of the i-th word, j≤n 1 , k≤n 2 , i≤n, and i, j, k, n, n 1,i and n 2,i All are positive integers.

[0057] According to one embodiment of the present invention, in formula (1), is the absolute value of the slope of the connecting line between the position corresponding to the first peak frequency and the first amplitude in the throat spectrum and the position of the j-th first noise frequency and its corresponding amplitude in the throat spectrum. The larger the absolute value is, the more it represents that the energy of the sound wave of the voice signal segment is concentrated on the first peak frequency, and at other frequencies, the amplitude decreases faster, that is, the energy of the signal with the first noise frequency is smaller. In other words, the energy of the sound wave is mainly used for normal phonation, the energy consumed by noise is smaller, and the pronunciation efficiency is higher. is the average value of the absolute value of the above slope, which can represent the average amplitude decrease rate of the noise frequency in the voice spectrum, and can also be used to represent the concentration of energy in the voice spectrum at the first peak frequency. It can be used to indicate the average amplitude decrease rate of the noise frequency in the reference sound wave spectrum, and can also be used to indicate the concentration of energy in the reference sound wave spectrum at the second peak frequency. It is the relative energy concentration of the test child's voice signal segment relative to the reference sound wave signal segment of vocal cord health. The higher the relative energy concentration, the closer the energy concentration of the test child's voice is to the healthy level, and the higher the pronunciation efficiency is.

[0058] According to one embodiment of the present invention, It is the ratio of the second noise frequency to the first noise frequency. Since the vocalization of the test child usually has more noise relative to the reference sound wave signal segment, the ratio is usually less than 1. The closer the ratio is to 1, the closer the number of the first noise frequency is to the number of the second noise frequency, which means that the energy concentration of the test child's voice signal segment is closer to the reference sound wave signal and the pronunciation efficiency is higher.

[0059] According to one embodiment of the present invention, is the ratio between the first amplitude and the second amplitude. Since the reference sound wave signal segment is generated based on the maximum amplitude and the average amplitude of the voice signal segment, the total energy of the sound wave of the voice signal segment is equal to the total amount of the sound wave of the voice signal segment. In this case, the higher the amplitude concentrated on the peak frequency, the more the energy of the sound wave is concentrated on the peak frequency, and the higher the pronunciation efficiency. Therefore, this ratio can also represent the relative energy concentration of the voice signal segment of the test child relative to the reference sound wave signal segment with healthy vocal cords, and can also represent the pronunciation efficiency.

[0060] According to one embodiment of the present invention, the three items that can represent the pronunciation efficiency are multiplied to obtain the relative pronunciation efficiency of the voice signal segment relative to the reference sound wave signal segment obtained from the acoustic perspective, and the average value of the relative pronunciation efficiency of each voice signal segment is solved as the child's voice acoustic evaluation score. The higher the child's voice acoustic evaluation score, the higher the voice pronunciation efficiency of the test child obtained from the acoustic perspective, the healthier the vocal cords, and the lower the possibility of voice disorders.

[0061] In this way, the relative pronunciation efficiency of the voice signal segment relative to the reference sound wave signal segment can be determined from three aspects: the amplitude decrease rate of the noise frequency, the number of noise frequencies, and the amplitude of the peak frequency. This can comprehensively analyze the voice pronunciation efficiency of the test child and objectively and accurately evaluate the possibility of the test child having a voice disorder.

[0062] According to an embodiment of the present invention, in step S108, in addition to analyzing the possibility of the child having a voice disorder from an acoustic perspective as described above, the possibility of the child having a voice disorder may also be analyzed from an aerodynamic perspective.

[0063] According to one embodiment of the present invention, step S108 includes: determining the pronunciation start time and pronunciation end time of each word according to the voice signal segment corresponding to each word; determining the pronunciation start video frame and pronunciation end video frame of each word in the test video according to the pronunciation start time and the pronunciation end time; sampling multiple video frames in the pronunciation start video frame and the pronunciation end video frame to obtain the sampled video frame corresponding to each word, and the timestamp of the sampled video frame; determining the air flow rate data corresponding to the sampled video frame according to the timestamp of the sampled video frame; determining the child's voice aerodynamic assessment score according to the sampled video frame, the air flow rate data corresponding to the sampled video frame and the amplitude average.

[0064] According to one embodiment of the present invention, the test video and the voice signal to be tested can be recorded simultaneously, so that the start time of the voice signal segment (i.e., the pronunciation start time of a word) is the same as the pronunciation start time of the corresponding word in the test video, and the end time of the voice signal segment (i.e., the pronunciation end time of a word) is the same as the pronunciation end time of the corresponding word in the test video. Based on this, the pronunciation start video frame and the pronunciation end video frame of each word can be determined in the test video.

[0065] According to an embodiment of the present invention, there may be multiple video frames between the pronunciation start video frame and the pronunciation end video frame of a word, and these video frames may be sampled, for example, uniformly sampled at a preset time interval, and the sampled video frames corresponding to each word may be obtained, and the sampled video frames include the pronunciation start video frame and the pronunciation end video frame of the word. In addition, the timestamp of the sampled video frame may be recorded.

[0066] According to an embodiment of the present invention, the air flow sensor can measure the air flow rate data in real time. After obtaining the above timestamp, the air flow rate data obtained at the same time as the time represented by the timestamp can be determined. The air flow rate data is the air flow rate data obtained at the same time as the sampled video frame. Based on this method, the air flow rate data obtained at the same time as each sampled video frame can be determined.

[0067] According to one embodiment of the present invention, a child's voice aerodynamic assessment score is determined based on the sampled video frames, the air flow velocity data corresponding to the sampled video frames and the amplitude average value, including: determining the mouth area of ​​the test child in the sampled video frames; determining the actual mouth opening area corresponding to each sampled video frame based on the mouth area and the external parameters of the camera; determining the air flow data corresponding to each sampled video frame based on the actual mouth opening area corresponding to each sampled video frame and the air flow velocity data corresponding to each sampled video frame; determining the child's voice aerodynamic assessment score based on the air flow data corresponding to each sampled video frame and the amplitude average value.

[0068] According to one embodiment of the present invention, a plurality of image recognition methods can be used to identify the area where the mouth is located in the sampled video frame, and then based on the external parameters of the camera and the positions of the various points in the area where the mouth is located in the sampled video frame, the corresponding positions of the various points in the area where the mouth is located in the real world are determined, and then based on the relative position relationship between the corresponding positions, the area of ​​the area where the mouth is located, that is, the actual mouth opening area, is determined, which is the cross-sectional area of ​​the path through which air can flow out of the mouth. Further, based on the actual mouth opening area and the air flow rate data corresponding to the sampled video frame (that is, the air flow rate data obtained at the same time as the sampled video frame), the air flow data corresponding to each sampled video frame is determined, that is, the actual mouth opening area is multiplied by the corresponding air flow rate data to determine the air flow data at the moment represented by the timestamp of the sampled video frame.

[0069] According to one embodiment of the present invention, the child voice aerodynamic assessment score is determined based on the air flow data corresponding to each sampled video frame and the amplitude average value, including: determining the child voice aerodynamic assessment score S according to formula (2): D ,

[0070]

[0071] in, is the average amplitude corresponding to the i-th word, R i,s is the air flow data corresponding to the sth sampled video frame of the ith word, n i is the number of sampled video frames of the i-th word, E p is the preset efficiency, n is the number of words in the preset text, s≤n i , i≤n, and i, s, n and n i All are positive integers.

[0072] According to one embodiment of the present invention, is the average value of the air flow data corresponding to multiple sampled video frames of the i-th word, that is, the average air flow in front of the mouth when the i-th word is pronounced, It can represent the amplitude of the sound wave generated by the unit air flow, and can also represent the actual pronunciation efficiency when pronouncing the i-th word. Therefore, It is the ratio of actual pronunciation efficiency to preset efficiency, i.e., relative pronunciation efficiency. The relative pronunciation efficiency of each word is averaged to obtain the child's voice aerodynamic assessment score, which can describe the test child's voice pronunciation efficiency from an aerodynamic perspective. The higher the child's voice aerodynamic assessment score, the higher the test child's voice pronunciation efficiency obtained from aerodynamic analysis, the healthier the vocal cords, and the lower the possibility of voice disorders.

[0073] In this way, the pronunciation efficiency of the test child when speaking can be determined from an aerodynamic perspective by measuring the amplitude of the sound waves generated by the unit air flow, thereby determining the child's voice aerodynamic assessment score, which can accurately and objectively assess the possibility of voice disorders in the test child.

[0074] According to one embodiment of the present invention, in step S109, the child voice acoustic evaluation score and the child voice aerodynamic evaluation score may be weighted and summed to obtain the child voice disorder score. The child voice disorder score may combine the two perspectives of acoustics and aerodynamics to more comprehensively evaluate the pronunciation efficiency of the tested child and the possibility of having a voice disorder, that is, the higher the child voice disorder score, the higher the pronunciation efficiency of the tested child and the lower the possibility of having a voice disorder.

[0075] According to one embodiment of the present invention, the above sound wave generation neural network model can be trained before use. The training steps of the sound wave generation neural network model include: inputting the training text, the peak frequency data of each word in the training text, the amplitude average value and the amplitude maximum value into the sound wave generation neural network model to obtain the training sound wave signal; according to the training text, the peak frequency data of each word in the training text, the amplitude average value and the amplitude maximum value, obtain the theoretical sound wave signal; inputting the training sound wave signal and the trainer's vocal signal of the training text into the human voice discrimination model to obtain the authenticity discrimination result of the training sound wave signal and the vocal signal; according to the training sound wave signal, the theoretical sound wave signal and the authenticity discrimination result, determine the comprehensive loss function of the human voice discrimination model and the sound wave generation neural network model; according to the comprehensive loss function, the human voice discrimination model and the sound wave generation neural network model are subjected to generative adversarial training to obtain the trained sound wave generation neural network model and the trained human voice discrimination model.

[0076] According to one embodiment of the present invention, the training personnel may be those who have undergone medical examinations to determine that their vocal cords are healthy, and may read aloud each word of the training text to obtain a vocalization signal. In addition, the peak frequency data, the average amplitude, and the maximum amplitude of each word in the vocalization signal may also be determined. The determination method of these parameters is similar to the method of obtaining the first peak frequency, the average amplitude, and the maximum amplitude of each word of the voice signal to be tested above, and will not be repeated here.

[0077] According to one embodiment of the present invention, the training text, the peak frequency data, the amplitude average value and the amplitude maximum value of each word can be input into the sound wave generation neural network model to obtain the training sound wave signal. The training sound wave signal is a sound wave signal that simulates a healthy human voice. The pronunciation efficiency of the sound wave signal of a healthy human voice is high, but there is still a certain amount of noise. On the other hand, the training text, the peak frequency data, the amplitude average value and the amplitude maximum value of each word in the training text can also be used to generate a theoretical sound wave signal. For example, a sound wave signal that is completely pronounced using the peak frequency can be generated, that is, only the peak frequency is used for the pronunciation of a word. There is only a difference in amplitude during the pronunciation process, and there is no noise frequency at all. Although the pronunciation efficiency of the theoretical sound wave signal has reached the maximum value, it does not conform to medical laws and does not really exist in reality. It is only used in the process of training the sound wave generation neural network model.

[0078] According to one embodiment of the present invention, in the process of training using theoretical sound wave signals, in addition to maintaining a high pronunciation efficiency of the training sound wave signals generated by the sound wave generation neural network model, it is also necessary to improve the authenticity of the training sound wave signals, that is, to make them closer to the real human voice, rather than directly making the training sound wave signals close to the theoretical sound wave signals. In other words, instead of making the sound wave generation neural network model generate a completely noise-free sound wave signal, reasonable noise is retained to make the training sound wave signals more authentic.

[0079] According to one embodiment of the present invention, in order to achieve the above-mentioned purpose, a human voice discrimination model can be used for auxiliary training. The human voice discrimination model can discriminate whether the sound wave signal of the data is a real human voice or a machine-generated sound, thereby outputting the authenticity discrimination result, for example, a result in the form of probability. Using the human voice discrimination model, the authenticity of the training sound wave signal can be discriminated after the training sound wave signal is generated by the sound wave generation neural network model. If the authenticity is low, the parameters of the sound wave generation neural network model can be adjusted during the training process, so that the sound wave generation neural network model not only improves the pronunciation efficiency of the training sound wave signal, but also improves the authenticity of the training sound wave signal. In this way, in the process of repeated training, the human voice discrimination model and the sound wave generation neural network model can reach a balanced state, and in the process of continuously improving the discrimination accuracy of the human voice discrimination model, the sound wave generation neural network model is prompted to improve the authenticity of the training sound wave signal, and in the process of adjusting the parameters of the sound wave generation neural network model itself, the pronunciation efficiency of the training sound wave signal is improved.

[0080] According to one embodiment of the present invention, based on the training sound wave signal, the theoretical sound wave signal and the authenticity discrimination result, a comprehensive loss function of a human voice discrimination model and a sound wave generation neural network model is determined, including: performing time domain division on the training sound wave signal to determine the training sound wave segment corresponding to each word in the training text; performing time domain division on the theoretical sound wave signal to determine the theoretical sound wave segment corresponding to each word in the training text; performing fast Fourier transform on the training sound wave segment to obtain a training spectrum of the training sound wave segment; performing fast Fourier transform on the theoretical sound wave segment to obtain a theoretical spectrum of the theoretical sound wave segment; determining the training spectrum , a first training amplitude corresponding to the peak frequency data, and determining a first theoretical amplitude corresponding to the peak frequency data in a theoretical spectrum; setting a preset ratio, and screening out a first training noise frequency with an amplitude greater than or equal to the product of the first training amplitude and the preset ratio from multiple maximum values ​​of the training spectrum; determining a first training noise amplitude corresponding to the first training noise frequency; determining a comprehensive loss function of a human voice discrimination model and a sound wave generation neural network model according to the first training amplitude, the first training noise frequency, the first training noise amplitude, the first theoretical amplitude and the authenticity discrimination result.

[0081] According to one embodiment of the present invention, the process of obtaining the first training amplitude and the first theoretical amplitude is similar to the above process of obtaining the first amplitude, and the process of obtaining the first training noise frequency is similar to the above process of obtaining the first noise frequency, which will not be repeated here.

[0082] According to an embodiment of the present invention, determining the comprehensive loss function of the human voice discrimination model and the sound wave generation neural network model according to the first training amplitude, the first training noise frequency, the first training noise amplitude, the first theoretical amplitude and the authenticity discrimination result includes: determining the comprehensive loss function LOSS of the human voice discrimination model and the sound wave generation neural network model according to formula (3),

[0083]

[0084] Among them, A x,1,th is the first theoretical amplitude of the theoretical sound wave fragment of the xth word, A x,1,tr is the first training amplitude of the training sound wave segment of the xth word, F no,1,x,y is the yth first training noise frequency of the training sound wave segment of the xth word, A no,1,x,y is the first training noise amplitude corresponding to the yth first training noise frequency of the training sound wave segment of the xth word, F f,x is the peak frequency data of the xth word, n 1,x is the number of the first training noise frequency of the training sound wave segment of the xth word, N is the number of words in the training text, and p th To obtain the authenticity discrimination result by inputting the training sound wave signal into the human voice discrimination model, p vo The authenticity judgment result obtained by inputting the vocal signal into the human voice discrimination model, α and β are preset weights, y≤n 1,x , x≤N, and x, y, N and n 1,x All are positive integers.

[0085] According to one embodiment of the present invention, in formula (3), the first term of the comprehensive loss function can be used to adjust the parameters of the sound wave generation neural network model to improve the pronunciation efficiency of the training sound wave signal output by it, wherein, is the average amplitude decrease rate of the noise frequency in the training spectrum, which can represent the pronunciation efficiency of the training sound wave segment. The average amplitude decrease rate is used as the denominator of the fraction in the first term. As the training process progresses, the loss function decreases, the fraction in the first term decreases, and the average amplitude decrease rate increases, that is, the pronunciation efficiency will gradually increase. The numerator of the fraction in the first term is That is, the relative deviation between the first training amplitude and the first theoretical amplitude, as the training process progresses, the loss function decreases, the fraction in the first term decreases, then the relative deviation decreases, the first training amplitude gradually approaches the first theoretical amplitude, and the pronunciation efficiency will gradually improve. Averaging the above fractions, we get As the first term of the comprehensive loss function.

[0086] According to one embodiment of the present invention, the second term of the comprehensive loss function is used to improve the authenticity of the training sound wave signal output by the sound wave generation neural network model. vo ) is a function term obtained based on the authenticity discrimination result obtained by inputting the vocal signal into the human voice discrimination model. In the training process of improving the accuracy of the human voice discrimination model, the authenticity discrimination result can be gradually increased, thereby reducing the term in the loss function. log(1-p th ) is a function term obtained based on the authenticity discrimination result obtained by inputting the training sound wave signal into the human voice discrimination model. In the process of improving the authenticity of the training sound wave signal output by the sound wave generation neural network model, the authenticity discrimination result can be gradually increased, thereby reducing the term in the loss function.

[0087] According to one embodiment of the present invention, the first item and the second item may be weighted and summed to obtain a comprehensive loss function. During the training process, the comprehensive loss function may be reduced.

[0088] After the above training process, the comprehensive loss function can be reduced during the training process, that is, the first and second items are reduced, thereby improving the accuracy of the human voice discrimination model while prompting the sound wave generation neural network model to improve the authenticity of the output training sound wave signal, thereby improving the probability that the human voice discrimination model will discriminate the training sound wave signal as a real human voice, that is, increasing the authenticity discrimination result of the training sound wave signal. At the same time, the pronunciation efficiency of the training sound wave signal generated by the sound wave generation neural network model can also be improved. After multiple rounds of training, the pronunciation efficiency and authenticity of the training sound wave signal generated by the sound wave generation neural network model can be balanced, so that the reference sound wave signal generated by the trained neural network model is close to a healthy human voice.

[0089] In this way, through generative adversarial training, the pronunciation efficiency and authenticity of the training sound wave signals generated by the sound wave generation neural network model can be balanced, so that it has the characteristics of high pronunciation efficiency and high authenticity, and is close to healthy human voice, providing an important basis for generating accurate reference sound wave signals.

[0090] According to the artificial intelligence-based evaluation method for children's voice disorders of an embodiment of the present invention, a reference sound wave signal can be generated through a trained sound wave generation neural network model, thereby comparing and evaluating the voice signal to be tested of the test child, obtaining the child's voice acoustic evaluation score, and determining the child's voice aerodynamic evaluation score based on measured data such as the video frame and air velocity data when the test child pronounces, and then comprehensively evaluating the severity of the child's voice disorder through the child's voice acoustic evaluation score and the child's voice aerodynamic evaluation score, thereby improving the accuracy and objectivity of the evaluation. When determining the child's voice acoustic evaluation score, the relative pronunciation efficiency of the voice signal segment relative to the reference sound wave signal segment can be determined from three aspects: the amplitude decrease rate of the noise frequency, the number of noise frequencies, and the amplitude of the peak frequency, thereby comprehensively analyzing the test child's voice pronunciation efficiency and objectively and accurately evaluating the possibility of the test child having a voice disorder. When determining the aerodynamic assessment score of a child's voice, the amplitude of the sound waves generated by the unit air flow can be used to determine the pronunciation efficiency of the test child from an aerodynamic perspective, thereby determining the aerodynamic assessment score of the child's voice, which can accurately and objectively assess the possibility of the test child having a voice disorder. When training the sound wave generation neural network model, the sound wave generation neural network model can be trained through adversarial training to achieve a balance between the pronunciation efficiency and authenticity of the training sound wave signal, so that it has the characteristics of high pronunciation efficiency and high authenticity, and is close to healthy human voice, providing an important basis for generating accurate reference sound wave signals.

[0091] Figure 2 A block diagram of an evaluation system for children's voice disorders based on artificial intelligence according to an embodiment of the present invention is exemplarily shown, the system comprising:

[0092] A receiving module, used for receiving the pronunciation of a preset text by a test child to obtain a voice signal to be tested;

[0093] a measuring module, used for measuring the air velocity data in front of the test child's mouth in real time through an airflow sensor when the test child pronounces, and taking a test video of the front of the test child through a camera, wherein the field of view of the camera includes the mouth of the test child;

[0094] A division module, used for performing time domain division on the voice signal to be tested to obtain voice signal segments corresponding to each word in a preset text;

[0095] A sampling module is used to sample the voice signal segment of each word to obtain the maximum amplitude and the average amplitude of the voice signal segment of each word;

[0096] A peak frequency module, used for performing a fast Fourier transform on the voice signal segment corresponding to each word, obtaining the voice spectrum of the voice signal segment, and determining a first peak frequency corresponding to each word;

[0097] A reference sound wave signal module, used to input the preset text, the first peak frequency, the maximum amplitude and the average amplitude of the voice signal segment of each word into a trained sound wave generation neural network model to obtain a reference sound wave signal of the preset text;

[0098] A child voice acoustic assessment score module, configured to obtain a child voice acoustic assessment score according to the first peak frequency, the voice spectrum, the voice signal segment of each word and the reference sound wave signal;

[0099] A child voice aerodynamic assessment score module, used to determine a child voice aerodynamic assessment score based on the test video, the air velocity data, the amplitude average value and the voice signal segment;

[0100] The child voice disorder scoring module is used to determine the child voice disorder score based on the child voice acoustic assessment score and the child voice aerodynamic assessment score.

[0101] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An artificial intelligence-based method for assessing children's voice disorders, characterized in that: include: Receiving the pronunciation of a preset text by the test child to obtain a voice signal to be tested; When the test child pronounces, the air flow velocity data in front of the test child's mouth is measured in real time by an air flow sensor, and a test video of the front of the test child is taken by a camera, wherein the field of view of the camera includes the mouth of the test child; Performing time domain division on the voice signal to be tested to obtain voice signal segments corresponding to each word in the preset text; Sampling the voice signal segment of each word to obtain the maximum amplitude and the average amplitude of the voice signal segment of each word; Performing a fast Fourier transform on the voice signal segment corresponding to each word to obtain the voice spectrum of the voice signal segment, and determining the first peak frequency corresponding to each word; Input the preset text, the first peak frequency, the maximum amplitude and the average amplitude of the voice signal segment of each word into the trained sound wave generation neural network model to obtain a reference sound wave signal of the preset text; Obtaining a child's voice acoustic assessment score according to the first peak frequency, the voice spectrum, the voice signal segment of each word, and the reference sound wave signal; Determine a child's voice aerodynamic assessment score based on the test video, the air velocity data, the amplitude average value, and the voice signal segment; A child's voice disorder score is determined based on the child's voice acoustic assessment score and the child's voice aerodynamic assessment score.

2. The method for evaluating children's voice disorders based on artificial intelligence according to claim 1, characterized in that: Obtaining a child's voice acoustic assessment score according to the first peak frequency, the voice spectrum, the voice signal segment of each word, and the reference sound wave signal, including: Performing time domain division on the reference sound wave signal to obtain reference sound wave signal segments corresponding to each word in the preset text; Performing a fast Fourier transform on the reference sound wave signal segment to obtain a reference sound wave spectrum of the reference sound wave signal segment; determining a second peak frequency of the reference sound wave spectrum; determining a first amplitude corresponding to the first peak frequency and a second amplitude corresponding to the second peak frequency; Setting a preset ratio, and screening out a first noise frequency having an amplitude greater than or equal to a product of a first amplitude and the preset ratio from a plurality of maximum values ​​of the voice spectrum; Screening out a second noise frequency having an amplitude greater than or equal to a product of the second amplitude and the preset ratio from among a plurality of maximum values ​​of the reference sound wave spectrum; A child voice acoustic assessment score is determined based on the first peak frequency, the first amplitude, the first noise frequency, the second peak frequency, the second amplitude, and the second noise frequency.

3. The method for evaluating children's voice disorders based on artificial intelligence according to claim 2, characterized in that: Determining a child voice acoustic assessment score based on the first peak frequency, the first amplitude, the first noise frequency, the second peak frequency, the second amplitude, and the second noise frequency comprises: According to the formula Determine the acoustic assessment score S for children's voice A , where n is the number of words in the preset text, F f,2,i is the second peak frequency of the i-th word, F f,1,i is the first peak frequency of the i-th word, A 1,i is the first amplitude of the i-th word, A 2,i is the second amplitude of the i-th word, F no,1,i,j is the jth first noise frequency of the ith word, A no,1,i,j is the amplitude corresponding to the jth first noise frequency of the ith word, F no,2,i,k is the kth second noise frequency of the ith word, A no,2,i,k is the amplitude corresponding to the kth second noise frequency of the ith word, n 1,i is the number of the first noise frequency of the voice spectrum of the i-th word, n 2,i is the number of the second noise frequencies of the reference sound wave spectrum of the i-th word, j≤n1, k≤n2, i≤n, and i, j, k, n, n 1,i and n 2,i All are positive integers.

4. The method for evaluating children's voice disorders based on artificial intelligence according to claim 1, characterized in that: Determining a child's voice aerodynamic assessment score according to the test video, the air velocity data, the amplitude average value, and the voice signal segment, including: Determine the pronunciation start time and pronunciation end time of each word according to the voice signal segments corresponding to each word; Determine a pronunciation start video frame and a pronunciation end video frame of each word in the test video according to the pronunciation start time and the pronunciation end time; Sampling a plurality of video frames in the pronunciation start video frame and the pronunciation end video frame to obtain a sampled video frame corresponding to each word and a timestamp of the sampled video frame; determining air velocity data corresponding to the sampled video frame according to the timestamp of the sampled video frame; A child voice aerodynamic assessment score is determined based on the sampled video frames, the air velocity data corresponding to the sampled video frames, and the amplitude average value.

5. The method for evaluating children's voice disorders based on artificial intelligence according to claim 4, characterized in that: Determining a child's voice aerodynamic assessment score according to the sampled video frame, the air velocity data corresponding to the sampled video frame, and the amplitude average value, comprising: Determining the area where the mouth of the test child is located in the sampled video frame; Determine the actual mouth opening area corresponding to each sampled video frame according to the area where the mouth is located and the external parameters of the camera; The actual opening area corresponding to each sampled video frame and the air flow rate data corresponding to each sampled video frame are used to determine the air flow rate data corresponding to each sampled video frame; A child voice aerodynamic assessment score is determined based on the air flow data corresponding to each sampled video frame and the amplitude average.

6. The method for evaluating children's voice disorders based on artificial intelligence according to claim 5, characterized in that: Determine a child voice aerodynamic assessment score based on the air flow data corresponding to each sampled video frame and the amplitude average value, including: According to the formula Determine the child's voice aerodynamic assessment score S D ,in, is the average amplitude corresponding to the i-th word, R i,s is the air flow data corresponding to the sth sampled video frame of the ith word, n i is the number of sampled video frames of the i-th word, E p is the preset efficiency, n is the number of words in the preset text, s≤n i , i≤n, and i, s, n and n i All are positive integers.

7. The method for evaluating children's voice disorders based on artificial intelligence according to claim 1, characterized in that: The training steps for the sound wave generation neural network model include: Inputting the training text, the peak frequency data of each word in the training text, the amplitude average value and the amplitude maximum value into the sound wave generation neural network model to obtain a training sound wave signal; Obtaining a theoretical sound wave signal according to the training text, the peak frequency data of each word in the training text, the amplitude average value and the amplitude maximum value; Input the training sound wave signal and the vocalization signal of the training personnel to the training text into a human voice discrimination model to obtain authenticity discrimination results of the training sound wave signal and the vocalization signal; Determine a comprehensive loss function of a human voice discrimination model and a sound wave generation neural network model according to the training sound wave signal, the theoretical sound wave signal and the authenticity discrimination result; Generative adversarial training is performed on the human voice discrimination model and the sound wave generation neural network model according to the comprehensive loss function to obtain a trained sound wave generation neural network model and a trained human voice discrimination model.

8. The method for evaluating children's voice disorders based on artificial intelligence according to claim 7, characterized in that: Determining the comprehensive loss function of the human voice discrimination model and the sound wave generation neural network model according to the training sound wave signal, the theoretical sound wave signal and the authenticity discrimination result includes: Performing time domain division on the training sound wave signal to determine the training sound wave segment corresponding to each word in the training text; Divide the theoretical sound wave signal in the time domain to determine the theoretical sound wave segment corresponding to each word in the training text; Performing fast Fourier transform on the training sound wave segment to obtain a training spectrum of the training sound wave segment; Performing fast Fourier transform on the theoretical sound wave fragment to obtain the theoretical spectrum of the theoretical sound wave fragment; Determine a first training amplitude corresponding to the peak frequency data in the training spectrum, and determine a first theoretical amplitude corresponding to the peak frequency data in the theoretical spectrum; Setting a preset ratio, and screening out a first training noise frequency having an amplitude greater than or equal to a product of a first training amplitude and the preset ratio from a plurality of maximum values ​​of the training spectrum; determining a first training noise amplitude corresponding to the first training noise frequency; A comprehensive loss function of a human voice discrimination model and a sound wave generation neural network model is determined according to the first training amplitude, the first training noise frequency, the first training noise amplitude, the first theoretical amplitude and the authenticity discrimination result.

9. The method for evaluating children's voice disorders based on artificial intelligence according to claim 8, characterized in that: Determining a comprehensive loss function of a human voice discrimination model and a sound wave generation neural network model according to the first training amplitude, the first training noise frequency, the first training noise amplitude, the first theoretical amplitude, and the authenticity discrimination result includes: According to the formula Determine the comprehensive loss function LOSS of the human voice discrimination model and the sound wave generation neural network model, where A x,1,th is the first theoretical amplitude of the theoretical sound wave fragment of the xth word, A x,1,tr is the first training amplitude of the training sound wave segment of the xth word, F no,1,x,y is the yth first training noise frequency of the training sound wave segment of the xth word, A no,1,x,y is the first training noise amplitude corresponding to the yth first training noise frequency of the training sound wave segment of the xth word, F f,x is the peak frequency data of the xth word, n 1,x is the number of the first training noise frequency of the training sound wave segment of the xth word, N is the number of words in the training text, and p th To obtain the authenticity discrimination result by inputting the training sound wave signal into the human voice discrimination model, p vo The authenticity discrimination result obtained by inputting the vocal signal into the human voice discrimination model, α and β are preset weights, y≤n 1,x , x≤N, and x, y, N and n 1,x All are positive integers.

10. An artificial intelligence-based assessment system for children's voice disorders, characterized in that: include: A receiving module, used for receiving the pronunciation of a preset text by a test child to obtain a voice signal to be tested; a measuring module, used for measuring the air velocity data in front of the test child's mouth in real time through an airflow sensor when the test child pronounces, and taking a test video of the front of the test child through a camera, wherein the field of view of the camera includes the mouth of the test child; A division module, used for performing time domain division on the voice signal to be tested to obtain voice signal segments corresponding to each word in a preset text; A sampling module is used to sample the voice signal segment of each word to obtain the maximum amplitude and the average amplitude of the voice signal segment of each word; A peak frequency module, used for performing a fast Fourier transform on the voice signal segment corresponding to each word, obtaining the voice spectrum of the voice signal segment, and determining a first peak frequency corresponding to each word; A reference sound wave signal module, used to input the preset text, the first peak frequency, the maximum amplitude and the average amplitude of the voice signal segment of each word into a trained sound wave generation neural network model to obtain a reference sound wave signal of the preset text; A child voice acoustic assessment score module, configured to obtain a child voice acoustic assessment score according to the first peak frequency, the voice spectrum, the voice signal segment of each word and the reference sound wave signal; A child voice aerodynamic assessment score module, used to determine a child voice aerodynamic assessment score based on the test video, the air velocity data, the amplitude average value and the voice signal segment; The child voice disorder scoring module is used to determine the child voice disorder score based on the child voice acoustic assessment score and the child voice aerodynamic assessment score.

Images