Video image segmentation classification network and tongue movement ultrasonic video segmentation classification method

Abstract

The application provides a video image segmentation and classification network and a tongue movement ultrasonic video segmentation and classification method.The video image segmentation and classification network comprises a segmentation module and a classification module.The segmentation module comprises a multi-layer U-Net neural network in which each layer of convolution layers is replaced by a multi-branch group level modular attention mechanism segmentation module.The segmentation module is used to output a segmentation result and a multi-view same-dimension feature map group based on a to-be-tested video image.The classification module identifies the multi-view same-dimension feature map group to output a classification result.The video image segmentation and classification network can restore the feature map group to the same size of the original video image, the image classification accuracy is high, the prediction accuracy of the output result is effectively improved, and the video image segmentation and classification network can be widely applied to tongue movement ultrasonic image trajectory prediction and other modalities and segmentation and classification of medical video images of lesions, so as to assist doctors in more accurate diagnosis and improve the efficiency of related disease diagnosis and treatment scheme selection.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to video image segmentation and classification networks and a method for segmenting and classifying tongue-movement ultrasound videos. Background Technology

[0002] Language is a natural form of communication, and incorrect pronunciation can lead to misunderstandings. To pronounce a complete word or sentence, one needs to correctly represent a complex combination of phonemes. Phonemes are generally composed of vowels and consonants.

[0003] Language disorders are among the most common childhood illnesses, and early diagnosis and supportive training during the language learning period can greatly help children integrate into normal life. Visual biofeedback is crucial for speech therapists, allowing patients to correct their pronunciation by comparing their own tongue movement trajectories with those of the therapist. Therefore, segmenting tongue movement trajectories is particularly essential in visual biofeedback. Traditionally, language learners learn the pronunciation of a new word solely through hearing. New research indicates that visual feedback technologies, such as ultrasound imaging, can help people learn new languages ​​more efficiently.

[0004] Due to the performance, affordability, non-invasiveness, and portability of real-time ultrasound systems, it has become one of the most popular imaging methods in many clinical applications. When applied to the tongue, ultrasound imaging effectively addresses issues such as tongue visualization, tongue trajectory tracking, and numerous educational surveys. Furthermore, these technologies can assist speech therapists in implementing pronunciation rehabilitation training for many individuals with speech disorders.

[0005] In existing technologies, tongue movement ultrasound images have the characteristics of low contrast, difficulty in distinguishing ROI (Region of Interest) from the morphological features of the image, random direction and speed of tongue movement, and the ability to use the spatiotemporal semantic information of the context to assist in the segmentation of future frames when combined into a video. Therefore, if the classic model of tongue movement ultrasound images predicts the tongue movement ultrasound images separately and then stitches them into a video, edge jitter will be found, which will greatly affect the video presentation quality. Summary of the Invention

[0006] To overcome the shortcomings of the existing technology, this invention utilizes a multi-branch group-level modular attention segmentation mechanism in the feature extraction stage. By assigning different weights to different channels in the channel dimension, it extracts the channel dimension information of video images more accurately, resulting in more accurate and coherent images. A video image segmentation and classification network is proposed, comprising: The segmentation module comprises a multi-layer U-Net neural network in which each convolutional layer is replaced by a multi-branch group-level modular attention mechanism segmentation module. The segmentation module is used to output segmentation results and multi-view, same-dimensional feature maps based on the ultrasound video image to be tested. The U-Net neural network has a symmetrical encoder-decoder structure, and the attention segmentation module is obtained by adding a multi-branch group-level modular attention mechanism during feature extraction. The classification module is a downsampling part of the symmetric structure of the U-Net neural network. It is used to identify multi-view feature maps of the same dimension based on the parameters of the U-Net downsampling part trained during segmentation training, so as to output classification results.

[0007] Specifically, the classification module includes: The stitching unit is used to stitch together feature maps of the same dimension from multiple perspectives to obtain a multi-view fused feature map set; A dimensionality reduction unit is used to reduce the dimensionality of the multi-view fused feature map group. A global average pooling unit is used to perform global average pooling on the dimensionality-reduced multi-view fused feature map group to output the classification result.

[0008] Preferably, the multi-view feature map group of the same dimension includes a feature map group of the first view and a feature map group of the same dimension; the first view and the second view are perpendicular to each other.

[0009] This invention also proposes a method for segmenting and classifying ultrasound videos of tongue movements, the method comprising: The ultrasound video image to be tested is input into the segmentation module, which obtains the segmentation result and a set of feature maps of the same dimension from multiple perspectives. The ultrasound video image to be tested includes an image obtained from a tongue movement ultrasound video obtained through ultrasound imaging. The segmentation module includes a multi-layer U-Net neural network in which the convolutional layer is replaced by an attention segmentation module. The encoder-decoder structure of the U-Net neural network is a symmetrical structure. The attention segmentation module is obtained by adding a multi-branch group-level modular attention mechanism during feature extraction. The same-dimensional feature map group from multiple perspectives is fused to obtain a multi-view fused feature map group. The multi-view fused group is then dimensionality reduced, and global average pooling is performed on the dimensionality-reduced multi-view fused feature map group to output the classification result. Based on the segmentation results and the classification results, a prediction is made to obtain the tongue movement trajectory image and pronunciation problem classification of the source of the ultrasound video image to be tested.

[0010] Preferably, the multi-view feature map group of the same dimension includes a feature map group of the first view and a feature map group of the same dimension; the first view and the second view are perpendicular to each other.

[0011] Preferably, the ultrasound video images to be tested include multiple ultrasound video images of tongue movement acquired from the same object, with a time axis set.

[0012] Furthermore, the method also includes: The system displays a comparison image of the user's own tongue movement trajectory with a standard movement trajectory image to assist the user in making corrections.

[0013] Optionally, the standard motion trajectory image includes a tongue motion trajectory image obtained by using an ultrasound image of the speech therapist's tongue movement as the ultrasound video image to be tested.

[0014] Furthermore, the "obtaining segmentation results through the segmentation module" includes: The segmentation module obtains a segmented image based on the ultrasound video image to be tested, and scans the entire area of ​​the segmented image by moving the scan lines from the first column direction to the second column direction. If the scan line passes through at least two segmented regions simultaneously during the scanning process, the segmented region with the largest area is retained, and the other segmented regions are deleted to obtain the segmentation result.

[0015] Preferably, the method further includes: The number of connected regions in a frame is determined. If it is greater than or equal to 2, a skeleton line is extracted for each connected region. Hermitian interpolation is then performed using the points on the skeleton line, and the tongue movement image is re-segmented using the connected curve trajectories. This ensures that the obtained segmentation results are consistent and consistent.

[0016] The present invention has at least the following beneficial effects: The image segmentation and classification network proposed in this invention can perform segmentation and classification simultaneously and effectively. By replacing the convolutional layers of the neural network with an attention splitting mechanism, it can obtain image information from different channels. Through a symmetrical encoder-decoder structure, the image segmentation and classification network can restore the ultrasound video image under test to the same feature map group size as the original image after the feature encoding and feature decoding stages, thereby improving the accuracy of the segmentation module. At the same time, the classification model is trained using a multi-view fusion method, which improves the accuracy of the classification module and thus improves the prediction accuracy of the output results. Furthermore, the image segmentation and classification network proposed in this invention can be applied to medical image segmentation of various modalities and lesions to assist in diagnosis and treatment. For example, when using the tongue movement ultrasound video segmentation and classification method proposed in this invention, the contrast of tongue movement ultrasound images can be enhanced, the video presentation quality is high, and it can effectively help speech therapists achieve pronunciation rehabilitation training for many people with speech disorders. Preferably, the tongue movement ultrasound video segmentation and classification method proposed in this invention can also avoid the occurrence of abnormal segmentation regions due to the low resolution of ultrasound images and the randomization of tongue movements, thus ensuring the accuracy of the images.

[0017] Therefore, this invention provides a video image segmentation and classification network and a method for segmenting and classifying tongue movement ultrasound videos. The video image segmentation and classification network proposed in this invention can restore feature map groups to the same size as the original video image, with high image classification accuracy, effectively improving the prediction accuracy of the output results. It can be widely applied to the segmentation and classification of medical video images of modalities and lesions, such as tongue movement ultrasound image trajectory prediction, to assist doctors in making more accurate diagnoses and improve the efficiency of diagnosis and treatment plan selection for related diseases. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the module structure of the video image segmentation and classification network provided in Example 1; Figure 2 This is a schematic diagram of the overall network structure of the video image segmentation and classification network provided in Example 1; Figure 3 A network structure that replaces simple convolutional layers for the attention segmentation module; Figure 4 A network structure for attention segmentation; Figure 5 This is a flowchart illustrating the tongue movement ultrasound video segmentation and classification method provided in Example 2. Figure 6(a) is a schematic diagram of the segmented image obtained by the segmentation module based on the ultrasound video image under test, and Figure 6(b) is a schematic diagram of the segmented image after processing.

[0020] Figure label: 1-Segmentation module; 2-Classification module; 21-Segmentation unit; 22-Dimensionality reduction unit; 23-Global average pooling unit. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] Various embodiments of the invention will be described more fully below. The invention may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the invention to the specific embodiments disclosed herein, but rather the invention should be understood to cover all modifications, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of the invention.

[0023] In the following, the terms “comprising” or “may include” as used in various embodiments of the invention indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of the invention, the terms “comprising,” “having,” and their cognates are intended only to indicate a specific feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of the foregoing.

[0024] In various embodiments of the invention, the expression "or" or "at least one of A and / or B" includes any combination or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A and / or B" may include A, may include B, or may include both A and B.

[0025] The expressions used in the various embodiments of the present invention (such as "first," "second," etc.) may modify various constituent elements in the various embodiments, but do not limit the corresponding constituent elements. For example, the above expressions do not limit the order and / or importance of the elements. The above expressions are only used for the purpose of distinguishing one element from other elements. For example, a first user device and a second user device refer to different user devices, although both are user devices. For example, a first element may be referred to as a second element without departing from the scope of the various embodiments of the present invention, and similarly, a second element may also be referred to as a first element.

[0026] It should be noted that, in this invention, unless otherwise explicitly specified and defined, terms such as "installation," "connection," and "fixation" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0027] In this invention, those skilled in the art should understand that the terms indicating orientation or positional relationship in the text are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the purpose of facilitating the description of this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0028] The terminology used in the various embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments of the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. The terms (such as those defined in a generally used dictionary) are to be interpreted as having the same meaning as in the context of the relevant technical field and are not to be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0029] Example 1 This embodiment proposes a video image segmentation and classification network, which is based on attention segmentation mechanism and multimodal fusion technology. On the basis of U-Net model, attention splitting mechanism is added to the feature extraction process. Experiments show that attention splitting mechanism can effectively extract image information from different channels, and can achieve better performance in classification and segmentation tasks while accelerating training.

[0030] Please refer to Figure 1 The video image segmentation and classification network proposed in this embodiment includes a segmentation module 1 and a classification module 2; The segmentation module 1 comprises a multi-layer U-Net neural network in which each convolutional layer is replaced by a multi-branch group-level modular attention mechanism segmentation module. The segmentation module is used to output segmentation results and multi-view feature maps of the same dimension based on the video image to be tested. The U-Net neural network has a symmetrical encoder-decoder structure, and the attention segmentation module is obtained by adding a multi-branch group-level modular attention mechanism during feature extraction. In this embodiment, the U-Net neural network includes a 3D U-Net neural network. Classification module 2 is the downsampling part of the symmetric structure of the U-Net neural network. It is used to identify multi-view feature maps of the same dimension based on the parameters of the U-Net downsampling part trained during segmentation training, so as to output the classification result. The classification module includes a fully connected layer. During classification training, a transfer learning strategy is adopted, and the parameters of the U-Net downsampling part remain unchanged to train the parameters of the fully connected layer.

[0031] It should be noted that by using the attention-based segmentation mechanism to replace the simple convolutional layers in each layer of the U-Net neural network with the attention segmentation module, image information on different channels can be obtained. By setting the encoder-decoder structure of the U-Net neural network to a symmetrical structure, the ultrasound video image under test can be restored to the same feature map group size as the original image after the feature encoding and feature decoding stages, thereby improving the accuracy of the segmentation module.

[0032] Specifically, classification module 2 includes: The stitching unit 21 is used to stitch together feature maps of the same dimension from multiple perspectives to obtain a multi-view fused feature map set; Dimensionality reduction unit 22 is used to reduce the dimensionality of multi-view fused feature map groups; Global average pooling unit 23 is used to perform global average pooling on the dimensionality-reduced multi-view fused feature map group to output classification results.

[0033] It should be noted that global average pooling is essentially averaging the values ​​of all pixels in each channel image to obtain a new 1. Channel diagram of 1.

[0034] In this embodiment, the feature map group of the same dimension from multiple perspectives includes the feature map group of the first perspective and the feature map group of the second perspective, which are of the same dimension. The segmentation module outputs the feature map group of the first perspective and the feature map group of the second perspective based on the image to be detected. The first perspective and the second perspective are perpendicular to each other. The classification module's stitching unit stitches together the feature maps from the first and second perspectives to obtain a dual-view fusion map. The dimensionality reduction unit reduces the dimensionality of the dual-view fusion map, and the global average pooling unit performs global average pooling on the dimensionality-reduced dual-view fusion map to output the classification result.

[0035] Specifically, the dimensionality reduction unit includes a convolutional layer; in this embodiment, the dimensionality reduction unit includes 1... A convolutional layer of 1, through 1 The 1 convolutional layer reduces the dimensionality of the multi-view fused feature map group, and can change the number of channels of the multi-view fused feature map group without changing the size of the multi-view fused feature map group.

[0036] See Figure 2 , Figure 2 In this embodiment, (a) is the main 5-layer U-Net segmentation branch, and (b) is the classification branch. The segmentation module 1 includes a main 5-layer U-Net neural network in which each convolutional layer is replaced by an attention segmentation module. In the feature decoding stage of the U-Net neural network, the feature maps of the same layer are spliced ​​in the channel dimension through skip connections. The numbers above the square indicate the number of channels, the numbers on both sides indicate the size of the feature map groups, arrow 1 indicates the attention segmentation module, arrow 2 indicates the skip connection, arrow 3 indicates the downsampling convolution module, arrow 4 indicates the corresponding upsampling convolution module, arrow 5 indicates the 1x1 convolutional layer, and arrow 6 indicates global average pooling.

[0037] See Figure 3 , Figure 3 This paper demonstrates a network structure that replaces the simple convolutional layer with an attention segmentation module. The input is split into k groups in the channel dimension using the idea of ​​grouped convolution, namely 1-k groups (cardinal 1-k). Within each group (cardinal), the channel dimension is further split into 1-r units (radix 1-r). The weights corresponding to each unit (radix) are calculated, and finally, a weighted sum is performed.

[0038] See Figure 4 Channel-attention provides a network mechanism for capturing the intrinsic relationships between features. This is achieved by introducing a soft attention mechanism to allocate weights between feature channels. The weight coefficients can be obtained by referring to [reference needed]. Figure 3 First, the features of the radix group are summed, then global average pooling is performed to obtain a vector with the same dimensions as a single radix group. Then, two sets of 1... The first convolutional layer assigns weights; specifically, the output dimension of the first convolutional layer is max(in_channels). radix / / reduction_factor,32); Wherein, reduction_factor is the scaling factor, which is set to 4 in this embodiment to reduce the number of parameters; the output dimension of the second group of convolutions is channels. The radix maintains the correspondence with the dimension of the input feature layer of the unit (radix). To ensure that the weights of the feature layer splits between unit groups (radix groups) are independently distributed, r-softmax is used to calculate the softmax function for the weights of each unit group (radix group) to obtain attention. Finally, the feature layer corresponding to each group is multiplied by the attention coefficients and then summed.

[0039] Example 2 This embodiment proposes a method for segmenting and classifying tongue movement ultrasound videos. It is implemented using a video image segmentation and classification network, which includes a segmentation module and a classification module. Please refer to [reference needed]. Figure 5 The method includes: S100: Input the ultrasound video image to be tested into the segmentation module, and obtain the segmentation result and a set of feature maps of the same dimension from multiple perspectives through the segmentation module; the ultrasound video image to be tested includes the ultrasound image of tongue movement through ultrasound imaging.

[0040] Specifically, the segmentation module includes a multi-layer U-Net neural network in which the convolutional layers are replaced with attention segmentation modules. The segmentation module is used to output segmentation results and multi-view feature maps of the same dimension based on the ultrasound video image to be tested. The encoder-decoder structure of the U-Net neural network is a symmetrical structure. The attention segmentation module is obtained by a multi-branch group-level modular attention mechanism during feature extraction. In this embodiment, the U-Net neural network includes a 3D U-Net neural network.

[0041] S200: Fuse the same-dimensional feature map groups from multiple perspectives to obtain a multi-view fused feature map group. Reduce the dimensionality of the multi-view fused feature map group and perform global average pooling on the dimensionality-reduced multi-view fused feature map group to output the classification result.

[0042] It should be noted that step S200 is completed by the classification module. Global average pooling is to calculate the average value of all pixel values ​​in each channel image and then obtain a new 1. Channel diagram of 1.

[0043] Optionally, the multi-view feature map set of the same dimension includes a feature map set of the first view and a feature map set of the second view in the same dimension. The segmentation module outputs the feature map set of the first view and the feature map set of the second view based on the image to be detected. The first view and the second view are perpendicular to each other. The classification module concatenates the feature maps from the first and second perspectives to obtain a dual-view fusion map. The dual-view fusion map is then dimensionality-reduced, and global average pooling is performed on the dimensionality-reduced dual-view fusion map to output the classification result.

[0044] In this embodiment, the first viewpoint and the second viewpoint are the horizontal and vertical viewpoints relative to a fixed two-dimensional image plane.

[0045] S300: Based on the segmentation and classification results, prediction is made to obtain the tongue movement trajectory image and pronunciation problem classification of the source of the ultrasound video image to be tested.

[0046] Preferably, the method further includes: S400: Displays a comparison between the user's own tongue movement trajectory image and a standard movement trajectory image to assist the user in making corrections based on the comparison results.

[0047] Classic image segmentation and classification models only perform a single function. Furthermore, because these models are designed to process two-dimensional data, individually predicting each frame of the tongue movement ultrasound image and then stitching them together into a video results in edge jitter, significantly impacting video quality. Therefore, in this embodiment, the ultrasound video image to be tested includes multiple tongue movement ultrasound images acquired from the same object, each with a set time axis. By setting a time axis for the ultrasound video image to be tested, the accuracy of the tongue movement trajectory can be effectively improved, and it also facilitates users comparing their own tongue movement trajectory images with standard movement trajectory images.

[0048] It should be noted that the video image segmentation and classification network can complete the segmentation and classification tasks in three-dimensional spacetime. It can record tongue movement ultrasound as video during data acquisition and process it into three-dimensional data. That is, a time axis T is added to the two-dimensional image, so that each frame of two-dimensional image is stacked into three-dimensional data.

[0049] Specifically, the standard motion trajectory image may include a pre-set motion trajectory image, or a tongue motion trajectory image obtained by inputting a tongue movement ultrasound image with no abnormalities in the tongue motion trajectory into a video image segmentation and classification network as the ultrasound video image to be tested. For example, the ultrasound video image to be tested may include an ultrasound image of the patient's tongue movement acquired through ultrasound imaging, and the standard motion trajectory image may include an ultrasound image of the speech therapist's tongue movement acquired through ultrasound imaging. This is used as the ultrasound video image to be tested and input into a video image segmentation and classification network to obtain the tongue motion trajectory image. By comparing the segmentation and classification results obtained from the patient and speech therapist respectively using the ultrasound video image to be tested as inputs into the segmentation and classification network, accurate tongue movement prediction probabilities can be output. This also helps patients use visual biofeedback to correct their pronunciation based on the difference between their own tongue motion trajectory and the speech therapist's tongue motion trajectory.

[0050] The segmentation results obtained by the segmentation module may contain abnormal segmentation regions due to the low resolution of ultrasound images and the randomization of tongue movements, leading to inaccurate segmentation results. This affects the tongue movement prediction probability and the effectiveness of patients using visual biofeedback to correct pronunciation. To solve this problem, preferably, the steps for the segmentation module to obtain the segmentation results include: The segmentation module obtains a segmented image based on the ultrasound video image to be tested. The scanning line moves from the first column direction to the second column direction to scan the entire area of ​​the segmented image. If the scanning line passes through at least two segmented areas at the same time during the scanning process, the segmented area with the largest area is retained and the other segmented areas are deleted to obtain the segmentation result.

[0051] For specific processing results, please refer to Figure 6(a) and Figure 6(b).

[0052] In one specific embodiment, this can be achieved by acquiring ultrasound images of the patient's tongue movements for the same vowel.

[0053] In this embodiment, the scan line is a vertical scan line with a first direction to the left and a second direction to the right.

[0054] Furthermore, to prevent a situation where a segmented image has at least two segmented regions, but the scan line does not simultaneously cross at least two segmented regions during the scanning process, the method further includes: The number of connected regions in a frame is determined. If it is greater than or equal to 2, a skeleton line is extracted for each connected region. Hermitian interpolation is then performed using the points on the skeleton line, and the tongue movement image is re-segmented using the connected curve trajectories. This ensures that the obtained segmentation results are consistent and consistent.

[0055] Furthermore, in this embodiment, the points within the retained segmented region are fitted using a univariate quartic function to obtain the tongue motion trajectory image from the source of the ultrasound video image to be tested.

[0056] The method proposed in this embodiment was tested on a tongue movement ultrasound image dataset and showed good feasibility. The comparison results with commonly used medical image segmentation methods are shown in Tables 1-3. Three evaluation metrics were used to evaluate the segmentation results, including Dice similarity coefficient (DSC), class pixel accuracy (CPA), and intersection-union ratio (IOU).

[0057] Table 1. Segmentation results of U-Net, ResU-Net, KiU-Net, and SplitAtten-UNet networks from a horizontal perspective.

[0058] Table 2. Segmentation results of U-Net, ResU-Net, KiU-Net, and SplitAtten-UNet networks from a vertical perspective.

[0059] Table 3. Classification results of dual-view fusion U-Net, ResU-Net, KiU-Net, and SplitAtten-UNet networks

[0060] In the classification results, Acc represents Accuracy, Pre represents Precision, Recall represents Recall, and Spe represents Specificity.

[0061] In summary, this invention provides a video image segmentation and classification network and a method for segmenting and classifying tongue movement ultrasound videos. The image segmentation and classification network proposed in this invention can restore feature image groups to the same size as the original video image, achieving high image classification accuracy and effectively improving the prediction accuracy of the output results. It can be widely applied to the segmentation and classification of medical video images of modalities and lesions, such as tongue movement ultrasound image trajectory prediction, to assist doctors in making more accurate diagnoses and improve the efficiency of diagnosis and treatment plan selection for related diseases.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for segmenting and classifying ultrasound video of tongue movement, characterized in that, The method includes: The ultrasound video image to be tested is input into the segmentation module, which obtains the segmentation result and a set of feature maps of the same dimension from multiple perspectives. The ultrasound video image to be tested includes multiple ultrasound video images of tongue movement acquired from the same object with a set time axis, obtained through ultrasound imaging of tongue movement. The segmentation module includes a multi-layer U-Net neural network in which the convolutional layers are replaced with attention segmentation modules. The encoder-decoder structure of the U-Net neural network is a symmetrical structure. The attention segmentation module is obtained by adding a multi-branch group-level modular attention mechanism during feature extraction. The set of feature maps of the same dimension from multiple perspectives includes a set of feature maps of the first perspective and a set of feature maps of the second perspective, which are perpendicular to each other. The same-dimensional feature map group from multiple perspectives is fused to obtain a multi-view fused feature map group. The multi-view fused group is then dimensionality reduced, and global average pooling is performed on the dimensionality-reduced multi-view fused feature map group to output the classification result. Based on the segmentation results and the classification results, a prediction is made to obtain the tongue movement trajectory image and pronunciation problem classification of the source of the ultrasound video image to be tested; The segmentation result obtained through the segmentation module includes: The segmentation module obtains a segmented image based on the ultrasound video image to be tested, and uses scan lines to move from the first column direction to the second column direction to scan the entire area of ​​the segmented image; If the scan line passes through at least two segmented regions simultaneously during the scanning process, the segmented region with the largest area is retained, and the other segmented regions are deleted to obtain the segmentation result.

2. The tongue movement ultrasound video segmentation and classification method according to claim 1, characterized in that, The method further includes: The system displays a comparison image of the user's own tongue movement trajectory with a standard movement trajectory image to assist the user in making corrections.

3. The tongue movement ultrasound video segmentation and classification method according to claim 2, characterized in that, The standard motion trajectory image includes a tongue motion trajectory image obtained by using an ultrasound image of the speech therapist's tongue movement as the ultrasound video image to be tested.

4. The tongue movement ultrasound video segmentation and classification method according to claim 1, characterized in that, The method further includes: Determine the number of connected regions in a frame of image. If it is greater than or equal to 2, extract the skeleton line for each connected region, perform Hermit interpolation using the points on the skeleton line, and re-segment the tongue movement image through the connected curve trajectory to ensure that the obtained segmentation result has continuity and uniformity.

Images