Ultrasonic operation intelligent training method and equipment based on multiple modes

Abstract

The invention relates to the technical field of medical simulation training, in particular to an ultrasonic operation intelligent training method and device based on multiple modalities, and the method comprises the steps: obtaining real-time six-degree-of-freedom pose data of an ultrasonic probe held by an operator, and generating an ultrasonic image in real time through a first deep learning model in cooperation with scene parameters; wherein the first deep learning model is trained to learn and establish a continuous mapping relation from an ultrasonic probe pose space to an ultrasonic image space; performing section classification on the ultrasound image by using a second deep learning model, and determining deviation information for a non-standard section; and based on the real-time six-degree-of-freedom pose data and a division result of a preset standard section and an ultrasonic probe pose, generating visual guide information for guiding an operator to adjust the probe pose, and displaying the visual guide information. In this way, the problems that an existing virtual training image is discontinuous and guiding is inaccurate are solved, and meanwhile the training efficiency and the reality sense of ultrasonic operation are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of medical simulation training technology, specifically to a multimodal intelligent training method and device for ultrasound operation. Background Technology

[0002] Prenatal ultrasound screening is a key technology for preventing birth defects, and its core lies in the operator's ability to accurately obtain standard cross-sections of the fetus. However, the training of qualified ultrasound physicians is lengthy and costly, and highly dependent on practical operation on real pregnant women, leading to a general scarcity of high-quality training resources.

[0003] To address this challenge, virtual ultrasound training systems have emerged. Currently, existing virtual training systems can be broadly categorized into two types. The first type is physical model-based simulation systems, which generate ultrasound images by simulating the propagation and reflection of ultrasound waves within a digital human body model. While the images generated by these systems are physically realistic, their computational process is extremely complex, resulting in rendering times of hundreds of milliseconds for a single frame. This falls far short of meeting the real-time interactivity required for ultrasound scanning, leading to a significant difference in operational feel compared to actual scanning and severely impacting the training effectiveness of the operator's hand-eye coordination.

[0004] The second type is image database-based retrieval systems. These systems pre-collect and store a large number of "probe pose-ultrasound image" data pairs. During training, based on the pose of the simulated probe currently held by the operator, they retrieve and display the closest image frame from the database. While this method has a fast response speed, its fundamental drawback is that the six degrees of freedom pose of the probe constitute a vast continuous space, which no finite database can fully cover. When the probe pose is in a "blind zone" not captured by the database, the system displays discontinuous jumps in the images, severely damaging the realism and smoothness of the scanning operation, and failing to help learners effectively train continuous and smooth scanning techniques.

[0005] In addition, some auxiliary systems exist that can analyze real-time acquired ultrasound images and provide operational guidance. These systems can identify probe posture, determine whether the current image is within the target standard section, and generate guidance information superimposed on the image to guide the operator. However, these systems require the operator to use real ultrasound equipment on a real patient, and they do not solve the core problem of virtual training: how to generate continuous, realistic ultrasound images that respond to different clinical scenarios (such as different fetal positions and gestational ages) in a simulated environment without real patients. Furthermore, the guidance function of these systems often remains at the level of qualitative directional indication, lacking the ability to accurately quantify operational deviations and provide specific corrective instructions. Therefore, current technology still cannot provide a virtual training solution that can both generate continuous, realistic images and provide precise quantitative guidance. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a multimodal intelligent training method and device for ultrasound operation, so as to overcome the problems of discontinuous virtual training images and inaccurate guidance in the current virtual training.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, this application provides a multimodal intelligent training method for ultrasound manipulation, comprising: Acquire real-time six-degree-of-freedom pose data of the ultrasonic probe held by the operator; Based on the real-time six-degree-of-freedom pose data and preset scene parameters, an ultrasound image corresponding to the real-time six-degree-of-freedom pose data and scene parameters is generated in real time using a first deep learning model; wherein, the first deep learning model is trained to learn and establish a continuous mapping relationship from the ultrasound probe pose space to the ultrasound image space. The ultrasound images generated by the first deep learning model are intelligently evaluated using a second deep learning model. The intelligent evaluation includes section classification and, when the section classification result is a non-standard section, determining the deviation information between the current probe pose and the first target standard section. Based on real-time six-degree-of-freedom pose data and the division results of preset standard sections and ultrasound probe pose, visual guidance information is generated to guide the operator to adjust the probe pose, and the visual guidance information is superimposed on the ultrasound image.

[0008] Furthermore, in some embodiments of this application, the real-time six-degree-of-freedom pose data includes three-dimensional spatial coordinates and three-axis rotation angles; The scenario parameters include the mother's gestational age, fetal position type, amniotic fluid volume, the mother's BMI index, and placental location.

[0009] Furthermore, in some embodiments of this application, the first deep learning model includes an encoder module, a condition generator module, and a decoder module; The encoder module is used to extract spatiotemporal features from real-time six-degree-of-freedom pose data based on a temporal convolutional network architecture, and to determine the motion trend of the ultrasound probe and the operator's intention based on the spatiotemporal features; and to output a trajectory semantic vector, which includes the current probe pose, the motion trend, and the operator's intention. The condition generator module is used to determine the fine-grained information of the fetus based on the scene parameters, so as to construct the fetal skeletal topology through a graph convolutional network; and to generate a pose-scene joint semantic code by combining the scene constraints corresponding to the current probe pose. The decoder module is used to determine the positional boundary information of organs in the ultrasound image based on the output of the condition generator module and the multi-scale U-shaped convolutional network, and to generate texture details, and then fuse the positional boundary information and texture details to generate the ultrasound image.

[0010] Furthermore, in some embodiments of this application, the process by which the decoder generates texture details includes: Using a pre-defined ultrasonic physical characteristic modeling layer, ultrasonic physical phenomenon information is generated based on the beam propagation principle of ultrasonic imaging; and ultrasonic artifact information is generated by calculating the echo intensity of different tissue interfaces through a differentiable acoustic simulation module; wherein, the ultrasonic physical phenomenon information includes beam angle, tissue attenuation and multiple reflection information, and the ultrasonic artifact information includes reverberation artifact, sound shadow and enhancement effect information.

[0011] Furthermore, in some embodiments of this application, the section classification of the ultrasound image generated by the first deep learning model using the second deep learning model includes: The feature extraction and fusion module extracts features from ultrasound images and fuses them to obtain fused features; The fused features are mapped to a preset number of dimensional output spaces using a fully connected layer, and the probability distribution of each dimension is output through a softmax function. The dimension corresponds to one non-standard facet type and 11 standard facets. The feature extraction and fusion module includes a local scale extraction branch, an intermediate scale extraction branch, a global scale extraction branch, and an attention fusion module. The feature extraction and fusion module extracts features from the ultrasound image and fuses them to obtain fused features. This includes: extracting fine features of anatomical landmarks from the ultrasound image through the local scale extraction branch, extracting organ-level structural features from the ultrasound image through the intermediate scale extraction branch, and extracting overall layout features from the ultrasound image through the global scale extraction branch; and fusing the fine features of the anatomical landmarks, organ-level structural features, and overall layout features through the attention fusion module to obtain the fused features.

[0012] Furthermore, in some embodiments of this application, the method further includes: outputting a score for the ultrasound image based on a preset regression branch and scoring criteria using a second deep learning model; wherein the scoring criteria include image clarity, anatomical landmark integrity, and cross-sectional angle accuracy.

[0013] Furthermore, in some embodiments of this application, determining the deviation information between the current probe pose and the first target standard section when the section classification result is a non-standard section includes: When the ultrasound image is classified as a non-standard section, the first target standard section is determined based on the posterior probability distribution of the section type, and the deviation information between the current probe pose and the first target standard section is calculated. The deviation information includes a three-dimensional spatial deviation vector and a three-axis rotational deviation angle.

[0014] Furthermore, in some embodiments of this application, the step of generating visual guidance information to guide the operator in adjusting the probe pose based on real-time six-degree-of-freedom pose data and the division results of preset standard sections and ultrasound probe pose includes: For each standard section, the pose space is divided, and the three-dimensional position space and three-dimensional rotation space reachable by the ultrasound probe are discretized into a voxel mesh; Label each voxel mesh with the cross-section type information and the distance field information to the nearest standard cross-section; The voxel grid corresponding to the real-time six-DOF pose data is determined, and the gradient vector pointing to the second target standard section is calculated. Visual guidance information is generated based on the gradient vector. The second target standard section includes the nearest standard section corresponding to the voxel grid corresponding to the real-time six-DOF pose data.

[0015] Furthermore, in some embodiments of this application, it also includes: A skill profile is generated for the operator based on preset profile indicators; The profiling metrics include the success rate of acquiring each standard cross-section, average positioning time, the ratio of path length to optimal path, and the number of repeated errors. The scene parameters are adjusted based on the skill profile to generate ultrasound training cases of varying difficulty.

[0016] Secondly, this application provides a multimodal ultrasonic manipulation intelligent training device, including a processor and a memory, wherein the processor is connected to the memory: The processor is used to call and execute the program stored in the memory; The memory is used to store the program, which is at least used to execute the above-described intelligent training method for ultrasound manipulation based on multimodality.

[0017] Compared with the prior art, the technical solution provided in this application has the following beneficial effects: 1. It achieves continuous and realistic image generation, enhancing the realism of training. By establishing a continuous mapping from arbitrary probe pose to ultrasound images through the first deep learning model, it completely solves the image jump problem caused by incomplete pose space coverage in existing technologies, providing learners with smooth visual feedback that is highly consistent with real scanning.

[0018] 2. It provides precise and quantitative intelligent assessment and guidance, improving assessment accuracy and training efficiency. The second deep learning model not only determines whether the cross-section is standard, but also accurately calculates the deviation information of the current probe pose, providing assessors with quantitative assessment information and generating visual guidance information to improve operator training effectiveness.

[0019] 3. Supports flexible training across multiple scenarios, enhancing generalization ability. By introducing scene parameters as conditions for image generation, it can simulate clinical scenarios such as different fetal positions and gestational weeks, greatly enhancing the breadth and depth of training.

[0020] 4. Personalized adaptive training has been implemented, optimizing the learning curve. By analyzing learners' historical operational data, the system can automatically adjust the difficulty of training scenarios, enabling personalized, progressive learning and thus accelerating the cultivation of professional talent. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the intelligent training method for ultrasonic manipulation based on multimodality provided in an embodiment of the present invention. Figure 2This is a schematic diagram illustrating the principle of the intelligent training method for ultrasonic manipulation based on multimodality provided in this embodiment of the invention. Figure 3 This is a schematic diagram of the structure of the intelligent training device for ultrasonic manipulation based on multimodal operation provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0024] Figure 1 This is a flowchart illustrating the intelligent training method for ultrasound manipulation based on multimodality provided in this embodiment of the invention. Please refer to [link / reference]. Figure 1 This embodiment may include the following steps: S101. Acquire real-time six-degree-of-freedom pose data of the ultrasonic probe held by the operator.

[0025] Specifically, real-time six-DOF pose data includes three-dimensional spatial coordinates and three-axis rotation angles.

[0026] S102. Based on real-time six-degree-of-freedom pose data and preset scene parameters, the first deep learning model is used to generate an ultrasound image corresponding to the real-time six-degree-of-freedom pose data and scene parameters in real time.

[0027] The first deep learning model was trained to learn and establish a continuous mapping from the ultrasound probe pose space to the ultrasound image space. Scene parameters included information such as the mother's gestational age, fetal position, amniotic fluid volume, the mother's BMI, and placental location.

[0028] S103. Use the second deep learning model to intelligently evaluate the ultrasound images generated by the first deep learning model.

[0029] Intelligent assessment includes section classification, and when the section classification result is a non-standard section, determining the deviation information between the current probe pose and the first target standard section.

[0030] S104. Based on real-time six-degree-of-freedom pose data and the division results of preset standard sections and ultrasound probe pose, generate visual guidance information to guide the operator to adjust the probe pose, and overlay the visual guidance information on the ultrasound image.

[0031] The multimodal ultrasound operation intelligent training method provided in this application acquires real-time six-degree-of-freedom pose data of the ultrasound probe held by the operator, and generates ultrasound images in real time using a first deep learning model in conjunction with scene parameters. The first deep learning model is trained to learn and establish a continuous mapping relationship from the ultrasound probe pose space to the ultrasound image space. A second deep learning model is used to classify sections of the ultrasound images and determine deviation information for non-standard sections. Based on the real-time six-degree-of-freedom pose data and the pre-defined standard sections and ultrasound probe pose classification results, visual guidance information is generated and displayed to guide the operator in adjusting the probe pose. This solves the problems of discontinuous virtual training images and inaccurate guidance in existing methods, while significantly improving the training efficiency and realism of ultrasound operation.

[0032] Understandably, the core challenge in prenatal ultrasound screening training lies in the highly random and individually varied pose trajectories of operators, or learners, when manipulating the ultrasound probe in a virtual environment. This makes it impossible to meet real-time interactive requirements using pre-recorded videos or static image libraries. This application addresses this by constructing a multimodal paired dataset and developing a deep learning network to achieve a continuous mapping from any ultrasound probe pose space to the corresponding ultrasound image space. This enables the system to dynamically generate smooth and realistic ultrasound image feedback based on the operator's real-time actions.

[0033] Figure 2 This is a schematic diagram illustrating the principle of the multimodal-based intelligent training method for ultrasonic manipulation provided in this embodiment of the invention. Figure 2 As shown, in this application, the first deep learning model includes an encoder module, a condition generator module, and a decoder module.

[0034] First, during the training of the first deep learning model, this application records the six-degree-of-freedom pose data of the ultrasound probe in real time during the prenatal ultrasound examination using a high-precision spatial positioning device. Specifically, this includes three-dimensional spatial coordinates and three-axis rotation angles. The sampling frequency can be greater than or equal to 60Hz. Simultaneously, image frames output by the ultrasound device are acquired, with timestamps accurate to the millisecond level, ensuring a strict correspondence between the pose data of the ultrasound probe and the ultrasound image.

[0035] Based on this, ultrasound experts with prenatal screening qualifications perform multi-dimensional annotation on the acquired sequence ultrasound images, including section type annotation, i.e., marking which section the ultrasound image belongs to in the prenatal screening standard section or marking it as a non-standard section; and marking the location of important anatomical structures through key anatomical landmarks. Complete scene parameters for each ultrasound examination are also recorded, including gestational age, fetal position, amniotic fluid volume, maternal BMI index, and placental location. During the training phase, these parameters are input as conditional variables into the first deep learning model.

[0036] In practical applications, a dataset containing no less than 50,000 valid samples is ultimately constructed to cover the key time windows of screening at different stages of pregnancy. The sample distribution ensures a balanced number of various standard sections, while also including diverse scenarios with different fetal positions, different pregnant women's body types, and different amniotic fluid conditions.

[0037] Based on this, a deep learning network model for joint encoding of pose and ultrasound image is developed, namely the first deep learning model, based on the above multimodal dataset. This enables the first deep learning model to learn the continuous mapping from ultrasound probe pose space to ultrasound image space, allowing the system to generate reasonable ultrasound images for ultrasound probe poses that have not appeared in the training set, thus achieving generalization of the relationship between pose and image.

[0038] The encoder module is used to extract spatiotemporal features from real-time six-degree-of-freedom pose data based on a temporal convolutional network architecture, and to determine the motion trend of the ultrasonic probe and the operator's intention based on the spatiotemporal features; and to output a trajectory semantic vector, which includes the current probe pose, motion trend and the operator's intention.

[0039] Specifically, the encoder module employs a temporal convolutional network architecture, treating the six-DOF pose data of the ultrasound probe as a temporal sequence. A sliding window mechanism is used to acquire the current probe pose and its trajectory data for the preceding and following 10 frames. Multi-layer one-dimensional convolution and pooling operations are then used to extract the spatiotemporal features of the probe motion. In practical applications, considering that the probe movement during prenatal screening is not isolated but rather part of a continuous scanning action, the encoder module in this application can capture the dynamic trend of the probe motion through recurrent units. This allows the network model to understand operational intentions such as "the probe is moving towards the head" or "the probe is rotating to find a cross-section." Finally, the encoder module outputs a 1024-dimensional trajectory semantic vector, which not only contains the current probe pose but also the operator's scanning intentions and motion trends.

[0040] The condition generator module is used to determine the fine-grained information of the fetus based on scene parameters, so as to construct the fetal skeletal topology through graph convolutional networks, and generate the joint semantic code of the output pose scene by combining the scene constraints corresponding to the current probe pose.

[0041] Specifically, the condition generator module adopts a cross-modal fusion architecture. This application designs a fetal posture decoupling sub-network to address the decisive influence of fetal position on images during prenatal screening. This sub-network decomposes the fetal position-related information in the scene parameters into fine-grained parameters such as spinal orientation, head position, and limb posture. Then, a graph convolutional network is used to model the fetal skeletal topology.

[0042] In practical applications, the condition generator module adopts a cross-modal attention mechanism to deeply interact with the output of the encoder module and the scene parameters, dynamically retrieving the scene constraints most relevant to the current probe pose. For example, when the probe is located in the lower abdomen of the pregnant woman and the fetal position is breech, the attention mechanism will strengthen the scene constraint of "fetal head is on top" and suppress the generation tendency of "displaying cranial structure". Finally, the fetal skeletal topology and scene constraints mentioned above are used as condition vectors and passed through the gating fusion unit to form a 2048-dimensional condition code containing pose and scene joint semantics, namely pose and scene joint semantic code.

[0043] The decoder module is used to determine the positional boundary information of organs in ultrasound images and generate texture details based on the output of the conditional generator module, using a multi-scale U-shaped convolutional network. The positional boundary information and texture details are then fused to generate an ultrasound image.

[0044] Specifically, in this application, the decoder module employs an iterative denoising generation strategy to generate high-fidelity ultrasound images through progressive refinement. The decoder module utilizes a multi-scale U-shaped convolutional network architecture, comprising five progressive upsampling stages to gradually generate high-resolution images from a coarse 32×32 layout to 512×512.

[0045] In the decoder module, this application constructs an ultrasound physical property modeling layer. This layer is used to determine the beam angle, tissue attenuation, and multiple reflection information during the ultrasound image generation process based on the beam propagation principle of ultrasound imaging, so as to explicitly model physical phenomena such as beam angle, tissue attenuation, and multiple reflections. It also calculates the echo intensity of different tissue interfaces through a differentiable acoustic simulation module to generate ultrasound artifact information, and then generates an image with real ultrasound artifact characteristics, including reverberation artifacts, sound shadows, and enhancement effects.

[0046] In some embodiments of this application, the decoder module may employ a dual-path parallel generation mechanism. One path determines the positional boundary information of organs in the ultrasound image based on the results of the conditional generator, including generating a semantic segmentation map of the anatomical structure to clearly identify the organ's position and boundary. The other path generates texture details, including generating texture details based on the semantic segmentation map and the output of the conditional generator module. Then, the two paths work together through a feature fusion module to ensure the consistency between structural accuracy and texture realism.

[0047] It is understandable that the above process is the principle of training the first deep learning model based on historical data, that is, the principle of generating ultrasound images based on real-time data by the trained model.

[0048] In practical applications, during the training phase, this application adopts a multi-objective collaborative optimization framework. The application of loss functions includes using diffusion loss to train denoising capabilities, using adversarial loss to evaluate image authenticity and temporal coherence through a spatiotemporal discriminator, using multi-scale structural similarity loss to constrain the consistency between generated images and real images, using perceptual loss to ensure the accuracy of anatomical structures through a medical image coding network, and using physical consistency loss to constrain the acoustic rationality of ultrasound imaging.

[0049] Meanwhile, training can employ a course-based learning strategy, gradually transitioning from simple to difficult samples. The first deep learning model, once trained, achieves real-time inference on a single RTX 4090 GPU. Experiments demonstrate that the first deep learning model can generate reasonable ultrasound images in response to arbitrary ultrasound probe pose inputs. Even if the scene corresponding to the pose appears with extremely low frequency in the training data, the model can still make reasonable inferences based on the learned mapping rules between pose and image. This generalization ability enables the virtual training system to support the operator's free exploratory learning, rather than being limited to fixed operation paths.

[0050] Furthermore, in some embodiments of this application, the cross-sectional classification of the ultrasound image generated by the first deep learning model using the second deep learning model includes: extracting features from the ultrasound image using a feature extraction and fusion module, and fusing them to obtain fused features; and mapping the fused features to a preset number of dimensional output spaces using a fully connected layer, and outputting the probability distribution of each dimension using a softmax function, wherein each dimension corresponds to one non-standard cross-sectional type and 11 standard cross-sectional types.

[0051] It should be noted that the core task of the second deep learning model provided in this application is to determine in real time whether the ultrasound image generated by the first deep learning model is a standard section for prenatal screening, and to quantitatively evaluate the quality of the section, so as to provide learners with accurate operational feedback. The second deep learning model adopts a multi-task learning framework, which simultaneously completes the associated tasks of section classification, quality scoring and bias calculation by sharing a feature extraction backbone network. Compared with independently trained single-task models, multi-task learning can improve the overall performance by utilizing the inherent correlation between tasks.

[0052] The feature extraction and fusion module includes local scale extraction branches, intermediate scale extraction branches, global scale extraction branches, and an attention fusion module. The feature extraction and fusion module extracts features from ultrasound images and fuses them to obtain fused features. These features include: extracting fine features of anatomical landmarks from ultrasound images through the local scale extraction branch, extracting organ-level structural features from ultrasound images through the intermediate scale extraction branch, and extracting overall layout features from ultrasound images through the global scale extraction branch; and fusing the fine features of anatomical landmarks, organ-level structural features, and overall layout features through the attention fusion module to obtain fused features.

[0053] Specifically, the feature extraction part of the second deep learning model adopts a pyramid-shaped multi-scale convolutional architecture, which is optimized for the multi-scale characteristics of anatomical structures in ultrasound images. This part specifically includes five scale branches, which extract feature information from local details to global structure respectively.

[0054] The algorithm comprises several branches: a local-scale extraction branch focuses on extracting fine features of anatomical landmarks, such as the opening and closing status of the atrioventricular valves, the clarity of the valve leaflets, and the continuity of the endocardium in a four-chamber view; a mid-scale extraction branch captures organ-level structural features, such as the morphology of the four chambers of the heart, the continuity of the spine, and the positional relationship of the gastric bubble; and a global-scale extraction branch extracts overall layout features, determining whether the spatial relationships of anatomical structures conform to standard section requirements. This global-scale extraction branch acquires a wide range of contextual information through dilated convolution and pooling operations. The results from these five scale branches are then integrated using an attention fusion module. This module dynamically assigns weights to features at different scales based on the characteristics of the current image, emphasizing detailed features for high-quality samples and global structural features for less clear samples.

[0055] Based on this, the tasks of section classification, quality scoring, and deviation calculation are completed respectively through the section classification task head, the quality scoring task head, and the deviation calculation task head.

[0056] Specifically, the facet classification task head uses a fully connected layer to map the fused features to a 12-dimensional output space, corresponding to 11 standard facets for product screening plus 1 non-standard facet category, and outputs the probability distribution of each category through a softmax function.

[0057] The quality scoring task head is designed with a regression branch, outputting a continuous score from 0 to 100. The scoring criteria include three dimensions: image sharpness, anatomical landmark integrity, and cross-sectional angle accuracy (more detailed scoring indicators can be determined based on these three dimensions). In practical applications, to ensure that the scoring results are consistent with the subjective judgment of prenatal screening experts, the quality scoring task head can use expert-annotated scores as supervision signals during training, and a ranking loss function can be introduced to ensure that the output score can accurately reflect the relative relationship between different image qualities.

[0058] The deviation calculation task head is specifically designed for the analysis of non-standard sections and is only activated when the ultrasound image classification result is a non-standard section. Once activated, it outputs a three-dimensional spatial deviation vector and a three-axis rotational deviation angle via regression, indicating the difference between the current probe pose and the first target standard section (e.g., the standard section closest to the current probe pose). In some embodiments, deviation calculation can be based on the posterior probability distribution of section type (standard or non-standard) to calculate the probe adjustment required to reach the first target standard section (e.g., the second-highest probability standard section).

[0059] In this application, the training of the second deep learning model adopts a weighted multi-task loss function. The cross-entropy loss is used for the section classification task, the smoothing L1 loss combined with the ranking loss is used for the quality scoring task, and the mean squared error loss is used for the bias calculation task.

[0060] The loss weights for the three tasks are adaptively adjusted using an uncertainty-weighted method, allowing the model to automatically balance the optimization progress of each task during training. In practical applications, training data can include real historical clinical images and simulated ultrasound images generated by the first deep learning model, ensuring that the second deep learning model has good recognition capabilities for images from both sources. In actual experiments, the second deep learning model achieved a 96.8% accuracy rate in section classification on an independent test set, with a correlation coefficient of 0.93 between quality scores and expert scores, and angular errors of less than 3 degrees and positional errors of less than 5 mm in bias calculation. The inference speed of the second deep learning model can reach 200fps on a single GPU, with a single frame inference time of only 5 milliseconds. Even when concatenated with the first deep learning model, it maintains real-time performance, providing learners with instant feedback without delay.

[0061] Furthermore, in some embodiments of this application, the intelligent training of ultrasound operation based on multimodal methods also includes real-time interactive training and multimodal feedback.

[0062] The real-time interactive training scheme provided in this application aims to achieve a high-frequency closed loop of pose acquisition, image generation, cross-section evaluation, and feedback display, ensuring that the end-to-end latency is less than 20 milliseconds below the human eye's perception threshold. In practical applications, the system partially adopts an asynchronous multi-threaded pipeline architecture, including a pose acquisition thread, an image generation thread, a cross-section evaluation thread, and a rendering thread.

[0063] The pose acquisition thread reads real-time six-degree-of-freedom pose data of the ultrasound probe from the input device at a frequency of 120Hz and stores it in a circular buffer of shared memory. The image generation thread retrieves the latest data from the circular buffer and calls the first deep learning model for inference. The section evaluation thread calls the second deep learning model in parallel to analyze the generated image. The rendering thread synthesizes the image and the evaluation results and outputs them to the display.

[0064] It should be noted that, in terms of the dynamic scheduling mechanism of GPU resources, the inference tasks of the first deep learning model and the second deep learning model are executed concurrently through CUDA streams, and the parallel computing capabilities of the GPU are used to achieve time overlap of model inference. At the same time, model quantization and operator fusion technology are used to compress the inference time of the first deep learning model from 12 milliseconds to 8 milliseconds and the second deep learning model from 5 milliseconds to 3 milliseconds. The rendering module adopts a double buffering mechanism to avoid screen tearing.

[0065] Based on this, intelligent guided generation and adaptive scene generation functions are implemented.

[0066] The intelligent guidance generation process includes dividing the pose space of each standard section and discretizing the three-dimensional position space and three-dimensional rotation space reachable by the ultrasound probe into voxel grids; labeling each voxel grid with section type information and distance field information to the nearest standard section; determining the voxel grid corresponding to the real-time six-degree-of-freedom pose data and calculating the gradient vector pointing to the second target standard section; and generating visual guidance information based on the gradient vector. The second target standard section includes the nearest standard section corresponding to the voxel grid corresponding to the real-time six-degree-of-freedom pose data.

[0067] Specifically, a guided decision tree can be constructed based on the knowledge base of prenatal screening experts. The knowledge base contains the pose space division information of each standard section, as well as the three-dimensional position space and three-dimensional rotation space that the ultrasound probe can reach are discretized into voxel grids. Each voxel grid cell is labeled with the section type that should be displayed at that position and the distance field to the nearest standard section.

[0068] Using a pre-defined guidance generation module and a fast voxel lookup algorithm, the corresponding voxel grid is retrieved in O(1) time complexity based on the current probe pose of the operator. The gradient vector pointing to the second target standard section (such as the nearest standard section corresponding to the voxel grid corresponding to the current probe pose or the standard section based on user input) is calculated. After smoothing filtering and visualization rendering, the vector is superimposed on the ultrasound image in the form of arrows or trajectory lines, intuitively indicating the direction and amplitude of the ultrasound probe movement.

[0069] For complex rotational adjustments of the ultrasound probe, a quaternion interpolation algorithm can be used to calculate the shortest rotation path, decomposing the three-axis rotation into a sequence of single-axis dominant movements, thus reducing the operator's cognitive burden. Furthermore, the guidance display can employ a layered transparency design. The opacity of the visualized guidance information can be proportional to the operator's deviation (such as the result of deviation calculation based on the second deep learning model mentioned above, or the deviation between the current probe pose and the second target standard section). For example, with slight deviations, the guidance is barely visible and does not interfere with observation; with severe deviations, the guidance is highlighted to forcefully attract attention.

[0070] For adaptive scene generation, a skill profile can be generated for the operator based on preset profile indicators; scene parameters can be adjusted based on the skill profile to generate ultrasound training cases of different difficulty; among them, the profile indicators include the success rate of acquiring each standard section, the average positioning time, the ratio of path length to the optimal path, and the number of repeated errors.

[0071] Specifically, the learner behavior analysis module is used to evaluate the operator's skill level in real time, generate and maintain a multi-dimensional skill profile, and the corresponding indicators may include the success rate of acquiring each standard aspect, average positioning time, the ratio of path length to the optimal path, the number of repeated errors, etc. Among them, the current skill value is calculated by using a sliding window to count the data of the last 50 operations.

[0072] Based on this, the preset scene generation module calls the conditional generation capability of the first deep learning model according to the current skill value, dynamically modifies the scene parameters to generate training cases of different difficulties. In terms of technical implementation, the difficulty can be continuously adjusted by interpolating the scene encoding vector. For example, the encoding vector of the fetal position is linearly interpolated from the encoding vector of the head position LOA to the encoding vector of the breech position to generate an oblique position scene of intermediate difficulty.

[0073] Furthermore, in practical applications, the skills assessment report generation module can also use data visualization technology to project multi-dimensional data such as the aforementioned quality scores, deviations, and current skill values ​​onto a two-dimensional radar chart. Principal component analysis can be used to extract key skill factors, generating a structured report that includes numerical scores, chart analysis, and textual suggestions, thereby generating a report that meets the standard requirements for production screening qualification assessment.

[0074] This multimodal intelligent training method for ultrasound operation firstly achieves a continuous mapping from probe pose to ultrasound image. The first deep learning model can generate reasonable ultrasound images for any probe pose not present in the training data, solving the image transition problem caused by incomplete pose space coverage in image library retrieval methods. When the probe moves smoothly, the generated image sequence maintains a continuous transition, allowing learners to obtain visual feedback consistent with operating real equipment, effectively establishing hand-eye coordination muscle memory. Experimental verification shows that the peak signal-to-noise ratio of the generated images exceeds 35 dB, the structural similarity index is higher than 0.92, and the blind test accuracy of prenatal screening experts is only 52%, demonstrating the high fidelity of the images.

[0075] Secondly, this system enables flexible combination and generation of parameters for multiple scenarios. Through a cross-modal attention mechanism that generates scenario conditions, the system can dynamically generate differentiated images based on parameters, simulating a full spectrum of scenarios from ideal conditions to difficult cases, allowing learners to gain rich scenario experience in a virtual environment.

[0076] Furthermore, an intelligent assessment and guidance system conforming to professional standards for prenatal screening was established. The second deep learning model not only identifies the type of cross-section but also provides quantitative scores based on dimensions such as image quality, anatomical landmark integrity, and cross-section angle accuracy, and calculates spatial deviation vectors to provide precise operational suggestions. Simultaneously, the intelligent guidance component can reduce the average time for learners to obtain standard cross-sections by 45% and the number of erroneous operations by 62%, significantly improving training efficiency.

[0077] In addition, the adaptive scenario mechanism can dynamically adjust the difficulty of the scenario according to the learner's skill level, realize personalized and progressive training, and generate multi-dimensional skill assessment reports that can be used as auxiliary materials for prenatal screening qualification assessment. This breaks the dependence of prenatal screening training on real pregnant women and instructors, compresses the skill development cycle to 1-2 months, promotes the popularization of prenatal diagnostic technology in primary medical institutions, and helps reduce the rate of missed diagnosis of fetal malformations.

[0078] Based on the same inventive concept, the present invention also provides a multimodal ultrasonic manipulation intelligent training device for implementing the above-described method embodiments. Figure 3 This is a schematic diagram of the structure of the intelligent training device for ultrasonic manipulation based on multimodal operation provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the multimodal ultrasound manipulation intelligent training device of this embodiment includes a processor 11 and a memory 12, with the processor 11 connected to the memory 12. The processor 11 is used to call and execute the program stored in the memory 12; the memory 12 is used to store the program, which is at least used to execute the multimodal ultrasound manipulation intelligent training method described in the above embodiments.

[0079] The specific implementation scheme of the multimodal ultrasonic manipulation intelligent training device provided in this application can refer to the implementation scheme of the multimodal ultrasonic manipulation intelligent training method in any of the above embodiments, and will not be repeated here.

[0080] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0081] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0082] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0083] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0084] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0085] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0086] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0087] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0088] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A multimodal intelligent training method for ultrasonic manipulation, characterized in that, include: Acquire real-time six-degree-of-freedom pose data of the ultrasonic probe held by the operator; Based on the real-time six-degree-of-freedom pose data and preset scene parameters, an ultrasound image corresponding to the real-time six-degree-of-freedom pose data and scene parameters is generated in real time using a first deep learning model; wherein, the first deep learning model is trained to learn and establish a continuous mapping relationship from the ultrasound probe pose space to the ultrasound image space. The ultrasound images generated by the first deep learning model are intelligently evaluated using a second deep learning model. The intelligent evaluation includes section classification and, when the section classification result is a non-standard section, determining the deviation information between the current probe pose and the first target standard section. Based on real-time six-degree-of-freedom pose data and the division results of preset standard sections and ultrasound probe pose, visual guidance information is generated to guide the operator to adjust the probe pose, and the visual guidance information is superimposed on the ultrasound image.

2. The intelligent training method for ultrasound operation based on multimodal operation according to claim 1, characterized in that, The real-time six-degree-of-freedom pose data includes three-dimensional spatial coordinates and three-axis rotation angles; The scenario parameters include the mother's gestational age, fetal position type, amniotic fluid volume, the mother's BMI index, and placental location.

3. The intelligent training method for ultrasound operation based on multimodal operation according to claim 2, characterized in that, The first deep learning model includes an encoder module, a condition generator module, and a decoder module; The encoder module is used to extract spatiotemporal features from real-time six-degree-of-freedom pose data based on a temporal convolutional network architecture, and to determine the motion trend of the ultrasonic probe and the operator's intention based on the spatiotemporal features. And output a trajectory semantic vector, the trajectory semantic vector including the current probe pose, the motion trend and the operation intention; The condition generator module is used to determine the fine-grained information of the fetus based on the scene parameters, so as to construct the fetal skeletal topology through a graph convolutional network; and to generate a pose-scene joint semantic code by combining the scene constraints corresponding to the current probe pose. The decoder module is used to determine the positional boundary information of organs in the ultrasound image based on the output of the condition generator module and the multi-scale U-shaped convolutional network, and to generate texture details, and then fuse the positional boundary information and texture details to generate the ultrasound image.

4. The intelligent training method for ultrasound operation based on multimodal operation according to claim 3, characterized in that, The process by which the decoder generates texture details includes: Using a pre-defined ultrasonic physical characteristic modeling layer, ultrasonic physical phenomenon information is generated based on the beam propagation principle of ultrasonic imaging; and ultrasonic artifact information is generated by calculating the echo intensity of different tissue interfaces through a differentiable acoustic simulation module; wherein, the ultrasonic physical phenomenon information includes beam angle, tissue attenuation and multiple reflection information, and the ultrasonic artifact information includes reverberation artifact, sound shadow and enhancement effect information.

5. The intelligent training method for ultrasound manipulation based on multimodal operation according to claim 4, characterized in that, The ultrasound images generated by the first deep learning model are classified into sections using a second deep learning model, including: The feature extraction and fusion module extracts features from ultrasound images and fuses them to obtain fused features; The fused features are mapped to a preset number of dimensional output spaces using a fully connected layer, and the probability distribution of each dimension is output through a softmax function. The dimension corresponds to one non-standard facet type and 11 standard facets. The feature extraction and fusion module includes a local scale extraction branch, an intermediate scale extraction branch, a global scale extraction branch, and an attention fusion module. The feature extraction and fusion module extracts features from the ultrasound image and fuses them to obtain fused features. This includes: extracting fine features of anatomical landmarks from the ultrasound image through the local scale extraction branch, extracting organ-level structural features from the ultrasound image through the intermediate scale extraction branch, and extracting overall layout features from the ultrasound image through the global scale extraction branch; and fusing the fine features of the anatomical landmarks, organ-level structural features, and overall layout features through the attention fusion module to obtain the fused features.

6. The intelligent training method for ultrasound manipulation based on multimodal operation according to claim 5, characterized in that, Also includes: The second deep learning model outputs a score for the ultrasound image based on a preset regression branch and scoring criteria; wherein the scoring criteria include image clarity, integrity of anatomical landmarks, and accuracy of section angles.

7. The intelligent training method for ultrasound manipulation based on multimodal operation according to claim 5, characterized in that, When the section classification result is a non-standard section, the deviation information between the current probe pose and the first target standard section is determined, including: When the ultrasound image is classified as a non-standard section, the first target standard section is determined based on the posterior probability distribution of the section type, and the deviation information between the current probe pose and the first target standard section is calculated. The deviation information includes a three-dimensional spatial deviation vector and a three-axis rotational deviation angle.

8. The intelligent training method for ultrasound manipulation based on multimodal operation according to claim 5, characterized in that, The method, based on real-time six-degree-of-freedom pose data and the division results of preset standard sections and ultrasound probe pose, generates visual guidance information to guide the operator in adjusting the probe pose, including: For each standard section, the pose space is divided, and the three-dimensional position space and three-dimensional rotation space reachable by the ultrasound probe are discretized into a voxel mesh; Label each voxel mesh with the cross-section type information and the distance field information to the nearest standard cross-section; The voxel grid corresponding to the real-time six-DOF pose data is determined, and the gradient vector pointing to the second target standard section is calculated. Visual guidance information is generated based on the gradient vector. The second target standard section includes the nearest standard section corresponding to the voxel grid corresponding to the real-time six-DOF pose data.

9. The intelligent training method for ultrasound operation based on multimodal operation according to claim 1, characterized in that, Also includes: A skill profile is generated for the operator based on preset profile indicators; The profiling metrics include the success rate of acquiring each standard cross-section, average positioning time, the ratio of path length to optimal path, and the number of repeated errors. The scene parameters are adjusted based on the skill profile to generate ultrasound training cases of varying difficulty.

10. A multimodal ultrasonic manipulation intelligent training device, characterized in that, It includes a processor and a memory, wherein the processor is connected to the memory: The processor is used to call and execute the program stored in the memory; The memory is used to store the program, which is at least used to execute the multimodal ultrasonic manipulation intelligent training method according to any one of claims 1-9.

Images