Generation of ultrasound self-scanning guidance video
By generating personalized ultrasound self-scan guided videos, showing the probe movement path and setting steps, the problem of ultrasound imaging training in non-professional environments is solved, the coverage of ultrasound examinations is expanded, and the ability and frequency of ultrasound examinations in health care is improved. .
Patent Information
- Application Number
- CN202380043477.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-31
- Filing Date
- 2023-05-29
- Publication Date
- 2025-01-10
AI Technical Summary
In non-professional settings, especially in homes or tertiary care and rehabilitation centers, the lack of skilled ultrasound technicians leads to difficulties in obtaining and interpreting images of ultrasound imaging, and traditional methods require high levels of knowledge and expertise, which are difficult to cost-effectively and Effective training is achieved in terms of time efficiency.
By capturing the landmark position, the time stamped tracking data of the ultrasound probe and user interaction during the initial ultrasound imaging scan, the probe movement path corresponding to the landmark to the keyframe is planned, and a personalized ultrasound self-scan guide video is generated to display the probe. Move paths and setup steps to help unskilled users perform ultrasound scans.
Expand the range of personnel that can perform ultrasound examinations, reduce the need for ultrasound scans of unskilled personnel, improve ultrasound imaging capabilities in non-clinical settings, reduce dependence on professional care, and improve ultrasound examinations in healthcare Frequency and mass.
Smart Images
Figure CN119300770A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of ultrasound imaging, and more particularly to generating ultrasound self-scanning instructional videos for users (who are not skilled sonographers) to perform subsequent diagnostic ultrasound imaging. Background Art
[0002] As medical diagnosis moves toward evidence support that is objectively and measurably accessible, the need for and reliance on technical assistance from physicians, especially in medical imaging modalities, is increasing. The increase in the number of medical imaging examinations faces increasing challenges, first in terms of the availability of operators and physicians skilled enough to perform the examinations and interpret their results. Second, despite the almost complete commoditization of imaging technology suites, financial pressures are also increasing in terms of the huge labor costs of providing care.
[0003] This has resulted in many tasks once performed strictly in highly specialized care being relegated to less expensive settings, from skilled workers dressing wounds at home to remote reading of radiology images in low-cost countries. In ultrasound imaging, the challenge of using unskilled workers has traditionally been the high level of knowledge and expertise required to acquire and interpret the images. Summary of the invention
[0004] There is an increasing exploration of performing medical diagnostic examinations in non-professional environments and without professional caregivers, such as at home or in tertiary care and rehabilitation centers. There is now extremely portable ultrasound imaging hardware that technically and logistically supports "self-scanning" and similar usage scenarios. However, for this mode, the high level of knowledge and expertise required to acquire and interpret images is a barrier to use in non-professional environments. When the scan is performed by an unskilled user (e.g., the patient himself or a family member, etc.), there is a considerable challenge in obtaining images with satisfactory diagnostic value. The inventors have determined that: (a) in the process of enabling non-professionals to perform ultrasound examinations, realistic, detailed, task-specific and patient-specific training will be necessary; (b) personalized video guidance is the preferred way to perform such training; and (c) the creation of such content should be cost-effective and time-efficient, and minimize interference with the daily professional tasks of ultrasound technicians. The present invention provides a method for automatically generating patient-specific training video materials for patients to perform self-scans.
[0005] While all other medical imaging devices are still too expensive and / or too large in footprint and fixed for use outside of clinics and doctors’ offices, for ultrasound (US) imaging there now exists extremely portable hardware that technically and logistically supports “self-scanning” and similar use cases. It is a perfect representation of this trend.
[0006] The smaller size of the product will support increased production, and may be accompanied by lower profits per sale. If the resulting staffing bottleneck can be overcome, higher technical capabilities in ultrasound in healthcare may lead to better care, for example, enabling more frequent follow-up imaging. One way to achieve this is to expand the range of people who can perform ultrasound, either by raising more people to the required skill standard or lowering the standard, or a combination of both.
[0007] By providing automatically generated, personalized (ie, task-specific and patient-specific) instructional videos that summarize the most critical steps of an expert scan, the present invention expands the possibilities of using ultra-mobile devices (eg, ) to include non-professionals (including patients themselves). The present invention eliminates the need for unskilled personnel to perform ultrasound scans in order to view and interpret ultrasound images (a task that requires a lot of skills).
[0008] In the present invention, an initial ultrasound imaging scan performed by a skilled ultrasound technician in a clinical setting is used to generate an ultrasound self-scanning instructional video. Using the self-scanning instructional video, an unskilled user does not need to evaluate the image, but can simply replicate the clinical process to perform subsequent ultrasound imaging scans in a non-clinical environment. In fact, it may not be necessary to show the user / patient the actual ultrasound images obtained during the process, as an unskilled user cannot accurately interpret the image. Therefore, the instructional video may only show how to (self) perform an ultrasound scan (i.e., moves and settings) without actually showing the scanned image.
[0009] According to a first aspect of the present invention, a method for generating an ultrasound self-scanning video is provided. The method includes capturing landmark locations, time-stamped tracking data of an ultrasound probe, and time-stamped user interactions during an initial ultrasound imaging scan, wherein the user interactions include identification of key frames. The method also includes planning a probe movement path from the landmark to an ultrasound probe position corresponding to the key frame using the landmark location, tracking data, identified key frames, and timestamps. After planning the probe movement path, an ultrasound self-scanning guidance video is generated that displays the landmark and the planned probe movement path to the probe position corresponding to the key frame.
[0010] According to an exemplary embodiment of the present invention, the user interaction further includes ultrasound probe settings, and the self-scanning training video further indicates the ultrasound probe settings.
[0011] According to an exemplary embodiment of the present invention, capturing landmark locations and tracking data of the ultrasound probe includes recording a video of an initial ultrasound imaging scan and extracting landmark locations and ultrasound probe position data from the video.
[0012] According to an exemplary embodiment of the present invention, capturing user interactions includes extracting user interaction data from a log file.
[0013] According to an exemplary embodiment of the present invention, capturing the user interaction includes recording a video of the user interface and extracting the user interaction from the video.
[0014] According to an exemplary embodiment of the present invention, the step of generating an ultrasound self-scan training video includes video images recorded during an initial ultrasound imaging scan.
[0015] According to an exemplary embodiment of the present invention, a video image is edited to show the ultrasound probe following a planned path from a landmark to a location corresponding to a key frame.
[0016] According to an exemplary embodiment of the present invention: a key area is identified around each position corresponding to a key frame to compensate for tracking inaccuracies; and the step of generating an ultrasound self-scanning training video includes creating an animation of an ultrasound probe following a planned path from a landmark to a position corresponding to the key frame.
[0017] According to an exemplary embodiment of the present invention, the step of generating an ultrasound self-scanning training video includes graphical or auditory instructions for replicating probe settings.
[0018] According to an exemplary embodiment of the present invention, the step of generating an ultrasound self-scanning training video further includes graphical or auditory instructions for practicing the planned path one or more times when the ultrasound scanner is turned off before performing the ultrasound self-scanning with the ultrasound scanner turned on.
[0019] The suggested use of the instructional video is to encourage the user not only to watch it but also to follow the instructions as closely as possible, holding the device and positioning it for several dry runs in order to perform a final run with the ultrasound device actually turned on. Therefore. According to an exemplary embodiment, the step of generating the ultrasound self-scanning instructional video also includes: configuring the ultrasound self-scanning instructional video so as to obtain tracking data of the self-scanning ultrasound probe; and superimposing a position indicator mark of the self-scanning probe on the self-scanning ultrasound instructional video so as to provide feedback to an unskilled user.
[0020] According to a second aspect of the present invention, there is provided a computer program product, comprising a machine-readable medium, on which is encoded a machine-executable instruction program, and when the instruction program is executed by a processor, the instruction program performs the following steps:
[0021] During an initial ultrasound imaging scan, capturing landmark locations, time-stamped tracking data of the ultrasound probe, and time-stamped user interactions, wherein the user interactions include identification of key frames;
[0022] planning a path from the landmark to an ultrasound probe position corresponding to the keyframe using the landmark position, the tracking data, the identified keyframe, and the time stamp;
[0023] Generate an ultrasound self-scanning guidance video showing landmarks and the planned path to the probe position corresponding to the keyframes.
[0024] According to a third aspect of the present invention, a system for generating an ultrasound self-scanning training video is provided. The system includes: a processor; and a memory, which is operably connected to the processor and has a computer executable program code encoded thereon, the program code being configured to: receive tracking data, the tracking data identifying the position and orientation of the ultrasound probe relative to the anatomical landmark during ultrasound imaging; receive interaction data from a log file and identify key frames from the interaction data; plan a path from the landmark to a position corresponding to each key frame; and generate an ultrasound self-scanning training video showing the landmark and the planned path to the position corresponding to each key frame.
[0025] According to an exemplary embodiment of the present invention, the system also includes: an ultrasound probe and a camera, wherein the probe and the camera are operably connected to the processor; wherein the camera is configured to capture the position of anatomical landmarks and the position of the ultrasound probe during an initial scan; and the computer executable program code is also configured to: control the ultrasound probe to transmit and receive ultrasound signals and receive ultrasound image data from the ultrasound probe; and receive a video stream from the camera, identify landmarks on the video image stream, determine the position of the ultrasound probe relative to the anatomical landmarks, and provide the position of the ultrasound probe relative to the anatomical landmarks as tracking data.
[0026] According to an exemplary embodiment of the present invention, the system further comprises: an ultrasound probe and a tracking sensor mounted on the ultrasound probe. The probe and the tracking sensor are operably connected to the processor. The tracking sensor is configured to generate tracking data capturing the position of the anatomical landmark and the position of the ultrasound probe during the initial scan. The computer executable program code is further configured to: control the ultrasound probe to transmit and receive ultrasound signals and receive tracking data from the ultrasound probe and from the tracking sensor, and determine the position of the ultrasound probe relative to the anatomical landmark.
[0027] As used herein, the term "key frame" shall mean an image that defines the start and end points of a transition, ie, an image that provides a preferred view of a patient's features for diagnostic purposes.
[0028] As used herein, the term "animation" shall mean the manipulation of electronic images by computer to create moving images. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The features and advantages of the present invention will be more clearly understood by reading the following detailed description of the preferred embodiments in conjunction with the accompanying drawings. The accompanying drawings include the following figures:
[0030] Figure 1 A system for generating ultrasound self-scanning guidance videos according to an embodiment of the present invention is shown;
[0031] Figure 2 is a block diagram of a system for generating an ultrasound self-scanning guidance video according to an embodiment of the present invention;
[0032] Figure 3 is a diagram showing a view of a patient with anatomical landmarks and a planned ultrasound path according to one embodiment of the present invention;
[0033] Figure 4 is a diagram showing key areas according to one embodiment of the present invention;
[0034] Figure 5 is a block diagram of a method for generating an ultrasound self-scanning guidance video according to an embodiment of the present invention; and
[0035] Figure 6 yes Figure 5 A flow chart of a method for generating an ultrasound self-scanning guidance video. DETAILED DESCRIPTION
[0036] The present invention provides a method and system for generating ultrasound self-scanning instructional videos to guide unskilled users through the ultrasound imaging process.
[0037] Reference Figure 1 According to one embodiment of the present invention, a system for automatically generating ultrasound self-scanning guidance videos is shown. The system includes an ultrasound system for generating an ultrasound image of a patient 10 in a clinical environment. The ultrasound system includes an ultrasound probe 200 operably connected to an ultrasound controller 121 in a computer 100.
[0038] The ultrasound probe 200 includes one or more ultrasound transducers for transmitting and receiving ultrasound signals and transmitting the received ultrasound signals to the ultrasound controller 121 in the computer 100. The computer 100 may be an ultrasound system console and may include hardware circuits or executable program codes in software, the computer 100 including the ultrasound controller 121. The ultrasound probe 200 may be any ultrasound probe currently commercially available or developed in the future.
[0039] The ultrasound controller 121 includes program code executable by the processor 110 to control the ultrasound probe to transmit and receive ultrasound signals, transmit received signals, and process transmitted signals to generate sonogram images. The ultrasound controller can be implemented in any ultrasound system available on the market or any ultrasound system developed in the future. The ultrasound probe 200 can be operably connected to the ultrasound controller 121 via a cable or wireless communication between the ultrasound probe 200 and the processor 110 executing the program code of the ultrasound controller 121.
[0040] The system for automatically generating a self-scanning guidance video also includes a tracking system for tracking the position of the probe during an initial ultrasound scan in a clinical environment. According to one embodiment of the present invention, the tracking system is an optical tracking system, including a camera 300 and a video processor 122. The camera can be any camera with a field of view sufficient to capture the ultrasound probe 200 and anatomical landmarks 400 and provide a video data stream to the video processor 122. The video processor 122 includes program code executable by the processor 110 to process the video data stream from the camera 300 to identify and locate the anatomical landmarks 400 and determine the time-stamped position of the ultrasound probe relative to the landmarks. The anatomical landmarks 400 can be any identifiable anatomical feature close to the area to be scanned, such as a umbilicus or a nipple.
[0041] The video processor 122 utilizes deep learning based image processing, using pattern matching methods to reliably and automatically determine the types of landmarks visible in the video frames and the position and orientation of the US probe for each frame. The deep learning process can be unsupervised, using statistical analysis results. For example, landmark recognition is optimized for well-recognizable optical features that are common to the training images and points common to the planned probe movement path. The deep learning process can also be supervised. That is, the operator marks the landmarks in the training data.
[0042] The video processor 122 defines a patient-specific and landmark-specific image coordinate system that allows the position of the probe relative to the landmark (or the nearest one if there are multiple landmarks) corresponding to the keyframe to be determined. The patient-specific and landmark-specific image coordinate system can be defined by registering the landmarks and the patient orientation. In order to define this orientation, it is necessary to capture the prominent body axis of the patient 10. In most cases, this will be parallel to the spine. The tracking camera will typically view the scene from an oblique angle (rather than straight on), so there will be a perspective tilt that must be corrected ("capture the ruler"). In order to capture the ruler, either the real size of the probe is known, or a ruler (such as a simple crosshair or 2D barcode pattern) is attached to the probe so that at least one of them is always visible to the video camera. As is known from augmented reality (AR) technology, the distance and orientation of the pattern can then be easily determined by AR image processing software.
[0043] According to one embodiment, the video camera 300 is placed on top of an ultrasound cart equipped with a video controller so that the camera's field of view covers the area to be examined and the anatomical landmarks 400. The camera can be a self-rotating camera that can automatically follow the movement of the probe. Alternatively, the camera angle can be adjusted by the ultrasound operator or ultrasound technician. In another embodiment, the camera can be mounted on the ultrasound technician's head. In yet another embodiment, the camera includes two or more spatially separated co-registered cameras to avoid obstructed areas in the field of view.
[0044] When the initial clinical ultrasound examination begins, video capture also begins and begins recording video footage (images) of the sonographer's hands holding the probe and moving and tilting the probe over the patient's skin surface. The camera's field of view is wide enough so that the video footage also includes one or more easily identifiable landmarks, such as the navel or nipple. The video of the ultrasound examination provides an example for the unskilled user to replicate the movement of the ultrasound probe to perform patient / diagnosis specific follow-up scans. The video footage of the video camera is processed to extract the probe movements relative to the landmarks, including translation, tilt, and pitch movements. These movements are then simplified to generate the easiest movements for obtaining images for each key frame position (the position corresponding to each diagnostically valuable image or frame), as will be explained below.
[0045] In an alternative embodiment, the tracking system does not use a camera, but instead uses another modality (e.g., inertial sensors, magnetic sensors, etc.). The tracking sensors 195 (inertial, magnetic, etc.) are located on the ultrasound probe used during the initial scan. Sensors mounted on ultrasound probes are known in the art. The sensor positions must be registered with the patient, such as by the sonographer tapping anatomical landmarks and moving the probe along the main axis of the patient before starting the initial scan. As the sonographer taps anatomical landmarks, the positions can be registered by an input / output device 150 (e.g., a mouse, keyboard, foot pedal, microphone to capture verbal signals, or any other suitable input / output device).
[0046] The system for automatically generating a self-scanning guidance video also includes an interaction detector 123. The interaction detector 123 may include program code configured to search for user interactions in the log file 132, such as screenshot capture, ultrasound beam manipulation, and ultrasound mode changes with event timestamps. Alternatively, the interaction detector 123 may be a program code configured to intercept user input directly from one or more user input devices (e.g., keyboard, mouse, foot pedal, etc.). The interaction detector will also obtain a timestamp when intercepting the user interaction. In another alternative embodiment, the interaction detector may include a microphone and a voice recognition program code to detect verbal instructions of user interactions provided by the ultrasound technician and a timestamp for marking the user interaction. In yet another alternative embodiment, the interaction detector 123 may include a camera pointed at a control for the ultrasound system and a video processing program code for identifying interactions. This may be the same camera 300 as the camera used for tracking, with a field of view wide enough to also capture user interactions at the user input device, or it may be a separate camera.
[0047] The video processor 121 matches the timestamps of the detected screenshots with the timestamps of the probe positions in order to identify the probe positions corresponding to the key frames (the image frames for which the sonographer initiated the screenshot due to their diagnostic value).
[0048] The system for automatically generating a self-scanning instructional video also includes a path planner 126. The path planner may include a program configured to calculate one or more direct paths 401-404 (e.g., Figure 3 ) program code.
[0049] During the initial ultrasound imaging examination, the ultrasound technician may perform probe movement and orientation while observing the ultrasound image on the display 140 operably connected to the ultrasound probe 200 through the ultrasound controller 121 in order to find an image with high diagnostic value, and the process is not always simple, that is, the ultrasound technician usually moves the probe and reorients it multiple times. These movements are defined as search movements, which are performed to search for the best position and orientation. Advantageously, in the present invention, an unskilled user does not have to evaluate the diagnostic value of the ultrasound image. Instead, the unskilled user only needs to copy the probe position and orientation of each key frame. Therefore, during the self-scan, the patient does not need to copy these search movements, and the probe movement and orientation path displayed to the patient can be a simplified path that only shows those movements required to move the probe to the key frame. Since the position and orientation of each key frame are known, the system and method can replace the movement of the ultrasound technician with a movement directly pointing to the position of the key frame.
[0050] To avoid accumulation of movement errors, the ultrasound self-scanning guidance video can instruct the patient or other unskilled user to always return to a "safe point", the anatomical landmark 400, during a key frame and restart from there. This creates a star-shaped movement path with the anatomical landmark at the center of the star. The anatomical landmark does not have to be in the center of the pattern. Any easily identifiable location will do, which may result in an "unbalanced" star shape. For very simple examinations, the method can be changed to go directly from one key frame to the next.
[0051] Since a patient or unskilled user may not be able to accurately replicate the position and orientation of the keyframe, the position of the keyframe may be extended to one or more key regions 411-414, such as Figure 5 For example, if the key frame is located 10 cm to the left of the anatomical landmark, the key region will extend from 7 cm to 13 cm. By incrementally scanning the entire key region, the system and method increases the likelihood of capturing images with high diagnostic value.
[0052] For each key area 411-414, the ultrasound self-scanning guidance video graphically instructs the patient or the scan performer to position the probe at the anatomical landmark and then slowly move toward the key area. Once the edge of the key area is reached, the instruction is to slowly tilt and rotate the probe to match the orientation taken by the ultrasound technician during the initial scan, and then continue to move outward incrementally during the opposite end of the key area. The guidance video can use an animation rendering from the tracking system to display the probe orientation, which displays the orientation from the perspective of an unskilled user. If the tracking system is the video source of the initial scan, the camera angle can be very different from the perspective of the unskilled user. Alternatively, if the video is generated by a camera positioned to replicate the perspective of an unskilled user (e.g., mounted on the head or glasses of the ultrasound technician), where the unskilled user is not a patient, the video from the initial scan can be used to display the orientation. The ultrasound self-scanning guidance video can also use a combination of animation and video images from the initial scan. At each increment, a small forward and backward tilt and rotation are specified. This ensures that any inaccuracies in positioning can be balanced and ultrasound images with high diagnostic value are captured with a high probability. After covering each critical area, the user is instructed to reposition the probe on top of the anatomical landmarks without retracing the path backwards.
[0053] The video processing code can be configured to extract and display a series of user interactions based in part on information obtained during a previous examination as part of the ultrasound self-scanning instructional video. That is, the video can display a captured image of a user manipulating a user interface device (e.g., a keyboard). Alternatively, the video processing code can use pattern recognition to detect user interactions from a video stream from a camera. Timestamps of user interactions extracted from the camera video stream will also be obtained.
[0054] The system for automatically generating a self-scanning instructional video also includes an instructional video generator 127. The instructional video generator automatically generates an instructional video showing the probe moving along the planned paths 401-404 (eg, Figure 3 The planned path is directly from the anatomical landmark 400 to each key frame location or key area 411-414. The self-scanning guidance video can use the images captured by the camera 300 to generate the video and edit out the frames that are not in the planned path because of the repetitive nature of clinical scanning. Alternatively, the guidance video generator can generate animations based on the planned paths 401-404, as described below.
[0055] The self-scanning instructional video may also include instructions for ultrasound scan settings, such as mode changes and beam manipulation (e.g., steering, focusing, shaping) captured by the interactive detector 123. These instructions may be provided in the form of audio instructions, video or animation of the user interface.
[0056] In one embodiment, pre-recorded verbal instructions from a pre-recorded voice clip are added to the ultrasound self-scan instructional video. These instructions may provide guidance for repeating the initial scan, such as "slowly move the probe 5 cm to the left" or "tilt the probe slightly towards you."
[0057] The ultrasound self-scanning guidance video can be stored on a computer-readable medium. The medium can be an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system (or device or equipment). Examples of computer-readable media include semiconductor or solid-state memory, tape, removable computer disk, thumb drive, random access memory (RAM), read-only memory (ROM), hard disk, MPEG file and optical disk. Current examples of optical disks include read-only compact disk (CD-ROM), read / write compact disk (CD-R / W) and DVD. Alternatively, the ultrasound self-scanning guidance video can be implemented in a video stream transmitted via the Internet or a cable or broadcast television program, or stored on a DVD, VHS tape.
[0058] The home user (patient or other unskilled user) loads the media with the instructional video into an appropriate device and plays the instructional video. Depending on the media used, the instructional video can be uploaded to a laptop, smartphone, or tablet. The tablet can be part of the home user's ultrasound system. The home user plays the instructional video, watches and, if possible, listens to the instructional video, and follows the setup instructions embedded in the instructional video. The home user performs probe movements, mimicking the instructional video, which shows the planned path from the home user's perspective. The home user then performs the completion instructions provided on the instructional video, including uploading the scanned imaging data for review by a trained sonographer.
[0059] According to one embodiment, the ultrasound probe 200 is equipped with position sensors 195, such as optical, magnetic, inertial or other known position sensors. These position sensors increase the accuracy of capturing the probe position and / or orientation during key frames.
[0060] In some embodiments, the video camera captures video from a position / perspective that is not similar to the patient's perspective or the ultrasound technician's perspective. In these embodiments, the video cannot be directly used as the input of the guidance video. Instead, an animated movie clip is automatically generated, showing a representation of a skin patch similar to the target scanning area (e.g., abdomen) and a series of star-shaped probe movements and orientations from the ultrasound technician's perspective and / or the patient's perspective. Animation can be generated using the patient-specific and landmark-specific image coordinate system and ruler defined by the video processor 122. The guidance video generator 127 can generate animation by superimposing the images of anatomical landmarks and ultrasound probes on simulated skin patches, which are located at their relative positions provided in the tracking data of each video frame. Orientation references can also be provided, such as showing the patient side of a prominent body axis or an axis corresponding to a prominent body axis. The planning path can also be superimposed on the video.
[0061] In one embodiment, pre-recorded verbal instructions from a pre-recorded voice clip are added to the ultrasound self-scan instructional video. These instructions may provide guidance for repeating the initial scan, such as "slowly move the probe 5 cm to the left" or "slightly tilt the probe toward you."
[0062] The instructional video can be expanded by adding pre-made video clip sequences that show how to prepare for the exam, such as preparing the ultrasound equipment, sitting or lying down, applying coupling gel, starting the exam, and after the core section, appending pre-made video clip sequences that show how to complete the exam, post the transfer date, power off the equipment, etc.
[0063] Reference Figure 5 and Figure 6 , a method for implementing the present invention is now described. An initial ultrasound imaging scan is performed by a skilled ultrasound technician in a clinical setting. During the initial ultrasound scan, the tracking system captures time-stamped tracking data of the landmark locations and the position of the ultrasound probe relative to the landmarks at step 601. According to one embodiment, the tracking system includes a camera 300 and a video processor 122. The video processor 122 obtains a video stream 501 from the camera 300. The video processor uses a deep learning algorithm as described above to identify anatomical landmarks. The video processor also identifies the ultrasound probe and calculates the time-stamped 503 position of the probe relative to the anatomical landmark for each video frame. The video processor 122 provides the time-stamped position of the probe relative to the anatomical landmark to the path planner 126 as tracking data 505.
[0064] Also, during the initial ultrasound scan, the interaction detector 123 detects user interactions at step 602 and annotates the interactions with timestamps 507. The interaction detector 123 provides the interaction identification results (i.e., interaction types) and timestamps to the keyframe identifier 125 as interaction data 509. Based on the user interaction data, the keyframe identifier 125 identifies keyframes at step 603. Keyframes can be identified by the ultrasound technician taking screenshots, or by the ultrasound technician providing input through an input device (e.g., providing verbal signals through a microphone, or providing tactile signals through a keyboard or mouse). The keyframe data 511 (timestamps of the keyframes) is provided to the path planner 126. The interaction detector 123 also provides ultrasound settings and mode changes 513 to the guidance video generator 127.
[0065] Using the landmark positions, tracking data 505, identified keyframe data 511, and timestamps, the path planner 126 plans a path 401-404 directly from the landmark to the ultrasound probe position corresponding to each keyframe at step 604. The path planner 126 matches the probe position from the tracking data to the identified keyframe by matching the timestamps. Using the now known keyframe positions, the path planner calculates a planned path 401-404 that includes a direct path from the anatomical landmark to each keyframe position.
[0066] To maximize the likelihood of obtaining images with high diagnostic value, the planned path may also provide incremental translation of the probe over key areas 411-414 and incremental orientation adjustment at each incremental translation step, i.e., key areas may be provided to obtain denser ultrasound data capture coverage at many positions and many probe orientations.
[0067] Using the planned path and the interaction data (e.g., mode changes and beam steering), the guidance video generator 127 generates a guidance video at step 605 to guide an unskilled user through the ultrasound scan. The generated ultrasound self-scan guidance video shows landmarks and the planned path to and through key areas. The planned path can be displayed using animation, video captured during the initial ultrasound scan, or a combination thereof.
[0068] According to one embodiment of the present invention, the generated ultrasound self-scanning instruction video further includes instructions for the user to perform one or more scanning exercises before turning on the ultrasound scanner to make the user familiar with the movement of the probe.
[0069] According to one embodiment, the ultrasound self-scanning instructional video is configured to obtain tracking data of the ultrasound probe during practice, during actual scanning by an unskilled user, or both. Position indicator marks may then be superimposed on the ultrasound self-scanning instructional video animation to provide feedback to the unskilled user, showing how well the movements correspond to the desired movements on the instructional video.
[0070] The present invention can adopt the form of complete hardware embodiment, complete software embodiment or embodiment comprising hardware and software elements simultaneously.In exemplary embodiments, the present invention is implemented in the form of software (including but not limited to firmware, resident software, microcode, etc.).
[0071] Furthermore, the invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system or device.
[0072] The medium can be an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system (or device or apparatus) or a propagation medium. Examples of computer readable media include semiconductor or solid-state memory, magnetic tape, a removable computer floppy disk or thumb drive, random access memory (RAM), read-only memory (ROM), hard disk, and optical disk. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read / write (CD-R / W), and DVD.
[0073] The above method can be implemented by a program product, which includes a machine-readable medium having a machine-executable instruction program, and when the instruction program is executed by a machine (such as a computer), the steps of the method are performed. The program product can be stored on various known machine-readable media, including but not limited to optical disks, floppy disks, USB storage devices, etc.
[0074] The foregoing description and drawings are intended to illustrate the present invention rather than to limit the present invention. The scope of the present invention is intended to encompass modifications and configurations equivalent to the full scope of the following claims.
[0075] Reference numerals list
[0076] 10 patients
[0077] 100 computers (workstations)
[0078] 110 Processor
[0079] 120 Memory
[0080] 121 Ultrasonic Controller
[0081] 122 Video Processor
[0082] 123 Interaction Detector
[0083] 125 Keyframe Recognizer
[0084] 126 Path Planner
[0085] 127 Instructional Video Generator
[0086] 131 video files
[0087] 132 Log files
[0088] 140 Display
[0089] 150 Input / Output Devices
[0090] 300 cameras
[0091] 200 Ultrasound Probe
[0092] 195 Position Sensor
[0093] 400 Anatomical landmarks
[0094] 401-404 Planning Path
[0095] 411-414 Key Area
[0096] 501 Video Stream
[0097] 503 Video Timestamp
[0098] 505 Tracking Data
[0099] 507 Ultrasound timestamp
[0100] 509 User interaction data
[0101] 511 key frame data
[0102] 513 U / S Setting / Mode Change
[0103] 601 Capture landmarks and probe positions
[0104] 602 Capturing User Interactions
[0105] 603 Identify key frames
[0106] 604 Planning Path
[0107] 605 Generate instructional video
Claims
1. A method for generating an ultrasound self-scanning video, comprising the following steps: During an initial ultrasound imaging scan, capturing landmark locations and time-stamped tracking data (601) of the ultrasound probe (200); capturing user interactions with timestamps (602), wherein the user interactions include identification of key frames (603); planning (604) a path from the landmark to an ultrasound probe position corresponding to the keyframe using the landmark position, the tracking data, the identified keyframe, and the time stamp; as well as The ultrasound self-scanning guidance video is generated (605) showing the landmarks and the planned path to the probe position corresponding to the key frame.
2. The method according to claim 1, wherein: The user interaction further includes ultrasound probe settings, and the self-scan training video further indicates the ultrasound probe settings.
3. The method according to claim 1, wherein: Capturing landmark locations and ultrasound probe tracking data includes recording a video of the initial ultrasound imaging scan and extracting the landmark locations and ultrasound probe position data from the video.
4. The method according to claim 1, wherein: Capturing user interactions includes extracting user interaction data from log files.
5. The method according to claim 1, wherein: Capturing user interactions includes recording a video of the user interface and extracting the user interactions from the video.
6. The method according to claim 1, wherein: The step of generating the ultrasound self-scan training video includes video images recorded during the initial ultrasound imaging scan.
7. The method according to claim 6, wherein: The video image is edited to show the ultrasound probe following the planned path from the landmark to the location corresponding to the keyframe.
8. The method according to claim 1, wherein: identifying a key region around each position corresponding to the key frame to compensate for tracking inaccuracies; and The step of generating the ultrasound self-scanning training video includes creating an animation of the ultrasound probe following the planned path from the landmark to the position corresponding to the key frame.
9. The method according to claim 2, wherein: The step of generating the ultrasound self-scan training video includes graphical or auditory instructions for replicating the probe settings.
10. The method according to claim 9, wherein: The step of generating the ultrasound self-scan training video also includes providing graphical or auditory instructions for practicing the planned path one or more times when the ultrasound scanner is turned off before performing the ultrasound self-scan with the ultrasound scanner turned on.
11. The method according to claim 10, wherein: The step of generating the ultrasound self-scanning guidance video also includes: configuring the ultrasound self-scanning guidance video to obtain tracking data of a self-scanning ultrasound probe; and A position indication mark of the self-scanning probe is superimposed on the self-scanning ultrasound guidance video to provide feedback to unskilled users.
12. A computer program product comprising a machine-readable medium having encoded thereon a machine-executable instruction program, wherein when the instruction program is executed by a processor, the instruction program performs the following steps: During an initial ultrasound imaging scan, capturing landmark locations (601), time-stamped tracking data of an ultrasound probe (601), and time-stamped user interactions (602), wherein the user interactions include identification of key frames (603); planning a path from the landmark to an ultrasound probe position corresponding to the keyframe using the landmark location, the tracking data, the identified keyframe, and the timestamp (604); and An ultrasound self-scanning guidance video is generated that displays the landmarks and the planned path to the probe position corresponding to the key frame (605).
13. A system for generating ultrasound self-scanning training videos, comprising: - a processor (110); and - a memory (120) operatively connected to the processor and having encoded thereon a computer executable program code configured to: receiving tracking data identifying a position and orientation of an ultrasound probe relative to anatomical landmarks during ultrasound imaging; receiving interaction data from a log file and identifying key frames from the interaction data; planning a path from the landmark to a location corresponding to each keyframe; and The ultrasound self-scanning training video is generated showing the landmarks and the planned path to the location corresponding to each key frame.
14. The system according to claim 13, wherein: The system further comprises: - an ultrasound probe (200); and - Camera (300); wherein the probe and the camera are operably connected to the processor; wherein the camera is configured to generate a video stream capturing anatomical landmarks and the ultrasound probe during an initial scan; and Wherein, the computer executable program code is further configured to: controlling the ultrasound probe to transmit and receive ultrasound signals and receiving ultrasound image data from the ultrasound probe; and The video stream from the camera is received, the landmarks on the video image stream are identified, the position of the ultrasound probe relative to the anatomical landmarks is determined, and the position of the ultrasound probe relative to the anatomical landmarks is provided as tracking data.
15. The system of claim 13, wherein: The system further comprises: - an ultrasound probe (200); and - a tracking sensor (195) mounted on the ultrasound probe; wherein the probe and the tracking sensor are operably connected to the processor; wherein the tracking sensor is configured to generate tracking data during an initial scan, the tracking data capturing the positions of anatomical landmarks and the position of the ultrasound probe; and Wherein, the computer executable program code is further configured to: controlling the ultrasound probe to transmit and receive ultrasound signals and receiving from the ultrasound probe and tracking data from the tracking sensor; and The position of the ultrasound probe relative to the anatomical landmark is determined.