Ultrasound imaging device for displaying ultrasound video and method of displaying ultrasound video

Abstract

An ultrasound imaging apparatus for displaying an ultrasound video and a method of displaying an ultrasound video are provided. Specifically, an ultrasound imaging apparatus and a method of displaying an ultrasound video are provided, the ultrasound imaging apparatus configured to: adjust a playback speed of a portion of interest in an ultrasound video; change a frame rate of the ultrasound video including the portion of interest from a first frame rate to a second frame rate based on the adjusted playback speed; input a plurality of frame images included in the portion of interest to an artificial intelligence model; generate at least one interpolated frame between the plurality of frame images by using the artificial intelligence model; generate an interpolated frame ultrasound video based on the plurality of frame images and the at least one interpolated frame; and display the interpolated frame ultrasound video through a display.

Description

Technical Field

[0001] This disclosure relates to an ultrasound imaging apparatus and a method for displaying ultrasound video. More specifically, this disclosure relates to a technique for providing enhanced quality ultrasound video by estimating the motion of objects in the ultrasound video using frame interpolation techniques based on artificial intelligence (such as deep learning). Background Technology

[0002] In the medical field, various medical imaging devices are now widely used to image and obtain information about living human tissue for the early diagnosis of various diseases or for surgery. Representative examples of such medical imaging devices include ultrasound imaging devices, computed tomography (CT) devices, and magnetic resonance imaging (MRI) devices.

[0003] An ultrasound imaging device emits ultrasound signals generated by a transducer in a probe toward a subject and receives information about signals reflected from the subject, so as to non-invasively obtain at least one video of an internal region of the subject (e.g., soft tissue or blood flow). Ultrasound imaging devices can be used for medical purposes (such as observing the interior of a subject, detecting foreign bodies, or assessing damage). Compared to imaging devices that use X-rays, ultrasound imaging devices offer advantages such as greater stability, real-time video display, and safety without radiation exposure, and are therefore widely used along with other imaging devices.

[0004] Ultrasonic imaging equipment can display ultrasound video. Ultrasonic video can be a video in which multiple frames are displayed sequentially. When displaying ultrasound video using existing ultrasonic imaging equipment, intermediate frames are generated between multiple consecutive frames using an optical flow-based frame interpolation technique. The optical flow-based frame interpolation technique calculates the optical flow between two consecutive frames, causing the two frames to be distorted relative to each other, and then blends the distorted frames together to generate an intermediate frame.

[0005] When using optical flow-based frame interpolation, it becomes difficult to display objects that are visible in one frame but not in another. Furthermore, it is challenging to calculate the optical flow of objects with complex or rapid motion using optical flow-based frame interpolation. Therefore, image quality degradation may occur when displaying rapidly changing ultrasound video using optical flow-based frame interpolation. Summary of the Invention

[0006] Other aspects will be set forth in part in the description which follows, and in part will be readily understood from the description, or may be learned by practicing the embodiments presented in this disclosure.

[0007] According to an embodiment, an ultrasound imaging device for displaying ultrasound video includes: an ultrasound transceiver module configured to acquire volumetric data for displaying the ultrasound video; a display configured to display the ultrasound video; a memory storing at least one instruction; and at least one processor electrically connected to the ultrasound transceiver module, the display, and the memory, wherein the at least one processor is configured to execute the at least one instruction to cause the ultrasound imaging device to: adjust the playback speed of a portion of interest in the ultrasound video; change the frame rate of the ultrasound video of the portion of interest from a first frame rate to a second frame rate based on the adjusted playback speed; input a plurality of frame images included in the portion of interest into an artificial intelligence model; generate at least one interpolated frame between the plurality of frame images using the artificial intelligence model; generate an interpolated ultrasound video based on the plurality of frame images and the at least one interpolated frame; and display the interpolated ultrasound video through the display.

[0008] According to an embodiment, a method for displaying ultrasound video performed by an ultrasound imaging device includes: adjusting the playback speed of a portion of interest in the ultrasound video; changing the frame rate of the ultrasound video of the portion of interest from a first frame rate to a second frame rate based on the adjusted playback speed; inputting a plurality of frame images including the portion of interest into an artificial intelligence model; generating at least one interpolated frame between the plurality of frame images using the artificial intelligence model; generating an interpolated ultrasound video based on the plurality of frame images and the at least one interpolated frame; and displaying the interpolated ultrasound video. Attached Figure Description

[0009] This disclosure can be readily understood through the combination of the following detailed description and the accompanying drawings, wherein reference numerals denote structural elements.

[0010] The above and other aspects, features, and advantages of specific embodiments of this disclosure will be more readily understood from the following description taken in conjunction with the accompanying drawings, in which: Figure 1A and Figure 1B This is a block diagram illustrating the configuration of an ultrasound imaging system according to an embodiment; Figure 2A , Figure 2B , Figure 2C and Figure 2D These are diagrams illustrating ultrasound imaging apparatuses according to embodiments; Figure 3 This is a block diagram illustrating the processor of an ultrasound imaging apparatus according to an embodiment; Figure 4 This is a flowchart of a method for displaying ultrasound video performed by an ultrasound imaging device according to an embodiment; Figure 5This is a flowchart of a method for processing ultrasound video performed by an ultrasound imaging device according to an embodiment; Figure 6 This is a diagram illustrating the original ultrasound video displayed by an ultrasound imaging device according to an embodiment; Figure 7 This is a diagram illustrating an ultrasound imaging apparatus according to the present disclosure, showing an interpolated ultrasound video according to a comparative example; Figure 8 This is a diagram illustrating an ultrasound imaging apparatus according to the present disclosure, showing interpolated ultrasound video according to an embodiment; Figure 9 This is a diagram illustrating an ultrasound imaging apparatus that interpolates a portion of an ultrasound video according to an embodiment; Figure 10 This is a flowchart of a method for extracting a portion of interest performed by an ultrasound imaging device according to an embodiment; Figure 11 This is a flowchart of a method for changing the frame rate performed by an ultrasound imaging device according to an embodiment; Figure 12 This is a flowchart of a method for generating an insert frame performed by an ultrasound imaging device according to an embodiment; Figure 13 This is a diagram illustrating an ultrasound imaging apparatus for setting the region of interest according to an embodiment; Figure 14 This is a diagram illustrating an ultrasound imaging apparatus for displaying interpolated M-mode video according to an embodiment; Figure 15 This is a diagram illustrating an ultrasound imaging apparatus for displaying color Doppler video according to an embodiment; Figure 16 This is a flowchart illustrating the result of displaying a combination of elastic video and B-mode video using an ultrasound imaging apparatus according to an embodiment; and Figure 17 This is a flowchart illustrating frame interpolation of contrast agent enhanced ultrasound (CEUS) video using an ultrasound imaging apparatus according to an embodiment. Detailed Implementation

[0011] To clarify the scope of the claims of this disclosure and to enable those skilled in the art to practice the embodiments, the principles of the embodiments will be described and disclosed. The embodiments may be implemented in various forms.

[0012] Throughout this specification, the same reference numerals denote the same elements. The specification does not describe all elements of the embodiments, and general content within the art to which this disclosure pertains or identical content between embodiments will be omitted. The term "module" or "unit" as used herein may be implemented in software, hardware, firmware, or a combination thereof, and according to embodiments, multiple "modules" or "units" may be implemented as a single element, or a single "module" or "unit" may comprise multiple elements.

[0013] Unless the relevant context clearly indicates otherwise, the singular form of the noun corresponding to the item may include one or more things.

[0014] As used herein, each of the phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B or C” may include any one or all possible combinations of the items listed together in the phrase.

[0015] As used herein, the term “and / or” includes any one or a combination of the plurality of related listed elements.

[0016] As used herein, terms such as “first” and “second” or “first” and “second” can be used to simply distinguish one component from another without limiting the components in any other respect (e.g., importance or order).

[0017] Additionally, as used herein, terms such as “front surface,” “rear surface,” “upper surface,” “lower surface,” “side surface,” “left,” “right,” “upper,” or “lower” are defined relative to the accompanying drawings, and the shape and position of each component are not limited by these terms.

[0018] As used herein, terms such as “comprising,” “including,” or “having” specify the presence of the stated features, numbers, stages, operations, components, parts, or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, stages, operations, components, parts, or combinations thereof.

[0019] When an element is described as being “connected to” another element, “integrated into” another element, “supported by” another element, or “in contact with” another element, it means that the element is directly connected to, integrated into, supported by, or in contact with another element, or that the element is indirectly connected to, integrated into, supported by, or in contact with another element via a third element.

[0020] When an element is said to be "on" another element, it means that the element is in contact with the other element, or that there are other elements between the element and the other element.

[0021] In the following description, an ultrasound imaging apparatus according to various embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the drawings, the same or corresponding components may be given similar reference numerals, and redundant descriptions may be omitted.

[0022] In this disclosure, the term "video" may include medical video obtained by medical imaging equipment, such as magnetic resonance imaging (MRI) equipment, computed tomography (CT) equipment, ultrasound imaging equipment, or X-ray imaging equipment.

[0023] In this disclosure, the term "object" refers to a target to be imaged and may include a person, an animal, or a part thereof. For example, the term "object" may include a part of the human body (e.g., an organ), a phantom, etc.

[0024] In this disclosure, the term "ultrasound video" refers to a video about an object generated or processed based on ultrasound signals sent to and reflected from the object.

[0025] In the following description, embodiments will be described in detail with reference to the accompanying drawings.

[0026] Figure 1A and Figure 1B This is a block diagram illustrating the configuration of an ultrasound imaging system according to an embodiment.

[0027] Reference Figure 1A and Figure 1B The ultrasound imaging system 100 may include a probe 20 and an ultrasound imaging device 40.

[0028] The ultrasound imaging device 40 can be implemented not only as a pushcart model but also as a portable model. Examples of portable ultrasound imaging devices may include, but are not limited to, smartphones, laptops, personal digital assistants (PDAs), or tablet computers (PCs) that include a probe and an application. The ultrasound imaging device 40 can also be implemented as a probe-integrated model.

[0029] The probe 20 may include a wired probe configured to be connected to and communicate with the ultrasound imaging device 40 via a wire, a wireless probe configured to be wirelessly connected to and communicate with the ultrasound imaging device 40, and / or a hybrid probe configured to be connected to and communicate with the ultrasound imaging device 40 via a wire or wirelessly.

[0030] According to various embodiments, such as Figure 1A As shown, the ultrasound imaging device 40 may include an ultrasound transceiver module 110, or as... Figure 1B As shown, probe 20 may include an ultrasonic transceiver module 110. According to various embodiments, both the ultrasonic imaging device 40 and probe 20 may also include an ultrasonic transceiver module 110.

[0031] According to various embodiments, probe 20 may further include at least one of video processor 130, display 140, and input interface 170. The descriptions of the ultrasound transceiver module 110, video processor 130, display 140, or input interface 170 included in the ultrasound imaging device 40 in this disclosure also apply to the ultrasound transceiver module 110, video processor 130, display 140, or input interface 170 included in probe 20.

[0032] Figure 1A This is a block diagram showing the configuration of the ultrasound imaging system 100 when the probe 20 is a wired probe or a hybrid probe.

[0033] The probe 20 may include multiple transducers. Multiple transducers may be arranged in a specific configuration to form a transducer array. The transducer array may correspond to a one-dimensional (1D) array or a two-dimensional (2D) array. The multiple transducers may transmit ultrasonic signals to the object 10 according to a transmission signal applied from the transmission module 113. The multiple transducers may receive ultrasonic signals reflected from the object 10 (echo signals) to form a received signal. Alternatively, the probe 20 may be integrated with the ultrasound imaging device 40, or implemented as a separate unit connected to the ultrasound imaging device 40 via wires. Furthermore, the ultrasound imaging device 40 may be connected to one or more probes 20, depending on the implementation.

[0034] In the case where probe 20 is a wired probe or a hybrid probe, probe 20 may include a cable and a connector capable of connecting to the connector of the ultrasound imaging device 40.

[0035] The probe 20 according to the embodiment can be implemented as a 2D probe. When the probe 20 is implemented as a 2D probe, a plurality of transducers included in the probe 20 can be arranged in two dimensions to form a 2D transducer array.

[0036] For example, a 2D transducer array may include multiple subarrays, each subarray including multiple transducers arranged along a first direction, and the subarrays are arranged along a second direction different from the first direction.

[0037] Additionally, when the probe 20 according to the embodiment is implemented as a 2D probe, the ultrasonic transceiver module 110 may include at least one of an analog beamformer or a digital beamformer. Furthermore, according to the embodiment, the 2D probe may include at least one of an analog beamformer and a digital beamformer, depending on the implementation.

[0038] The processor 120 controls the transmitting module 113 to form a transmitting signal to be applied to each transducer 115, taking into account the position and focus of the multiple transducers included in the probe 20.

[0039] The processor 120 can control the receiving module 117 by taking into account the position and focus of multiple transducers to perform analog-to-digital conversion on the received signal received from the probe 20 and generate ultrasound data by summing the received signal that has undergone analog-to-digital conversion.

[0040] When probe 20 is implemented as a 2D probe, processor 120 can calculate a time delay value for digital beamforming for each of the multiple subarrays included in the 2D transducer array based on each subarray. Additionally, processor 120 can calculate a time delay value for analog beamforming for each transducer included in any of the multiple subarrays. Processor 120 can control the analog beamformer and digital beamformer based on the time delay values ​​for analog and digital beamforming to form the transmitted signals to be applied to the multiple transducers, respectively. Furthermore, processor 120 can control the analog beamformer based on the time delay value for analog beamforming to sum the signals received from the multiple transducers in each subarray. Additionally, processor 120 can control the ultrasonic transceiver module 110 to perform analog-to-digital conversion on the result of summing the signals from each subarray. Furthermore, processor 120 can control the digital beamformer based on the time delay value for digital beamforming to generate ultrasonic data by summing the signals that have already undergone analog-to-digital conversion.

[0041] The video processor 130 generates or processes ultrasound video by using the generated ultrasound data.

[0042] The display 140 can display the generated ultrasound video and various information processed by the ultrasound imaging device 40 or the probe 20. The probe 20 or the ultrasound imaging device 40 may include one or more displays 140, depending on the implementation. Alternatively, the display 140 may include a touch panel or a touchscreen. Additionally, the display 140 may include a flexible display.

[0043] The processor 120 can control the overall operation of the ultrasound imaging device 40 and the operation of its components. The processor 120 can run programs or instructions stored in the memory 150 to perform or control various operations or functions of the ultrasound imaging device 40. Additionally, the processor 120 can receive control signals from the input interface 170 or external devices to control the operation of the ultrasound imaging device 40.

[0044] The ultrasound imaging device 40 may include a communication module 160, and can be connected to and communicate with external devices (e.g., probe 20, server, medical device, or portable device (e.g., smartphone, tablet PC, or wearable device)) via the communication module 160.

[0045] The communication module 160 may include one or more components that enable communication with external devices. The communication module 160 may include at least one of, for example, a short-range communication module, a wired communication module, or a wireless communication module.

[0046] The communication module 160 can receive control signals or data from an external device. The processor 120 can control the operation of the ultrasound imaging device 40 based on the control signals received through the communication module 160. Additionally, the processor 120 can send control signals to an external device through the communication module 160 to control the external device based on the sent control signals. The external device can operate based on the control signals received from the ultrasound imaging device 40, or it can process the data received from the ultrasound imaging device 40.

[0047] The program or application associated with the ultrasound imaging device 40 may be installed on an external device. The program or application installed on the external device may control the ultrasound imaging device 40 or may operate based on control signals or data received from the ultrasound imaging device 40.

[0048] External devices can receive or download programs or applications associated with the ultrasound imaging device 40 from the ultrasound imaging device 40, probe 20, or server, and can install and run the programs or applications on the external device. The ultrasound imaging device 40, probe 20, or server providing the programs or applications may include recording media storing instructions, commands, installation files, executable files, data, etc., associated with the programs or applications. External devices may also be sold pre-installed with the programs or applications.

[0049] The memory 150 can store various data or programs, input / output ultrasound data, ultrasound video, etc., used to drive and control the ultrasound imaging device 40.

[0050] Input interface 170 can receive user input for controlling ultrasound imaging device 40. For example, user input may include, but is not limited to, input via operation buttons, keypad, mouse, trackball, micro switch, knob, etc., input via touchpad or touch screen, voice input, motion input, biometric information input (e.g., iris recognition or fingerprint recognition), etc.

[0051] Figure 1B This is a control block diagram of the ultrasound imaging system 100 when probe 20 is a wireless probe or a hybrid probe.

[0052] According to various embodiments, Figure 1B The ultrasound imaging device 40 shown above can be used with reference to the above. Figure 1A The described ultrasound imaging device 40 is replaced.

[0053] According to various embodiments, Figure 1A The probe 20 shown above can be used as a reference. Figure 1B The described probe 20 is replaced.

[0054] The probe 20 may include a display 112, a transmitting module 113, a battery 114, a transducer 115, a charging module 116, a receiving module 117, an input interface 109, a processor 118, and a communication module 119. Figure 1B The probe 20 is shown to include both a transmitting module 113 and a receiving module 117. However, depending on the implementation, the probe 20 may include only a portion of the transmitting module 113 and the receiving module 117, and a portion of the transmitting module 113 and the receiving module 117 may be included in the ultrasound imaging device 40. Additionally, according to an embodiment, the probe 20 may also include a video processor 130.

[0055] Transducer 115 may include multiple transducers. Multiple transducers may be arranged in a specific configuration to form a transducer array. The transducer array may correspond to a 1D array or a 2D array. The multiple transducers may transmit ultrasonic signals to object 10 according to a transmission signal applied from transmission module 113. Additionally, the multiple transducers may receive ultrasonic signals reflected from object 10 to form or generate an electrical reception signal.

[0056] The charging module 116 can charge the battery 114. The charging module 116 can receive power from an external source. According to an embodiment, the charging module 116 can receive power wirelessly. Alternatively, according to an embodiment, the charging module 116 can also receive power via a wire. The charging module 116 can transfer the received power to the battery 114.

[0057] The processor 118 controls the transmitting module 113 to generate or form a transmitting signal to be applied to each of the multiple transducers, taking into account the position and focus of the multiple transducers.

[0058] Processor 118 controls receiver module 117 to perform analog-to-digital conversion on the received signals received from transducer 115, taking into account the positions and focal points of multiple transducers, and generates ultrasound data by summing the received signals that have undergone analog-to-digital conversion. According to an embodiment, when probe 20 includes video processor 130, ultrasound video can be generated using the generated ultrasound data.

[0059] When probe 20 is implemented as a 2D probe, processor 118 can calculate a time delay value for digital beamforming for each of the multiple subarrays included in the 2D transducer array based on each subarray. Additionally, processor 118 can calculate a time delay value for analog beamforming for each transducer included in any of the multiple subarrays. Processor 118 can control the analog beamformer and digital beamformer based on the time delay values ​​for analog and digital beamforming to form the transmitted signals to be applied to the multiple transducers, respectively. Furthermore, processor 118 can control the analog beamformer based on the time delay value for analog beamforming to sum the signals received from the multiple transducers in each subarray. Additionally, processor 118 can control the ultrasonic transceiver module 110 to perform analog-to-digital conversion on the result of summing the signals from each subarray. Furthermore, processor 118 can control the digital beamformer based on the time delay value for digital beamforming to generate ultrasonic data by summing the signals that have already undergone analog-to-digital conversion.

[0060] The processor 118 can control the overall operation of the probe 20 and the operation of its components. The processor 118 can execute programs or instructions stored in the memory 111 to perform or control various operations or functions of the probe 20. Additionally, the processor 118 can receive control signals from the input interface 109 of the probe 20 or from an external device (e.g., ultrasound imaging equipment 40) to control the operation of the probe 20. The input interface 109 can receive user input for controlling the probe 20. For example, user input may include, but is not limited to, input via buttons, keypad, mouse, trackball, microswitch, knob, etc.; input via touchpad or touchscreen; voice input; motion input; biometric information input (e.g., iris recognition or fingerprint recognition); etc.

[0061] The display 112 can display ultrasound video generated by the probe 20, ultrasound video generated by processing ultrasound data generated by the probe 20, ultrasound video received from the ultrasound imaging device 40, and various information processed by the ultrasound imaging system 100. Additionally, the display 112 can also display status information about the probe 20. This status information may include at least one of the following: device information about the probe 20, battery status information about the probe 20, frequency band information about the probe 20, output information about the probe 20, information regarding whether the probe 20 is malfunctioning, setting information about the probe 20, or temperature information about the probe 20.

[0062] The probe 20 may include one or more displays 112, depending on the implementation. Additionally, the display 112 may include a touch panel or a touchscreen. Alternatively, the display 112 may include a flexible display.

[0063] The communication module 119 can wirelessly transmit the generated ultrasound data or ultrasound video to the ultrasound imaging device 40 via a wireless network. In addition, the communication module 119 can receive control signals and data from the ultrasound imaging device 40.

[0064] The ultrasound imaging device 40 can receive ultrasound data or ultrasound video from the probe 20.

[0065] In an embodiment, if the probe 20 includes a video processor 130 capable of generating ultrasound video using ultrasound data, the probe 20 can send ultrasound data or ultrasound video generated by the video processor 130 to the ultrasound imaging device 40.

[0066] In an embodiment, where the probe 20 does not include a video processor 130 capable of generating ultrasound video using ultrasound data, the probe 20 may transmit ultrasound data to the ultrasound imaging device 40. The ultrasound data may include raw ultrasound data, and the ultrasound video may refer to ultrasound video data.

[0067] The ultrasound imaging device 40 may include a processor 120, a video processor 130, a display 140, a memory 150, a communication module 160, and an input interface 170.

[0068] The video processor 130 generates or processes ultrasound video using ultrasound data received from the probe 20.

[0069] The display 140 can display ultrasound video received from the probe 20, ultrasound video generated by processing ultrasound data received from the probe 20, and various information processed by the ultrasound imaging system 100. The ultrasound imaging device 40 may include one or more displays 140 depending on the implementation. Additionally, the display 140 may include a touch panel or a touchscreen. Furthermore, the display 140 may include a flexible display.

[0070] The processor 120 can control the overall operation of the ultrasound imaging device 40 and the operation of its components. The processor 120 can run programs or applications stored in the memory 150 to perform or control various operations or functions of the ultrasound imaging device 40. Additionally, the processor 120 can receive control signals from the input interface 170 or external devices to control the operation of the ultrasound imaging device 40.

[0071] The ultrasound imaging device 40 may include a communication module 160, and can be connected to and communicate with external devices (e.g., probe 20, server, medical device, or portable device (e.g., smartphone, tablet PC, or wearable device)) via the communication module 160.

[0072] The communication module 160 may include one or more components that enable communication with external devices. The communication module 160 may include at least one of, for example, a short-range communication module, a wired communication module, or a wireless communication module.

[0073] The communication module 160 of the ultrasound imaging device 40 and the communication module 119 of the probe 20 can communicate using a network and can also communicate using short-range wireless communication methods. For example, the communication module 160 of the ultrasound imaging device 40 and the communication module 119 of the probe 20 can communicate using any of the following wireless data communication methods: Wireless Local Area Network (LAN), Wi-Fi, Bluetooth, Zigbee, Wi-Fi Direct (WFD), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (WiBro), Global Microwave Access Interoperability (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), Radio Frequency (RF) communication, or 60 GHz millimeter wave (mmWave) short-range communication.

[0074] Therefore, the communication module 160 of the ultrasound imaging device 40 and the communication module 119 of the probe 20 may include at least one of the following modules: wireless LAN communication module, Wi-Fi communication module, Bluetooth communication module, Zigbee communication module, WFD communication module, IrDA module, BLE communication module, NFC communication module, WiBro communication module, WiMAX communication module, SWAP communication module, WiGig communication module, RF communication module, or 60 GHz millimeter wave (mmWave) short-range communication module.

[0075] In an embodiment, probe 20 can transmit device information (e.g., identifier (ID) information) about probe 20 to ultrasound imaging device 40 using a first communication method (e.g., BLE) and wirelessly pair with ultrasound imaging device 40. Additionally, probe 20 can transmit ultrasound data and / or ultrasound video to the paired ultrasound imaging device 40.

[0076] Device information about probe 20 may include various information associated with probe 20’s serial number, model name, or battery status.

[0077] The ultrasound imaging device 40 can receive device information (e.g., ID information) about the probe 20 from the probe 20 using a first communication method (e.g., BLE) and wirelessly pair with the probe 20. Additionally, the ultrasound imaging device 40 can send an activation signal to the paired probe 20 and receive ultrasound data and / or ultrasound video from the probe 20. Here, the activation signal may include a signal for controlling the operation of the probe 20.

[0078] In an embodiment, probe 20 can transmit device information (e.g., ID information) about probe 20 to ultrasound imaging device 40 using a first communication method (e.g., BLE) and wirelessly pair with ultrasound imaging device 40. Additionally, probe 20 can transmit ultrasound data and / or ultrasound video to ultrasound imaging device 40 paired with probe 20 using a second communication method (e.g., 60 GHz millimeter wave or Wi-Fi).

[0079] The ultrasound imaging device 40 can receive device information (e.g., ID information) about the probe 20 from the probe 20 using a first communication method (e.g., BLE) and wirelessly pair with the probe 20. Additionally, the ultrasound imaging device 40 can send an activation signal to the paired probe 20 and receive ultrasound data and / or ultrasound video from the probe 20 using a second communication method (e.g., 60 GHz millimeter wave or Wi-Fi).

[0080] According to an embodiment, the first communication method for pairing the probe 20 with the ultrasound imaging device 40 may have a lower frequency band than the second communication method used by the probe 20 to transmit ultrasound data and / or ultrasound video to the ultrasound imaging device 40.

[0081] The display 140 of the ultrasound imaging device 40 can display a user interface (UI) indicating device information about the probe 20. For example, the display 140 can display identification information about the probe 20, the method of pairing with the probe 20, the data communication status between the probe 20 and the ultrasound imaging device 40, the method of performing data communication with the ultrasound imaging device 40, the battery status of the probe 20, etc.

[0082] When the probe 20 includes a display 112, the display 112 of the probe 20 can display a UI indicating device information about the probe 20. For example, the display 112 can display UI indicating identification information about the probe 20, the method of pairing with the probe 20, the data communication status between the probe 20 and the ultrasound imaging device 40, the method of performing data communication with the ultrasound imaging device 40, the battery status of the probe 20, etc.

[0083] The communication module 160 can receive control signals or data from external devices. The processor 120 can control the operation of the ultrasound imaging device 40 based on the control signals received through the communication module 160.

[0084] Additionally, the processor 120 can send control signals to external devices via the communication module 160 to control the external devices according to the sent control signals. The external devices can operate according to the control signals received from the ultrasound imaging equipment 40, or they can process the data received from the ultrasound imaging equipment 40.

[0085] External devices can receive or download programs or applications associated with the ultrasound imaging device 40 from the ultrasound imaging device 40, probe 20, or server, and can install and run the programs or applications on the external device. The ultrasound imaging device 40, probe 20, or server providing the programs or applications may include recording media storing instructions, commands, installation files, executable files, data, etc., associated with the programs or applications. External devices may also be sold pre-installed with the programs or applications.

[0086] The memory 150 can store various data or programs, input / output ultrasound data, ultrasound video, etc., used to drive and control the ultrasound imaging device 40.

[0087] The following will refer to Figure 2A , Figure 2B , Figure 2C and Figure 2D An example of an ultrasound imaging system 100 according to an embodiment is described.

[0088] Figure 2A , Figure 2B , Figure 2C and Figure 2D These are diagrams illustrating ultrasound imaging devices according to embodiments.

[0089] Reference Figure 2A and Figure 2B The ultrasound imaging devices 40a and 40b may each include a main display 121 and a secondary display 122. The main display 121 and the secondary display 122 may correspond to... Figure 1A and Figure 1BThe display 140. At least one of the main display 121 or the secondary display 122 can be implemented as a touch screen. At least one of the main display 121 or the secondary display 122 can display ultrasound video or various information processed by the ultrasound imaging device 40a or 40b. In addition, at least one of the main display 121 or the secondary display 122 can be implemented as a touch screen and can provide a graphical user interface (GUI) to receive data from the user for controlling the ultrasound imaging device 40a or 40b. For example, the main display 121 can display ultrasound video, and the secondary display 122 can display a control panel for controlling the display of ultrasound video in GUI form. The secondary display 122 can receive data for controlling the display of video through the control panel displayed in GUI form. For example, a time gain compensation (TGC) button, a lateral gain compensation (LGC) button, a freeze button, a trackball, a micro switch, a knob, etc., can be provided on the secondary display 122 as a GUI.

[0090] Ultrasonic imaging devices 40a and 40b can control the display of ultrasound video on the main display 121 using control data received as input. Additionally, ultrasound imaging devices 40a and 40b can be connected to probe 20 via wires or wirelessly to send and receive ultrasound signals to and from the object.

[0091] Reference Figure 2B In addition to the main display 121 and the secondary display 122, the ultrasound imaging device 40b may also include a control panel 165. The control panel 165 may include buttons, a trackball, microswitches, knobs, etc., and may receive data from the user for controlling the ultrasound imaging device 40b. For example, the control panel 165 may include a TGC button 171, a freeze button 172, etc. The TGC button 171 is used to set the TGC value for each depth of the ultrasound video. Furthermore, when input from the freeze button 172 is detected while scanning the ultrasound video, the ultrasound imaging device 40b may maintain the display of the frame video at the corresponding time point, capture the frame video at the corresponding time point, or store the frame video at the corresponding time point.

[0092] Additionally, the buttons, trackball, microswitches, knobs, etc., included in the control panel 165 can be configured as a GUI on the main display 121 or the secondary display 122. Furthermore, the ultrasound imaging devices 40a and 40b can be connected to the probe 20 to send ultrasound signals to and receive ultrasound signals from the object.

[0093] In addition, ultrasound imaging devices 40a and 40b may include various types of input / output interfaces (such as speakers, light-emitting diodes (LEDs), or vibration devices). For example, ultrasound imaging devices 40a and 40b can output various information in the form of graphics, sound, or vibration through the input / output interfaces. Furthermore, ultrasound imaging devices 40a and 40b can output various notifications or data through the input / output interfaces.

[0094] Reference Figure 2C and Figure 2D The ultrasound imaging devices 40c and 40d can be implemented as portable types. Examples of portable ultrasound imaging devices 40c and 40d may include, but are not limited to, smartphones, laptops, PDAs, or tablet PCs containing probes and applications.

[0095] The ultrasound imaging device 40c may include a main body 41. (Refer to...) Figure 2C The probe 20 can be connected to one side of the body 41 via a wire. For this purpose, the body 41 may include a connection terminal to which the cable connected to the probe 20 can be detachably connected. The probe 20 may include a cable with a connection terminal that can be connected to the body 41.

[0096] Reference Figure 2D The probe 20 can be wirelessly connected to the ultrasound imaging device 40d. The main body 41 may include an input / output interface (e.g., a touch screen). The input / output interface can display ultrasound video, various information processed by the ultrasound imaging device 40d, a GUI, etc.

[0097] The ultrasound imaging device 40d and the probe 20 can establish communication or pair with each other using short-range wireless communication. For example, the ultrasound imaging device 40d and the probe 20 can communicate using Bluetooth, BLE, Wi-Fi, WFD, etc.

[0098] Ultrasonic imaging devices 40c and 40d can run programs or applications associated with probe 20 to control probe 20 and output information associated with probe 20. Ultrasonic imaging devices 40c and 40d can perform operations associated with probe 20 while communicating with a specific server. Probe 20 can be registered to ultrasonic imaging devices 40c or 40d or to a specific server. Ultrasonic imaging devices 40c and 40d can communicate with the registered probe 20 and perform operations associated with probe 20.

[0099] In addition, ultrasound imaging devices 40c and 40d may include various types of input / output interfaces (such as speakers, LEDs, or vibration devices). For example, ultrasound imaging devices 40c and 40d can output various information in the form of graphics, sound, or vibration through the input / output interfaces. Furthermore, ultrasound imaging devices 40c and 40d can output various notifications or data through the input / output interfaces.

[0100] According to embodiments, ultrasound imaging devices 40a, 40b, 40c, or 40d can process ultrasound video or obtain additional information from ultrasound video using an artificial intelligence (AI) model. According to embodiments, ultrasound imaging devices 40a, 40b, 40c, or 40d can generate ultrasound video or perform processing (such as correction, image quality enhancement, encoding, or decoding) on ​​ultrasound video using an AI model. Additionally, according to embodiments, ultrasound imaging devices 40a, 40b, 40c, or 40d can perform processing (such as defining baselines, obtaining anatomical information, obtaining lesion information, extracting surfaces, defining boundaries, measuring length, measuring area, measuring volume, or generating annotations) on ultrasound video using an AI model.

[0101] The AI ​​model can be set in ultrasound imaging devices 40a, 40b, 40c or 40d or in a server.

[0102] AI models can be implemented using various artificial neural network models or deep neural network models. Additionally, AI models can be trained and generated using various machine learning algorithms or deep learning algorithms. AI models can be implemented using models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or long short-term memory (LSTM).

[0103] Figure 3 This is a block diagram illustrating the processor 120 of the ultrasound imaging device 40 according to an embodiment.

[0104] The ultrasound imaging device 40 can display ultrasound video. The ultrasound imaging device 40 can acquire ultrasound data via the ultrasound transceiver module 110. The ultrasound data can be data used to display the ultrasound video. For example, the ultrasound data can be data representing the structure of the object 10 obtained by the ultrasound transceiver module 110.

[0105] The ultrasound imaging device 40 can display ultrasound video via a display 140. The memory 150 of the ultrasound imaging device 40 can store at least one instruction. The processor 120 of the ultrasound imaging device 40 can be electrically connected to the ultrasound transceiver module 110, the display 140, and the memory 150.

[0106] The processor 120 can execute at least one instruction to cause the ultrasound imaging device 40 to acquire ultrasound data. The processor 120 can execute at least one instruction to cause the ultrasound transceiver module 110 to transmit ultrasound signals to the object 10. The processor 120 can execute at least one instruction to obtain an echo signal, which is an ultrasound signal reflected from the object 10 and then received by the ultrasound transceiver module 110.

[0107] Processor 120 can execute at least one instruction to display ultrasound video on display 140. Processor 120 can obtain ultrasound data based on echo signals. Processor 120 can generate ultrasound video based on ultrasound data. Processor 120 can display the generated ultrasound video on display 140.

[0108] Processor 120 can execute at least one instruction to adjust the playback speed of a region of interest (ROI) in an ultrasound video. The ROI can be a portion of the ultrasound video whose importance is greater than or equal to a threshold. For example, the ROI can be a portion of the ultrasound video where motion is greater than or equal to a threshold. For example, the ROI can be a portion of the ultrasound video where the activity of object 10 is represented as a value greater than or equal to a threshold. Processor 120 can adjust the playback speed of the ROI to accurately represent it. For example, processor 120 can reduce the playback speed of the ROI to below normal speed.

[0109] The processor 120 according to an embodiment may include a frame rate adjustment module 310 and an insertion frame generation module 320. The processor 120 can process ultrasound video to improve its display quality. The processor 120 can process the ultrasound video using the frame rate adjustment module 310 and the insertion frame generation module 320.

[0110] The frame rate adjustment module 310 can adjust the frame rate of the ultrasound video. Based on the adjusted playback speed, the frame rate adjustment module 310 can change the frame rate of the ultrasound video of interest from a first frame rate to a second frame rate. The second frame rate can be lower than the first frame rate. For example, the first frame rate can be 60Hz, and the second frame rate can be 30Hz. The frame rate adjustment module 310 can adjust the frame rate of the ultrasound video to adjust the playback speed of the ultrasound video.

[0111] The processor 120 can execute at least one instruction to input multiple frame images, including those in the portion of interest, into the AI ​​model 321. The AI ​​model 321 may be included in the insertion frame generation module 320.

[0112] The insertion frame generation module 320 can generate at least one insertion frame between multiple frame images using AI model 321. At least one insertion frame can be at least one frame that supplements the interpolation between multiple frame images. For example, at least one insertion frame can be at least one frame that supplements the motion of an object between multiple frame images. At least one insertion frame enables a more continuous and natural playback of multiple frame images. The processor 120 can generate at least one insertion frame between multiple frame images using AI model 321.

[0113] Processor 120 can execute at least one instruction to generate an interpolated ultrasound video based on a plurality of frame images and at least one interpolated frame. Processor 120 can generate an interpolated ultrasound video in which at least one interpolated frame is inserted between the plurality of frame images.

[0114] The processor 120 can execute at least one instruction to display interpolated ultrasound video via the display 140. The display 140 can display the interpolated ultrasound video. The ultrasound imaging apparatus 40 according to this disclosure can easily display an object 10 with complex or rapid motion. The ultrasound imaging apparatus 40 according to this disclosure can display ultrasound video with high quality and rapid changes.

[0115] Figure 4 This is a flowchart of a method for displaying ultrasound video performed by an ultrasound imaging device 40 according to an embodiment.

[0116] In operation 410, the ultrasound imaging device 40 according to the embodiment can adjust the playback speed of the region of interest in the ultrasound video. The processor 120 of the ultrasound imaging device 40 can determine the region of interest based on the motion of an object displayed in the ultrasound video. For example, the processor 120 can determine the region of interest as having motion greater than or equal to a threshold. The processor 120 can adjust the playback speed of the determined region of interest.

[0117] In operation 420, the ultrasound imaging device 40 according to the embodiment may change the frame rate of the ultrasound video of the region of interest from a first frame rate to a second frame rate based on the adjusted playback speed. The processor 120 of the ultrasound imaging device 40 may reduce the playback speed of the region of interest. For example, the processor 120 of the ultrasound imaging device 40 may reduce the playback speed of the region of interest to 0.3 times, 0.5 times, or 0.7 times the normal speed.

[0118] In operation 430, the ultrasound imaging device 40 according to the embodiment can input multiple frame images, including the portion of interest, into the AI ​​model.

[0119] In operation 440, the ultrasound imaging device 40 according to the embodiment can generate at least one insert frame between multiple frame images by using an AI model. The processor 120 of the ultrasound imaging device 40 can predict at least one insert frame by using a machine learning model included in the AI ​​model. For example, the processor 120 can predict at least one insert frame suitable for insertion between multiple frame images by using a machine learning model. The processor 120 can generate at least one insert frame based on the presence of an abnormality in an object displayed in the ultrasound video. For example, when a lesion is found in an object displayed in the ultrasound video, the processor 120 can generate at least one insert frame.

[0120] In operation 450, the ultrasound imaging device 40 according to the embodiment can generate an interpolated ultrasound video based on multiple frame images and at least one interpolated frame.

[0121] In operation 460, the ultrasound imaging device 40 according to the embodiment can display interpolated ultrasound video. The ultrasound imaging device 40 according to the present disclosure can easily display an object 10 with complex or rapid motion. The ultrasound imaging device 40 according to the present disclosure can display ultrasound video with high quality and rapid changes.

[0122] Figure 5 This is a flowchart of a method for processing ultrasound video performed by an ultrasound imaging device 40 according to an embodiment.

[0123] In operation 510, the ultrasound imaging device 40 according to the embodiment can input scanned ultrasound video. The processor 120 of the ultrasound imaging device 40 can cause the ultrasound transceiver module 110 to send ultrasound signals to the object 10. The processor 120 can obtain echo signals, which are ultrasound signals reflected from the object 10 and then received by the ultrasound transceiver module 110. The processor 120 can obtain the results of scanning the object 10 based on the echo signals. The processor 120 can generate an ultrasound video based on the obtained scan results and input the ultrasound video into the memory 150.

[0124] In operation 520, the ultrasound imaging device 40 according to the embodiment can automatically extract the region of interest. The processor 120 of the ultrasound imaging device 40 can analyze the degree of motion in the ultrasound video. For example, the processor 120 can analyze the motion of objects included in the ultrasound video. The processor 120 can extract the portion in which the motion of the object is greater than or equal to a threshold as the region of interest.

[0125] In operation 530, the ultrasound imaging apparatus 40 according to the embodiment can adaptively apply a low playback speed to the extracted region of interest. The processor 120 of the ultrasound imaging apparatus 40 can adaptively change the playback speed according to the degree of motion in the region of interest. For example, the processor 120 can adaptively adjust the playback speed of the region of interest to 0.3 times, 0.5 times, or 0.7 times the normal speed.

[0126] In operation 540, the ultrasound imaging device 40 according to the embodiment can perform frame interpolation based on an AI model. The processor 120 of the ultrasound imaging device 40 can train the AI ​​model using multiple frames included in the ultrasound video. The processor 120 can generate at least one interpolated frame between the multiple frames using the trained AI model.

[0127] In operation 550, the ultrasound imaging device 40 according to the embodiment can perform detailed and accurate analysis and classification. The processor 120 of the ultrasound imaging device 40 can generate interpolated frame video based on multiple frames and at least one interpolated frame. The processor 120 can display the interpolated frame video via a display 140. The processor 120 can perform detailed and accurate analysis and classification of object motion using the interpolated frame video.

[0128] Figure 6 This is a diagram showing the original ultrasound video displayed by the ultrasound imaging device 40 according to an embodiment.

[0129] The processor 120 of the ultrasound imaging device 40 can display the raw ultrasound video via the display 140. The raw ultrasound video can be displayed at 30 frames per second (30 fps) for 2 seconds (2 sec). The raw ultrasound video can be a video with a large amount of motion (indicating the degree of motion within the ultrasound video). For example, the raw ultrasound video can be a video of a fetal heartbeat.

[0130] The processor 120 can store ultrasound video in frames. The processor 120 can adjust the playback speed of the ultrasound video for detailed examination. For example, after adjusting the playback speed to 0.5 times the normal playback speed, the processor 120 can store the ultrasound video in frames.

[0131] Figure 7 This is a diagram illustrating an ultrasound imaging apparatus 40 according to the present disclosure, which displays an interpolated ultrasound video according to a comparative example.

[0132] The processor 120 of the ultrasound imaging device 40 can display ultrasound video based on optical flow interpolation according to a comparative example. The processor 120 can replay the ultrasound video based on optical flow interpolation for 4 seconds at 0.5 times the normal playback speed.

[0133] Processor 120 may take approximately 37 seconds to interpolate ultrasound video frames based on optical flow. Processor 120 may require complex computations to generate the optically flow-interpolated ultrasound video. Optical flow-based frame interpolation of ultrasound video may exhibit limitations with respect to fast motion. Optical flow-based frame interpolation of ultrasound video may struggle with noisy ultrasound video.

[0134] Figure 8 This is a diagram illustrating an ultrasound imaging device 40 according to the present disclosure, which displays interpolated ultrasound video according to an embodiment.

[0135] The processor 120 of the ultrasound imaging device 40 can display AI-based frame-interpolated ultrasound video according to an embodiment. The processor 120 can replay the AI-based frame-interpolated ultrasound video for 4 seconds at 0.5 times the normal playback speed.

[0136] Processor 120 can spend approximately 10 seconds interpolating frames of ultrasound video based on AI. Processor 120 is capable of performing complex pattern learning to generate AI-interpolated ultrasound video. When generating AI-interpolated ultrasound video, processor 120 can perform frame interpolation more accurately than conventional algorithms.

[0137] Figure 9 This is a diagram illustrating frame interpolation of a portion of an ultrasound video 910 by an ultrasound imaging device 40 according to an embodiment.

[0138] The processor 120 of the ultrasound imaging device 40 can interpolate a portion of the ultrasound video 910. The processor 120 can interpolate the upper region of the ultrasound video 910. The processor 120 can replay the interpolated region 930, which has higher quality than the region 920 before interpolation.

[0139] Figure 10 This is a flowchart of a method for extracting a portion of interest performed by an ultrasound imaging device 40 according to an embodiment.

[0140] In operation 1010, the ultrasound imaging device 40 according to the embodiment can compare the motion and velocity of pixels in an ultrasound video. The processor 120 of the ultrasound imaging device 40 can apply algorithms to analyze the motion and velocity of pixels in the ultrasound video. For example, the processor 120 can apply at least one of an optical flow algorithm, a background subtraction algorithm, a frame differencing algorithm, or a phase correlation algorithm to obtain the motion and velocity of pixels in the ultrasound video.

[0141] In operation 1020, the ultrasound imaging device 40 according to the embodiment can compare motion and velocity with thresholds. The processor 120 of the ultrasound imaging device 40 can define thresholds for the motion and velocity of pixels in the ultrasound video.

[0142] In operation 1030, the ultrasound imaging device 40 according to the embodiment can extract portions where the motion and velocity exceed a threshold. The processor 120 of the ultrasound imaging device 40 can automatically extract portions where the motion and velocity of pixels exceed a threshold as regions of interest. For cases where automatic extraction is difficult or unnecessary, the processor 120 can provide an option to manually set the region of interest.

[0143] Figure 11 This is a flowchart of a method for changing the frame rate performed by an ultrasound imaging device 40 according to an embodiment.

[0144] In operation 1110, the ultrasound imaging apparatus 40 according to the embodiment can obtain the playback speed of the extracted region of interest. The processor 120 of the ultrasound imaging apparatus 40 can quantify the playback speed of the extracted region of interest.

[0145] In operation 1120, the ultrasound imaging apparatus 40 according to the embodiment can adjust the playback speed of the region of interest to be inversely proportional to the obtained playback speed. The processor 120 of the ultrasound imaging apparatus 40 can adjust the playback speed of the region of interest to be inversely proportional to the speed level of the quantized playback speed.

[0146] In operation 1130, the ultrasound imaging apparatus 40 according to the embodiment can adjust the playback speed of the region of interest (ROI) to be inversely proportional to the motion of the pixels within the ROI. The processor 120 of the ultrasound imaging apparatus 40 can adjust the playback speed of the ROI to be inversely proportional to the degree of pixel motion speed. For example, when the pixel motion speed is 10, the processor 120 can set the playback speed to 1 times the normal speed (i.e., the normal speed), and when the pixel motion speed is 20, the processor 120 can set the playback speed to 0.5 times the normal speed. Therefore, the processor 120 can reduce the playback speed of the ROI, which shows the portion of the ultrasound video where the scanned object (such as the heart or blood flow) has significant motion.

[0147] Figure 12 This is a flowchart of a method for generating an insert frame performed by an ultrasound imaging device 40 according to an embodiment.

[0148] In operation 1210, the ultrasound imaging device 40 according to the embodiment can generate multiple frames by dividing the portion of interest whose playback speed has been adjusted based on unit time. Taking into account the characteristics of ultrasound video (where motion is irregular and lacks continuity), the processor 120 of the ultrasound imaging device 40 can construct separate datasets. The processor 120 can divide each of the multiple frames in the portion of interest where the playback speed has been reduced along the time axis.

[0149] In operation 1220, the ultrasound imaging apparatus 40 according to the embodiment can acquire motion between a plurality of divided frames. The processor 120 of the ultrasound imaging apparatus 40 can group the plurality of frames in pairs. The processor 120 can define motion between the paired frames.

[0150] In operation 1230, the ultrasound imaging device 40 according to the embodiment can extract the vectors of pixels that have moved. The processor 120 of the ultrasound imaging device 40 can use the vectors of the pixels that have moved as labels. For example, the processor 120 can calculate the pixel motion that occurs between the first frame and the second frame.

[0151] In operation 1240, the ultrasound imaging apparatus 40 according to the embodiment can generate an insert frame based on the extracted vector. The processor 120 of the ultrasound imaging apparatus 40 can predict the insert frame to be inserted between multiple frames.

[0152] For example, processor 120 can predict the frame between the first and second frames (which may be referred to as frame 1.5) based on the calculation results.

[0153] Figure 13 This is a diagram illustrating an ultrasound imaging device 40 with a region of interest 1310 set according to an embodiment.

[0154] According to an embodiment, the ultrasound imaging device 40 can obtain a motion pattern (M-mode) video based on ultrasound video 1300 that has been interpolated after the playback speed has been reduced. The M-mode video can be a video resulting from obtaining information about motion in the ultrasound video 1300. For example, the M-mode video can be a video resulting from obtaining information about motion by acquiring images of lines included in the ultrasound video 1300 over time. For example, the ultrasound imaging device 40 can obtain the M-mode video based on interpolated fetal cardiac ultrasound video.

[0155] The ultrasound imaging device 40 according to an embodiment can obtain M-mode video by using at least some of the elements of the probe 20. The ultrasound imaging device 40 can obtain M-mode video by using fewer elements than those used for B-mode video (which is general ultrasound video 1300). For example, when the total number of elements in the probe 20 of the ultrasound imaging device 40 is 192, the ultrasound imaging device 40 can obtain B-mode video by using all 192 elements of the probe 20 through 192 channels. In contrast, the ultrasound imaging device 40 can obtain M-mode video by using only some of the elements of the probe 20. For example, the ultrasound imaging device 40 can obtain M-mode video for a single scan line using a single channel. For example, the ultrasound imaging device 40 can obtain M-mode video using five or fewer channels out of 192 elements.

[0156] Ultrasound imaging device 40 can acquire M-mode video by setting a region of interest 1310. The region of interest 1310 can be the area to be displayed in the M-mode video obtained using elements of probe 20. Ultrasound imaging device 40 can acquire interpolated M-mode video from ultrasound video data that has been interpolated by setting the region of interest 1310 in ultrasound video 1300. Ultrasound imaging device 40 can also acquire interpolated M-mode video from previously interpolated B-mode video data obtained by setting the region of interest 1310 in ultrasound video 1300. For example, ultrasound imaging device 40 can acquire interpolated M-mode video by setting the region of interest 1310 in an interpolated fetal cardiac ultrasound video.

[0157] Figure 14 This is a diagram of an ultrasound imaging device 40 showing an M-mode video 1410 with interpolated frames according to an embodiment.

[0158] The ultrasound imaging device 40 can display interpolated M-mode video 1410 corresponding to the region of interest 1310 on the display 140. The ultrasound imaging device 40 can also display both the interpolated M-mode video 1410 and the interpolated ultrasound video 1300 on the display 140. The interpolated M-mode video 1410 can present information about motion within the interpolated ultrasound video 1300. For example, the interpolated M-mode video 1410 can show the change in motion of an object within the interpolated ultrasound video 1300 over time.

[0159] The interpolated M-mode video 1410 can present the changes in motion of an object in the ultrasound video 1300 relative to the time axis. The time axis resolution of the interpolated M-mode video 1410 can be higher than that of the original M-mode video. The interpolated M-mode video 1410 can present the changes in motion relative to the time axis more clearly than the original M-mode video. The interpolated M-mode video 1410 can provide more accurate diagnostic results to the user of the ultrasound imaging device 40.

[0160] The ultrasound imaging device 40 can simultaneously display the M-mode video before frame interpolation and the interpolated M-mode video 1410. The ultrasound imaging device 40 allows the user of the ultrasound imaging device 40 to easily compare and identify diseases or abnormal symptoms that are not easily identifiable in the M-mode video before frame interpolation. Therefore, the ultrasound imaging device 40 provides diagnostic convenience to the user of the ultrasound imaging device 40.

[0161] Figure 15 This is a diagram illustrating an ultrasound imaging device 40 displaying color Doppler video 1510 according to an embodiment.

[0162] The ultrasound video displayed by the ultrasound imaging device 40 via the display 140 may include a color Doppler video 1510. The color Doppler video 1510 may be a video in which colors are assigned to distinguish the type and intensity of fluid flow present in the ultrasound video. The ultrasound imaging device 40 may display, via the display 140, both an ultrasound video 1300 corresponding to a general B-mode ultrasound video and a color Doppler video 1510 corresponding to the ultrasound video 1300.

[0163] The ultrasound imaging device 40 can adjust the playback speed of the color Doppler video 1510. The ultrasound imaging device 40 can adjust the playback speed of the color Doppler video 1510 by adjusting the playback speed of the region of interest 1310 in the ultrasound video 1300.

[0164] The ultrasound imaging device 40 can use the AI ​​model 321 to adjust the playback speed of the color Doppler video 1510. The ultrasound imaging device 40 can separate multiple frame images included in the ultrasound video into B-mode video frames and color Doppler video frames. For example, the ultrasound imaging device 40 can separate multiple frame images included in the ultrasound video 1300 into B-mode video frames and color Doppler video frames. The ultrasound imaging device 40 can input each of the B-mode video frames and color Doppler video frames into the AI ​​model 321.

[0165] The ultrasound imaging device 40 can generate interpolated color Doppler video based on ultrasound video frames and color Doppler video frames. For example, the ultrasound imaging device 40 can generate interpolated color Doppler video using AI model 321. AI model 321 can output interpolated B-mode video and interpolated color Doppler video 1510 based on B-mode video frames and color Doppler video frames.

[0166] The ultrasound imaging device 40 can provide interpolated color Doppler video 1510 to provide the user of the ultrasound imaging device 40 with accurate information related to motion in the ultrasound video 1300. For example, the ultrasound imaging device 40 can provide interpolated color Doppler video 1510 to provide the user of the ultrasound imaging device 40 with more accurate motion of blood flow in the ultrasound video 1300.

[0167] Figure 16 This is a flowchart illustrating the result of combining elastic video and B-mode video displayed by an ultrasound imaging device 40 according to an embodiment. The ultrasound imaging device 40 can fuse two different video pieces to simultaneously visualize the anatomical structure and biomechanical properties of a tissue. The ultrasound imaging device 40 can combine elastic video and B-mode video to represent the anatomical and biomechanical structure of the tissue as a single integrated ultrasound video.

[0168] In operation 1610, the ultrasound imaging apparatus 40 according to the embodiment can acquire B-mode video and elastography video. B-mode video can be general ultrasound video. B-mode video may not be able to directly represent the stiffness or elasticity of the object tissue. Elastography video can directly represent the degree of stiffness of the object tissue. For example, elastography video can provide the degree of tissue stiffness in the form of a color graph or an elastogram. Elastography video can visually identify pathological abnormalities in tissue (such as cancer).

[0169] The ultrasound imaging device 40 can perform preprocessing on the acquired B-mode video and elastic video. For example, the ultrasound imaging device 40 can align the acquired B-mode video with the elastic video. For example, the ultrasound imaging device 40 can remove noise from the acquired B-mode video and elastic video. For example, the ultrasound imaging device 40 can filter the acquired B-mode video and elastic video.

[0170] In operation 1620, the ultrasound imaging device 40 according to the embodiment can upscale the elastic video. The resolution of the elastic video data may be lower than that of the B-mode video data. The ultrasound imaging device 40 can upscale the elastic video data to a high resolution by using AI-based frame interpolation techniques. For example, the ultrasound imaging device 40 can upscale the resolution of a graph representing the hardness of a lesion included in the elastic video data to be equal to the resolution of the B-mode video data by using a super-resolution model.

[0171] In operation 1630, the ultrasound imaging apparatus 40 according to the embodiment can generate an elasticity map by interpolating frames of the magnified elastic video. The ultrasound imaging apparatus 40 can generate a continuous elasticity map by interpolating frames of the elastic video. The ultrasound imaging apparatus 40 can improve the temporal resolution of the elasticity map by using AI-based interpolation techniques.

[0172] In operation 1640, the ultrasound imaging apparatus 40 according to the embodiment can combine an elasticity map with B-mode video. The ultrasound imaging apparatus 40 can combine elasticity video data that has been magnified to high resolution with B-mode video. The ultrasound imaging apparatus 40 can generate interpolated elasticity video based on ultrasound video frames and elasticity video frames. For example, the ultrasound imaging apparatus 40 can generate interpolated elasticity video using AI model 321. AI model 321 can output interpolated B-mode video and interpolated elasticity video based on B-mode video frames and elasticity video frames.

[0173] The ultrasound imaging device 40 can provide interpolated elastic video to provide the user of the ultrasound imaging device 40 with accurate information relating to the motion and elasticity of anatomical tissues in the ultrasound video 1300. The ultrasound imaging device 40 can provide accurate clinical information to the user by accurately aligning the interpolated B-mode video and the interpolated elastic video. For example, the ultrasound imaging device 40 can provide interpolated elastic video to more accurately provide the user of the ultrasound imaging device 40 with information about the motion of tissues in the ultrasound video 1300.

[0174] Ultrasound imaging device 40 can visualize the anatomical structure and biomechanical properties of tissues in a video obtained by combining B-mode video with elastic video. For example, ultrasound imaging device 40 can examine whether tissue in which a lesion has occurred, which is included in ultrasound video 1300, matches the tumor boundary line in ultrasound video 1300. When tissue in which a lesion has occurred matches the tumor boundary line, ultrasound imaging device 40 can highlight the matching portion. Ultrasound imaging device 40 can assess the clinical significance of the lesion using AI. Ultrasound imaging device 40 can highlight the clinical significance of the lesion.

[0175] Figure 17 This is a flowchart illustrating the frame interpolation of contrast agent enhanced ultrasound (CEUS) video by the ultrasound imaging device 40 according to an embodiment.

[0176] In operation 1710, the ultrasound imaging apparatus 40 according to the embodiment can acquire CEUS video. CEUS video can be a video in which the tissues, organs, and blood circulation of a subject are improved by using a contrast agent. For example, the contrast agent can be a substance comprising inflated microbubbles. The ultrasound imaging apparatus 40 can acquire CEUS video in a low-power mode.

[0177] In operation 1720, the ultrasound imaging device 40 according to the embodiment may increase the frame rate during a first time period after contrast agent injection. The ultrasound imaging device 40 may need to rapidly capture changes in blood flow (e.g., perfusion and wash-out) in the CEUS video. The ultrasound imaging device 40 may temporarily increase the frame rate during a specific time period after the contrast agent is injected into the subject. For example, the ultrasound imaging device 40 may temporarily increase the frame rate during at least one of the early perfusion segment, peak concentration segment, or wash-out segment.

[0178] In operation 1730, the ultrasound imaging device 40 according to the embodiment may designate a portion of a first time period as a region of interest. The ultrasound imaging device 40 may switch the frame rate of the region of interest set by the user in the CEUS video to a high frame rate. The ultrasound imaging device 40 may also designate automatically detected important portions in the CEUS video as regions of interest and switch the frame rate of the designated region of interest to a high frame rate.

[0179] The ultrasound imaging device 40 can preprocess CEUS video data. Because the ultrasound imaging device 40 acquires CEUS video in low-power mode, the signal may be attenuated or noise may be present in some cases. The ultrasound imaging device 40 can perform noise reduction processing in the preprocessing operation before frame interpolation using AI. The ultrasound imaging device 40 can perform separation of tissue signals from contrast signals in the preprocessing operation. The ultrasound imaging device 40 can enhance the quality of the interpolated CEUS video and more clearly represent blood flow signals.

[0180] In operation 1740, the ultrasound imaging device 40 according to the embodiment can generate interpolated frames between multiple frames included in the region of interest. Even when the ultrasound imaging device 40 captures the region of interest in the CEUS video at a high frame rate, moments of motion of microvessels or microbubbles may be missed. The ultrasound imaging device 40 can apply an AI model to continuously generate interpolated frames of the CEUS video in order to reduce the occurrence of missed moments of motion in the CEUS video.

[0181] In operation 1750, the ultrasound imaging apparatus 40 according to the embodiment can display an interpolated CEUS video including the generated interpolated frames. The ultrasound imaging apparatus 40 can generate the interpolated CEUS video based on ultrasound video frames and CEUS video frames. For example, the ultrasound imaging apparatus 40 can generate the interpolated CEUS video by using AI model 321. AI model 321 can output the interpolated B-mode video and the interpolated CEUS video based on B-mode video frames and CEUS video frames.

[0182] The ultrasound imaging device 40 can provide interpolated CEUS video to provide the user of the ultrasound imaging device 40 with accurate information related to motion in the ultrasound video 1300. For example, the ultrasound imaging device 40 can provide interpolated CEUS video to provide the user of the ultrasound imaging device 40 with more accurate information about the motion of the contrast agent in the ultrasound video 1300.

[0183] The ultrasound imaging device 40 can generate interpolated frames to visualize the perfusion and clearance processes more smoothly in real time. The ultrasound imaging device 40 can also generate interpolated frames to easily represent detailed blood flow changes in CEUS videos. For example, the ultrasound imaging device 40 can generate interpolated frames to easily represent the distribution and disappearance patterns of microbubbles in CEUS videos.

[0184] This disclosure aims to provide a technique for improving the display quality of ultrasound video using deep learning-based frame interpolation techniques.

[0185] According to an embodiment, an ultrasound imaging device for displaying ultrasound video includes: an ultrasound transceiver module configured to acquire volumetric data for displaying ultrasound video; a display for displaying ultrasound video; a memory for storing at least one instruction; and at least one processor electrically connected to the ultrasound transceiver module, the display, and the memory, wherein the at least one processor is configured to execute at least one instruction to cause the ultrasound imaging device to adjust the playback speed of a portion of interest in the ultrasound video, change the frame rate of the ultrasound video of the portion of interest from a first frame rate to a second frame rate based on the adjusted playback speed, input multiple frame images included in the portion of interest into an artificial intelligence model, generate at least one interpolated frame between the multiple frame images using the artificial intelligence model, generate an interpolated ultrasound video based on the multiple frame images and at least one interpolated frame, and display the interpolated ultrasound video on the display.

[0186] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause the ultrasound imaging device to reduce the playback speed of the region of interest.

[0187] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause the ultrasound imaging device to predict at least one insert frame by using a machine learning model included in an artificial intelligence model.

[0188] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause the ultrasound imaging device to determine the region of interest based on the motion of an object displayed in the ultrasound video.

[0189] According to an embodiment, at least one processor may also be configured to execute at least one instruction to enable the ultrasound imaging device to obtain the motion and velocity of pixels in the ultrasound video, compare the motion and velocity with a threshold, and extract the portion of motion and velocity that is greater than the threshold.

[0190] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause the ultrasound imaging device to generate at least one insert frame based on whether an anomaly exists in an object displayed in the ultrasound video.

[0191] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause the ultrasound imaging device to adjust the playback speed of the region of interest to be inversely proportional to the playback speed of the obtained region of interest.

[0192] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause the ultrasound imaging device to adjust the playback speed of the region of interest to be inversely proportional to the motion of the pixels in the region of interest.

[0193] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause the ultrasound imaging device to generate multiple frames by dividing a portion of interest whose playback speed has been adjusted based on a unit time.

[0194] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause the ultrasound imaging device to obtain motion between a plurality of divided frames, extract vectors of pixels that have already moved, and generate at least one insert frame based on the extracted vectors.

[0195] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause an ultrasound imaging device to obtain interpolated M-mode video from ultrasound video data, the ultrasound video data being interpolated by setting a region of interest in the ultrasound video.

[0196] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause the ultrasound imaging device to separate multiple frame images included in the ultrasound video into ultrasound video frames and color Doppler video frames, and to generate interpolated color Doppler video based on the ultrasound video frames and color Doppler video frames.

[0197] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause the ultrasound imaging device to obtain a B-mode video and an elastic video that generally represent ultrasound video, to magnify the elastic video, to generate an elastic map by interpolating the magnified elastic video, and to combine the elastic map with the B-mode video.

[0198] According to an embodiment, at least one processor may also be configured to execute at least one instruction to cause the ultrasound imaging device to acquire CEUS video, increase the frame rate in a first time period after the injection of contrast agent, set at least a portion of the first time period as a part of interest, generate interpolated frames between multiple frames included in the part of interest, and display the CEUS video including the interpolated frames of the generated interpolated frames.

[0199] According to an embodiment, a method for displaying ultrasound video performed by an ultrasound imaging device includes: adjusting the playback speed of a portion of interest in the ultrasound video; changing the frame rate of the ultrasound video of the portion of interest from a first frame rate to a second frame rate based on the adjusted playback speed; inputting a plurality of frame images included in the portion of interest into an artificial intelligence model; generating at least one interpolated frame between the plurality of frame images using the artificial intelligence model; generating an interpolated ultrasound video based on the plurality of frame images and at least one interpolated frame; and displaying the interpolated ultrasound video.

[0200] According to an embodiment, changing the frame rate from a first frame rate to a second frame rate may include reducing the playback speed of the part of interest.

[0201] According to an embodiment, generating at least one insert frame among multiple frame images may include: predicting at least one insert frame by using a machine learning model included in an artificial intelligence model.

[0202] According to an embodiment, adjusting the playback speed of the region of interest may include determining the region of interest based on the motion of an object displayed in the ultrasound video.

[0203] According to an embodiment, adjusting the playback speed of the region of interest may include: obtaining the motion and velocity of pixels in the ultrasound video, comparing the motion and velocity with a threshold, and extracting the portion where the motion and velocity are greater than the threshold.

[0204] According to an embodiment, generating at least one insert frame between multiple frame images may include generating at least one insert frame based on whether an anomaly exists in an object displayed in an ultrasound video.

[0205] According to an embodiment, changing the frame rate from a first frame rate to a second frame rate may include: adjusting the playback speed of the part of interest to be inversely proportional to the playback speed of the obtained part of interest.

[0206] According to an embodiment, changing the frame rate from a first frame rate to a second frame rate may include adjusting the playback speed of the region of interest to be inversely proportional to the motion of the pixels in the region of interest.

[0207] According to this disclosure, by utilizing an AI model to optimize the frame interpolation process, the accuracy of frame generation can be improved, the computational load for generating interpolated frames can be reduced, and thus enhanced quality ultrasound video can be displayed in real time.

[0208] According to this disclosure, the overall quality of ultrasound video can be improved to enhance the diagnostic capabilities of ultrasound imaging equipment that displays ultrasound video, and accurate ultrasound video with enhanced quality can be provided to users of ultrasound diagnostic equipment.

[0209] The apparatus, method, or computer program according to the embodiments performs AI-related operations. The AI-related operations are performed via a processor and memory. A processor can perform the AI-related operations by using one or more processors. In this case, the one or more processors can be general-purpose processors (such as central processing units (CPUs), application processors (APs), or digital signal processors (DSPs)), dedicated graphics processors (such as graphics processing units (GPUs) or vision processing units (VPUs)), or dedicated artificial intelligence processors (such as neural processing units (NPUs)). The one or more processors process input data according to programs, instructions, AI models, etc., stored in memory.

[0210] AI-related programs, instructions, or AI models can be generated through machine learning. Here, "generating through learning" means training a base AI model using a learning algorithm with multiple training data sets, thereby generating programs, instructions, or AI models that perform desired characteristics (or purposes). This learning can be performed by the device itself that performs AI operations according to the embodiment, or it can be performed by a separate server and / or system. Examples of learning algorithms may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

[0211] AI models may include multiple neural network layers. Each neural network layer has multiple weight values, and neural network arithmetic operations are performed via arithmetic operations between the result of arithmetic operations on the previous layer and the multiple weight values. As a result of training the AI ​​model, the multiple weight values ​​in each of the multiple neural network layers may be optimized. Artificial neural networks may include, for example, deep neural networks (DNNs), and may include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), deep Q-networks, etc.

[0212] Machine-readable storage media may be provided in the form of non-transitory storage media. Here, the term "non-transitory storage media" refers to a tangible device and does not include signals (e.g., electromagnetic waves), and the term "non-transitory storage media" does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is temporarily stored. For example, "non-transitory storage media" may include a cache for temporarily storing data.

[0213] According to embodiments, the methods described herein may be included in and provided in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)), or distributed online through an app store (e.g., download or upload) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable application) may be temporarily stored in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a relay server).

Claims

1. An ultrasound imaging device for displaying ultrasound video, the ultrasound imaging device comprising: An ultrasonic transceiver module is configured to acquire volumetric data for displaying the ultrasonic video; The display is configured to display the ultrasound video; Memory, storing at least one instruction; as well as At least one processor is electrically connected to the ultrasonic transceiver module, the display, and the memory. The at least one processor is configured to execute the at least one instruction to cause the ultrasound imaging device to: adjust the playback speed of the portion of interest in the ultrasound video; change the frame rate of the ultrasound video of the portion of interest from a first frame rate to a second frame rate based on the adjusted playback speed; input a plurality of frame images included in the portion of interest into an artificial intelligence model; generate at least one interpolated frame between the plurality of frame images using the artificial intelligence model; and generate an interpolated ultrasound video based on the plurality of frame images and the at least one interpolated frame. The interpolated ultrasound video is then displayed on the monitor.

2. The ultrasound imaging device as described in claim 1, wherein, The at least one processor is also configured to execute the at least one instruction to cause the ultrasound imaging device to reduce the playback speed of the region of interest.

3. The ultrasound imaging device according to claim 1, wherein, The at least one processor is also configured to execute the at least one instruction to cause the ultrasound imaging device to predict the at least one inserted frame by using a machine learning model included in the artificial intelligence model.

4. The ultrasound imaging device as described in claim 1, wherein, The at least one processor is also configured to execute the at least one instruction to cause the ultrasound imaging device to determine the region of interest based on the motion of an object displayed in the ultrasound video.

5. The ultrasound imaging device as described in claim 1, wherein, The at least one processor is further configured to execute the at least one instruction to cause the ultrasound imaging device to obtain the motion and velocity of pixels in the ultrasound video, compare the motion and the velocity with a threshold, and extract the portion of the motion and the velocity that is greater than the threshold.

6. The ultrasound imaging device as described in claim 1, wherein, The at least one processor is also configured to execute the at least one instruction to cause the ultrasound imaging device to generate the at least one insert frame based on whether there is an anomaly in the object displayed in the ultrasound video.

7. The ultrasound imaging device as described in claim 1, wherein, The at least one processor is further configured to execute the at least one instruction to cause the ultrasound imaging device to adjust the playback speed of the region of interest to be inversely proportional to the playback speed of the obtained region of interest.

8. The ultrasound imaging device as described in claim 1, wherein, The at least one processor is further configured to execute the at least one instruction to cause the ultrasound imaging device to adjust the playback speed of the region of interest to be inversely proportional to the motion of the pixels in the region of interest.

9. The ultrasound imaging device as described in claim 1, wherein, The at least one processor is also configured to execute the at least one instruction to cause the ultrasound imaging device to generate multiple frames by dividing the portion of interest, whose playback speed has been adjusted, based on a unit time interval.

10. The ultrasound imaging device as described in claim 9, wherein, The at least one processor is further configured to execute the at least one instruction to cause the ultrasound imaging device to obtain motion between the generated plurality of frames, extract vectors of pixels that have undergone the motion, and generate the at least one insert frame based on the extracted vectors.

11. The ultrasound imaging device as claimed in claim 1, wherein, The at least one processor is further configured to execute the at least one instruction to cause the ultrasound imaging device to obtain interpolated M-mode video from ultrasound video data, the ultrasound video data being interpolated by setting a region of interest in the ultrasound video.

12. The ultrasound imaging device as claimed in claim 1, wherein, The at least one processor is further configured to execute the at least one instruction to cause the ultrasound imaging device to separate multiple frame images included in the ultrasound video into ultrasound video frames and color Doppler video frames, and to generate interpolated color Doppler video based on the ultrasound video frames and the color Doppler video frames.

13. The ultrasound imaging device as claimed in claim 1, wherein, The at least one processor is further configured to execute the at least one instruction to cause the ultrasound imaging device to obtain a B-mode video and an elastic video representing the ultrasound video in its original state, to magnify the elastic video, to generate an elastic map by interpolating the magnified elastic video, and to combine the elastic map with the B-mode video.

14. The ultrasound imaging device as claimed in claim 1, wherein, The at least one processor is further configured to execute the at least one instruction to cause the ultrasound imaging device to acquire contrast-enhanced ultrasound video, increase the frame rate in a first time period after contrast agent injection, set at least a portion of the first time period as the portion of interest, generate interpolated frames between a plurality of frames included in the portion of interest, and display the contrast-enhanced ultrasound video including the interpolated frames of the generated interpolated frames.

15. A method for displaying ultrasound video, the method comprising: Adjust the playback speed of the portion of interest in the ultrasound video; Based on the adjusted playback speed, the frame rate of the ultrasound video of the part of interest is changed from the first frame rate to the second frame rate. Multiple frame images, including those in the portion of interest, are input into the artificial intelligence model. At least one insert frame is generated between the multiple frame images using the artificial intelligence model; An interpolated ultrasound video is generated based on the plurality of frame images and the at least one interpolated frame; as well as Display the interpolated frame of the ultrasound video.

Images