Method and system for medical imaging device

Abstract

The present invention provides various systems for ultrasound imaging devices. In one example, a method includes indicating to a user a position to navigate an ultrasound probe in one or more of a horizontal plane and a vertical plane based on a first image previously acquired at a first anatomical location, while acquiring a second image at a second different anatomical location and automatically combining the first image and the second image together.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate to medical imaging and the facilitation of ultrasonic tissue scanning. Background Art

[0002] In the ultrasonic diagnostic apparatus, ultrasound is transmitted from an ultrasonic probe into a subject's body, and an echo signal reflected from the subject's body is received by the ultrasonic probe. Then, an ultrasonic image is generated based on the received echo signal and displayed on a display device.

[0003] In some ultrasound diagnostic apparatuses, ultrasound transmission / reception is performed on a 3D region to acquire ultrasound volume data. In some cases, the volume data is acquired by a 3D probe, wherein an operator changes the angle or pressure of the ultrasound probe or moves the ultrasound probe to perform ultrasound transmission / reception on a 3D region and acquire ultrasound volume data. Summary of the invention

[0004] In one embodiment, the method includes instructing a user to navigate an ultrasound probe via movement in a horizontal plane or pressure in a vertical plane based on a first image previously acquired at a first anatomical location, while simultaneously acquiring a second image at a second, different anatomical location and automatically combining the first image and the second image together.

[0005] It should be understood that the above brief description is provided to introduce selected concepts further described in the detailed description in a simplified form. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims following the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present invention will be better understood by reading the following description of non-limiting embodiments with reference to the accompanying drawings, in which:

[0007] Figure 1 An example of an ultrasound imaging system according to an embodiment of the present disclosure is shown;

[0008] Figure 2 An exemplary schematic diagram of various system components of an ultrasound imaging system according to one embodiment of the present disclosure is shown.

[0009] Figure 3 A high-level flow chart for combining two ultrasound images is shown.

[0010] Figure 4 A method for acquiring a first ultrasound image and a second ultrasound image is shown.

[0011] Figure 5 An example of ultrasound image acquisition for combination with a previously acquired ultrasound image is shown.

[0012] Figure 6 An example of combining a previously acquired ultrasound image with a current ultrasound image is shown.

[0013] Figure 7 The displacement of the ultrasound probe along the horizontal plane is shown.

[0014] Figure 8 A graph of the Huber function is shown. DETAILED DESCRIPTION

[0015] The following description relates to embodiments of an ultrasound device configured to combine two or more ultrasound images. Figure 1 An example of an ultrasound imaging system is shown in Figure 2 A schematic diagram of various system components of an ultrasound system is shown in FIG. Figure 2 The ultrasound system may be configured to acquire 3D ultrasound images. In one example, the 3D ultrasound image is generated via multiple slices of volume data. When scanning a larger area, due to the nature of the volume data and the image generation from the volume data, it may be difficult for the user to visualize how to position the ultrasound probe in order to generate adjacent images that can be stitched together. Therefore, in some examples, the user interface may provide instructions to the user on how to move, position, and apply pressure with the ultrasound probe in order to improve the ability to generate multiple images that can be stitched together from the overall volume data to form a larger image from the volume data acquired over time as the probe moves.

[0016] Each slice of the image may correspond to the temporal and spatial dimensions of the anatomical region scanned to generate the volumetric data. The ultrasound system may analyze slices acquired or formed from different sets of volumetric data, with reference points marked on each slice. Slices with matching and / or overlapping reference points may be stitched together to form a larger complete image of the multiple slices segmented relative to each other.

[0017] The ultrasound device may be configured to identify reference points in two or more diagnostic images generated from volume data based on a deep learning neural network and / or algorithmic approach. The diagnostic images may be generated from different sets of volume data acquired at different times, locations, and pressure levels. For example, more than two images may be generated from two sets of volume data, and two images may be selected from the images whose reference points meet selected criteria. The reference points may be used to identify a reference pair between the two or more images. Thus, if at least two images include a reference pair, the two images may be merged together along corresponding edges so that the images do not overlap but touch along the edges to form a final continuous image.

[0018] By combining volumetric data acquired at different times, locations, and pressure levels via stitching of 2-D images, the user can better observe quantitative measures such as ventricular volume, fluid accumulation, and dynamic movement of blood cells. Additionally or alternatively, the large image can be rotated or cropped in real time to reveal anatomical structures within the tissue volume, which may be advantageous in cancer treatment, etc. Still further, instructing the user how to position, apply, and / or manipulate the probe can provide improvements to the combined larger 2-D image.

[0019] Figure 3 and Figure 4 A method for combining these figures is shown in FIG. Figure 5 An embodiment of image acquisition is shown in which a match value between a first image and a second image of different sets of volume data is not greater than a threshold match value. The user is instructed to move or continue to move the probe a selected amount in each of the horizontal plane and the vertical plane to reduce the offset between the images, thereby increasing the match value. Figure 7 Another example of instructions provided to a user on a display device is shown. Figure 6 An embodiment of image acquisition is shown, wherein the matching value of the first image and the second image is greater than a threshold matching value. Figure 6 The images in an embodiment are joined along respective edges without overlapping.

[0020] Figure 1 An exemplary ultrasound system 100 configured for performing ultrasound is shown. In one example, the ultrasound system 100 is a handheld ultrasound system. The ultrasound system includes a probe 101 including a housing 102 in which an ultrasound transducer array 104 is positioned. The transducer array is positioned at a scan head and inserted into the tissue to be scanned, thereby producing a clearer image than an external probe due to the proximity of the transducer array to the inner layers of the tissue. The probe 101 may include a position tracking device including one or more position sensors ( Figure 2 ) to allow position and orientation sensing of the transducer. Suitable position sensors (e.g., gyroscopic, magnetic, optical, radio frequency (RF)) may be used.

[0021] A fully functional ultrasound engine for driving an ultrasound transducer and generating volumetric fetal, cardiac, transrectal, and intravascular ultrasound data from these scans along with associated position and orientation information may be coupled to the probe. In some examples, the volumetric data may be collected for use in obstetrics, cardiology, surgical guidance, vascular imaging, local anesthesia, and anesthesia. For example, the ultrasound engine may include a processor and a memory, and / or may be implemented using a processor and a memory. The ultrasound engine may be included as part of a scan processor 106 coupled to the probe. The volumetric scan data may be transmitted to another computer system for further processing using any of a variety of data transmission methods known in the art. A general-purpose computer that may be implemented on the same computer as the ultrasound engine is also provided for general user interaction and system control. The ultrasound system may be a self-contained, stand-alone unit that can be moved from one room to another, such as a cart-based system, a handheld system, or other portable system. In addition, it may be remotely controlled, configured, and / or monitored by a remote station connected via a network. Figure 1 Ultrasound system.

[0022] Figure 2 2 is a block diagram 200 schematically illustrating various system components of an ultrasound imaging system, including the probe 101, the scan processor 106, a client device 230, and a remote device 240. In some embodiments, the scan processor may also be coupled to a remote department information system, a hospital information system, and / or an internal or external network to allow operators at different locations to provide commands and parameters and / or access image data (as explained in more detail below, the remote device 240 is one non-limiting example of one such remote system to which the scan processor may be coupled).

[0023] First, referring to the probe 101, the probe 101 includes a transducer array 104. Figure 1 As explained, the transducer array may be positioned within the housing, and the housing and transducer array may be configured to be manually moved by an operator during an ultrasound examination. The transducer array may include an array of transducer elements (such as piezoelectric elements) that convert electrical energy into ultrasonic waves and then detect reflected ultrasonic waves.

[0024] The probe 101 may also include a memory 224. The memory 224 may be a non-transitory memory configured to store various parameters of the transducer array 104, such as transducer position information obtained from the probe's position tracking device 228, transducer usage data (e.g., the number of scans performed, the total amount of time spent scanning, etc.), and transducer specification data (e.g., the number of transducer array elements, array geometry, etc.) and / or identification information of the probe 101 (such as the probe's serial number). The memory 224 may include removable and / or permanent devices, and may include optical memory, semiconductor memory, and / or magnetic memory, etc. The memory 224 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, and / or additional memory. In one example, the memory 224 may include RAM. Additionally or alternatively, the memory 224 may include EEPROM.

[0025] The memory 224 may store non-transient instructions that may be executed by a controller or processor (such as the controller 226) to implement one or more methods or routines as described below. The controller 226 may receive outputs from various sensors of the tracking device 228 and trigger actuation of one or more actuators and / or communicate with one or more components in response to the sensor outputs. In one example, the tracking device 228 may include one or more position sensors, accelerometers, gyroscopes, pressure sensors, strain sensors, and / or temperature sensors. The tracking device 228 may also include a reference marker detector configured to detect a reference marker positioned on a patient receiving an ultrasound scan. The reference marker detector may be configured to detect a magnetic reference marker, an optical reference marker, or other suitable reference marker positioned inside or outside the patient. Before and / or during image acquisition, the position of the probe relative to the reference marker (in six degrees of freedom, including translation across three vertical axes and rotation around three vertical axes) may be determined by the output of one or more position sensors of the tracking device 228, and the position is stored in the memory 224 and / or sent to the scanning processor 106. The output from the sensor can be used to provide feedback to the operator of the probe 101 (via the scan processor 106, the client device 230, and / or a displayed user interface of another device). For example, if the probe is not in a predetermined position, the operator can be instructed to reposition the probe prior to image acquisition. In another example or in addition, the operator can be instructed to adjust the angle, depth, and / or position of the probe during scanning, such as further described herein.

[0026] For example, the probe 101 may communicate with the scan processor 106 to send raw scan data to the image processor 212. Additionally, in some examples, data stored in the memory 224 and / or output from the tracking device 228 may be sent to the scan processor 106. Furthermore, various actions of the probe 101 (e.g., activation of transducer elements) may be initiated in response to signals from the scan processor 106. The probe 101 may optionally communicate with a display of the scan processor and / or a display 236 of a client device, for example, to notify a user to reposition the probe 101 (as explained above) or to receive information from the user.

[0027] Turning now to the scan processor 106, it includes an image processor 212, a storage device 214, a display output 216, and an ultrasound engine 218, which is configured to send information to be displayed on a display device of the scan processor 106 and / or another display device (such as a display of a remote device 240). The ultrasound engine 218 can drive the activation of the transducer elements of the transducer array 104. In addition, the ultrasound engine 218 can receive raw image data (e.g., ultrasound echoes) from the probe 101. The raw image data can be sent to the image processor 212 and / or to a remote processor (e.g., via a network), and the raw image data is processed to form a displayable image of the tissue sample, which can be sent to a coupled display device via the display output 216. It should be understood that in some embodiments, the image processor 212 can be included with the ultrasound engine 218.

[0028] Information may be transmitted from the ultrasound engine 218 and / or the image processor 212 to a user of the ultrasound probe system via the display output 216 of the scan processor 106. In one example, the user may include an ultrasound technician, a nurse, or a doctor (such as a radiologist). For example, after real-time acquisition of imaging data (e.g., ultrasound imaging data), an image may be generated from the data and displayed on a display device (of the scan processor 106 or an operatively coupled device) via the display output 216. In another example, information related to the parameters of the scan (such as the progress of the scan) may be sent to the display via the display output 216. In another example, the display output 216 may output a user interface configured to display images or other information to the user. In addition, the user interface may be configured to receive input from the user and send the input to the scan processor 106. In one example, the user input may be via a touch screen of the display. However, other types of user input mechanisms are possible, such as a mouse, a keyboard, and the like.

[0029] The scan processor 106 may also include a storage device 214. Similar to the memory 224 of the probe, the storage device 214 may include removable and / or permanent devices, and may include optical memory, semiconductor memory, and / or magnetic memory, etc. The storage device 214 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, and / or additional memory. The storage device 214 may store non-transitory instructions that may be executed by a controller or processor (such as the ultrasound engine 218 or the image processor 212) to implement one or more methods or routines as described below. The storage device 214 may store position information of the ultrasound probe, such as transmitted via the tracking device 228. The storage device 214 may store raw image data received from the ultrasound probe, processed image data received from the image processor 212 or a remote processor, and / or additional information.

[0030] The scanning processor 106 and / or the probe 101 may be communicatively coupled to a client device 230. The client device 230 may include a suitable computing device, such as a laptop, a tablet computer, or a mobile device. The client device 230 may include a controller 232, a memory 234, a display 236, and a camera 238. The controller 232 may include an image processor and an ultrasound engine, which are similar to the image processor and ultrasound engine of the scanning processor 106. In this way, the client device 230 may drive the actuation of the elements of the transducer array 104 of the probe 101, receive raw image data from the probe 101, and / or process the raw image data into one or more images. The controller 232 may further execute instructions stored in the memory 234 to implement one or more of the methods described herein.

[0031] For example, the instructions may enable the controller to collect volume data in the form of multiple image slices. Each slice may correspond to a dimension and time when the ultrasound data was collected. The reference points of the individual slices may be analyzed so that instructions may be displayed to the user via a user interface, where the instructions may include how to arrange the probe and where to arrange the probe to collect a subsequent image slice. When the dimensions of the subsequent image sufficiently match the dimensions of the previous image slice through measurement of the matching value and comparison of the matching value relative to a threshold matching value, the image slices may be stitched together. The combination of the two image slices may produce a larger 3D image of the target anatomical location.

[0032] The memory 234 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, and / or additional memory. The memory 234 may store non-transitory instructions that may be executed by a controller or processor (such as the controller 232) to implement one or more methods or routines as described below. The memory 234 may store position information of the ultrasound probe, such as transmitted via the tracking device 228 of the probe. The memory 234 may store raw image data received from the ultrasound probe, processed image data received from the image processor, and / or additional information.

[0033] Display 236 may display suitable information, such as one or more user interfaces, images, etc. Display 236 may display content sent from controller 232 or other suitable processor. Camera 238 may include front-facing and rear-facing image sensors, each of which is configured to capture visible light images and / or light-based depth images. The visible light images and / or light-based depth images may be displayed on display 236 in the form of static images and / or videos. In some examples, data from the front-facing and / or rear-facing image sensors may also be used to determine direction / position and orientation data (e.g., from imaging environment features), thereby enabling position / motion tracking of client device 230 in a real-world environment.

[0034] The client device 230 may also include a tracking device 239. The tracking device 239 may be configured to provide position and / or orientation data of the client device 230 to the controller 232. In one example, the tracking device may be configured as an inertial motion unit (IMU) including a three-axis or three-degree-of-freedom (3DOF) position sensor system. The exemplary position sensor system may, for example, include three gyroscopes to indicate or measure changes in orientation of the client device around three orthogonal axes (e.g., roll, pitch, and yaw) in 3D space. The orientation derived from the sensor signals of the IMU may be used to display one or more augmented reality images via the display 236 in a realistic and stable position and orientation.

[0035] In another example, the IMU may be configured as a six-axis or six-degree-of-freedom (6DOF) position sensor system. Such a configuration may include three accelerometers and three gyroscopes to indicate or measure changes in position of the client device along three orthogonal spatial axes (e.g., x, y, and z) and changes in device orientation around three orthogonal rotational axes (e.g., yaw, pitch, and roll). In some embodiments, the position and orientation data from the image sensor and the IMU may be combined to determine the position and orientation of the client device.

[0036] The client device 230 may be communicatively coupled to a remote device 240. The remote device 240 may include a suitable computing device that may be used to store and / or display acquired ultrasound images, and may also be communicatively coupled to the scan processor 106. The remote device 240 may be part of a picture archiving and communication system (PACS) configured to store, for example, patient medical history, imaging data, test results, diagnostic information, management information, and / or scheduling information. The remote device 240 may include a PACS server that includes a computer-readable storage medium suitable for storing image data for later retrieval and viewing at, for example, a PACS workstation. The remote device 240 may include a controller 242, a memory 244, and / or a display 246. The memory 244 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, and / or additional memory. The memory 244 may store non-transitory instructions that may be executed by a controller or processor (such as the controller 242) to implement one or more methods or routines as described below. The memory 244 may store position information of the ultrasound probe as transmitted via the probe's tracking device 228. The memory 244 may store raw image data received from the ultrasound probe, processed image data received from the image processor, and / or additional information.

[0037] The methods described herein may illustrate a method for helping a user automatically combine multiple images. The method may include guiding the user to move the ultrasound probe, wherein the guidance may include the direction to move the probe and the pressure to apply to the probe. Instructions are provided in this manner for actuating the probe in three-dimensional space while scanning so that two or more images can be accurately combined.

[0038] The purpose of combining two or more images together is to acquire a wide image for a long scan area. For example, the purpose is to obtain a complete image of a relatively large tumor in a patient's abdomen and thus more accurately measure the size of the tumor.

[0039] The previous example of combining medical diagnostic images together forces the user to identify the region of interest in the first portion of the long scan area image. However, this step of the process can be difficult because the medical diagnostic images are gray and there is no clear boundary between the scanned object and the user (especially an untrained user). Therefore, marking the region of interest can be time consuming and / or inaccurate. Ultrasound data acquisition of the target area can be performed from multiple positions. For example, images can be acquired from anterior, posterior, and / or medial positions, thereby generating a large amount of data. It can be difficult for the user to combine each of these images in this large amount of data to generate a single large image (e.g., a long scan area image).

[0040] In addition, ultrasound images may depend on probe position, orientation, and pressure. Thus, moving the ultrasound probe along the patient's body may adjust the ultrasound image, holding the probe at various angles may adjust the ultrasound image, and using pressure applied to the patient's body with the ultrasound probe may adjust the ultrasound image, further complicating combining images acquired from different locations. By adjusting the ultrasound image, the plane, position, and / or angle at which the reference point of the anatomical location is visualized may change the appearance of the reference point. These three factors may increase the difficulty of aligning the second portion of a long scan area image with the first portion.

[0041] In order to solve the above problems, the ultrasonic diagnostic equipment can help the user to combine two or more images. Instructions can be provided to the user when the user collects volume data so that images from the volume data set can be combined more accurately. Adjacent parts of these images can be compared continuously during scanning to calculate matching values. The matching value can be compared with a threshold. If the matching value is greater than the threshold, images from multiple sets of volume data can be combined. If the matching value is not greater than the threshold, instructions can be provided to recapture at least one adjacent part of these images to increase the matching value. The instructions used may include one or more directions of movement during scanning, the amount of movement (in each of the one or more directions), one or more angles of movement, one or more axes around which to rotate, and / or the amount of pressure applied. Reference points between images collected at different positions can be more accurately matched and these images are stitched.

[0042] Now go to Figure 3 , which shows a high-level flow chart showing a method 300 for combining different medical diagnostic images. Instructions for implementing method 300 and the remaining methods included herein may be provided by a controller based on instructions stored on a memory of the controller and from sensors of the engine system (such as those described above in conjunction with Figure 2 The controller may use an engine actuator of the engine system to adjust the engine operation according to the following method.

[0043] The method 300 begins at 302, which includes acquiring a first image. The first image may be one of a plurality of images of a target area of ​​a patient. The first image may be acquired by an ultrasound probe (such as Figure 1 An ultrasound probe 101) captures a first image.

[0044] Method 300 proceeds to 304, which includes marking two or more reference points in the first image. The reference points may correspond to organs, bones, muscle insertions, etc. Thus, the reference points may correspond to known components of the patient's anatomy. For example, the reference points may include one or more of tendons, bones, joints, organs, nerves, blood vessels, fetuses, etc. The reference points may be marked on the image and / or may be stored in a memory of a controller of the ultrasound device. As will be described below in conjunction with Figure 4 In more detail, the first image may be compared to previously acquired images of the target area, wherein reference points may be compared to previously evaluated reference points to determine the legitimacy and quality of the first image.

[0045] The method 300 proceeds to 306, which includes displaying the real-time probe position. The probe position may be displayed in a first plane and a second plane. The first plane (also referred to as a horizontal plane) may show the direction of movement along the patient's body. The second plane (also referred to as a vertical plane) may show the pressure of the probe on the patient's body and may be perpendicular to the first plane. In this way, the user may be aware of both the direction and pressure of the probe on the patient's body, so that a similar direction and similar pressure may be applied during the acquisition of a subsequent image that is desired to be combined with the first image.

[0046] In one example, if the medical imaging is of the patient's abdomen, the first plane (including the direction of movement) may be parallel to the coronal plane. The second plane may be perpendicular to the coronal plane and parallel to the sagittal or transverse plane. It should be understood that the orientation of the first plane and the second plane may be adjusted based on the position of the medical imaging relative to the patient's body, so that the first plane may be parallel to one of the coronal plane, the sagittal plane, or the transverse plane.

[0047] The method 300 proceeds to 308, which includes acquiring a second image. In one example, the second image is an image of a target area that is directly adjacent to at least a portion of the first image. In one example, the first image may be a first half of a final image, and the second image may be a second half of the final image. Additionally or alternatively, the first image and the second image may be separate images to be combined together in the final image, wherein the first image and the second image share a common border. In one example, the first image may visualize a portion of a tumor or a portion of a reference point, and the second image may visualize a remaining portion of the tumor or a remaining portion of the reference point.

[0048] The method 300 proceeds to 310, which includes displaying a real-time match value. The match value may correspond to an accuracy of the combination of the first image and the second image. That is, as the difference between the first image and the second image increases, the match value decreases. Thus, a higher match value may indicate a smaller difference in the horizontal and vertical directions in which the first image and the second image were acquired. A higher match value may further correspond to a more accurate combination of the first image and the second image, so that the scanned portion of the patient's anatomy can be more reliably measured and / or analyzed.

[0049] Method 300 proceeds to 312, which includes determining whether the match value is greater than a threshold. The threshold may be based on a plurality of previously combined images. In one example, the threshold is a fixed value. For example, the threshold may be equal to a score of 9 in a scoring system where the highest value is 10 and the lowest value is 1. As another example, the threshold may be a percentage, such as 85%. Additionally or alternatively, the threshold may be a dynamic value, where the plurality of previously combined images continuously updates the threshold as different combinations and / or combinations of images are learned. Such examples will be described in more detail below.

[0050] If the match value is not greater than the threshold, the method 300 proceeds to 314, which includes rescanning the portion of the second image. The rescanning may include adjusting the force applied to the second image region during the recapture of the second image. By adjusting the force, the planes in which the first image and the second image were captured may be more closely matched, thereby increasing the match value.

[0051] If the match value is greater than the threshold, the method 300 may proceed to 316 to combine the first image and the second image. Combining the first image and the second image may include splicing the first image and the second image at a common boundary, wherein the common boundary is a central axis of a final image including the first image and the second image. That is, the area where the first image and the second image are combined is the middle of the final image. The boundaries of the first image and the second image may be adjacent to each other along a horizontal axis. Additionally or alternatively, the reference point may extend across the first image and the second image such that the boundaries divide the reference point across the first image and the second image. In this way, the correct boundary for combining the first image and the second image may be selected.

[0052] This specification describes in more detail herein the acquisition and combination of a first image and a second image based on one or more of an algorithmic method or a deep learning network. Broadly speaking, a spatial transformation matrix of two adjacent images is calculated in real time. Some values ​​in the matrix show translation, so the accumulation of these values ​​can provide a movement distance in the horizontal and vertical directions and / or planes. As described above, the horizontal direction and / or plane does not include pressure and is roughly parallel to the patient's skin (e.g., the patient's scan surface). The vertical direction and / or plane includes pressure and is perpendicular to the patient's scan surface. A matching value between the first image and the second current image is provided via one or more of an algorithmic method and a deep learning network.

[0053] During scanning (eg, image acquisition), an image may be obtained via periodic sampling. If the sampling period is equal to a certain duration (T), then the current image may be F(mT), m∈R + , and the first previously acquired image may be F(0). The feature point pair [(x x ,yi ),(x′ i ,y′ i )],i∈R.

[0054] Noise suppression may be based on a robust least squares algorithm, which may include calculating a spatial transformation matrix directly from the feature point pairs. If each of the feature point pairs is calculated accurately, a matrix as shown in Formula 1 below may be calculated.

[0055]

[0056] s x and y is a scale factor in the x-axis and y-axis, θ is the rotation angle between the two images, and pe,pf is the movement distance in the x-axis and y-axis. In one example, the horizontal direction is parallel to the x-axis, and the vertical direction is parallel to the y-axis. Formula 1 can be rewritten in a simplified homogeneous form in the example of Formula 2 below.

[0057]

[0058] The resulting matrix and / or vector can be represented by the notation shown below.

[0059]

[0060] There may be errors in the feature point pairs, and in some examples, one or more of the feature point pairs may be wrong. In order to prevent Formula 1 from representing all feature point pairs, a matrix can be generated to reduce the total error to the lowest possible value. Such a matrix can be represented by the following matrix 4.

[0061]

[0062] Calculate matrix 4 and discard the pairs that cannot be mapped by matrix 4. The discard can be based on the Huber function, which can reduce the impact and / or magnitude of the wrong pair. By doing so, the wrong or erroneous feature point pairs can be discarded to increase the accuracy of the spatial transformation matrix. The Huber cost function is represented by the combination of the following formula 3 and formula 4.

[0063]

[0064] Formula 3 can be a strictly convex function, where Newton's method is used to solve the matrix 4. The range of M and M is given by Figure 8 The curve graph is shown in .

[0065] Based on the determination of the following formula 5, it is concluded that the point pair is incorrect.

[0066] ||[x i ,y i ,1]X-[x′i ,y′ i ,1]||2>ξ (Formula 5); where ξ is a hyperparameter.

[0067] The discarded erroneous feature point pairs can provide clean feature point pairs B1, B2, and can be used to solve the spatial transformation matrix X2 with B1, B2 as part of the solution of the spatial transformation matrix based on L2-norm approximation. By omitting the erroneous pairs, X2 can have an analytical solution in the case of L2-norm approximation. The following formula 6 shows the problem to be solved

[0068]

[0069] Formula 7 shows the expanded form of Formula 6, where the cost function is a strictly convex function and takes a minimum value when the derivative is equal to 0, as shown below in Derivative 1.

[0070]

[0071] In one example, if the surface of the patient is considered to be in a first plane (e.g., an x-plane and / or a horizontal plane), the probe may be actuated from a first position to a second, different position along the x-plane. The value pe is the offset of the probe during the first cycle (e.g., the displacement of the probe from the first position to the second position). Therefore, the offset (e.g., mismatch) between the first image and the current image may be obtained by accumulating pe for each cycle based on the following formula 8.

[0072]

[0073] The probe may be moved in a second plane (e.g., the y-plane and / or the vertical plane), wherein the probe is pressed or pulled in a direction perpendicular to the x-plane at a fixed position along the y-plane. Additionally or alternatively, the orientation of the probe may cause the probe to be pressed or pulled in a direction at an angle to the x-plane. Movement along the y-plane at any rate may cause a change in the pressure applied to the probe parallel to the y-plane. Any movement of the probe that is not parallel to the y-plane may be measured along the x-plane.

[0074] Changes in the pressure applied to the probe may cause changes in the scanned image. The image scanned during the xxx-xxx depth corresponds well to the pressure change. In this way, the solution for the probe pressure change within a cycle may be similar to the probe offset. However, only the point pairs within the xxx-xxx depth participate in the calculation. In the example of monitoring pressure along the y plane, the image offset between the first image and the current image may be obtained based on the following formula 9.

[0075]

[0076] After determining the offset in the x-plane via Formula 8 and determining the offset in the y-plane via Formula 9, a match value may be determined based on a combination of the offsets in the x-plane and the y-plane. In one example, the match value is a measure of the accuracy of combining the first image and the second image. In one example, the accuracy is inversely proportional to the difference between the pixel intensities of the two images at the location where the two images are to be combined. Therefore, a relatively low difference between the pixel intensities (e.g., a higher similarity) may indicate a relatively high degree of match. The match value may be calculated via a combination of the following Formula 11 and Formula 10.

[0077]

[0078] Where D is a hyperparameter.

[0079] If the match value is greater than the threshold match value, the two images may be combined, as described above in connection with method 300. If the match value is not greater than the threshold match value, the two images may not be combined, and the operator may be required to adjust the movement and / or pressure of the probe to reduce the offset in the x-plane or y-plane.

[0080] The image combination may also be performed via a deep learning network structure, such as a deep learning neural network. The deep learning network structure may be used in combination with the previously described algorithmic methods or alone. The deep learning network structure may include a plurality of previously acquired image pairs, wherein each pair includes at least two ultrasound images at two different but proximal body parts. A proximal body part may be defined as a body part whose ultrasound images include an image pair that shares at least one common boundary.

[0081] The two images can be resized to a common image size (e.g., 448 height × 448 width) and then processed through a convolutional layer to generate a feature map and / or activation map. The convolutional layer can be a VGG, GoogLeNet, ResNet, and similar convolutional layers as known to those of ordinary skill in the art. As an example, if the convolutional layer is GoogLeNet, the size of the feature map is 14×14×1024. The two feature maps are combined together, and the final size is 12×14×2048. A 14×14×2048×2 convolution kernel is operated on the combined two feature maps without padding, and the two values ​​obtained are equal to the offset between the two images in the horizontal and vertical directions parallel to the x-plane and y-plane, respectively. A portion of each feature map is cut from the edge of each feature map, where the portion is 1×14×1024. The two parts are combined into a size of 1×14×2048, where a 1×14×2048×2 convolution kernel layer is operated on the two combined parts without padding to obtain a two-valued activated softmax function, which is essentially equal to the matching value.

[0082] For the offset output, smoothing L1 can be selected as the loss function expressed by the following formula 11.

[0083]

[0084] Loss offset =smooth l1 (xx t )+smooth l1 (yy t ) (Formula 12)

[0085] The total offset value including the offsets in the x-plane and the y-plane may be determined by Formula 12. The matching value may be calculated by selecting cross entropy as the loss function, as shown in the following Formula 13.

[0086]

[0087] The total loss may be equal to the weighted sum of Equations 12 and 13 as shown below in Equation 14.

[0088] Total loss = λLoss offset +Loss match (Formula 14)

[0089] The total loss may be compared to a threshold match value as described above.

[0090] Now go to Figure 4 , which shows a method 400 for determining reference points in two adjacent images, matching the reference points, and determining matching values ​​along a stitching edge of the adjacent images.

[0091] The method 400 starts at 402, which includes determining whether an ultrasound image is being acquired. If the probe is moving against the patient in a horizontal or vertical direction, an ultrasound image is being acquired. If an ultrasound image is not being acquired, the method 400 proceeds to 404, which includes maintaining the current operation and not displaying the ultrasound image on the display device.

[0092] If an ultrasound image is being acquired, method 400 proceeds to 406, which includes displaying the ultrasound probe position in real time on a screen of a display device. The probe may provide feedback to the display device regarding the acquired image (e.g., a slice of volume data) and the position of the probe relative to the patient to display a real-time image of the probe. For example, the display may show an image currently being acquired via the probe and a box or some other shape illustrating the precise position of the probe. In addition, the position of the probe may be shown relative to the x-plane and the y-plane. This may be shown via an arrow or other shape. In one example, the size of the box may increase in response to an increase in the pressure of the probe against the patient.

[0093] Method 400 proceeds to 408, which includes detecting reference points in the acquired images. Detecting the reference points may be performed via the deep learning neural network and / or algorithmic methods described above. The detected reference points may correspond to organs, blood vessels, nerves, joints, etc. Additionally or alternatively, if the patient has a tumor or other growth previously identified during diagnostic imaging, the tumor or other form of growth (e.g., a fetus) may be learned and added to the patient's reference point dataset.

[0094] Method 400 proceeds to 410, which includes matching the detected reference points between two adjacent images. Two adjacent images may be defined as two images that visualize shared reference points from different locations, wherein the two images share a border or other shared feature. Additionally or alternatively, the adjacent images may include a first image that visualizes a first anatomical location and a second image that visualizes a second anatomical location, and wherein the first anatomical location and the second anatomical location are consecutive. For example, the probe may visualize reference points from a primarily anterior position of the patient's body during the first image and from a primarily posterior position of the patient's body during the second image. Each of the first image and the second image may include a portion acquired from an outer portion of the patient's body. Therefore, the first image and the second image may include one or more reference points that may be matched at the outer portions of these images.

[0095] In one example, the first image and the second image correspond to slices of a volume of data. The images may be used in a transrectal ultrasound procedure where the first image is collected at a first time from a lateral position and the second image is collected at a second time after the first time from a posterior position.

[0096] Method 400 proceeds to 412, which includes using a robust least squares calculation to remove mismatched reference points. This may include a Huber calculation, as described in conjunction with equations 3, 4, and 5 above. If an algorithm calculation is used, this may increase the accuracy of equation 1.

[0097] Method 400 proceeds to 414, which includes calculating a spatial transformation matrix. The spatial transformation matrix may be substantially equal to Equation 1, wherein the accuracy of the spatial transformation matrix is ​​increased by taking into account the error in the reference alignment.

[0098] Method 400 proceeds to 416 , which includes determining offsets in the horizontal plane and the vertical plane relative to the adjacent image. The offset in the horizontal plane may be calculated via Equation 8, and the offset in the vertical plane may be calculated via Equation 9.

[0099] The method 400 proceeds to 418, which includes calculating a matching value. The matching value may be based on a combination of offsets in the horizontal plane and the vertical plane. Additionally or alternatively, the matching value may be based on a difference in pixel intensity at a splicing edge between adjacent images, wherein the splicing edge corresponds to an edge or boundary of the first image and the second image to be merged.

[0100] Once the match value is determined, the match value can be compared to a threshold match value, similar to 312 of method 300. If the match value is greater than the threshold match value, the two images can be joined together. When the user is scanning and the match value exceeds the threshold match value, the user can be instructed to briefly stop the movement of the probe so that the images can be merged together at their respective stitching edges. The user can then be instructed to continue actuating the probe after the merge.

[0101] Go to Figure 5 , which shows an embodiment 500 of a display 502, which shows the user a current probe position 504, written instructions 506, and visual instructions 508. The current probe position 504 may be based on feedback from a position sensor coupled to the probe, wherein the position sensor may provide feedback to a controller of the ultrasound device. Based on each of the position and image provided by the probe, the written instructions 506 and the visual instructions 508 may instruct the user to adjust the probe position to increase the match value between the first image and the second image. As shown, the visual instructions 508 include arrows, wherein the magnitude and / or size of the arrows may indicate the required commands. The first arrow 512 is parallel to the x-axis of the axis system 590, which is substantially similar to the above-mentioned x-plane and / or horizontal plane. The second arrow 514 is parallel to the y-axis of the axis system 590, which is substantially similar to the above-mentioned y-plane and / or vertical plane. The first arrow 512 is greater than the second arrow 514, thereby indicating that it is desired to move the probe more along the horizontal plane and less along the vertical plane to increase the match value.

[0102] Now go to Figure 7 , which shows an embodiment 700 showing an example of an instruction for a user to displace a probe along a horizontal plane. As known to those of ordinary skill in the art, the probe of an ultrasound device can scan an image in three dimensions shown via an axis system 790, which includes an x-axis, a y-axis, and a z-axis perpendicular to each of the x-axis and the y-axis. A horizontal plane can be arranged along the x-axis and the y-axis. A vertical plane can be arranged along the y-axis and the z-axis.

[0103] The width of the probe is shown via square bracket 702. The depth to which the probe sound waves can penetrate and acquire an image is shown via square bracket 704. The first image 710 represents an image previously acquired by the user. The ultrasound device can receive the image and identify reference points within the image. Figure 7 In the example of FIG. 7 , the first image 710 includes nine reference points.

[0104] After the acquisition of the first image 710, the user may select a long scan mode or a wide scan mode. In response, the ultrasound device may project the first image 710 onto a display device such as a Figure 2 The first image 710 and the second image 720 may be displayed on the first half of the display device 236 of the client device 230. The second image 720 that is desired to be acquired after the first image 710 may be displayed on the second half of the display device. Thus, the first image 710 and the second image 720 are displayed side by side on the display device for the user to easily visualize the final image including each of the first image 710 and the second image 720.

[0105] exist Figure 7 In the example of , the user is instructed to move the probe along a horizontal plane equal to one third of the probe width. Therefore, the second image 720 may no longer include reference points 1, 2, and 3, as shown in the dashed box 712. Due to the displacement of the probe in the horizontal direction, new reference points 10, 11, and 12 are obtained, as shown in the dashed box 722. Figure 7 In the example of FIG. 7 , the user is not instructed to adjust the displacement of the probe along the vertical plane in order to maintain a substantially uniform depth between the first image 710 and the second image 720 .

[0106] After the acquisition of the second image 720, a matching value may be calculated. If the matching value is greater than a threshold matching value, the first image 710 may be combined with the second image 720. Figure 7 In the example of , stitching the first image 710 to the second image 720 may be performed at an edge of the image between reference points 7 - 9 and reference points 10 - 12 .

[0107] An exemplary edge where the first image 710 and the second image 720 can be combined and / or spliced ​​can be shown via a dotted line 732 on the third image 730. In this way, redundant information can be omitted. Therefore, the third image 730 includes each of the first image 710 and the second image 720, wherein the images are combined side by side with each other without overlapping. The user can choose to continue imaging the patient, wherein additional images can be added to the third image 730 to further increase the width of the image.

[0108] Now go to Figure 6, which shows an embodiment 600 in which a first image 602 and a second image 604 are merged along a common stitching edge 606. In one example, the stitching edge 606 corresponds to the central axis of the final image that includes the merger of the first image and the second image. As shown, the first image and the second image are stitched together side by side without overlapping. In this way, the images are continuous with each other so that the anatomical locations visualized by each of the images are also continuous. In some examples, if portions of the first image and / or the second image are redundant so that overlap can occur, these portions can be cropped (e.g., removed). Additionally or alternatively, these portions can be used as another reference point to further match the first image and the second image.

[0109] Embodiment 600 also includes written instructions 608 and visual instructions 610. As shown, written instructions 608 indicate that the user can combine these images and stop the movement of the probe. Visual instructions 610 no longer provide arrows indicating the direction in which the probe may be desired to be moved. In response, these images can be automatically combined and / or spliced. Automatically combining and / or splicing these images can include situations where the user does not provide more input. Therefore, the user may not indicate a reference point or automatically combine these images.

[0110] Thus, a medical imaging device, such as an ultrasound device, is configured to merge two images of a reference point. In one example, the images are acquired from different vantage points relative to the patient's body. The technical effect of merging the two images of the reference point is to provide a single combined image of the reference point for medical analysis with high accuracy to accurately diagnose the patient's condition.

[0111] As used herein, the element or step that is listed in the singular and begins with the word "one" or "an" should be understood as not excluding a plurality of the elements or steps, unless such exclusion is explicitly stated. In addition, the reference to "an embodiment" of the present invention does not exclude the existence of the additional embodiment that also includes the cited feature. In addition, unless explicitly stated to the contrary, the embodiment of "including", "comprising" or "having" the element or multiple elements with a specific characteristic may include other such elements that do not have the characteristic. The terms "including" and "in..." are used as the popular language equivalents of the corresponding terms "including" and "wherein". In addition, the terms "first", "second" and "third" etc. are only used as marks, and are not intended to impose numerical requirements or specific position order on their objects.

[0112] The control methods and routines disclosed herein may be stored as executable instructions in a non-transitory memory and may be implemented by a control system (including a combination of a controller and various sensors, actuators, and other engine hardware). The specific routines described herein may represent one or more of any number of processing strategies (such as event-driven, interrupt-driven, multi-tasking, multi-threading, etc.). Therefore, the various actions, operations, and / or functions shown may be executed in the sequence shown, executed in parallel, or omitted in some cases. Similarly, the order of processing is not necessarily required to achieve the features and advantages of the exemplary embodiments described herein, but is provided for ease of illustration and description. One or more of the actions, operations, and / or functions shown may be repeatedly performed according to the specific strategy used. In addition, the actions, operations, and / or functions may be graphically represented as codes in a non-transitory memory of a computer-readable storage medium to be programmed into an engine control system, wherein the actions are implemented by executing instructions in a system including a combination of various engine hardware components and an electronic controller.

[0113] This written description uses examples to disclose the invention, including the best mode, and also to enable a person skilled in the relevant art to practice the invention, including making and using any devices or systems and performing any included methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to a person skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insignificant differences from the literal language of the claims.

Claims

1. A method for an imaging device, comprising: instructing a user to navigate the ultrasound probe via movement in a horizontal plane or pressure in a vertical plane based on a first image previously acquired at a first anatomical location while acquiring a second image at a second anatomical location; as well as automatically combining the first image and the second image together, wherein automatically combining the first image and the second image together is in response to a match value between the first image and the second image being greater than a threshold match value; The method also includes providing instructions to recapture at least one adjacent portion of the first image and the second image to increase the match value if the match value is not greater than a threshold match value, the instructions including at least one of one or more directions of movement, an amount of movement, one or more angles of movement, one or more axes of rotation about, and / or an amount of pressure applied during scanning. The method of claim 1 , wherein the first image and the second image are generated from volumetric data. 3 . The method of claim 2 , wherein the matching value is based on an offset of the first image and the second image along the horizontal plane and the vertical plane, wherein the matching value increases as the offset decreases. The method of claim 1 , wherein the horizontal plane is parallel to a scanning surface of the patient. The method of claim 1 , wherein the vertical plane is perpendicular to a scanning surface of the patient. The method according to claim 1 , wherein the first image is consecutive to the second image. The method of claim 1 , wherein the first anatomical location and the second anatomical location are consecutive.

8. A system for an imaging device, comprising: Ultrasonic equipment, comprising a probe and a display device; and A controller having computer readable instructions stored on its non-transitory memory, the computer readable instructions, when executed, enabling the controller to: displaying a first image on the display device; displaying a second image different from the first image on the display device; identifying a common reference point between the first image and the second image; Calculating a matching value includes along an edge of the first image and the second image; as well as In response to the matching value being greater than a threshold, stitching the first image and the second image along corresponding edges; The instructions further enable the controller to display instructions on the display device, wherein the instructions are configured to instruct a user to actuate the probe along a horizontal plane parallel to the surface of the patient's body and a vertical plane perpendicular to the surface of the patient's body to increase the match value if the match value is not greater than the threshold.

9. The system of claim 8, wherein the first image and the second image do not overlap.

10. The system of claim 8, wherein the instructions further enable the controller to display a position of the probe relative to the first image and the second image on the display device.

11. The system of claim 8, wherein the instructions further enable the controller to identify reference points in the first image and the second image, wherein the reference points include one or more of a tendon, a bone, a joint, an organ, a nerve, a blood vessel, a tumor, and a fetus.

12. The system of claim 11, wherein the first image and the second image share at least one reference point.

13. A method for an imaging device, comprising: Displaying a first image acquired via the ultrasound probe on a display screen; Displaying on the display screen the position of the ultrasound probe relative to the coordinate system of the first image; indicating to a user a direction in the first plane to navigate the ultrasound probe when acquiring a second image; indicating to the user a direction in which to navigate the probe in a second plane perpendicular to the first plane; calculating and displaying a matching value based on an offset between the first image and the second image in the first plane and the second plane; as well as The first image and the second image are combined side-by-side at an edge of the first image and the second image, wherein combining the first image and the second image is responsive to a match value being greater than a threshold match value.

14. The method of claim 13, wherein the direction indicating to the user in which to navigate the ultrasound probe in the first plane further comprises an arrow, the method further comprising increasing the size of the arrow to increase the indicated amount of desired ultrasound probe movement, wherein the first plane is parallel to a scanning surface.

15. The method of claim 13, wherein the direction indicating to the user in which to navigate the ultrasound probe in the second plane further comprises an arrow, the method further comprising reducing a size of the arrow to reduce an indication pressure desired to be applied to the ultrasound probe, wherein the second plane is perpendicular to the first plane, wherein the amount of size reduction is based on ultrasound volume data.

16. The method of claim 13, further comprising displaying the first image and the second image side-by-side on the display screen before combining the first image and the second image.

17. The method of claim 13, further comprising identifying a reference point in each of the first image and the second image.

18. The method of claim 13, wherein the match value is further based on a difference in pixel intensity at an edge of the first image and the second image, wherein the match value decreases as the difference increases.

Images