Image comparison system based on natural language

Abstract

The present disclosure provides systems and methods for an automatic image comparison system (200) for generating a natural language description of an image. In an example, the image comparison system (200) is configured to perform a method (500) that includes: acquiring (510) a current image of a patient during a current examination; generating (512) a first compressed representation (CR) of the current image using a computer vision-enabled large language model (CV / LLM (206); obtaining (518) a second CR of a similar image similar to the current image and acquired in a previous check; generating (518) a text-based comparison of the current image and the similar image using the CV / LLM (206) by inputting the first CR and the second CR as inputs to the CV / LLM (206); and outputting (520) the text-based comparison.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate generally to medical imaging, and more particularly, to image comparison systems that rely on large language models. Background Art

[0002] Medical ultrasound is an imaging modality that uses ultrasound waves to detect the internal structures of a patient's body and produce corresponding images. For example, an ultrasound probe including multiple transducer elements transmits ultrasound pulses, which are reflected or transmitted back, refracted, or absorbed by structures in the body. The ultrasound probe then receives the reflected echoes, which are processed into an image. Ultrasound images of internal structures can be saved for later analysis by clinicians to aid in diagnosis, and / or the images can be displayed on a display device in real time or near real time. Because ultrasound does not utilize ionizing radiation, ultrasound can be used in longitudinal studies, where physicians can compare images taken over a series of patient visits to examine the evolution of structures and / or assess the effectiveness of treatment. Summary of the invention

[0003] In one example, a method includes: acquiring a current image of a patient during a current examination; generating a first compressed representation (CR) of the current image using a computer vision enabled large language model (CV / LLM); obtaining a second CR of a similar image that is similar to the current image and was acquired in a previous examination; generating a text-based comparison of the current image and the similar image using the CV / LLM by inputting the first CR and the second CR as inputs to the CV / LLM; and outputting the text-based comparison.

[0004] The above advantages and other advantages and features of the present specification will be apparent from the following detailed description when considered alone or in conjunction with the accompanying drawings. It should be understood that the above summary is provided to introduce a series of concepts further described in the detailed description in a simplified form. This is not meant to identify key features or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims that follow the detailed description. In addition, the claimed subject matter is not limited to specific implementations that address any shortcomings mentioned above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various aspects of the present disclosure may be better understood by reading the following detailed description and referring to the accompanying drawings, in which:

[0006] Figure 1 shows a block diagram of an ultrasound system according to one embodiment;

[0007] Figure 2A schematic block diagram of an image comparison system according to one or more embodiments of the present disclosure is shown;

[0008] Figure 3 One or more embodiments of the present disclosure are shown. Figure 2 A schematic block diagram of an image processing system of an image comparison system;

[0009] Figure 4 A schematic block diagram illustrating a workflow of a computer vision / large language model (CV / LLM) for training an image comparison system according to one or more embodiments of the present disclosure is shown;

[0010] Figure 5 is a flow chart illustrating an exemplary method for performing image comparison using an image comparison system according to one or more embodiments of the present disclosure;

[0011] Figure 6 is a flow chart illustrating an exemplary method for training a CV / LLM of an image comparison system according to one or more embodiments of the present disclosure;

[0012] Figure 7 An exemplary graphical user interface (GUI) including an image is shown according to one or more embodiments of the present disclosure;

[0013] Figure 8 shows a GUI having an image and a first exemplary output of an image comparison function for the image and similar images according to one or more embodiments of the present disclosure;

[0014] Fig. 9 showing a GUI having an image and a second exemplary output of an image comparison function for the image and similar images according to one or more embodiments of the present disclosure; and

[0015] Fig.10 A GUI with exemplary output of image playback and image comparison functionality for image playback and similar image playback is shown in accordance with one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Provided herein is an automatic image comparison system that can generate a natural language comparison of two or more similar medical images using a computer vision / large language model (CV / LLM). The CV / LLM can be a private implementation of a commercial LLM (such as IMAGECHAT) with computer vision capabilities that can be trained to generate a compressed representation (CR) of a medical image (such as an ultrasound image). The CV / LLM can be further trained to describe the similarities and / or differences between the two or more medical images using natural language using previously created CRs of the two or more medical images. The CV / LLM can be trained using multiple patient examinations, including two or more consecutive examinations for each patient for at least a subset of the patients imaged in the multiple patient examinations. After the CV / LLM is trained, the image comparison system can call the CV / LLM to generate CRs of the acquired medical images and a natural language comparison between two or more similar images.

[0017] In order to accurately monitor a patient's condition, a clinician may schedule multiple rounds of medical imaging sessions over time. For example, a patient's heart defect may be detected during a first imaging session (also referred to as a checkup), and a second imaging session may be performed on the patient after a certain period of time (e.g., one month, six months, one year) from the first imaging session to monitor the heart defect (e.g., to see if the defect has changed / progressed or remained the same). In order to fully assess the heart defect, the clinician may desire to view one or more images from the first imaging session and one or more images from the second imaging session. Specifically, the clinician may want to view one or more images from the first imaging session during the second imaging session in order to ensure that a complete and comprehensive assessment of the heart defect can be performed based on the images obtained during the second imaging session while further images of the patient are still available.

[0018] However, medical images from patient examinations are often relatively large files that are stored in an image archive, typically a picture archive and communication system (PACS) operating in accordance with the Digital Imaging and Communications in Medicine (DICOM) standard, along with additional information related to the image in the study (clinical findings, measurements, etc.). Due to the size of the image files and the constraints set by the DICOM standard, obtaining the desired images from past examinations can be time consuming. For example, it can take an hour or more to download images from a study to a computing device associated with the medical imaging system. Therefore, it is not possible to access previous images of a patient during real-time scanning of the patient in a current examination utilizing the medical imaging system.

[0019] The embodiments disclosed herein solve this problem by utilizing CV / LLM to generate a description of each medical image as the medical image is acquired. These descriptions can be text-based and can therefore be relatively small files compared to the image itself. The description, referred to as a compressed representation, can be saved in a DICOM structured report (DICOM SR) object of the study saved on the PACS. The DICOM SR object is typically used to store measurements of anatomical features present in the image of the study, but can actually include any text-based information. Then, when the clinician desires to view the patient's previous images in order to assess the anatomical changes from the previous examination to the current examination, the CR of the previous image can be retrieved from the PACS (e.g., by retrieving the DICOM SR object and extracting the CR from the DICOM SR object). Because the CR is small and stored separately from the image, the CR within the DICOM SR object can be sent to a computing device associated with the medical imaging system in a relatively short amount of time (e.g., on the order of seconds or minutes) without the previous image itself. The CV / LLM can be further called to generate a natural language description of the difference between the selected previous image and the current image using the CR instead of the image. Thus, during performance of the current examination, the clinician may be provided with a clinically relevant description of any changes to the imaged anatomical feature of interest, which may allow the clinician to obtain additional images, if desired, to fully assess the anatomical feature of interest.

[0020] For example, if an image comparison between a current image and a previous image indicates that a change in the size of the left ventricle has occurred (e.g., the size of the left ventricle in the current image is different from the size of the left ventricle in the previous image), the clinician may determine that additional images of the left ventricle in various views should be obtained to fully assess the left ventricle. However, if the image comparison between the current image and the previous image indicates that there has been no change in the imaged anatomical features, the clinician may determine that the images obtained in the current examination are sufficient and no further images may be obtained. In doing so, clinical outcomes may be improved while reducing network bandwidth / traffic (e.g., by reducing the number of images sent from the PACS to the medical imaging system) and processing burden (e.g., by comparing CRs instead of images), thereby improving the performance of the medical imaging system.

[0021] Although ultrasound imaging systems are described herein as an exemplary imaging modality for acquiring medical images, it should be understood that other types of medical images (such as magnetic resonance images, x-ray images, etc.) may be utilized to perform the image comparisons described herein without departing from the scope of the present disclosure. However, because ultrasound does not employ ionizing radiation, can be used at the point of medical care, and is relatively inexpensive, ultrasound may be widely used in certain longitudinal studies, such as monitoring structural abnormalities of the heart. Further, given that the ultrasound probe is typically manipulated in a freehand manner, there may be variability in the acquired images from examination to examination based on, for example, operator experience or patient parameters. Still further, the resolution of ultrasound images may make the interpretation of ultrasound images variable between clinicians. Therefore, the image comparison operations described herein may be particularly suitable for ultrasound images because image comparison may promote consistency across examinations and clinicians, and enable a more rapid assessment of the current patient condition in real time while a patient examination is being performed.

[0022] refer to Figure 1 , a schematic diagram of an ultrasound imaging system 100 according to an embodiment of the present disclosure is shown. The ultrasound imaging system 100 includes a transmit beamformer 101 and a transmitter 102, which drives an element (e.g., a transducer element) 104 within a transducer array (referred to herein as a probe 106) to transmit a pulsed ultrasound signal (referred to herein as a transmit pulse) into a body (not shown). According to an embodiment, the probe 106 may be a one-dimensional transducer array probe. However, in some embodiments, the probe 106 may be a two-dimensional matrix transducer array probe. As further explained below, the transducer element 104 may be made of a piezoelectric material. When a voltage is applied to a piezoelectric crystal, the crystal physically expands and contracts, thereby emitting an ultrasonic spherical wave. In this way, the transducer element 104 can convert an electronic transmit signal into an acoustic transmit beam.

[0023] After the elements 104 of the probe 106 transmit the pulsed ultrasound signals into the body (of the patient), the pulsed ultrasound signals are backscattered from structures inside the body (such as blood cells or muscle tissue) to produce echoes that return to the elements 104. The echoes are converted into electrical signals or ultrasound data by the elements 104, and the electrical signals are received by the receiver 108. The electrical signals representing the received echoes pass through the receive beamformer 110 which outputs radio frequency (RF) data. Additionally, the transducer elements 104 may generate one or more ultrasound pulses based on the received echoes to form one or more transmit beams.

[0024] According to some embodiments, the probe 106 may contain electronic circuitry to perform all or part of the transmit beamforming and / or receive beamforming. For example, all or part of the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110 may be located within the probe 106. In the present disclosure, the term "scan" may also be used to refer to the acquisition of data by the process of sending and receiving ultrasound signals. In the present disclosure, the term "data" may be used to refer to one or more data sets acquired with the ultrasound imaging system. The user interface 115 may be used to control the operation of the ultrasound imaging system 100, including controlling the input of patient data (e.g., patient medical history), changing scanning or display parameters, starting a specific acquisition mode such as shear wave imaging, etc. The user interface 115 may include one or more of the following: a rotating element, a mouse, a keyboard, a trackball, hard keys linked to specific actions, soft keys that can be configured to control different functions, and a graphical user interface displayed on the display device 118.

[0025] The ultrasound imaging system 100 also includes a processor 116 that controls the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110. The processor 116 is in electronic communication (e.g., communicatively connected) with the probe 106. For the purposes of this disclosure, the term "electronic communication" may be defined as including both wired communication and wireless communication. The processor 116 may control the probe 106 to acquire data according to instructions stored in the processor's memory and / or the memory 120. The processor 116 controls which of the elements 104 are active and the shape of the beam emitted from the probe 106. The processor 116 also communicates electronically with the display device 118, and the processor 116 may process data (e.g., ultrasound data) into an image for display on the display device 118. According to an embodiment, the processor 116 may include a central processing unit (CPU). According to other embodiments, the processor 116 may include other electronic components capable of performing processing functions, such as a digital signal processor, a field programmable gate array (FPGA), or a graphics board. According to other embodiments, the processor 116 may include a plurality of electronic components capable of performing processing functions. For example, the processor 116 may include two or more electronic components selected from a list of electronic components, including: a central processing unit, a digital signal processor, a field programmable gate array, and a graphics board. According to another embodiment, the processor 116 may also include a composite demodulator (not shown) that demodulates RF data and generates IQ data pairs representing echo signals. In another embodiment, demodulation may be performed earlier in the processing chain. The processor 116 is adapted to perform one or more processing operations according to a plurality of optional ultrasound modalities on the data. In one example, data may be processed in real time during a scanning session because the echo signals are received by the receiver 108 and sent to the processor 116. For the purposes of this disclosure, the term "real time" is defined as including programs executed without any intentional delay. For example, an embodiment may acquire images at a real-time rate of 7 frames / second to 20 frames / second. The ultrasound imaging system 100 may acquire 2D data of one or more planes at a significantly faster rate. However, it should be understood that the real-time frame rate may depend on the length of time it takes to acquire each frame of data for display. Therefore, when a relatively large amount of data is collected, the real-time frame rate may be slower. Therefore, some embodiments may have a real-time frame rate significantly faster than 20 frames / second, while other embodiments may have a real-time frame rate less than 7 frames / second. The data may be temporarily stored in a buffer (not shown) during a scanning session and processed in less than real time in real-time or off-line operation. Some embodiments of the present invention may include multiple processors (not shown) to handle the processing tasks handled by processor 116 according to the exemplary embodiments described above.For example, a first processor may be utilized to demodulate and extract the RF signal prior to displaying the image, while a second processor may be utilized to further process the data, such as by expanding the data. It should be appreciated that other embodiments may utilize different processor arrangements.

[0026] The ultrasound imaging system 100 can continuously acquire data at a frame rate of, for example, 10 Hz to 30 Hz (e.g., 10 to 30 frames per second). The image generated based on the data can be refreshed at a similar frame rate on the display device 118. Other embodiments can acquire and display data at different rates. For example, depending on the size of the frame and the intended application, some embodiments can acquire data at a frame rate less than 10 Hz or greater than 30 Hz. A memory 120 is included to store processed frames of the acquired data. In an exemplary embodiment, the memory 120 has sufficient capacity to store at least a few seconds of ultrasound data frames. The data frames are stored in a manner that is easy to retrieve according to their acquisition order or time. The memory 120 may include any known data storage medium.

[0027] In various embodiments of the present invention, the processor 116 can process data through different mode-related modules (e.g., B-mode, color Doppler, M-mode, color M-mode, spectral Doppler, elastic imaging, TVI, strain, strain rate, etc.) to form two-dimensional (2D) or three-dimensional (3D) data. For example, one or more modules can generate B-mode, color Doppler, M-mode, color M-mode, spectral Doppler, elastic imaging, TVI, strain, strain rate, and combinations thereof, etc. As an example, one or more modules can process color Doppler data, which can include traditional color blood flow Doppler, power Doppler, HD flow, etc. The image lines and / or frames are stored in the memory and can include timing information indicating the time when the image lines and / or frames are stored in the memory. These modules can include, for example, a scan conversion module that performs a scan conversion operation to convert the acquired image from beam space coordinates to display space coordinates. A video processor module can be provided that reads the acquired image from the memory and displays the image in real time while a procedure (e.g., ultrasound imaging) is performed on the patient. The video processor module may include a separate image memory, and the ultrasound image may be written to the image memory for reading and display by the display device 118 .

[0028] In various embodiments of the present disclosure, one or more components of the ultrasound imaging system 100 may be included in a portable handheld ultrasound imaging device. For example, the display device 118 and the user interface 115 may be integrated into the external surface of the handheld ultrasound imaging device, which may further contain a processor 116 and a memory 120. The probe 106 may include a handheld probe that electronically communicates with the handheld ultrasound imaging device to collect raw ultrasound data. The transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110 may be included in the same or different parts of the ultrasound imaging system 100. For example, the transmit beamformer 101, the transmitter 102, the receiver 108, and the receive beamformer 110 may be included in a handheld ultrasound imaging device, a probe, and a combination thereof.

[0029] After performing a two-dimensional ultrasound scan, a data block including scan lines and their samples is generated. After applying the back-end filter, a process called scan conversion is performed to transform the two-dimensional data block into a displayable bitmap image with additional scanning information (such as depth, the angle of each scan line, etc.). During scan conversion, interpolation techniques are applied to fill in the missing holes (i.e., pixels) in the resulting image. These missing pixels occur because each element of the two-dimensional block should generally cover many pixels in the resulting image. For example, in current ultrasound imaging systems, bicubic interpolation is applied, which utilizes adjacent elements of the two-dimensional block. Therefore, if the two-dimensional block is relatively small compared to the size of the bitmap image, the scan-converted image will include areas of poor or low resolution, especially for areas with greater depth.

[0030] Figure 2 An image comparison system 200 including an image processing system 202 is shown according to an embodiment. In some examples, the image processing system 202 may be a computing device included as part of the ultrasound imaging system 100. For example, and as explained in more detail below, the image processing system 202 may include a memory and one or more processors, which may be non-limiting examples of the memory 120 and the processor 116, respectively. In other examples, the image processing system 202 may communicate with the ultrasound imaging system 100 or otherwise be configured to obtain images acquired using the ultrasound imaging system 100. The image processing system 202 may be configured to generate a compressed representation of the acquired image and / or generate a text-based image comparison between two or more similar images using CV / LLM.

[0031] Image processing system 202 may be communicatively coupled (e.g., via a network, such as network 203) to one or more image archives, such as a picture archiving and communication system (PACS) 204. PACS 204 may store medical images (e.g., using Figure 1 The PACS 204 may include an ultrasound image obtained by an ultrasound imaging system 100 of the PACS 204), data associated with the medical image (such as measurements, findings indicated by a clinician, artificial intelligence (AI) model output, etc.), and a compressed representation (CR) of the medical image. When requested, the PACS 204 may retrieve the CR of the image and send the CR to the image processing system 202 to facilitate the image comparison operations described herein. The PACS 204 may obtain, store, and send images according to the Digital Imaging and Communications in Medicine (DICOM) standard. DICOM is a common standard for medical imaging and lists command sets, data formats, interface specifications, communication protocols, and command syntax. DICOM describes information objects (data types, such as computed tomography, magnetic resonance, x-ray, ultrasound, etc.), service categories (actions on data, such as sending, receiving, printing, etc.), and data transfer protocols. The DICOM application service provides the ability to transfer images and image-related data (e.g., measurements, findings) between DICOM applications. Although in Figure 2 204, it should be understood that the image comparison system 200 may include more than one PACS and may additionally or alternatively include one or more vendor neutral archives (VNAs) that may be similar in functionality to the PACS 204.

[0032] The image processing system 202 may utilize one or more LLMs supporting computer vision, such as CV / LLM 206, to generate a CR of an image and / or generate a text-based comparison of two or more similar images. CV / LLM 206 may be a trained version of a commercially available LLM, such as GPT or IMAGECHAT. CV / LLM 206 may be multimodal, for example, configured to generate output based on both text-based input (e.g., natural language prompts) and image-based input. To some extent, LLMs can understand human language. This makes LLMs a good candidate for comparing two text descriptions / representations, which may be grammatically different, but still conceptually / semantically represent the same meaning. Additional details about training and deploying CV / LLM 206 are provided below. In some examples, as shown, the image processing system 202 may be communicatively coupled to CV / LLM 206 via a network (e.g., network 203). In other examples, CV / LLM 206 may be stored in a memory of the image processing system 202.

[0033] The image processing system 202 may obtain one or more images 208, generate a CR 210 for each image or one or more selected images, and output an image comparison 212 upon request. For example, the one or more images 208 may be ultrasound images obtained during a scanning session of a patient. In some examples, the one or more images may include an image playback (e.g., a video, also referred to as a movie playback). In some examples, the image processing system 202 may obtain ultrasound data from an ultrasound probe (e.g., 106), process the ultrasound data into images and / or image playbacks, output the images and / or image playbacks for display on a display device, and save the selected images and / or image playbacks in a temporary storage device. The image processing system 202 may use the CV / LLM 206 to generate a CR 210 for each image and / or each playback. For example, the one or more images 208 (and playbacks, when included) may be input (separately / individually) to the CV / LLM 206, and the CV / LLM 206 may output a CR 210 for each image and / or playback. The CR 210 for an image or playback may be a collection of significant, relevant, or otherwise selected features of the image / playback in a text-based and / or numeric character format that may be saved as part of a DICOM Structured Report (DICOM SR) object 211, which is a DICOM object that may be saved with the image on the PACS 204 (e.g., as part of a study). As explained in more detail below, the CV / LLM 206 may be trained to generate a CR for each image and / or playback in a supervised manner such that clinically relevant image / video features (e.g., anatomical structures, anatomical functions, presence or absence of motion) are included in the CR, while background information and non-clinically relevant image / video features are excluded from the CR.

[0034] Each image in the one or more images 208 and the DICOM SR object 211 (including each CR 210) may be sent to the PACS 204 for long-term storage. For example, during an active imaging session for a patient (e.g., in the case of acquiring images using the ultrasound imaging system 100, also referred to as an examination), each acquired image selected for saving by the clinician conducting the examination may be input as input to the CV / LLM 206 to generate a corresponding CR for each saved image. The saved images and corresponding CRs may be saved locally in a database of the image processing system 202 (e.g., as described below with respect to Figure 3Once the examination is completed, the saved images and corresponding CRs may be sent to PACS 204. Specifically, each CR may be saved in a DICOM SR object that accompanies the images of the examination saved in the PACS. The DICOM SR objects may be viewable via a DICOM / PACS workstation (e.g., image processing system 202 or another computing device) and include patient information, measurements, clinician findings, etc. for the examination.

[0035] Further, each image may include metadata stored in the PACS 204 (e.g., stored in a DICOM SR object and / or in a DICOM header of the image and / or in other DICOM files of the examination / study), and the metadata may include classification tags, modes, etc. Classification tags may include information for grouping or classifying images based on properties of the images. For example, each image and / or playback may be subject to one or more models configured to identify the viewing plane of the image (e.g., apical four-chamber view, two-chamber view, etc.). The output of the model (e.g., the identified viewing plane) may be saved as a view classification tag and used to identify other images expected to show the same anatomical structure. The mode may include information about how the image was acquired, such as the dimensionality of the image (2D or 3D) and / or whether the image is a brightness mode (B-mode) image, a Doppler image, or a color Doppler image. Other examples of stored metadata are various measurements of anatomical features or functions (e.g., ejection fraction, area, volume), type of ultrasound probe used, probe imaging presets, patient heart rate, imaging frame rate, scan depth, scan width, scan rotation, image / playback location within the examination, etc.

[0036] When performing an image comparison operation, the image processing system 202 may output an image comparison 212 that includes, for example, a text-based description of similarities and differences between the two images. The image comparison operation may be initiated in response to a user request, such as a user selection of a "compare" element on a graphical user interface (GUI). The image comparison operation may include a selection of a current image. The current image (e.g., which may be an image in the one or more images 208) may be, for example, an image displayed on the GUI when the comparison element is selected. The image processing system 202 may initiate identification of one or more similar images that are similar to the selected current image. The one or more similar images that are similar to the current image may be images of the same patient acquired in a previous image session and stored in the PACS 204. In other examples, the one or more similar images that are similar to the current image may be images of other patients. In either example, similar images may be identified based on metadata of the images. For example, when an image comparison operation is initiated, metadata of all previous images / examinations of the patient stored in the PACS 204 may be retrieved (e.g., sent to the image processing system 202). The metadata of each retrieved image can be compared with the metadata of the current image, and similar images can be identified based on those images having the same classification tags and patterns as the current image. In some examples, similar images can be identified based on additional metadata / tags such as heart rate, imaging frame rate, scan depth, scan width, scan rotation, and / or position of the image / playback within the examination.

[0037] The CR of the identified similar image may be retrieved from the PACS 204. For example, all DICOM SR objects of the previous exam may be retrieved, and the CR of the identified similar image may be extracted from the DICOM SR object storing the CR. The CV / LLM 206 may then be called to generate an image comparison 212 that describes the difference between the current image and the similar image using the CR of the current image and the CR of the similar image. For example, the CR of the current image and the CR of the similar image may be input as input to the CV / LLM 206 to generate the image comparison 212. The image comparison 212 may be text summarizing the difference in anatomical features, image acquisition parameters, or other features of the current image relative to the similar image.

[0038] Figure 3is a schematic block diagram of an image processing system 202. In some embodiments, at least a portion of the image processing system 202 is disposed at a device (e.g., a workstation, an edge device, a server, etc.) that is communicatively coupled to one or more healthcare and / or hospital networks and computer systems via a wired and / or wireless connection (e.g., network 203), and can receive or access medical data (including patient data) stored in one or more healthcare and / or hospital computer systems. The medical data may include medical images acquired using the ultrasound imaging system 100. The image processing system 202 may also be operably / communicatively coupled to a user input device 316 and a display device 314. As previously explained, the image processing system 202 may be part of the ultrasound imaging system 100, and thus the display device 314 may be Figure 1 The display device 118 and the user input device 316 may be Figure 1 User interface 115.

[0039] The image processing system 202 includes a processor 302 configured to execute machine-readable instructions stored in a non-transitory memory 304. In some examples, the processor 302 is a non-limiting example of the processor 116. The processor 302 may be a single-core or multi-core processor, and the program executed thereon may be configured for parallel processing or distributed processing. In some embodiments, the processor 302 may optionally include separate components distributed on two or more devices, which may be located at a distance and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 302 may be virtualized and performed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0040] The non-transitory memory 304, which may be a non-limiting example of the memory 120, may store the image and compressed representation (CR) storage 306, the training module 308, and the inference module 310. In the example shown, the non-transitory memory 304 also stores the CV / LLM 206, but it should be understood that in other examples, the CV / LLM 206 may be stored away from the image processing system 202 (e.g., on one or more servers communicatively coupled to the image processing system 202). The image and CR storage 306 may store medical images (e.g., ultrasound images) and image playbacks acquired using one or more imaging modalities. For example, during a patient examination, the ultrasound imaging system 100 may be employed to acquire ultrasound images and playbacks of the patient. One or more of the ultrasound images and / or playbacks acquired during the patient examination may be saved in the image and CR storage 306 (e.g., a clinician performing the patient examination may enter a user input indicating that one or more ultrasound images are to be saved).

[0041] In addition, as described above, a CR may be generated for each saved image and the CR may be saved in the image and CR storage 306. Upon completion of the patient examination, one or more images and / or replays and associated CRs saved in the image and CR storage 306 may be sent to an image archive (e.g., PACS 204) for long-term storage. Further, the image and CR storage 306 may store CRs of one or more images determined to be similar to the current image in order to perform image comparison operations. In examples where the image processing system 202 performs training of the CV / LLM 206, the image and CR storage 306 may additionally store images from previous patient examinations (including images and objects associated with the images, such as text-based findings, measurements, etc.).

[0042] The training module 308 may include instructions for training the CV / LLM 206. The training module 308 may include instructions that, when executed by the processor 302, cause the image processing system 202 to perform Figure 6 One or more of the steps of method 600 may be used to train CV / LLM 206. However, in other examples, training module 308 may be stored on a different computing device.

[0043] The inference module 310 may include instructions for deploying the CV / LLM 206 to generate CRs of images and text-based image comparisons. Specifically, the inference module 310 may include instructions that, when executed by the processor 302, cause the image processing system 202 to perform Figure 5 One or more of the steps of method 500.

[0044] The user input device 316 may include one or more of a touch screen, a keyboard, a mouse, a trackpad, a microphone, a motion sensing camera, or other devices configured to enable a user to interact with the image processing system 202. In one example, the user input device 316 may enable a user to submit a request to the image processing system 202 to identify an image similar to a current image (e.g., to initiate an image comparison operation). For example, a user (e.g., a clinician such as a sonographer operator) may select a "Compare" element / button using a keyboard, a touch screen, a microphone, etc.

[0045] The display device 314 may include one or more display devices utilizing almost any type of technology. In some embodiments, the display device 314 may include a computer monitor. The display device 314 may be combined with the processor 302, the non-transient memory 304, and / or the user input device 316 in a common housing, or may be a peripheral display device, and may include a monitor, a touch screen, a projector, or other display devices known in the art, which may enable a user to view images, CR of images, and image comparisons and / or interact with various data stored in the non-transient memory 304.

[0046] It should be understood that Figure 3 The image processing system 202 shown is for illustration and not for limitation. Another suitable image processing system may include more, fewer, or different components.

[0047] Figure 4 A schematic diagram of a workflow 400 for training a computer vision-enabled LLM (eg, CV / LLM 206) to generate natural language image comparisons of two or more images and / or image playbacks is shown. Figure 6 A method for training a CV / LLM is described, which may follow a workflow 400. The steps indicated in the workflow 400 may be based on a non-transitory memory stored in the image processing system 202 (such as a memory stored in Figure 3 The training module 308 of the non-volatile memory 304 of the image processing system 202, or instructions stored on another suitable computing device) are executed.

[0048] Specifically, the workflow 400 may include a two-part training process using two training submodules to fine-tune an initial CV / LLM 406 to generate CRs for medical images and / or image replays, and then train the fine-tuned CV / LLM to generate a natural language comparison of two or more CRs for two or more similar images or image replays. The initial CV / LLM 406 may be fine-tuned using data obtained from an examination database 402 and organized using a data set generator 404. The examination database 402 may include multiple patient examinations for multiple patients. Each patient examination in the examination database 402 may include a medical image, and in some examples, may include an image replay (e.g., an ultrasound image and / or replay) of the corresponding patient and a report, the report including clinician-entered findings, computer-assisted findings (e.g., from a computer-aided diagnostic tool, an artificial intelligence model, etc.), measurements (whether obtained by a clinician or automatically generated), patient information (e.g., age, gender, previous diagnosis), and examination information (e.g., scan protocol, acquisition settings). Images and replays included in an exam may include metadata such as classification tags, modality, patient heart rate, imaging frame rate, etc. Exam database 402 may be an example of PACS 204 or exam database 402 may be separate from PACS 204 but include patient exams obtained from PACS 204 .

[0049] The data set generator 404 can collate the data from the examination database 402 to generate training data 408 and test data 410, which can be used by the first training submodule 409 to train (e.g., fine-tune) the initial CV / LLM 406. The training data 408 and the test data 410 can each include a plurality of training pairs, wherein each training pair includes an image or image playback and a description of the image or image playback generated from a report associated with the image or image playback. In some examples, the description of the image or image playback can include clinical findings visible in the image or image playback, measurements of anatomical features in the image or image playback, etc. In some examples, the training data 408 can include a majority of the training pairs (e.g., 90%) formed by the data set generator 404, and the remaining training pairs can form the test data 410.

[0050] The first training submodule 409 may generate a fine-tuned CV / LLM 412 via a supervised learning process using the training data 408. For example, for a given training pair of the training data 408, an image or image playback may be input as an input to an initial CV / LLM 406, and the parameters of the initial CV / LLM 406 may be adjusted based on the output from the initial CV / LLM 406 relative to the description of the image or playback for the given training pair. The initial CV / LLM 406 may be a pre-trained CV / LLM trained with a general domain dataset to recognize and describe images, but may not be able to describe medical images such as echocardiography images. The first stage of the training process may fine-tune the CV / LLM so that the CV / LLM is able to provide clinically relevant descriptions of medical images (e.g., echo images) that simulate the descriptions that a clinician would make of such images.

[0051] In some examples, the validator 414 may validate the performance of the fine-tuned CV / LLM 412 on the test data 410. The validator 414 may take as input a trained or partially trained CV / LLM (e.g., the fine-tuned CV / LLM 412) and the test data 410, and may output an evaluation of the performance of the fine-tuned CV / LLM 412 on the test data 410.

[0052] Once the validator 414 determines that the fine-tuned CV / LLM is sufficiently trained, the second step of the training process may be performed using the second training submodule 416 using additional training data 418 and test data 420. To generate the additional training data 418 and test data 420, the dataset generator 404 may collate data from the exam database 402 to form exam sets, where each exam set includes two or more exams from the same patient. For example, some patients may undergo a longitudinal study, where the same anatomical features (e.g., heart, lesions) may be imaged multiple times with the same imaging modality (e.g., ultrasound) over a period of time (e.g., days, weeks, months, years). The dataset generator 404 may identify patients who have more than one exam saved in the exam database, and group each exam from the same patient into an exam set. In some examples, other exams in the exam database 402 (e.g., not part of a longitudinal study, so only one exam for a given patient is saved in the exam database) may be omitted from the training and test data.

[0053] The data set generator 404 may also select each exam set to identify and form one or more similar image pairs or similar image replay pairs. In some examples, DICOM fields and / or header information may be used to identify similar images, as described above. Each image (or image replay) pair may include a first image (or first replay) from a first (e.g., earlier) exam of a patient and a second image (or second replay) from a second (e.g., later) exam of the patient, wherein the first image (or replay) and the second image (or replay) are similar because the first image and the second image (or the first replay and the second replay) have the same viewing plane and / or include the same anatomical features. For example, the first image and the second image may both be apical two-chamber views. At least a portion of the similar image or replay pair may include a change from the first image to the second image (or a change from the first replay to the second replay) such that the first and second images (or replays) are similar but not identical. For example, the second image or replay may exhibit changes in anatomical features relative to the first image or replay, such as changes in wall thickness, changes in valve function, changes in ventricular or atrial size, etc. In some examples, the image pair may be partially synthetically generated, where a first image may be adjusted to form a second image, such as changing wall thickness, masking or adding anatomical features, etc.

[0054] Once similar images and / or similar image-playback pairs are identified, each similar image and / or similar image-playback pair may be processed, such as via a fine-tuned CV / LLM 412, to generate a CR for each image or playback. Thus, similar CR pairs may be formed that include a first CR for a first image or playback (e.g., which may be an earlier image of the patient) and a second CR for a second image or playback (e.g., which may be a later image of the patient in the same viewing plane as the first image). The dataset generator 404 may form training triples, where each training triple includes a similar CR pair (e.g., a first CR for a first image from a first exam of the patient and a second CR for a second image from a second exam of the patient, where the first image and the second image are the same viewing plane) and a description of the similar images / playbacks that highlights the differences, if any, between the similar images / playbacks. In some examples, the descriptions may be generated from reports associated with the first and second images / playbacks. For example, for the purpose of training the CV / LLM, the descriptions may be generated by an expert based on the reports.

[0055] It should be understood that the data set generator 404 may include instructions stored in the memory, which, when executed, automatically generate various training data and test data described herein. However, in some examples, the data set generator 404 may be an interface, via which one or more experts may sort and select data from the inspection database 402 to form training data and test data.

[0056] The second training submodule 416 may generate a final CV / LLM 422 via a supervised learning process using the additional training data 418. For example, for a given training triplet of the additional training data 418, similar CR pairs may be input as input to the fine-tuned CV / LLM 412, and the parameters of the fine-tuned CV / LLM 412 may be adjusted based on the output from the fine-tuned CV / LLM 412 relative to the description of the similar CRs for the given training triplet.

[0057] In some examples, the validator 414 may validate the performance of the final CV / LLM 422 against the test data 420. The validator 414 may take as input a trained or partially trained CV / LLM (e.g., the final CV / LLM 422) and the test data 420, and may output an evaluation of the performance of the final CV / LLM 422 on the test data 420. Specifically, the test data 420 may include: some training triplets (e.g., half of the training triplets of the test data 420) including similar CR pairs where the images or playbacks used to create the CRs are substantially the same (e.g., there is no change in anatomical features in the second image relative to the first image); and some training triplets (e.g., the other half) including similar CR pairs where the images or playbacks used to create the CRs are not the same (e.g., there is a change in anatomical features in the second image relative to the first image). The validator 414 may determine whether the final CV / LLM 422 can distinguish between similar CR pairs without any anatomical changes and similar CR pairs with anatomical changes, and reflect this in the output from the final CV / LLM 422. Once the validator 414 determines that the final CV / LLM 422 is sufficiently trained, the final CV / LLM 422 may be stored in the image processing system 202 (e.g., as the CV / LLM 206) or another suitable location. In this way, the final CV / LLM 422 may be trained to generate CRs for medical images and playbacks (e.g., echocardiographic images), and also trained to generate natural language comparisons of similar CRs.

[0058] Figure 5 A method 500 is shown for providing image comparison via an image comparison system, such as image comparison system 200, where the image comparison system relies on a trained CV / LLM to generate a natural language image comparison between two or more images. The trained CV / LLM may be a non-limiting example of CV / LLM 206. The method 500 may be executed by a processor of the image comparison system, such as Figure 3 The processor 302 of the image processing system 202) executes based on instructions stored in the non-volatile memory of the image processing system.

[0059] At 502, method 500 includes obtaining images and / or image playback (eg, a series of sequential images, which thereby form a video) during a first examination of a patient. The images and / or image playback may be acquired using an ultrasound imaging system (such as Figure 1 ) obtained by an ultrasound imaging system 100 of the ultrasound imaging system. During a first examination, an operator of the ultrasound imaging system may select a subset of the acquired images to be saved as part of the first examination. For example, the first examination may be an echocardiography examination that indicates that a specified set of images is acquired and saved in a plurality of different scanning planes. At 504, a compressed representation (CR) is optionally generated for each image and / or playback of the first examination using CV / LLM. Each CR is saved together with the image or playback described by the CR. For example, when the operator of the ultrasound imaging system selects an image to be saved as part of the first examination, the selected image may be input to the CV / LLM together with a prompt indicating that the CV / LLM generates a CR for the image as input. The CV / LLM may output a CR of the image describing the relevant features present in the image. The description will describe the image in detail as if an experienced physician had described the image in detail. The CR may include a description of which cavities and structures are visible, the extent to which they are fully or partially visible, a description of any artifacts in the image, and any clinical assessment of the normality or abnormality / pathology of the cavities and structures present. The CR may be saved in the DICOM SR object of the examination. It should be understood that more than one CR may be saved to the same DICOM SR object and that an exam / study may include multiple DICOM SR objects.

[0060] At 506, method 500 includes receiving a request to start a current examination of the patient. The current examination may be performed after the first examination and spaced a suitable time period after the first examination, such as a week, a month, or longer. The request to start the current examination may be received via user input, for example, a clinician performing the current examination may enter patient information and / or select a scanning protocol (e.g., echocardiogram of adult patients), which may indicate that the current examination for the patient is starting.

[0061] At 508, method 500 determines whether a previous exam (e.g., a first exam) for the patient includes acquired images and / or a replayed CR for the patient. In some examples, such as when the previous exam was performed before the CV / LLM was available to generate a CR, the previous exam for the patient may not include an acquired image / replayed CR. In such examples (e.g., no at 508), method 500 includes acquiring images / replays of the previous exam and generating CRs of the images / replays in the previous exam at 509. For example, during the first exam of the patient performed at 502, the CV / LLM may not yet be available, and the CRs of the images / replays may not yet have been generated.

[0062] Thus, at the start of the current examination, a process of obtaining images and / or playback and generating CRs may be initiated so that the CRs are available at least near the end of the current examination to enable image comparison operations. However, as previously explained, downloading images from the PACS to the image processing system may be time consuming, and therefore, in some examples, the process of generating CRs for previous examinations may be initiated in a more proactive manner to ensure that the CRs for the images / playbacks of the patient's previous examinations are available at the start of the current examination. For example, the image processing system may access the patient schedule every morning (or the patient schedule for the next day every evening) to determine which patients will be scanned on that day (or the next day), and initiate the process of generating CRs for the images / playbacks of the previous examinations of each scheduled patient before the scheduled examination begins. In still other examples, a computing device (which may be an image processing system or another computing device) may be designated to systematically generate CRs for each recent examination of each patient stored on the PACS, and store the CRs in the SR of each examination / study.

[0063] If the prior exam (e.g., the first exam) did include a CR of the images / playbacks obtained (e.g., "yes" at 508), method 500 proceeds to 510 to obtain one or more images and / or playbacks during the current exam of the patient. In some examples, the one or more images and / or playbacks obtained during the current exam (referred to as current images and / or playbacks) may include at least some images or playbacks obtained in the same scanning / viewing plane and having the same anatomical features as the images or playbacks obtained in the first exam. For example, the first exam and the current exam may be exams intended to monitor structural and / or functional changes in the same anatomical features of the patient (such as monitoring the growth of a lesion, cardiac abnormalities, etc.).

[0064] At 512, a CR is generated for each current image and / or playback of the current exam using the CV / LLM. Each CR is saved with the current image or playback described by the CR. As previously explained, when an operator of the ultrasound imaging system selects a current image to be saved as part of the current exam, the selected image may be input as input to the CV / LLM along with a prompt instructing the CV / LLM to generate a CR for the image. The CV / LLM may output a CR for the current image that includes selected features of the current image, such as the scan / viewing plane (e.g., four-chamber view) of the current image, normal and / or abnormal features, etc. The CR may be saved in a DICOM SR object.

[0065] At 514, method 500 includes determining whether an image comparison operation has been requested for the selected image or playback. For example, and as explained in more detail below, the ultrasound imaging system / image processing system may include or be operably coupled to a display device that displays a graphical user interface (GUI) during the current examination, the graphical user interface including a first display area for displaying a current image or a current image playback. The GUI may also include one or more user interface elements, including a comparison element. When a user selects the comparison element, the current image or playback (e.g., the image or playback currently displayed on the GUI, or another image or playback specified by the user) may be designated as the selected image or playback for performing an image comparison.

[0066] If an image comparison operation has not been requested, the method 500 continues to 522 to determine whether the current examination is complete, which will be explained in more detail below. If an image comparison operation has been requested, the method 500 proceeds to 516 to identify similar images or playbacks and retrieve CRs associated with the similar images or playbacks. The similar images or playbacks may be similar to the current / selected image or playback (e.g., the image selected for the image comparison operation), but were obtained during a previous examination (e.g., the first examination). In other examples, the similar images or playbacks may be images or playbacks of another patient (e.g., a reference image). CRs for similar images are retrieved, but the similar images themselves may not be retrieved.

[0067] Similar images or replays may be similar to the selected image because the similar images or replays may be obtained in the same scan / view plane as the selected image or replay, include the same anatomical features, imaged using the same acquisition / scan parameters, etc. Similar images or replays may be identified based on having metadata that matches the metadata of the selected image, such as the same classification tags and mode, and in some examples, similar other features captured in the metadata, such as imaging frame rate, etc. For example, each image saved during an examination of a patient may be input into a model configured to identify the scan / view plane of the image (e.g., apical two-chamber view, apical three-chamber view, apical four-chamber view, parasternal short-axis view, etc.), and the image may be tagged with a classification tag indicating the identified view plane (e.g., the identified view plane may be included in the metadata of the image). When an image comparison operation is initiated, the view plane of the current image may be determined, and the patient's most recent previous examinations may be searched to identify images from the previous examinations that are tagged with the same view plane as the current image. Once a similar image or playback is identified, the CR for the similar image or playback can be extracted (e.g., from a DICOM SR object). In this way, the CR for the similar image or playback is retrieved from long-term storage (e.g., from a PACS), but the similar image or playback is not retrieved.

[0068] In some examples, when an image comparison operation is initiated, the most recent previous exam for the patient (e.g., the first exam) may be accessed in the PACS, and a DICOM SR object for the exam may be sent from the PACS to the image processing system. Because the DICOM SR object includes CRs for images / replays from the previous exam, similar images or replays may be identified by searching the CRs and / or information included in the DICOM header, as the DICOM header and / or each CR may similarly include a classification tag, mode, etc. for each image, or otherwise include searchable information to enable determination of which CRs represent images / replays in the viewing plane of the current / selected image. The identified CRs (e.g., CRs for similar images) may be extracted from the DICOM SR object.

[0069] At 518, an image comparison is generated from the CR of the current / selected image or playback and the CR of the similar image or playback using the CV / LLM. For example, the CR of the similar image and the CR of the selected / current image may be input as input to the CV / LLM along with a prompt (e.g., a command from an image processing system) instructing the CV / LLM to generate a natural language description of the similarities and differences between the selected image and the similar image. In examples where more than one similar image or playback is identified, each CR of each similar image may be individually compared to the CR of the selected image to thereby generate multiple image comparisons.

[0070] By generating an image comparison using the CR of the identified image (e.g., the CR of the selected image and the CR of similar images) rather than the image itself, network bandwidth can be reduced and the processing efficiency of the image processing system can be improved. Medical images (e.g., ultrasound images) acquired during previous examinations can be relatively large files stored in the PACS. During the current patient examination, if an image comparison is to be performed on similar images stored on the PACS, the similar images must be retrieved from the PACS and sent to the image processing system. Due to the constraints of the DICOM standard on the PACS, it is not possible to identify and retrieve a single similar image. Instead, the entire examination / study can be retrieved, which can include tens or hundreds of images in addition to the similar images. The process of downloading the entire examination / study to the image processing system to enable the identification of similar images may take a relatively long time, such as multiple hours. By sending the SR instead of the entire examination, and generating the image comparison with the CR instead of the image, the image comparison can be generated in a much shorter period of time (e.g., seconds instead of hours) while using fewer processing resources and reducing network traffic.

[0071] As previously explained, images obtained in the current examination (and in some examples, the first examination) may be analyzed to determine various features of the image, such as identification of anatomical features in the image, measurement of anatomical features, evaluation of anatomical functions performed by anatomical features, etc. These various features may be identified by a clinician and manually input via user input and / or identified by one or more artificial intelligence models. For example, when saving the current image, a first artificial intelligence model may be called to automatically identify anatomical features in the current image, and then a second artificial intelligence model may be called to perform an automatic measurement of one of the anatomical features identified in the current image. In some examples, the image comparison may include outputs from one or more of the artificial intelligence models. For example, the image comparison may include measurements of anatomical features in the current image output by the second artificial intelligence model (and when the second artificial intelligence model is called to measure anatomical features in similar images, corresponding measurements of anatomical features in similar images may also be included). In some examples, the outputs from one or more artificial intelligence models (and / or measurements / findings input by clinicians) may be used as part of a quality check of the image comparison output by the CV / LLM. For example, the outputs of measurements or findings from one or more artificial intelligence models or clinician input may be compared with statements made by the CV / LLM in the image comparison. If a statement made in the image comparison does not match the output from the artificial intelligence model, it can be determined that the image comparison has not passed the quality check, and the image comparison can be updated (e.g., by the image processing system) to ensure that the image comparison is consistent with the artificial intelligence model output and / or clinician findings. As a specific example, if the image comparison output by the CV / LLM indicates that a particular measurement of an anatomical feature in the current image is different from a corresponding measurement output by an artificial intelligence model (e.g., the second artificial intelligence model described above), the image processing system can determine that the image comparison output by the CV / LLM has not passed the quality check, and the image comparison can be updated by the image processing system. For example, a statement that includes a measurement output by the CV / LLM can be modified to include a measurement from the artificial intelligence model.

[0072] At 520, the image comparison output by the CV / LLM is output, such as displaying the image comparison on a display device and / or outputting an audio narration of the sentences in the image comparison from a speaker. In some examples, the image comparison may be displayed with the selected image. An exemplary GUI including the selected image and the image comparison is shown in FIG. Figure 8 It should be understood that during the current examination, similar images are not displayed (nor are any images from the previous / first examination).

[0073] At 522, the method 500 determines whether the current examination is complete. The current examination is determined to be complete when the scan plan indicating which images are to be acquired for the current examination indicates that each image of the scan plan has been acquired. In other examples, the current examination may be determined to be complete in response to user input, for example, the user may select a user interface element indicating that the examination is complete. If the examination is not complete, the method 500 proceeds to 510 to continue acquiring images / playbacks, generating CRs for each saved image / playback, and generating image comparisons when indicated. If the examination is complete, the method 500 proceeds to 524 to send the saved images / playbacks and associated CRs (e.g., within one or more DICOM SR objects) from the current examination to an image archive (e.g., a PACS) for long-term storage. If an image comparison is performed, the image comparison may also be sent to the image archive. Further, any measurements taken during the current examination, clinician findings entered during the examination, etc. may also be sent to the image archive. The method 500 then ends.

[0074] Reference now Figure 6 , a method 600 for training a CV / LLM to generate a compressed representation of a medical image and generate a comparison of two compressed representations via an image comparison system including an image processing system, such as the image processing system 202, is shown. The CV / LLM may be a non-limiting example of the CV / LLM 206. The method 600 may be executed by a processor of the image processing system (e.g., the processor 302) based on instructions stored in a non-transitory memory of the image processing system (e.g., the training module 308).

[0075] At 602, method 600 includes obtaining a computer vision-enabled LLM pre-trained using a general domain natural language processing (NLP) dataset including an image. The general domain NLP dataset can be a publicly available dataset associated with the CV / LLM that can be used to train the CV / LLM to respond to general questions in an intelligent manner. Further, the general domain NLP dataset can include images and text associated with the images, so that the CV / LLM is trained to describe the images. However, the general domain NLP dataset may not include training data specific to medical images. For example, the CV / LLM can be an LLM of the GPT-3.5 series of IMAGECHAT or OPENAI, and the general domain NLP dataset can be an InstructGPT dataset provided by OPENAI. In other examples, different CV / LLMs can be used, and different training datasets can be provided for different CV / LLMs.

[0076] At 604, by checking the database (such as Figure 4The CV / LLM may be fine-tuned by training the CV / LLM on training data from a medical examination database 402 of the present invention. The CV / LLM may be trained on the training data to produce a fine-tuned CV / LLM that is specifically fine-tuned on medical images and generates CRs for medical images. Training the CV / LLM to produce a fine-tuned CV / LLM may include training the CV / LLM on training data that includes images (e.g., ultrasound images) and / or image playbacks from patient examinations, as indicated at 606. Further, the training data may include a description of each image / playback that includes text (e.g., findings), measurements, and / or model outputs (e.g., segmentation, view plane recognition) associated with each image or playback, as indicated at 608. Thus, the training data may include training pairs, each training pair including an image or playback and an associated description. In order to train the CV / LLM to produce a fine-tuned CV / LLM, a training pair is selected and the images or playbacks from the training pair are input as input to the CV / LLM. The CV / LLM outputs a description of the image or playback that is compared to the description from the training pairs. The parameters of the CV / LLM may be updated based on the difference (e.g., a loss function) between the description output by the CV / LLM and the description from the training pairs. The CV / LLM may be trained with a predetermined set of training pairs (e.g., hundreds or thousands of training pairs), or until the loss function reaches a threshold, at which point the CV / LLM may be considered fine-tuned.

[0077] At 610, the fine-tuned CV / LLM is validated using test data to ensure that the fine-tuned CV / LLM can adequately generate CRs for medical images (such as echocardiography images) and image playbacks (e.g., echocardiography cine playbacks). The test data may include test pairs of images / playbacks and associated descriptions.

[0078] Once the fine-tuned CV / LLM is verified, the second stage of training is performed on the fine-tuned CV / LLM using the collated training data from the examination database. The training data used to perform the second stage of training to produce the final CV / LLM can be a highly collated version of the data from the examination database to ensure that CRs of similar images / playbacks from the same patient and CRs of similar images / playbacks in which changes visible in later images or playbacks (e.g., changes in anatomical structure and / or function) have occurred are included in the training data. Therefore, the collated training data may include CRs of images and / or playbacks from longitudinal patient examinations, as indicated at 614. As previously explained, a longitudinal examination may be an examination in which more than one imaging session / examination is performed on the same anatomical region of the same patient within a certain period of time. For example, a patient may undergo two echocardiography examinations within a year to monitor the development of a heart defect / condition or to monitor the effectiveness of a treatment. The two echocardiography examinations may be longitudinal examinations, which may be selected to be included in the training data. Images / replays from the second echocardiography examination that are in the same viewing plane or are otherwise determined to be similar to images / replays from the first echocardiography examination may be selected (along with corresponding images / replays from the first echocardiography examination), and CRs for the selected images / replays may be included in the training data. Thus, each training data triplet may include a CR pair, wherein the CR pair includes a first CR of a first image or replay from a first, earlier examination of the patient and a second CR of a second image or replay from a second, later examination of the patient, wherein the first image or replay is in the same viewing plane or is otherwise determined to be similar to the second image or replay. Further, at least a portion (e.g., at least 50%) of the CR pairs included in the training data may include CRs for images or replays that include visible structural or functional changes relative to corresponding similar images of the pair. For example, at least some of the CR pairs include a first CR of a first image or playback from a first, earlier examination of the patient and a second CR of a second image or playback from a second, later examination of the patient, wherein the first image or playback is in the same viewing plane or is otherwise determined to be similar to the second image or playback, and the second image or playback includes structural or functional changes relative to the first image or playback.

[0079] The collated training data may also include a ground truth comparison for each CR pair. The ground truth comparison may include a description of the images / playbacks used to form the CR pair, the description including text, measurements, and / or other outputs associated with each image or playback, as indicated at 616. The description (e.g., text (e.g., findings), measurements, and / or other outputs (e.g., segmentation, visual plane identification)) may be extracted from a report associated with each image / playback. The description extracted from the report may be included in the training data triple along with the CR pair. Thus, each training data triple includes similar CR pairs (from the same patient, but from different imaging sessions / examinations) and text descriptions / comparisons of the images / playbacks in the similar CR pairs. In some examples, only information from the report associated with the second image / playback may be included in the description of the training data triple.

[0080] The collated training data (e.g., the training data triples described above) can be used to update the parameters of the fine-tuned CV / LLM. For example, the CR pairs from the training data triples and the prompts used to generate the comparisons of the CR pairs can be input as input to the fine-tuned CV / LLM. The fine-tuned CV / LLM can output a comparison including a natural language description of the CR pairs, which can be compared with the comparison / description from the training data triples. The parameters of the fine-tuned CV / LLM can be updated based on the difference (e.g., loss function) between the comparison / description output by the CV / LLM and the comparison / description of the training data triples. The fine-tuned CV / LLM can be trained with a predetermined set of training triples (e.g., hundreds or thousands of training triples), or until the loss function reaches a threshold, at which point the fine-tuned CV / LLM can be considered finalized. Therefore, the second stage of training the CV / LLM can include training the CV / LLM to output a comparison of two CRs. In contrast, the first stage of training the CV / LLM may include training the CV / LLM to output a CR of an image or image playback.

[0081] At 618, the finalized CV / LLM is validated using test data to ensure that the final CV / LLM can adequately generate descriptions of the CRs of medical images (such as echocardiography images) and image playbacks (e.g., echocardiography movie playbacks), including in particular any differences between the images / playbacks. The test data may include test triplets of CR pairs and associated descriptions. Once the final CV / LLM is validated, it may be deployed to generate CRs and image comparisons as described herein (e.g., to form the CV / LLM 206).

[0082] Figure 7An exemplary GUI 700 is shown that may be displayed on a screen 702 of a display device of an image comparison system, such as display device 118 and / or display device 314. GUI 700 may be displayed during a current patient exam, in which a patient is scanned and a medical image of the patient is displayed as the image is acquired. In the example shown, the medical image is an ultrasound image acquired using an ultrasound imaging system.

[0083] The GUI 700 includes a first display area 704 in which a thumbnail image of the patient is displayed. The thumbnail image may be an image of the current examination previously acquired and saved by an operator of the ultrasound imaging system. The GUI 700 also includes a main display area in which a current image 706 is displayed.

[0084] GUI 700 also includes interface elements 710, and in particular, a comparison element. Interface element 710, when selected, may trigger an image comparison operation to be performed on current image 706. As previously explained, the image comparison operation may include identifying images similar to current image 706 and generating a natural language comparison of the similar images and the current image.

[0085] Figure 8 The second state 800 is shown. Figure 7 . In the second state 800, the current image 706 is displayed. Further, a first image comparison 802 is displayed. The first image comparison 802 is generated using the final CV / LLM described herein and summarizes the differences between the current image 706 and similar images in natural language. For example, the first image comparison 802 indicates differences between anatomical features imaged in the current image 706 and similar images (e.g., features (particularly the left ventricular septum) have different sizes (e.g., increased thickness) in the current image relative to the similar images).

[0086] Fig. 9 The third state 900 is shown. Figure 7 In the third state 900, the current image 706 is displayed. Further, a second image comparison 902 is displayed. The second image comparison 902 is generated using the final CV / LLM described herein and summarizes the differences (if any) between the current image 706 and similar images in natural language. Fig. 9 In the example shown in , the second image comparison 902 indicates that no functional or anatomical differences were detected between the anatomical features imaged in the current image 706 and the similar image.

[0087] Fig.10 The fourth state 1000 is shown. Figure 7 In the fourth state 1000, the current image playback 1002 is displayed (in Fig.10). Further, a third image comparison 1004 is displayed. The third image comparison 1004 is generated using the final CV / LLM described herein and summarizes the differences (if any) between the current image playback 1002 and the previous similar image playback in natural language. Fig.10 In the example shown in , the third image comparison 1004 indicates that a functional change (e.g., a decrease in contractility) is observed in the current playback relative to the previous playback, and an anatomical feature observed in the previous playback (e.g., a P2 segment prolapse) is also observed in the current playback but has not changed since the previous examination.

[0088] A technical effect of generating and displaying an image comparison of two similar images using the CR of the two images via a trained CV / LLM is that the image comparison can be generated in a relatively short amount of time (e.g., in a few seconds or less) without having to access the actual images to perform the image comparison, which can improve the efficiency of the computing device performing the image comparison.

[0089] The present disclosure also provides support for a method comprising: acquiring a current image of a patient during a current examination; generating a first compressed representation (CR) of the current image using a computer vision-enabled large language model (CV / LLM); obtaining a second CR of a similar image that is similar to the current image and was acquired in a previous examination; using the CV / LLM to generate a text-based comparison of the current image and the similar image by inputting the first CR and the second CR as inputs to the CV / LLM; and outputting the text-based comparison. In a first embodiment of the method, the similar image is an image of the patient and includes the same anatomical features imaged in the current image. In a second embodiment of the method that optionally includes the first embodiment, the second CR is generated using the CV / LLM, and wherein outputting the text-based comparison comprises outputting the text-based comparison to a display device. In a third embodiment of the method, optionally including one or both of the first and second embodiments, the method further comprises: when the similar image was acquired during the previous examination, inputting the similar image as input to the CV / LLM to generate the second CR; saving the second CR in a DICOM Structured Report (DICOM SR) object; and sending the DICOM SR object and the similar image to an image archive for long-term storage. In a fourth embodiment of the method, optionally including one or more or each of the first to third embodiments, generating the text-based comparison of the current image and the similar image using the CV / LLM by inputting the first CR and the second CR as input to the CV / LLM comprises: obtaining the DICOM SR object from the image archive; identifying that the similar image is similar to the current image based on a search of metadata describing the image included in the DICOM SR object; and extracting the second CR from the DICOM SR object. In a fifth embodiment of the method, which optionally includes one or more or each of the first to fourth embodiments, the method further includes: inputting the similar image and the current image as input into one or more artificial intelligence models, each artificial intelligence model being trained to generate corresponding outputs related to anatomical features in the similar image and the current image. In a sixth embodiment of the method, which optionally includes one or more or each of the first to fifth embodiments, generating the text-based comparison further includes including one or more corresponding outputs from one or more of the one or more artificial intelligence models in the text-based comparison.In a seventh embodiment of the method that optionally includes one or more or each of the first to sixth embodiments, generating the text-based comparison also includes: determining that one or more statements of the text-based comparison do not match the corresponding one or more outputs of the one or more artificial intelligence models; and adjusting the one or more statements of the text-based comparison to match the corresponding one or more outputs of the one or more artificial intelligence models.

[0090] The present disclosure also provides support for an image processing system, the image processing system including a processor and a non-volatile memory, the non-volatile memory storing instructions, which when executed cause the processor to: acquire a current ultrasound image of a patient during a current examination; generate a first compressed representation (CR) of the current ultrasound image using a computer vision-enabled large language model (CV / LLM) trained on a dataset of patient examinations; identify a similar ultrasound image of the patient, the similar ultrasound image being similar to the current ultrasound image and acquired in a previous examination of the patient; obtain a second CR of the similar ultrasound image; generate a natural language comparison of the current ultrasound image and the similar ultrasound image by inputting the first CR and the second CR as inputs to the CV / LLM; and output the natural language comparison. In a first embodiment of the system, the instructions, when executed, also cause the processor to automatically identify a viewing plane of the current ultrasound image, and wherein identifying the similar ultrasound image includes identifying the similar ultrasound image based on the similar ultrasound image being labeled with the same viewing plane as the current ultrasound image. In a second embodiment of the system optionally including the first embodiment, obtaining the second CR of the similar ultrasound image includes: retrieving a DICOM structured report (DICOM SR) object associated with the previous examination from an image archive; and extracting the second CR from the DICOM SR object. In a third embodiment of the system optionally including one or both of the first and second embodiments, the CV / LLM is trained by: performing a first training phase, the first training phase including fine-tuning the CV / LLM to generate CRs of ultrasound images and ultrasound image playbacks, the first training phase including training the CV / LLM using training pairs generated from the dataset of patient examinations, each training pair including a corresponding ultrasound image or a corresponding ultrasound image playback and a corresponding description of the ultrasound image or ultrasound image playback; and performing a second training phase, the second training phase including finalizing the CV / LLM. The CV / LLM is configured to generate a comparison of CRs of ultrasound images and ultrasound image playbacks, the second training phase comprising training the CV / LLM using training triplets generated from the dataset of patient examinations, each training triplet comprising a corresponding CR pair and a corresponding description of the CR pair, wherein each CR pair comprises a first training CR of a first ultrasound image or ultrasound image playback from a first earlier examination of a given patient and a second training CR of a second ultrasound image or ultrasound image playback from a second later examination of the given patient, wherein the first ultrasound image or ultrasound image playback is in the same viewing plane as the second ultrasound image or ultrasound image playback. In a fourth embodiment of the system that optionally includes one or more or each of the first to third embodiments, in at least some of the CR pairs of the training triplets, the second ultrasound image or ultrasound image playback of the CR pair comprises a structural or functional change relative to the first ultrasound image or ultrasound image playback of the CR pair.In a fifth embodiment of the system, optionally including one or more or each of the first to fourth embodiments, the image processing system is operably coupled to an ultrasound probe, and wherein acquiring the current ultrasound image of the patient during the current examination includes receiving ultrasound data from the ultrasound probe and processing the ultrasound data to form the current ultrasound image. In a sixth embodiment of the system, optionally including one or more or each of the first to fifth embodiments, outputting the natural language comparison includes displaying the natural language comparison on a display device, and wherein during the current examination, the similar ultrasound image is not displayed on the display device.

[0091] The present disclosure also provides support for a method for an image processing system for an ultrasound imaging system, the method comprising: acquiring a current ultrasound image of a patient during a current examination; generating a first compressed representation (CR) of the current ultrasound image using a computer vision-enabled large language model (CV / LLM), the CV / LLM being trained to generate a CR of the ultrasound image based on training pairs, the training pairs including an ultrasound image and a first description of the ultrasound image for each training pair; identifying a similar ultrasound image of the patient, the similar ultrasound image being similar to the current ultrasound image and acquired in a previous examination of the patient; obtaining a second CR of the similar ultrasound image; generating a natural language comparison of the current ultrasound image and the similar ultrasound image by inputting the first CR and the second CR as input to the CV / LLM, wherein the CV / LLM is further trained to generate a natural language comparison of similar CRs based on training triples, the training triples including a similar CR pair and a second description of the similar CR pair for each training triple; and displaying the natural language comparison on a display device. In a first embodiment of the method, the method further comprises: automatically identifying a viewing plane of the current ultrasound image, and wherein identifying the similar ultrasound image comprises identifying the similar ultrasound image based on the similar ultrasound image being tagged with the same viewing plane as the current ultrasound image. In a second embodiment of the method optionally including the first embodiment, obtaining the second CR of the similar ultrasound image comprises: retrieving a DICOM structured report (DICOM SR) object associated with the previous examination from an image archive instead of the similar ultrasound image; and extracting the second CR from the DICOM SR object. In a third embodiment of the method optionally including one or both of the first and second embodiments, displaying the natural language comparison on the display device comprises displaying the natural language comparison with the current ultrasound image and not displaying the similar ultrasound image. In a fourth embodiment of the method optionally including one or more or each of the first to third embodiments, each similar CR pair comprises a first training CR for a first ultrasound image and a second training CR for a second ultrasound image, the second ultrasound image being acquired after the first ultrasound image and comprising structural or functional changes in anatomical features relative to the first ultrasound image.

[0092] When introducing the elements of various embodiments of the present disclosure, the articles "one", "a kind of" and "the" are intended to mean that there are one or more such elements. The terms "first", "second", etc. do not represent any order, amount or importance, but are used to distinguish one element from another element. The terms "include", "comprises", and "have" are intended to be inclusive, and mean that additional elements may also exist in addition to the listed elements. As used herein, the terms "connected to", "coupled to", etc., an object (e.g., material, element, structure, member, etc.) may be connected to or coupled to another object, regardless of whether the one object is directly connected or coupled to another object, or whether there are one or more intervening objects between the one object and another object. In addition, it should be understood that reference to "an embodiment" or "embodiment" of the present disclosure is not intended to be interpreted as excluding the existence of additional embodiments that are also combined with the cited features.

[0093] In addition to any previously indicated modifications, those skilled in the art may devise numerous other variations and alternative arrangements without departing from the spirit and scope of the present specification, and the appended claims are intended to cover such modifications and arrangements. Therefore, although the information has been described in detail and in detail as described above in conjunction with what are currently considered to be the most practical and preferred aspects, it will be apparent to those of ordinary skill in the art that many modifications, including but not limited to form, function, mode of operation, and use, may be made without departing from the principles and concepts set forth herein. Likewise, as used herein, the examples and embodiments are meant to be illustrative in all respects and should not be construed as limiting in any way.

Claims

1. A method (500), comprising: acquiring (510) a current image of the patient during a current examination; generating (512) a first compressed representation (CR) of the current image using a computer vision enabled large language model (CV / LLM (206)); obtaining (516) a second CR of a similar image that is similar to the current image and was acquired in a previous examination; generating (518) a text-based comparison of the current image and the similar image using the CV / LLM (206) by inputting the first CR and the second CR as input to the CV / LLM (206); and The text-based comparison is output (520).

2. The method (500) of claim 1, wherein the similar image is an image of the patient and includes the same anatomical features imaged in the current image.

3. The method (500) of claim 1, wherein the second CR is generated (504) using the CV / LLM (206), and wherein outputting the text-based comparison comprises outputting the text-based comparison to a display device (314, 118).

4. The method (500) according to claim 3, further comprising: when the similar image was acquired during the previous examination, inputting (504) the similar image as input to the CV / LLM (206) to generate the second CR; Saving (504) the second CR in a DICOM Structured Report (DICOM SR) object; as well as The DICOM SR object and the similar images are sent to an image archive for long term storage.

5. The method (500) of claim 4, wherein generating the text-based comparison of the current image and the similar image using the CV / LLM (206) by inputting the first CR and the second CR as input to the CV / LLM (206) comprises: obtaining (516) the DICOM SR object from the image archive; identifying the similar image as being similar to the current image based on a search of metadata describing the image included in the DICOM SR object; And extracting the second CR from the DICOM SR object.

6. The method (500) according to claim 1 further includes inputting the similar images and the current image as input into one or more artificial intelligence models, each artificial intelligence model being trained to generate corresponding outputs related to anatomical features in the similar images and the current image.

7. The method (500) of claim 6, wherein generating the text-based comparison further comprises including one or more corresponding outputs from one or more of the one or more artificial intelligence models in the text-based comparison.

8. The method (500) of claim 6, wherein generating the text-based comparison further comprises: determining that one or more statements of the text-based comparison do not match corresponding one or more outputs of the one or more artificial intelligence models; and adjusting the one or more statements of the text-based comparison to match the corresponding one or more outputs of the one or more artificial intelligence models.

9. An image processing system (202), the image processing system comprising a processor (302) and a non-transitory memory (304), the non-transitory memory storing instructions, the instructions when executed causing the processor (302) to: acquiring (510) a current ultrasound image of the patient during a current examination; generating (512) a first compressed representation (CR) of the current ultrasound image using a computer vision enabled large language model (CV / LLM (206)) trained on a dataset of patient examinations; identifying (516) a similar ultrasound image of the patient, the similar ultrasound image being similar to the current ultrasound image and being acquired in a previous examination of the patient; obtaining (516) a second CR of the similar ultrasound image; generating (518) a natural language comparison of the current ultrasound image and the similar ultrasound image by inputting the first CR and the second CR as input to the CV / LLM (206); and The natural language comparison is output (520).

10. The image processing system (202) of claim 9, wherein the instructions, when executed, further cause the processor (302) to automatically identify a viewing plane of the current ultrasound image, and wherein identifying the similar ultrasound image comprises identifying the similar ultrasound image based on the similar ultrasound image being marked with the same viewing plane as the current ultrasound image.

11. The image processing system (202) according to claim 9, wherein obtaining the second CR of the similar ultrasound image comprises: retrieving a DICOM Structured Report (DICOM SR) object associated with the previous examination from an image archive; And extract the second CR from the DICOMSR object.

12. The image processing system (202) of claim 9, wherein the CV / LLM (206) is trained by: performing (604) a first training phase, the first training phase comprising fine-tuning the CV / LLM (206) to generate CRs for ultrasound images and ultrasound image replays, the first training phase comprising training the CV / LLM (206) using training pairs generated from the dataset of patient examinations, each training pair comprising a corresponding ultrasound image or a corresponding ultrasound image replay and a corresponding description of the ultrasound image or ultrasound image replay; and Executing (612) a second training phase, the second training phase comprising finalizing the CV / LLM (206) to generate a comparison of CRs of ultrasound images and ultrasound image replays, the second training phase comprising training the CV / LLM (206) using training triplets generated from the data set of patient examinations, each training triplet comprising a corresponding CR pair and a corresponding description of the CR pair, wherein each CR pair comprises a first training CR of a first ultrasound image or ultrasound image replay from a first earlier examination of a given patient and a second training CR of a second ultrasound image or ultrasound image replay from a second later examination of the given patient, wherein the first ultrasound image or ultrasound image replay is in the same viewing plane as the second ultrasound image or ultrasound image replay.

13. The image processing system (202) according to claim 12, wherein in at least some CR pairs of the training triplets, the second ultrasound image or ultrasound image playback of the CR pair includes structural or functional changes relative to the first ultrasound image or ultrasound image playback of the CR pair.

14. The image processing system (202) of claim 9, wherein the image processing system (202) is operably coupled to an ultrasound probe (106), and wherein acquiring the current ultrasound image of the patient during the current examination includes receiving ultrasound data from the ultrasound probe (106) and processing the ultrasound data to form the current ultrasound image.

15. The image processing system (202) of claim 9, wherein outputting the natural language comparison comprises displaying the natural language comparison on a display device (314, 118), and wherein during the current examination, the similar ultrasound images are not displayed on the display device (314, 118).

Images