Computer-implemented method and system for reducing a data volume of image recordings

Abstract

The invention relates to a computer-implemented method for reducing a data volume of image recordings, to be saved, of an image recording sequence, having the steps of: providing one or more image recordings of an image recording sequence of a patient with a first image resolution; creating, on the basis of the provided image recordings of the image recording sequence, corresponding preprocessed image recordings of the image recording sequence by means of burst image processing, wherein the preprocessed image recordings of the image recording sequence have reduced image noise and / or reduce image artefacts; generating, for each created preprocessed image recording of the image recording sequence, an image recording with a second image resolution by means of super-resolution image processing, wherein the second image resolution is higher than the first image resolution; and compressing the generated highly resolved and / or the created preprocessed image recordings of the image recording sequence in order to reduce the data volume, to be saved, of the image recording sequence.

Description

[0001] Computer-implemented method and system for reducing the amount of data from image recordings

[0002] Technical field of the invention

[0003] The present invention relates to a computer-implemented method for reducing the data volume of image recordings, particularly those originating from an endoscope or exoscope, using super-resolution. Furthermore, the present invention relates to a system for reducing the data volume of image recordings using super-resolution.

[0004] Background of the invention

[0005] For documentation purposes during medical procedures and surgical interventions, it is helpful to rotate and store image recordings (such as photos or videos) in this format. If incidents occur during or after a medical procedure, this allows for a more precise determination of the cause, thus providing protection for physicians, patients, and hospitals.

[0006] Generating video recordings of medical activities with satisfactory resolution and quality currently requires large amounts of data. This results in high storage requirements, which are not always available.

[0007] Merely compressing the data is often unsatisfactory. Recording quality is particularly critical for endoscope video recordings, where the image contrast between bright and dark areas can be strong. In particular, unwanted artifacts such as luminance and chrominance noise can be caused by hardware limitations and known physical effects of the recording technique. Data compression involves processing not only the desired signal but also a noise signal (caused by artifacts). The more artifacts present, the poorer the compression result. Since artifacts counteract strong data compression, their presence leads to either significantly lower compression factors or, in the case of lossy methods, to significant losses in the desired signal.

[0008] Conventional methods for storing video recordings in medical and surgical settings require a lot of storage space. Furthermore, the recording quality is sometimes poor, meaning that some delicate physical structures, such as veins or nerves, whose identification is important for surgical procedures, are not clearly visible in the video recordings.

[0009] Accordingly, there is a need to provide improved methods to visualize image recordings with the resolution requirements, particularly of endoscopic video recordings, and to store them with reduced storage space requirements.

[0010] Summary of the invention

[0011] It is therefore an object of the present invention to introduce a system and a method which provide a reduction in the amount of data of image recordings to be stored.

[0012] The object of the present invention is achieved by a computer-implemented method having the features of claim 1 and by a system having the features of claim 9. Preferred embodiments of the invention with advantageous features are specified in the dependent claims. According to a first aspect, the invention comprises a computer-implemented method for reducing the amount of data to be stored from images of an image acquisition sequence of a patient, comprising the steps:

[0013] Providing one or more image acquisitions of an image acquisition sequence of a patient with a first image resolution;

[0014] Generating, based on the provided image recordings of the image recording sequence, corresponding pre-processed image recordings of the image recording sequence by means of burst image processing, wherein the pre-processed image recordings of the image recording sequence have reduced image noise and / or reduced image artifacts;

[0015] Generating, for each generated preprocessed image acquisition of the image acquisition sequence, an image acquisition with a second image resolution using super-resolution image processing, wherein the second image resolution is higher than the first image resolution; and

[0016] Compressing the generated high-resolution and / or the generated pre-processed images of the image acquisition sequence to reduce the amount of data of the image acquisition sequence to be stored.

[0017] Images from an image recording sequence are images of physical structures captured by a still camera and / or a video camera. Such images have an image resolution associated with the number of pixels of the recording device (still camera and / or video camera). The images exhibit image noise and other image artifacts and generally exhibit quality limitations. These are related to the recording technology (performance of the camera and / or video camera) and are also referred to as noise. Image artifacts include, among others: image noise, graininess, color casts, ringing, blooming, color distortions, blurring, or fringing.

[0018] Providing includes capturing and / or reading in images.

[0019] The invention particularly relates to real-time data from medical activities and / or surgical interventions, as well as to data from previous activities and / or interventions. In both cases, the images of the image acquisition sequence can originate from either a still camera and / or a video camera (e.g., the video camera at the distal end of an endoscope).

[0020] Data reduction refers to a reduction in the data volume of the image recordings, achieved by removing or filtering out image artifacts and / or compressing the data volume. In other words, data reduction refers in particular or exclusively to noise reduction in the data volume.

[0021] Burst image processing essentially refers to methods for image processing with which a large number of details can be reconstructed in an image recording, usually based on a combination of signal information from several shifted images, ie from a sequence of image recordings arranged one after the other in time within an image recording sequence.

[0022] Super-resolution essentially refers to image processing methods that increase the resolution of an image. Super-resolution image processing essentially involves increasing the number of pixels in an image through interpolation, which creates new pixels. The resolution increase factor can be at least 2, preferably at least 4.

[0023] Compression involves reducing the amount of data using a conventional method (e.g. mp4 / jpeg).

[0024] According to a second aspect, the invention provides a system for reducing the amount of data to be stored from image recordings of an image recording sequence, comprising: a provision device configured to provide one or more image recordings in an image recording sequence of a patient with a first image resolution; a burst image processing device configured to generate, based on the provided image recordings of the image recording sequence, corresponding preprocessed image recordings of the image recording sequence by means of burst image processing, wherein the preprocessed image recordings of the image recording sequence have reduced image noise and / or reduced image artifacts;a super-resolution image processing device configured to generate, for each generated preprocessed image of the image acquisition sequence, an image with a second image resolution using super-resolution image processing, wherein the second image resolution is higher than the first image resolution; and an image compression device configured to compress the generated high-resolution and / or the generated preprocessed image recordings of the image acquisition sequence to reduce the amount of data to be stored in the image acquisition sequence.

[0025] The various devices can each be implemented as a device comprising or consisting of at least one central processing unit (CPU) and / or at least one graphics processing unit (GPU) and / or at least one field-programmable gate array (FPGA) and / or at least one application-specific integrated circuit (ASIC) and / or any combination of the aforementioned elements. Furthermore, the devices can comprise a programming interface (API) via which signals and information can be exchanged with each other via wired and / or wireless means.

[0026] Each element of the system of the invention may further comprise a memory operatively connected to the at least one CPU and / or a non-volatile memory operatively connected to the at least one CPU and / or the memory. Each element may be implemented partially and / or entirely in a local device and / or partially and / or entirely in a remote system, such as a cloud computing platform.

[0027] The burst image processing device, the super-resolution image processing device, and the image compression device may execute or be configured as software, an app, or an algorithm with different data processing capabilities. These devices may be implemented in hardware and / or software, wired and / or wireless, and any combination thereof. They may further include an interface to an intranet or the internet, to a cloud computing service, to a remote server, and / or the like.

[0028] The provisioning device can have various sensors that can capture (or record) images. Furthermore, the provisioning device can receive, process, and evaluate sensor data.

[0029] In particular, the computer-implemented method according to the first aspect of the invention can be performed with the system according to the second aspect of the invention. The features and advantages described herein in connection with the system are therefore also applicable to the method, and vice versa.

[0030] According to a third aspect, the invention provides a computer program product comprising an executable program code which, when executed, is adapted to carry out the method according to the first aspect of the present invention.

[0031] According to a fourth aspect, the invention provides a non-volatile computer-readable data storage medium comprising executable program code configured, when executed, to perform the method according to the first aspect of the present invention. The non-volatile, computer-readable data storage medium may comprise or consist of any type of computer memory, in particular a semiconductor memory, such as a solid-state memory. The data storage medium may also comprise or consist of a CD, a DVD, a Blu-ray disc, a USB memory stick, or the like.

[0032] According to a fifth aspect, the invention provides a data stream comprising or being adapted to generate executable program code which, when executed, is adapted to carry out the method according to the first aspect of the present invention.

[0033] One idea underlying the invention is to introduce a computer-implemented method with which image recordings, and in particular video recordings, could be stored (or transmitted) with reduced data storage space. According to the invention, this reduction is achieved by combining two effects: On the one hand, artifacts (e.g., those caused by hardware limitations) are removed from the image recordings to improve the recording quality. This is achieved, for example, by using burst image processing.

[0034] This creates preprocessed, denoised images from images captured by a camera. On the other hand, the resolution of the preprocessed images is increased through super-resolution image processing.

[0035] The method can be used in two modes. In a first mode, the preprocessed images are compressed and then processed using super-resolution. The super-resolution is then applied as post-processing, which can be performed, for example, during the visualization of the video recording. In a second mode, the preprocessed images are processed using super-resolution and then compressed. In this case, the compressed file package contains super-resolution images, and no post-processing is required. The computer-implemented method described above advantageously enables the implementation of a system for reducing the amount of data to be stored in image recordings.The system comprises a provision unit by which images of a medical activity or scene are captured (for example, when the provision device comprises the camera of an endoscope) or read in (for example, when the captured images were received from the camera of an endoscope). A burst image processing device is used to perform image processing that generates denoised (or interference-free) images (images with reduced image artifacts) based on combinations of the images. A super-resolution image processing device is used to generate super-resolution images using super-resolution. An image compression device is configured to compress the images generated by the burst image processing device or the images generated by the super-resolution image processing device.

[0036] One advantage of the present invention is that the combination of burst image processing and super-resolution image processing significantly reduces the data volume of the captured images without any loss in the useful signal. In other words, the removal of image noise and / or various image artifacts ensures that compression essentially only affects the useful signal, which allows for higher compression factors to be achieved. Super-resolution ensures that the resolution of the denoised images is increased. The result is a high-resolution video recording with good image quality and low storage requirements.

[0037] Another advantage of the invention is that it allows for improved low-light performance. This allows for patient protection, in particular, by allowing the use of thinner endoscopes with smaller camera lenses for medical / surgical procedures.

[0038] A further advantage of the invention is that the increase in image quality and resolution of the processed video recordings is not dependent on the hardware of an endoscope. The invention is achieved by optimizing data processing and does not require any hardware improvements.

[0039] Advantageous embodiments and further developments emerge from the dependent claims and from the description of the various preferred embodiments shown in the attached figures.

[0040] According to some preferred embodiments, variants, or further developments of embodiments, it is provided that several underexposed raw images are used to generate the preprocessed images of the image acquisition sequence. In order to perform the image processing, information from the raw images is required. Thus, the system according to the invention can be used in an image processing method that leads from raw images (for example, from the camera of an endoscope) to finished images or videos.

[0041] In endoscopic imaging, the light-dark contrast is particularly strong. Therefore, it is advantageous to be able to work with underexposed raw images. Undesirable effects such as image noise or motion blur can be minimized and filtered and removed using burst image processing. This contributes to improving low-light performance.

[0042] According to some preferred embodiments, variants, or further developments of embodiments, the computer-implemented method further comprises storing the compressed, generated high-resolution and / or the compressed, generated preprocessed image recordings of the image recording sequence. According to the invention, the data can be stored after compression and stored in a computer, on a server, or on a cloud computing platform. The compressed files can have at least one identifier indicating whether the file contains compressed, generated preprocessed image recordings or compressed, generated high-resolution image recordings.

[0043] According to some preferred embodiments, variants, or further developments of embodiments, the generation includes reading in the stored, compressed, generated, preprocessed images of the image acquisition sequence. Reading in can be performed using an interface that, for example, has the super-resolution image processing device. This allows the stored file to be retrieved and processed for visualization with increased resolution.

[0044] According to some preferred embodiments, variants or further developments of embodiments, it is provided that the generation of the preprocessed image recordings of the image recording sequence by means of burst image processing comprises the following steps:

[0045] Grouping the provided image recordings of the image recording sequence into image recording groups, depending on an image recording rate;

[0046] Segmenting each provided image capture of the image capture group into image segments; and

[0047] Combining corresponding spatially correlated image segments of the different image recordings of the image recording group into a pre-processed image recording.

[0048] Burst image processing produces images with a variety of details of the recorded physical structures. The generated images are based on a combination of signal information from a sequence of chronologically arranged images. Burst image processing can be performed using a burst algorithm. Accordingly, the images are first grouped, bundled, or clustered. The number of images in each group depends on the image capture rate (the number of frames captured per second), which in turn depends on the camera. Typically, a group can contain between 3 and 20 images. The different images within each group are then segmented. The size of the image segments can vary. An image segment can contain one pixel, but also a large area of ​​an image. The segments of an image do not have to be identical.It is important that spatially correlated segments of the different image acquisitions in a group are comparable so that they can be combined into a processed image segment. The processed image segments create a processed image acquisition.

[0049] According to some preferred embodiments, variants or further developments of embodiments, it is provided that the number of corresponding spatially correlated image segments that are combined to form a preprocessed image recording depends on the brightness of the corresponding image segments.

[0050] One goal of combining different image recordings is to overlap the underexposed raw image recordings to create a normally exposed image recording. In some embodiments, the processed image recording can be generated more quickly by making the overlap of the image segments dependent on the brightness of the image segments. The brighter an image segment, the fewer image recordings of the image recording group must be used to generate the corresponding processed image segment. Thus, the brightest image segment can use only the last image recording (in a chronological order of the image recordings) of the image recording group, while the darkest image segment can use all image recordings of the image recording group. According to some preferred embodiments, variants, or further developments of embodiments, it is provided that combining corresponding spatially correlated image segments comprises:

[0051] Aligning the images provided by the image acquisition group with each other;

[0052] Overlaying the image segments of the aligned images of the image acquisition group; and

[0053] Hue mapping of the superimposed image segments of the aligned images of the image acquisition group.

[0054] Overlapping the various image segments of an image acquisition group can involve at least three steps: First, the image segments are aligned, for example, based on pixel identification. The image segments are then merged or overlaid. This can be done adaptively by performing the overlay step by step. For example, in an image acquisition group with 20 images, overlaying can be performed with only four images at a time. Once the fifth image has been taken, the first image is no longer considered, so the algorithm always works with four images simultaneously. Overlaying serves to filter out image noise.

[0055] Color mapping can then be performed. Color mapping refers to various image processing methods, such as black level subtraction, white balance, Bayer demosaicing, bilinear chroma denoising, sRGB color correction, tone mapping, gamma correction, global contrast adjustment, and unsharp mask sharpening, and / or chromatic aberration correction.

[0056] According to some preferred embodiments, variants or further developments of embodiments, it is provided that the generation of a high-resolution image recording is carried out by means of super-resolution image processing with the aid of a trained artificial intelligence model, wherein the trained artificial intelligence model corresponds to a convolutional neural network, CNN, or a deep learning architecture, in particular a transformer-based architecture, or the like.

[0057] Super-resolution image processing essentially involves increasing the number of pixels in an image using an interpolation algorithm that generates new pixels. This interpolation algorithm can be implemented with an artificial intelligence model, or at least partially supported by it. Particularly suitable models are artificial intelligence models adapted for pattern recognition, particularly CNNs. Models with vision transformer (ViT)-based architectures, such as a Swin transformer, are becoming increasingly important. This allows, for example, the generated images to be created with a resolution four times that of the raw images.

[0058] According to some preferred embodiments, variants, or further developments of embodiments, the burst image processing device comprises a calculation unit that implements a burst algorithm designed to detect fine physical structures such as veins and nerves in the provided images of the patient's image acquisition sequence. Particularly in endoscopic image acquisitions, it is important that the burst image processing device is adapted to achieve the detection of fine structures. This is achieved by detecting these structures during the image acquisition sequence of an image acquisition group during their movements and enhancing them with a color contrast, so that the generated processed images represent these fine structures with satisfactory image quality.

[0059] According to some preferred embodiments, variants, or further developments of embodiments, the system's provision device is integrated into an endoscope or an exoscope. The provision device can comprise the video camera of an endoscope or exoscope and / or other image sensors present in an endoscope or an exoscope. The burst image processing device and / or the super-resolution image processing device can be implemented in the data processing device of an endoscope or an exoscope, which processes the signals from the endoscope / exoscope's camera into visualizable images.

[0060] According to some preferred embodiments, variants, or further developments of embodiments, the system further comprises a data storage device in which the image recordings of the image recording sequence compressed by the image compression device are stored and from which they can be read out. The data storage device can store the compressed, generated, processed image recordings and / or the compressed, generated, high-resolution image recordings.

[0061] According to some preferred embodiments, variants, or further developments of embodiments, the system further comprises a video interface designed to transmit the generated preprocessed images of the image acquisition sequence and / or the generated high-resolution images of the image acquisition sequence to a display unit of the system. In some embodiments, the system can allow a user to visualize the processed images, either after burst image processing or after super-resolution image processing, on a display. If the images are processed real-time images originating directly from the video camera of an endoscope, at least the burst image processing device and the super-resolution image processing device can advantageously be implemented in the display unit of the endoscope.

[0062] Although some functions are described here and below as being performed by devices, this does not necessarily mean that these devices are provided as separate entities. In cases where one or more devices, or even a portion thereof, are provided as software, the devices may be implemented by program code sections or snippets that may be separate from each other, but may also be interwoven or integrated with each other.

[0063] Likewise, in cases where one or more devices are provided as hardware, the functions of one or more devices may be provided by one and the same hardware component, or the functions of several devices may be distributed among several hardware components that do not necessarily correspond to the devices. It is therefore to be assumed that any application, system, method, etc., that has all the features and functions attributed to a particular device includes or implements that device. In particular, it is possible that all the devices are implemented by program code executed by, for example, a server or a cloud computing platform.

[0064] All mentioned embodiments and implementations can be combined with each other as required, as long as this makes sense.

[0065] The further scope of applicability of the present method and system will become apparent from the following figures, detailed description, and claims. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the invention, are primarily illustrative, and various changes and modifications within the basic spirit and scope of the invention will be apparent to those skilled in the art.

[0066] Short description of the drawings

[0067] The invention will now be described with reference to advantageous embodiments thereof, with reference to the following drawings. In the drawings, identical or functionally similar elements are designated by the same reference numerals throughout the several views. The drawings serve to further illustrate embodiments of concepts incorporating the claimed invention and to explain various principles and advantages of those embodiments. Elements depicted in the drawings are not necessarily drawn to scale. This serves to clearly disclose the principles and principles of the invention.

[0068] In the drawings shows:

[0069] Fig. 1 is a schematic flow diagram of a computer-implemented method for reducing the amount of data to be stored from images of an image acquisition sequence according to an embodiment of the invention;

[0070] Fig. 2 is a schematic flow diagram of a computer-implemented method for reducing the amount of data to be stored from image recordings of an image recording sequence according to a further embodiment of the invention;

[0071] Fig. 3 shows a system for reducing the amount of data to be stored from images of an image acquisition sequence according to an embodiment of the invention;

[0072] Fig. 4 shows an endoscope in which a system for reducing the amount of data to be stored from images of an image acquisition sequence is implemented according to an embodiment of the invention; Fig. 5 shows a schematic block diagram illustrating a computer program product according to an embodiment of the third aspect of the present invention; and

[0073] Fig. 6 is a schematic block diagram showing a data storage medium according to an embodiment of the fourth aspect of the present invention.

[0074] In some cases, well-known structures and devices are depicted in block diagram form to illustrate the concepts of the present invention. The numbering of steps in the methods is also intended to facilitate their description. They do not necessarily imply a particular order of the steps. In particular, multiple steps may be performed concurrently or in an overlapping manner.

[0075] Description of the drawings

[0076] The detailed description of the accompanying drawings contains specific details in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details.

[0077] Fig. 1 shows a schematic flow diagram of a computer-implemented method for reducing a data volume of image recordings of an image recording sequence to be stored according to an embodiment of the invention.

[0078] In a step S1, images of an image acquisition sequence of a patient are provided at a first resolution. The images could originate from a video camera, for example, a video camera attached to the distal end of an endoscope. In this case, the images are read in or captured. The image acquisition sequence advantageously includes underexposed raw images.

[0079] In a step S2, the provided image recordings are preprocessed using burst image processing to generate corresponding image recordings of the image recording sequence with reduced image noise and / or reduced image artifacts. According to the embodiment of Fig. 1, step S2 comprises steps S20, S21, and S22.

[0080] In step S20, the provided images of the image acquisition sequence are grouped into image acquisition groups. The number of images per image acquisition group advantageously depends on the image acquisition rate (the number of recorded image frames per second). The image acquisition rate can be adjusted with the video camera, for example, by a user, depending on the physical structures to be recorded. For example, an image acquisition group can contain between 3 and 20 images.

[0081] In step S21, the various image recordings in each image recording group are segmented. The size of the image segments can be variable. An image segment can contain only one pixel or a significant area of ​​the image recording (i.e., several pixels, for example, over 30% or even over 50% of the pixels of the image recording). The image segments of an image recording do not have to be the same size. As a rule, the image segments should ensure that all details and properties of an image recording are optimally identified. Areas of an image recording that appear homogeneous can, for example, be associated with or displayed with an image segment. Areas of an image recording with different details generally require more than one image segment. It is advantageous if spatially correlated image segments of the various image recordings in an image recording group are of comparable size.In some embodiments of the invention, the number of spatially correlated image segments may depend on the brightness of the corresponding image segments. In step S22, the spatially correlated image segments are combined into a corresponding preprocessed image recording. According to the embodiment of Fig. 1, step S22 comprises steps S221, S222, and S223.

[0082] In step S221, an alignment of the image segments is carried out, which can be carried out, for example, with regard to pixel identification.

[0083] In step S222, the image segments are merged and / or superimposed. This can be done adaptively by performing the superimposition step by step. For example, in an image acquisition group with 20 images, the superimposition can be performed with only four images at a time. Once the fifth image has been taken, the first image is no longer considered so that four images are always processed simultaneously. Superimposition is usually used to filter out image noise.

[0084] In step S223, hue mapping is performed. Hue mapping refers to various image processing methods, such as black level subtraction, white balance, Bayer demosaicing, bilinear chroma denoising, sRGB color correction, tone mapping, gamma correction, global contrast adjustment, and unsharp mask sharpening, and / or chromatic aberration correction.

[0085] In step S3, an image is generated using super-resolution image processing for each preprocessed image of the image acquisition sequence. Due to the super-resolution, the generated images have a higher resolution (e.g., four times the resolution) than the provided images. In step S4, the generated high-resolution images of the image acquisition sequence are compressed. Compression can be performed using conventional methods (mp4 / jpeg).

[0086] In step S5, the compressed image recordings are saved. The image recordings can be stored and retained as compressed video files on a computer, a server, or a cloud computing platform.

[0087] Fig. 2 shows a schematic flow diagram of a computer-implemented method for reducing a data volume of image recordings of an image recording sequence to be stored according to a further embodiment of the invention.

[0088] In step S 1, images of an image recording sequence of a patient, which originate for example from a video camera of an endoscope, are provided with a first resolution.

[0089] In a step S2, the provided image recordings are preprocessed using burst image processing to generate corresponding image recordings of the image recording sequence with reduced image noise and / or reduced image artifacts. Step S2 may, for example, include steps S20, S21, and / or S22, which were described in connection with Fig. 1 (not shown in Fig. 2). Furthermore, step S22 may, if present, include steps S221, S222, and / or S223, which were described in connection with Fig. 1 (not shown in Fig. 2).

[0090] In Fig. 2, in a step S4, the preprocessed images of the image acquisition sequence generated in step S2 are compressed.

[0091] The compressed images are stored in step S5. The images can be stored and stored as compressed video files on a computer, a server, or a cloud computing platform. In step S3, the generated preprocessed images are processed using super-resolution, and corresponding images are generated. Due to the super-resolution, the generated images have a higher resolution (for example, four times the resolution) than the provided images and the generated preprocessed images. Step S3 includes step S31. In step S31, the stored compressed preprocessed images are read in.

[0092] Super-resolution image processing can process both compressed and uncompressed images. Therefore, prior decompression of the images compressed in step S4 is not required. However, in some embodiments, a decompression step may be performed before step S3.

[0093] The embodiment of the invention shown in Fig. 2 can be advantageous for further reducing storage space requirements. In this case, super-resolution is performed as post-processing of the image recordings during the visualization of the video file.

[0094] In the method shown in Fig. 1, the super-resolution images are compressed and then stored. After super-resolution image processing, the number of pixels increases, and accordingly, the generated high-resolution images require more storage space than the images generated using burst image processing. However, the generated high-resolution images still require less storage space than conventional images, such as those obtained from an endoscope's video camera. The advantage of the embodiment shown in Fig. 1 is that the stored video file contains super-resolution images, and no super-resolution post-processing is necessary. - TI -

[0095] Fig. 3 shows a system 100 for reducing the amount of data to be stored from an image acquisition sequence according to one embodiment of the invention. The various components and functions are schematically represented as blocks. The spatial arrangement of the blocks in Fig. 3 serves only to illustrate the illustrated embodiment.

[0096] As shown in Fig. 3, the system 100 comprises a provisioning device 10, a burst image processing device 20, a super-resolution image processing device 30, an image compression device 40, a data storage device 50, and a video interface 60.

[0097] The provision device 10 is configured to capture or read one or more image recordings D0 in an image recording sequence of a patient. The image recordings D0 can, for example, be or include underexposed raw image recordings that are or were recorded with a video camera. For example, the video camera can be a video camera attached to the end of an endoscope or exoscope. In some embodiments, the provision device 10 can comprise the video camera of an endoscope.

[0098] The burst image processing device 20 serves to generate corresponding preprocessed image recordings D1 based on the provided image recordings using burst image processing. The burst image processing device 20 can include a calculation unit 210 that implements a burst algorithm. The burst algorithm ensures that fine physical structures, such as veins and nerves, present in the image recordings D0 can be recognized in the generated preprocessed image recordings D1 after the burst image processing.

[0099] Furthermore, the burst image processing unit 20 may comprise various units configured to perform the various steps of the burst image processing. For example, a grouping unit, a segmenting unit, and a combining unit (not shown in Fig. 3) may each be used to perform one of the steps S20, S21, and S22 described in connection with Fig. 1. Similarly, the burst image processing unit 20 may comprise an alignment unit, an overlay unit, and a hue mapping unit, each configured to perform one of the steps S221, S222, and S223 described in connection with Fig. 1.

[0100] The super-resolution image processing device 30 is configured to generate a corresponding high-resolution (or: high-resolution) image recording D2 for each generated preprocessed image recording D1 of the image recording sequence. The super-resolution image processing device 30 is capable of reading in compressed as well as uncompressed files. The super-resolution image processing device 30 can increase the resolution with the aid of a trained artificial intelligence model. The artificial intelligence model can have a model adapted for pattern recognition, for example, a model based on or comprising convolutional neural networks (CNN) or vision transformer architectures.

[0101] The image compression device 40 serves to compress the generated high-resolution image recordings D2 and / or the generated preprocessed image recordings D1. The compression can comprise conventional compression using mp4 / jpeg or the like.

[0102] The data storage device 50 is configured to store compressed image recordings as files. From there, the files can be retrieved and / or readable. The data storage device 50 can be or comprise a data storage medium of a computer. Alternatively, the data storage device 50 can be implemented in a server or a cloud computing platform. The video interface 60 serves to visualize the processed image recordings, either the generated preprocessed image recordings D1 or the generated high-resolution image recordings D2. The video interface 60 can comprise at least one monitor. The video interface 60 can be implemented in portable user devices, for example, in a mobile phone or a tablet, or in non-portable devices, in particular in a video processor of an endoscope.

[0103] Fig. 4 shows an endoscope E in which a system 100 for reducing the amount of data to be stored from images of an image acquisition sequence is implemented according to an embodiment of the invention. The endoscope E shown in Fig. 4 has a provision device 10 (here: video camera K of the video endoscope I) and an image processing unit 150, which comprises the remaining elements of the system 100.

[0104] Fig. 5 shows a schematic block diagram illustrating a computer program product 300 according to an embodiment of the third aspect of the present invention. The computer program product 300 comprises executable program code 350 configured to perform the method according to any embodiment of the second aspect of the present invention, in particular as described in the preceding figures.

[0105] Fig. 6 shows a schematic block diagram illustrating a non-transitory computer-readable data storage medium 400 according to an embodiment of the fourth aspect of the present invention. The data storage medium 400 comprises executable program code 450 that, when executed, is configured to perform the method according to any embodiment of the second aspect of the present invention, in particular as described with reference to the preceding figures.

[0106] The non-volatile, computer-readable data storage medium may comprise or consist of any type of computer memory, in particular a semiconductor memory such as a solid-state memory. The data storage medium may also comprise or consist of a CD, a DVD, a Blu-ray disc, a USB memory stick, or the like.

[0107] List of reference symbols

[0108] 10 Provisioning facility

[0109] 20 Burst image processing device

[0110] 30 super-resolution image processing device

[0111] 40 image compression device

[0112] 50 Data storage facility

[0113] 60 video interface

[0114] 100 systems

[0115] 150 image processing unit

[0116] 210 Calculation unit

[0117] 300 computer program product

[0118] 350 program code

[0119] 400 data storage medium

[0120] 450 program code

[0121] E Endoscope

[0122] I Video endoscope

[0123] K Camera

[0124] D0 Image acquisition

[0125] Dl preprocessed image acquisition

[0126] D2 high-resolution image acquisition

[0127] S1..S223

[0128] Procedure steps

Claims

Patent claims:

1. Computer-implemented method for reducing the amount of data to be stored from images (DO) of an image acquisition sequence of a patient, comprising the steps: Providing (S l) one or more image recordings (DO) of an image recording sequence of a patient with a first image resolution; Generating (S2), based on the provided image recordings (DO) of the image recording sequence, corresponding preprocessed image recordings (Dl) of the image recording sequence by means of burst image processing, wherein the preprocessed image recordings (Dl) of the image recording sequence have reduced image noise and / or reduced image artifacts; Generating (S3), for each generated preprocessed image recording (Dl) of the image recording sequence, an image recording (D2) with a second image resolution by means of super-resolution image processing, wherein the second image resolution is higher than the first image resolution; and Compressing (S4) the generated high-resolution image recordings (D2) and / or the generated pre-processed image recordings (Dl) of the image recording sequence to reduce the amount of data to be stored in the image recording sequence.

2. Computer-implemented method according to claim 1, wherein a plurality of underexposed raw image recordings are used to generate (S2) the preprocessed image recordings (Dl) of the image recording sequence.

3. Computer-implemented method according to one of the preceding claims, further comprising storing (S5) the compressed generated high-resolution and / or the compressed generated preprocessed image recordings of the image recording sequence.

4. Computer-implemented method according to claim 3, wherein the generating (S3) comprises reading (S3 1) the stored compressed generated preprocessed image recordings of the image recording sequence.

5. Computer-implemented method according to one of the preceding claims, wherein the generation (S2) of the preprocessed image recordings (Dl) of the image recording sequence by means of burst image processing comprises the following steps: Grouping (S20) the provided image recordings (DO) of the image recording sequence into image recording groups, depending on an image recording rate; Segmenting (S21) each provided image acquisition (DO) of the image acquisition group into image segments; and Combining (S22) corresponding spatially correlated image segments of the different image recordings (DO) of the image recording group to form a pre-processed image recording (Dl).

6. A computer-implemented method according to claim 5, wherein the number of corresponding spatially correlated image segments combined to form a preprocessed image recording (Dl) depends on the brightness of the corresponding image segments.

7. A computer-implemented method according to claim 5 or 6, wherein combining (S22) corresponding spatially correlated image segments comprises: Aligning (S221) the provided image recordings (DO) of the image recording group with each other; Overlaying (S222) the image segments of the mutually aligned image recordings of the image recording group; and Hue mapping (S223) of the superimposed image segments of the aligned images of the image acquisition group.

8. Computer-implemented method according to one of the preceding claims, wherein the generation (S3) of a high-resolution image recording (D2) by means of Super-resolution image processing is performed using a trained artificial intelligence model, where the trained artificial intelligence model corresponds to a convolutional neural network, CNN, or a deep learning architecture, in particular a transformer-based architecture.

9. A system (100) for reducing the amount of data to be stored from image recordings (D0) of an image recording sequence of a patient, comprising: a provision device (10) configured to provide one or more image recordings (D0) in an image recording sequence of a patient with a first image resolution; a burst image processing device (20) configured to generate, based on the provided image recordings (D0) of the image recording sequence, corresponding preprocessed image recordings (D1) of the image recording sequence by means of burst image processing, wherein the preprocessed image recordings (D1) of the image recording sequence have reduced image noise and / or reduced image artifacts;a super-resolution image processing device (30) configured to generate, for each generated preprocessed image recording (D1) of the image recording sequence, an image recording (D2) with a second image resolution by means of super-resolution image processing, wherein the second image resolution is higher than the first image resolution; and an image compression device (40) configured to compress the generated high-resolution image recordings (D2) and / or the generated preprocessed image recordings (D1) of the image recording sequence to reduce the amount of data to be stored in the image recording sequence.

10. The system (100) according to claim 9, wherein the burst image processing device (20) comprises a calculation unit (210) implementing a burst algorithm designed to detect fine physical structures such as veins and nerves in the provided image recordings (D0) of the image recording sequence of the patient. 1 1. System (100) according to one of claims 9 or 10, wherein the provision device (10) is integrated in an endoscope (E) or in an exoscope.

12. System (100) according to one of claims 9 to 11, further comprising a data storage device (50) in which the image recordings of the image recording sequence compressed by the image compression device (40) are stored and can be read out from there.

13. System (100) according to one of claims 9 to 12, further comprising a video interface (60) which is designed to transmit the generated preprocessed image recordings (D1) of the image recording sequence and / or the generated high-resolution image recordings (D2) of the image recording sequence to a display unit (70) of the system (100).

14. A computer program product (300) comprising executable program code (350) which, when executed, is designed to perform the computer-implemented method according to claims 1 to 8.

15. A non-transitory computer-readable data storage medium (400) comprising executable program code (450) which, when executed, is adapted to perform the computer-implemented method according to claims 1 to 8.

Images