An image processing method and computer program product

Abstract

An image processing method and computer program product, images captured by different image capture devices can have different fields of view and / or resolutions. One or more of the images can be aligned based on an image template and a data recovery model using machine learning can be used to predict additional details of the adjusted image and add them to the adjusted image so that the images can have the same field of view or the same resolution.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method and a computer program product. Background Technology

[0002] Images captured by different image capture devices installed in the same environment may have different fields of view and / or different resolutions facing that environment. To use the information contained in these images for environment-related data processing tasks, it may be necessary to align and / or augment the images so that they have the same field of view, the same resolution, or pixel-level correspondence. Conventional methods for achieving this goal may crop images with a larger field of view to fit images with a smaller field of view, thereby (e.g., in at least some cases) producing an output image with a resolution equal to the minimum resolution of the input image. Therefore, systems and methods capable of automatically aligning and / or augmenting cross-modal images to achieve large FOV and / or high resolution may be desirable. Summary of the Invention

[0003] This document describes systems, methods, and apparatuses associated with automatic image alignment and augmentation. An apparatus configured to perform these tasks may include at least one processor configured to acquire images captured by a corresponding image capture device and adjust one or more images in the images based on an image template. The images acquired by the processor may differ from each other with respect to at least one of field of view (FOV) or resolution, and the adjustment based on the image template may align one or more images in the images with respect to at least one of the image resolution or aspect ratio (e.g., one or more images may be adjusted to have the same resolution or aspect ratio as the image template). The at least one processor may also be configured to determine additional details of the one or more adjusted images based on a machine learning (ML) data recovery model and utilize these additional details to augment the one or more adjusted images such that the one or more adjusted images may have the same field of view or the same resolution.

[0004] In the example, the images acquired by the device may include color images captured by a color image sensor, depth images captured by a depth sensor, and / or one or more medical scan images captured by a corresponding medical imaging device. The images may be captured at various (e.g., different) times and / or may have different fields of view (e.g., they may partially overlap). In the example, the image template used for aligning the images may be predefined or determined based on the images acquired by the device. The parametric model may be determined based on corresponding internal or external parameters of the image capture device and may include a corresponding projection matrix associated with the image capture device. The projection matrix can be used (e.g., during the image adjustment process) to project the image onto the image template.

[0005] In the example, the ML data recovery model can be implemented using at least one convolutional neural network (CNN) and can be trained on multiple image sets, each of which may include at least a first image captured by a first image capture device and a second image captured by a second image capture device. The first and second images can be aligned to conform to a training image template, and during the training of the ML data recovery model (e.g., a neural network), the ML data recovery model can be configured to predict missing details in the first image based on the first and second images. Attached Figure Description

[0006] The examples disclosed herein can be understood in more detail from the following description, which is given by way of example in conjunction with the accompanying drawings.

[0007] Figure 1 This is a diagram illustrating an example of an environment in which the techniques disclosed herein can be implemented.

[0008] Figure 2 This is a diagram illustrating example techniques for aligning and expanding cross-modal images according to one or more embodiments of the present disclosure.

[0009] Figure 3 This is a diagram illustrating an example of image alignment according to one or more embodiments of the present disclosure.

[0010] Figure 4 This is a diagram illustrating an example of missing details in a predicted image according to one or more embodiments of the present disclosure.

[0011] Figure 5 This is a flowchart illustrating example operations that can be associated with training a neural network to perform a task described according to one or more embodiments of this disclosure.

[0012] Figure 6 This is a flowchart illustrating example operations that can be associated with aligning and expanding cross-modal images according to one or more embodiments of the present disclosure.

[0013] Figure 7 This is a block diagram illustrating example components of a device that can be configured to perform tasks described according to one or more embodiments of the present disclosure. Detailed Implementation

[0014] The present disclosure is illustrated by way of example and not limitation in the accompanying drawings. A detailed description of illustrative embodiments will now be described with reference to the various drawings. Although detailed examples of possible implementations are provided in this specification, it should be noted that these details are intended to be illustrative and in no way limiting the scope of this application.

[0015] Figure 1An example environment 100 is illustrated in which the techniques disclosed herein can be implemented. Environment 100 can be Figure 1 The illustration is shown as part of a medical environment (e.g., a medical scanning room or operating room used for magnetic resonance imaging (MRI), X-ray, computed tomography (CT), etc.), but those skilled in the art will understand that the techniques disclosed herein can also be applied to other types of environments, including, for example, gaming environments, rehabilitation facilities, etc. Figure 1 As shown, environment 100 may be equipped with one or more image capture devices 102, which may be mounted at different locations within the environment and configured to capture images (e.g., including video) of the environment (e.g., including patient 104 and / or medical procedures performed within the environment). Image capture devices 102 may include one or more sensors, such as one or more color image sensors (e.g., color cameras), one or more depth sensors, one or more thermal sensors (e.g., infrared (FIR) or near-infrared (NIR) sensors), one or more radar sensors, one or more medical imaging devices (e.g., one or more CT, MRI, or X-ray scanners), etc. Depending on the type of image capture device mounted in environment 100, the images described herein may include, for example, one or more color images captured by a color image sensor, one or more depth images captured by a depth sensor, one or more thermal images captured by a thermal sensor, one or more medical scan images captured by a medical scanner, etc. Images may be captured by the different devices described herein at different times and / or from different viewpoints during a time period (e.g., some of which may overlap). Therefore, images can differ from one another with respect to at least one of (e.g., environment 100) field of view, resolution, size, or aspect ratio (e.g., 4:3, 16:9, etc.). Image capturing device 102 can be communicatively coupled (e.g., via communication network 106) to processing device 108 (e.g., computing device) and / or other devices in environment 100, and can be configured to send images captured by image capturing device 102 to processing device 108 and / or other devices. In the example, one or more image capturing devices 102 may themselves be equipped with a processing or functional unit (e.g., one or more processors) that can be configured to process the images captured by image capturing device 102.

[0016] Since the images captured by image capture device 102 can differ from each other in terms of field of view (FOV), resolution, size, and / or aspect ratio, processing device 108 (or a processing unit of image capture device 102) can be configured to adjust or modulate (e.g., align and / or expand) one or more images such that the images have the same FOV (e.g., in terms of the people and objects covered by the FOV), resolution, size, aspect ratio, etc. (e.g., it is not necessary to crop an image with a higher resolution to align with an image with a lower resolution or smaller FOV). The adjusted or modulated images can be used to facilitate various operations or tasks in environment 100. These operations or tasks may include, for example, automating medical procedures performed in environment 100 by recognizing, based on the adjusted or modulated images, the people (e.g., patient 104) and / or devices (e.g., surgical robot) involved in the medical procedure and their respective positions, such that navigation instructions can be automatically generated to move one or more devices toward the person (e.g., toward patient 104). As another example, the adjusted image can be used to reconstruct the patient's surface (e.g., a 3D mesh) and / or (e.g., the patient's organs) anatomical model, which can then be used for patient localization, image overlay, image analysis, and / or other medical procedures or applications.

[0017] Figure 2 Examples of aligned and / or expanded cross-modal images according to one or more embodiments of the present disclosure are illustrated. As shown, images can be obtained by various image capturing devices (e.g., Figure 1 The device 102) captures multiple images. The images may include, for example, a first image 202 captured by a color image sensor and a second image 204 captured by a depth sensor, and the images may differ from each other with respect to at least one of the field of view (FOV) (e.g., the size of the FOV) or resolution. For example, the first image 202 may be compared with the environment (e.g., Figure 1The first image 204 can be associated with a smaller FOV and lower resolution than the second image 202 (e.g., the corresponding FOVs of the first and second images may differ, but they may partially overlap). To adjust the images so that they all have the same FOV and / or resolution, the images can be aligned according to an image template (e.g., via image alignment process 206), which can be predefined or determined, for example, based on the acquired images (e.g., the template can be determined based on the image with the largest field of view). The image template can have a specific resolution and / or aspect ratio (e.g., aspect ratio, such as 4:3, 16:9, etc.), and it can be the same as or different from some images (e.g., the template can have the same resolution or aspect ratio as the second image 204 but different from the first image 202). Therefore, one or more images in the image set (e.g., images that do not conform to the image template, such as image 202) can be adjusted based on the image template to align the images with the image template (e.g., to adapt the images to the image template). One or more adjusted images can be further augmented (e.g., via image augmentation process 208) using additional details that may be missing from the original image, such that the resulting image has the same FOV (e.g., in terms of people and objects covered by the FOV) and / or resolution as the remaining images. Additional details can be determined, for example, using a machine learning (ML) data restoration model trained to predict additional details based on information learned from other images that may have a larger FOV and / or higher resolution (e.g., during the training of the ML data restoration model).

[0018] It should be noted that it may not be necessary to adjust (e.g., in 206) and / or expand (e.g., in 208) all images captured by the image capturing device. If some images (e.g.) Figure 2 If image 204) already conforms to the image template described herein and / or has a target FOV and / or resolution, then image alignment process 206 and / or image augmentation process 208 can be skipped for these images. It should also be noted that the terms "machine learning model," "machine learning model," "artificial intelligence model," and "neural network model" are used interchangeably herein.

[0019] Figure 3 Image alignment according to one or more embodiments of the present disclosure is illustrated (e.g., Figure 2An example of the image alignment process 206 is provided. As shown, the first image 302 and the second image 304, which can be captured by corresponding image capturing devices, may have different resolutions and / or aspect ratios. In 306, the images can be adjusted based on the image template 308 so that the images can be aligned with each other and / or aligned with the image template 308 (e.g., in terms of image resolution and / or aspect ratio). For example, if one or both of the first image 302 and the second image 304 have a different resolution or aspect ratio than the image template 308, then in 306, the image can be adjusted (e.g., projected) to fit the image template 308 (e.g., to have the same resolution or aspect ratio as the image template), as shown in the adjusted image 310 (e.g., if image 304 already has the same resolution and aspect ratio as the image template 308, then it can remain the same). Image template 308 can be determined based on first image 302 or second image 304 (e.g., if one of these images has a desired FOV, resolution, and / or aspect ratio), and the projection of the image onto image template 308 can be implemented based on a corresponding parametric model (e.g., parametric mathematical relationships between 3D points and their 2D projections) of the image capturing device used to capture the first and second images. For example, the parametric model of the image capturing device can be determined based on corresponding internal or external parameters of the image capturing device that can be acquired during installation of the image capturing device, and the parametric model can be used to determine the corresponding projection matrix for projecting the first image 302 and / or the second image 304 onto image template 308.

[0020] like Figure 3 For example, the adjusted image 310 (e.g., projected from image 302 based on image template 308) may lack details necessary to conform to the target FOV and / or resolution (e.g., the FOV and / or resolution of image 304) (e.g., as shown by blank areas in image 310). These details can be estimated using an ML data restoration model trained to predict details based on information learned from other images that may have the target FOV and / or resolution (e.g., during the training of the ML data restoration model).

[0021] Figure 4 Techniques for predicting missing details in an image, according to one or more embodiments of this disclosure, are illustrated. For example... Figure 4 As shown, the image 402 captured by the first image capturing device (e.g., Figure 3The adjusted image 310 may lack details used to achieve the target FOV and / or resolution. These details can be recovered (e.g., predicted or estimated) based on image 404 and an ML model 406 trained to predict details (e.g., a data recovery model). Image 404 may be captured by a different image capture device and may have a different FOV (e.g., a larger FOV) and / or a different resolution (e.g., a higher resolution) than image 402. For example, image 402 may be a color image (e.g., a red-green-blue or RGB image) captured by a color image sensor (e.g., a camera), while image 404 may be a depth image captured by a depth image sensor. The ML model 406 may be implemented and / or learned using an artificial neural network, which may include one or more feature extraction modules 406a and / or one or more detail prediction modules 406b. Feature extraction module 406a can be configured to extract features f1 and f2 from images 402 and 404, respectively, and detail prediction module 406b can be configured to predict missing details in at least one of images 402 or 404 based on the extracted features f1 and f2, and supplement at least one of images 402 or 404 with additional details to obtain augmented image 408. Although the example only shows the generation of an augmented image 408 for image 402, those skilled in the art will understand that similar augmentation can be performed on other input images (e.g., image 404) if they also lack details for achieving the target FOV and / or resolution.

[0022] Either or both of the feature extraction module 406a and the detail prediction module 406b can be implemented using at least one convolutional neural network (CNN), which may include multiple layers, such as one or more convolutional layers, one or more pooling layers, and / or one or more fully connected layers. Each convolutional layer may include multiple convolutional kernels or filters configured to extract features from the input image through a series of convolutional operations, followed by batch normalization and / or linear (or nonlinear) activation (e.g., Corrected Linear Unit (ReLU) activation). The features extracted by the convolutional layers may be downsampled by pooling layers and / or fully connected layers to reduce feature redundancy and / or size, thereby obtaining a representation of the downsampled features (e.g., in the form of feature vectors or feature maps). The neural network may also include one or more non-pooling layers and one or more transposed convolutional layers, which may be configured to upsample and deconvolve the features extracted by the above operations. As a result of upsampling and deconvolution, a dense feature representation of the input image (e.g., a dense feature map) can be derived, which can then be used to estimate missing details of the input image.

[0023] The ML model 406 can be trained using a training dataset comprising multiple sets of images. Each of the multiple image sets may include training images captured by a corresponding image capture device (e.g., a first training image and a second training image), and the training images may be aligned to conform to a training image template (e.g., in terms of resolution and / or aspect ratio). During training iterations, the ML model 406 may be configured to receive the aligned sets of training images, extract features from the training images included in the sets, and predict missing details of one or more training images based on the extracted features, such that the resulting images achieve the desired FOV and / or resolution. The predictions can then be compared to a gold standard (e.g., an actual image with the desired FOV and / or resolution) to calculate a loss based on the predictions, which can be used to adjust the parameters of the ML model (e.g., the weights of the neural network used to implement the ML model) to minimize the loss.

[0024] Figure 5 Example operations that can be associated with training a neural network (e.g., a neural network for implementing the ML data recovery model described herein) to perform one or more tasks described herein are illustrated. As shown, the training operation may include: at 502 initializing the execution parameters of the neural network (e.g., weights associated with the individual layers of the neural network), for example by sampling from a probability distribution or by replicating the parameters of another neural network with a similar structure. Training operation 500 may also include: at 504 processing the input (e.g., a training image) using the currently assigned parameters of the neural network; and at 506 making a prediction of the desired outcome (e.g., an augmented image with additional details). At 508, the prediction may be compared to a gold standard to determine, for example, a loss associated with the prediction based on a loss function (such as the mean squared error between the prediction and the gold standard, L1 norm, L2 norm, etc.). At 510, this loss may be used to determine whether one or more training termination criteria are met. For example, a training termination criterion may be determined to be met if the loss is below a threshold or if the change in loss between two training iterations is below a threshold. If the termination criterion is satisfied at 510, training can end; otherwise, at 512, for example, before training returns to 506, the currently assigned network parameters can be adjusted by backpropagating the gradient descent of the loss function through the network.

[0025] For the sake of simplicity, training operations are depicted and described in a specific order throughout this document. However, it should be understood that training operations can occur in various orders, simultaneously, and / or with other operations not presented or described herein. Furthermore, it should be noted that not all operations that may be included in the training methods are depicted and described herein, and not all exemplified operations need to be performed.

[0026] Figure 6 An example operation 600 is illustrated that can be associated with aligning and augmenting one or more images to achieve a common FOV and / or resolution of the images. As shown, operation 600 may include: at 602 obtaining images captured by respective image capture devices, wherein the images may differ from each other with respect to at least one of FOV or resolution. Operation 600 may also include: at 604 adjusting one or more images based on an image template; and at 606 determining additional details of one or more adjusted images based on a machine learning (ML) data restoration model. In the example, the adjustment at 604 may align one or more images with respect to at least one of the image resolution or aspect ratio (e.g., based on the resolution or aspect ratio of the image template), and the ML data restoration model used at 606 may be trained to predict additional details based on information learned from other images that may have a target FOV and / or resolution (e.g., during the training of the ML data restoration model). Once determined, at 608, additional details can be added to one or more adjusted images (e.g., details can be used to supplement one or more adjusted images) so that all images obtained at 602 can have the same FOV (e.g., in terms of people and objects covered by the FOV) and / or resolution.

[0027] The systems, methods, and / or apparatuses described herein may be implemented using one or more processors, one or more storage devices, and / or other suitable auxiliary devices (such as display devices, communication devices, input / output devices, etc.). Figure 7 This is a block diagram illustrating an example device 700 that can be configured to perform the image alignment and / or augmentation tasks described herein. As shown, device 700 may include a processor (e.g., one or more processors) 702, which may be a central processing unit (CPU), graphics processing unit (GPU), microcontroller, reduced instruction set computer (RISC) processor, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), or any other circuitry or processor capable of performing the functions described herein. Device 700 may also include communication circuitry 704, memory 706, mass storage device 708, input device 710, and / or communication link 712 (e.g., communication bus) through which one or more components shown in the figure exchange information.

[0028] Communication circuitry 704 can be configured to send and receive information using one or more communication protocols (e.g., TCP / IP) and one or more communication networks, including local area networks (LANs), wide area networks (WANs), the Internet, and wireless data networks (e.g., Wi-Fi, 3G, 4G / LTE, or 5G networks). Memory 706 may include a storage medium (e.g., a non-transitory storage medium) configured to store machine-readable instructions that, when executed, cause processor 702 to perform one or more functions described herein. Examples of machine-readable media may include volatile or non-volatile memory, including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, etc.). Mass storage device 708 may include one or more disks, such as one or more internal hard disks, one or more removable disks, one or more magneto-optical disks, one or more CD-ROMs or DVD-ROMs, etc., on which instructions and / or data may be stored for operation of processor 702. The input device 710 may include a keyboard, mouse, voice-controlled input device, touch-sensitive input device (e.g., touch screen), etc., for receiving user input from the device 700.

[0029] It should be noted that device 700 can operate as a standalone device or can be connected to other computing devices (e.g., networked or clustered) to perform the functions described herein. And even in Figure 7 Only one example of each component is shown in the figure, and those skilled in the art will understand that the device 700 may include multiple instances of one or more components shown in the figure.

[0030] Although this disclosure has been described according to certain embodiments and generally associated methods, changes and variations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above description of exemplary embodiments does not limit this disclosure. Other changes, substitutions, and modifications are possible without departing from the spirit and scope of this disclosure. Furthermore, unless specifically stated otherwise, discussions using terms such as “analyze,” “determine,” “enable,” “identify,” and “modify” refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data representing physical (e.g., electronic) quantities within the registers and memories of the computer system into other data representing physical quantities within the computer system's memory or other such information storage, transmission, or display devices.

[0031] It should be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will become apparent to those skilled in the art upon reading and understanding the above description. Therefore, the scope of this disclosure should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. An image processing method, comprising: A first image captured by a first image capturing device and a second image captured by a second image capturing device are obtained, wherein the first image and the second image are different from each other with respect to at least one of the field of view or resolution; Adjust at least one of the first image or the second image based on the image template; A machine learning-based ML data restoration model is used to predict additional details in the first image, wherein the ML data restoration model is trained to predict additional details in the first image based on a second image; and The first image is supplemented with the additional details so that the first image has the same field of view and the same resolution as the second image.

2. The method according to claim 1, wherein, The first image is a color image captured by a color image sensor, and the second image is a depth image captured by a depth sensor.

3. The method according to claim 1, wherein, The first image is a first medical scan image captured by a first medical imaging device, and the second image is a second medical scan image captured by a second medical imaging device.

4. The method according to claim 1, wherein, The first image and the second image were captured at different times or have overlapping fields of view.

5. The method according to claim 1, wherein at least one of the first image or the second image is adjusted based on the image template such that the first image and the second image have the same resolution or the same aspect ratio as the image template.

6. The method according to claim 1, further comprising: The image template is determined based on the first image and the second image.

7. The method according to claim 6, wherein, The parameter model associated with the first image capturing device is determined based on the internal or external parameters of the first image capturing device, and the parameter model associated with the second image capturing device is determined based on the internal or external parameters of the second image capturing device.

8. The method according to claim 7, wherein, The parameter model associated with the first image capture device includes a projection matrix associated with the first image capture device, the parameter model associated with the second image capture device includes a projection matrix associated with the second image capture device, and wherein adjusting at least one of the first image or the second image based on the image template includes: The first image is projected onto the image template based on the projection matrix associated with the first image capturing device; and / or The second image is projected onto the image template based on the projection matrix associated with the second image capturing device.

9. The method according to claim 1, wherein, The ML data recovery model is trained on multiple image sets, each of which includes at least a first training image captured by a first image capture device and a second training image captured by a second image capture device, the first training image and the second training image conforming to a training image template, and wherein, during the training of the ML data recovery model, the ML data recovery model is configured to predict missing details of the first training image based on the first training image and the second training image.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images