Object detection in image

Abstract

A method of object detection in a first image taken under suboptimal lighting conditions. The first image is input into a pre-processing neural network trained with training images taken under normalized lighting conditions, where the pre-processing neural network is configured to output a prediction for generating a second image containing content of the first image under normalized lighting conditions. Object detection is then performed on the second image.

Description

TECHNICAL FIELD

[0001] The present invention relates to the field of object detection in images. In particular, the present invention relates to body keypoint detection in images. BACKGROUND

[0002] Object detection algorithms are used to identify and locate objects within an image or video. These algorithms aim to automate the object recognition process, thereby enabling computers to interpret visual information. Object detection algorithms can be used for a variety of applications.

[0003] For example, the use of automated check preparation and object detection to improve clinical workflow has shown promise in increasing the efficiency of medical imaging departments. External camera devices have proven to be suitable tools for this purpose, allowing anatomical key points to be located accurately by object detection algorithms to automatically define scan positions. In most current implementations, a dedicated keypoint detection neural network is used to analyze the camera images.

[0004] However, it has been found that keypoint detection and, more generally, object detection for determining scan positions does not always provide the performance expected of the object detection algorithms used. There is therefore a need to improve object detection. SUMMARY

[0005] The invention is defined by the claims.

[0006] According to an example of an aspect of the invention, there is provided a computer- implemented method of object detection in a first image taken under suboptimal lighting conditions, the method comprising: inputting the first image into a pre-processing neural network trained with training images taken under normalized lighting conditions, wherein the pre-processing neural network is configured to output a prediction for generating a second image containing the content of the first image under normalized lighting conditions; and performing object detection on the second image, wherein: when performing object detection on an image taken under suboptimal lighting conditions, the resulting object detection confidence score does not satisfy a confidence threshold, and when performing object detection on an image taken under normalized lighting conditions, the resulting object detection confidence score satisfies the confidence threshold.

[0007] Object detection includes determining one or more points and / or regions indicative of the location of an object in the first image. For example, object detection can include determining key points, bounding boxes, heat maps, segmentation masks, etc.

[0008] Typically, neural networks have been used for object detection. However, it has been recognised that the networks are typically trained with images under optimal lighting conditions only. Therefore, when an image is input which is taken under sub-optimal lighting conditions, the object detection output can not be as good as the network expects.

[0009] It is therefore proposed to normalise the lighting conditions of the first image before performing object detection. This means that the second image (lighting normalised) will have improved lighting conditions compared to the original first image before performing object detection.

[0010] The neural network is trained to improve the lighting conditions of the first image.

[0011] Object detection typically outputs a confidence score for each detected object. The confidence score indicates the likelihood that the detected object is correct. Improving the lighting conditions of the taken image improves the confidence score.

[0012] Performing object detection means applying an object detection algorithm to the second image.

[0013] The pre-processing neural network can be a lighting normalisation network trained to output the second image.

[0014] The lighting normalisation network can be further trained to output a performance score for the second image.

[0015] The lighting normalisation network can be trained with performance scores for training images (under sub-optimal lighting conditions) and reference images (under optimal lighting conditions). Therefore, the lighting normalisation network can predict a performance score for the second image.

[0016] User feedback can be generated if the performance score is below a performance threshold.

[0017] The pre-processing neural network can be a parameter normalisation network trained to output one or more normalised lighting parameters which, when applied to the first image, normalise the lighting conditions of the first image.

[0018] The method can further comprise applying the normalised lighting parameters to the first image to generate the second image.

[0019] The parameter normalisation network can be trained to output one or more of: a normalised image brightness, a normalised image contrast, and a normalised gamma correction.

[0020] The method can further comprise generating user feedback in response to one or more of the normalised lighting parameters exceeding a first predetermined threshold.

[0021] The normalization lighting parameters that have to be applied will result in a large difference in the perceived lighting conditions between the first image and the second image, which can indicate that the lighting conditions of the first image are particularly poor. User feedback can provide an indication of the lighting conditions of the first image. The user can then decide to adjust the lighting of e.g. a room before obtaining a new first image.

[0022] The method can further comprise determining a difference between the first image and the second image, and generating user feedback in response to the difference exceeding a second predetermined threshold.

[0023] A large difference between the first image and the second image can indicate that the lighting conditions of the first image are particularly poor. User feedback can provide an indication of the lighting conditions of the first image. The user can then decide to adjust the lighting of e.g. a room before obtaining a new first image.

[0024] Performing object detection can comprise applying a body keypoint algorithm configured to output key points of a human body in the second image.

[0025] The invention also provides a computer program carrier comprising computer program code which, when executed on a computer, causes the computer to perform all the steps according to the preceding method.

[0026] The invention also provides a system for object detection in a first image taken under suboptimal lighting conditions, the system comprising a processor configured to: input the first image into a pre-processing neural network trained on training images taken under normalized lighting conditions, wherein the pre-processing neural network is configured to output a prediction for generating a second image containing the content of the first image under normalized lighting conditions; and perform object detection on the second image, wherein: when performing object detection on an image taken under suboptimal lighting conditions, the resulting object detection confidence score does not satisfy a confidence threshold, and when performing object detection on an image taken under normalized lighting conditions, the resulting object detection confidence score satisfies the confidence threshold.

[0027] The pre-processing neural network can be an illumination normalization network trained to output the second image.

[0028] The illumination normalization network can be further trained to output a performance score of the second image.

[0029] The pre-processing neural network can be a parameter normalization network trained to output one or more normalized lighting parameters that, when applied to the first image, normalize the lighting conditions of the first image.

[0030] The processor can be further configured to apply the normalized lighting parameters to the first image to generate the second image.

[0031] The parameter normalization network can be trained to output one or more of: a normalized image brightness, a normalized image contrast, and a normalized gamma correction.

[0032] The processor can be further configured to generate user feedback in response to one or more of the normalized lighting parameters exceeding a first predetermined threshold.

[0033] The processor can be further configured to determine a difference between the first image and the second image, and generate user feedback in response to the difference exceeding a second predetermined threshold.

[0034] The processor can be configured to perform object detection by applying a body keypoint algorithm configured to output key points of a human body in the second image.

[0035] These and other aspects of the application will become apparent and elucidated from the embodiments described hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0036] For a better understanding of the application, and to show how it can be carried into effect, there will now be described by way of example only, with reference to the accompanying drawings in which: Figure 1 A method of object detection using an illumination normalization neural network is shown; Figure 2 A method of object detection using a parameter normalization neural network is shown; Figure 3 Results of object detection using a pre-processing network are shown; and Figure 4 A method of object detection in a first image taken under sub-optimal lighting conditions is shown. DETAILED DESCRIPTION

[0037] The application will be described with reference to the drawings.

[0038] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of apparatuses, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the present application. These and other features, aspects, and advantages of the apparatuses, systems and methods of the present application will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the drawings are only schematic and are not drawn on scale. It should also be understood that the same reference numerals are used throughout the drawings for like or similar items.

[0039] The present application provides a method of object detection in a first image taken under suboptimal lighting conditions. The first image is input into a pre-processing neural network trained with training images taken under normalized lighting conditions, wherein the pre-processing neural network is configured to output a prediction for generating a second image containing the content of the first image under normalized lighting conditions. Object detection is then performed on the second image.

[0040] Neural networks for body keypoint detection are typically trained on large annotated datasets. It has been recognized that most available datasets represent images of various people under generally favorable lighting conditions. As a result, the resulting keypoint detection neural network exhibits best performance under such lighting conditions. When applied to other scenarios (e.g. low-light dark images, high noise conditions, rolling shutter streak artifacts from flickering light sources, strong local shadows, etc.), the performance of keypoint detection degrades substantially.

[0041] Without additional measures to prevent this, users of a camera-based workflow support system will not receive any feedback about this performance degradation, which can lead to frustration with the system’s false predictions. Classical methods (e.g. measurements of noise / signal-to-noise ratio, camera exposure, gain, etc.) can be used to detect unfavorable image characteristics, but can be difficult to analyze in the case of multiple simultaneous impairments and / or complex influences (e.g. spatially varying lighting (e.g. spotlights)).

[0042] To avoid such complex situations, an additional image pre-processing step is proposed to ensure optimal image characteristics for object detection. The degree of image modification performed in this pre-processing step can be used to provide feedback to the user. In the case of severe modifications, the user can be notified and asked to improve the lighting conditions of the scene.

[0043] The pre-processing step can be used by a camera-based workflow support system to improve object detection. For example, the system can comprise an optical camera mounted on the ceiling that provides a continuous data stream of an inspection table of objects. Alternatively, a rack or bore mounted camera with sufficient field of view of the objects can be used. The system can further comprise a dedicated image pre-processing module for optimizing image characteristics and a processor configured to input images from the camera to a body keypoint detection neural network.

[0044] After installation of the camera-based workflow support system at a clinical site, the pre-processing module is applied to all incoming images from the data stream of the camera.

[0045] In a first implementation, the pre-processing step can be implemented as a dedicated illumination normalization neural network for directly correcting the lighting conditions in the incoming raw camera images.

[0046] Figure 1 An object detection method 108 using an illumination normalization neural network 104 is shown. A raw camera image 102 is input into the illumination normalization neural network 104. The output of the neural network 104 consists of a corrected image 106 with the same image content as the raw camera image 102 but improved lighting conditions.

[0047] The illumination normalization neural network 104 is trained to provide normalized lighting conditions of the raw camera image 102. The corrected image 106 is further processed by an object detection network 108 to localize, for example, body key points 110 to be used for various workflow support applications.

[0048] For the illumination normalization network 104, any architecture for image-to-image translation described in the literature can be used (e.g., U-Net, HRNet, etc.). A simple way to create a suitable training dataset is to take a series of camera images with good lighting conditions (i.e., suitable for a body key point detection neural network) and retrospectively simulate various bad lighting conditions. Examples of such image modifications include variations in image brightness and / or image contrast, injection of artificial noise, and simulation of additional diffuse or directional light sources. Then, given the modified images as input, the network is trained to produce the original images with good lighting conditions.

[0049] Figure 2 An object detection method 210 using a parameter normalization neural network 204 is shown. In this second implementation, an illumination normalization module 206 and a parameter normalization neural network 204 are used for pre-processing the raw images 202. The illumination normalization module 202 consists of standard image transformations (e.g., image brightness, image contrast, gamma correction, etc.) applied to the raw images 202 to generate corrected images 208. The parameters of this module are estimated by a dedicated parameter normalization neural network 204. The resulting corrected images 208 can then be input into an object detection algorithm 210 to localize, for example, body key points 212.

[0050] For the parametric normalization neural network, any architecture for image classification / regression can be used (e.g. ResNet, EfficientNet, etc.). The training of the network can be achieved by using similar image modifications as described for the first implementation and using the modification parameters (e.g. brightness change, contrast change, etc.) as ground truth.

[0051] In case of sub-optimal lighting conditions, it can be useful to provide feedback to the operator / user, who can then decide to adjust the lighting. For the first implementation, the difference between the corrected image and the original camera image can be analyzed (e.g. a large uniform difference can indicate that a severe brightness adjustment was performed due to the original camera image being too dark / bright). For the second implementation, the user feedback can be based on the range of predicted parameters output by the parametric normalization network.

[0052] This network-based image correction pre-processing step can greatly improve the performance of the subsequent object detection network. However, in very poor (i.e. sub-optimal) lighting conditions, a performance drop in object detection is inevitable. The lighting normalization network can be extended to quantify this performance drop. To implement this feature, the training can be performed by simulating various deviating lighting conditions as described above for the first implementation. Since the reference image (with reference lighting) is also available in the training dataset (i.e. the training images), the reference performance drop can be quantified by applying the object detection network to each image pair and computing e.g. the mean squared error between the two object detection outputs.

[0053] The architecture of the lighting normalization network can be modified e.g. by inserting an additional output heat map and trained to predict the observed performance drop of the body keypoint detection network.

[0054] If the predicted performance score exceeds a predefined performance specification, visual and / or audio feedback can be provided to the user.

[0055] In an embodiment, the pre-processing neural network(s) and the object detection neural network(s) can be jointly trained using the same training images. To this end, the weights of the body keypoint detection network can be kept fixed and the resulting object localization error is used as loss function for the pre-processing neural network.

[0056] Figure 3 Results of object detection using the pre-processing network are shown. In this case, the first image 302 is corrected using the lighting normalization neural network. In this example, the reference image 310 is obtained from a magnetic resonance imaging (MRI) exam preparation with good lighting conditions. The reference heat map 312 output by the body keypoint network for the reference image 310 shows good performance.

[0057] The reference image 310 is modified to simulate poor lighting conditions (e.g. very low luminance / contrast, etc.). The heat map 304 output by the body keypoint network for the modified image 310 shows poor performance due to the (simulated) poor / sub-optimal lighting conditions in the modified image.

[0058] The modified image 302 is then fed to the lighting normalisation network, resulting in a corrected image 306. The heat map 308 output by the body keypoint detection network for the corrected image 306 shows an improvement in performance compared to the modified image 302. This demonstrates that the correction of the lighting conditions by the pre-processing network greatly improves object detection.

[0059] Figure 4 A method of object detection in a first image taken under sub-optimal lighting conditions is shown. The method comprises inputting 402 the first image into a pre-processing neural network trained on training images taken under normalised lighting conditions. The neural network is configured to output a prediction for generating a second image containing the content of the first image under normalised lighting conditions. Object detection is then performed 404 on the second image (i.e. the second image is input into an object detection neural network trained to locate objects (e.g. body keypoints, etc.) in the second image).

[0060] When performing object detection on an image taken under sub-optimal (i.e. poor) lighting conditions, the resulting object detection confidence score does not satisfy a confidence threshold.

[0061] When performing object detection on an image taken under normalised lighting conditions, the resulting object detection confidence score satisfies a confidence threshold.

[0062] The methods and systems described herein are particularly advantageous when applied to any object detection method (e.g. body keypoint detection, object segmentation, etc.) that is sensitive to lighting conditions.

[0063] Any of the methods described herein can be computer implemented, where computer implemented means that the steps of the method are performed by one or more computers, and where a computer is defined as a device adapted to process data according to specified instructions. The computer can be adapted to process data according to instructions specified by a software program.

[0064] A person of skill in the art would readily recognize that a processor for performing any of the methods described herein can be developed from the embodiments of the flowcharts. Each of the steps of the flowcharts can therefore represent a different action performed by the processor, and can be performed by a corresponding module of the processing processor.

[0065] One or more steps of any of the methods described herein can be performed by one or more processors. A processor comprises an electronic circuit adapted to process data.

[0066] As noted above, the system utilizes a processor to perform data processing. The processor can be implemented in numerous ways, with software and / or hardware, to perform the various functions required. The processor typically employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. The processor can be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

[0067] Examples of circuitry that can be employed in various embodiments of the disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0068] In various implementations, the processor can be associated with one or more storage media such as volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM. The storage media can be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform the required functions. Various storage media can be fixed within a processor or controller or can be transportable, such that the one or more programs stored thereon can be loaded into a processor.

[0069] A single processor or other unit can implement the functionality of several items recited in the claims.

[0070] A computer program carrier can comprise a computer memory (e.g., random access memory), a computer storage (e.g., hard disk drive, solid-state drive, etc.). The computer program carrier can be a bitstream carried by a carrier wave. A processor can be arranged to read the bitstream and thereby implement the computer program.

[0071] Computer program code can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state storage medium supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0072] Variations of the disclosed embodiments can become apparent to those of ordinary skill in the art upon reading the foregoing description of the disclosed embodiments with reference to the drawings. In the claims, the word “comprising” does not exclude other elements or steps, and the words “a” or “an” do not exclude a plurality. In the claims, the singular encompasses the plural unless otherwise indicated. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

[0073] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0074] If the term “adapted” is used in the claims and / or specification it should be taken as equivalent in meaning to the term “configured.”

[0075] Any reference signs in the claims should not be construed as limiting the scope.

[0076] Any of the methods described herein do not include the method for performing the mental act.

Claims

1. A computer-implemented method of object detection in a first image taken under sub-optimal lighting conditions, the method comprising: inputting (402) the first image into a pre-processing neural network trained with training images taken under normalized lighting conditions, wherein the pre-processing neural network is a parametric normalization network trained to output one or more normalization lighting parameters that, when applied to the first image, normalize the lighting conditions of the first image, and wherein the normalization lighting parameters are applied to the first image, and wherein the pre-processing neural network is configured to output a prediction for generating a second image containing the content of the first image under normalized lighting conditions; and performing (404) object detection on the second image, wherein: when performing object detection on an image taken under sub-optimal lighting conditions, the resulting object detection confidence score does not satisfy a confidence threshold, and when performing object detection on an image taken under normalized lighting conditions, the resulting object detection confidence score satisfies the confidence threshold.

2. The method of claim 1, wherein, the pre-processing neural network is an illumination normalization network trained to output the second image.

3. The method of claim 2, wherein, the illumination normalization network is further trained to output a performance score for the second image.

4. The method of claim 1, wherein, the pre-processing neural network and the object detection neural network have been jointly trained using the same training images.

5. The method of claims 1 to 4, wherein, the parametric normalization network is trained to output one or more of: a normalized image brightness; a normalized image contrast; and a normalized gamma correction.

6. The method of claims 1-5, further comprising generating user feedback in response to one or more of the normalization lighting parameters exceeding a first predetermined threshold.

7. The method of any of claims 1-6, further comprising: determining a difference between the first image and the second image; and generating user feedback in response to the difference exceeding a second predetermined threshold.

8. The method of any one of claims 1 to 7, wherein, performing object detection comprises applying a body keypoint algorithm configured to output key points of a human body in the second image.

9. A computer program carrier comprising computer program code which, when run on a computer, causes the computer to perform all the steps of any of claims 1-8.

10. A system for object detection in a first image taken under sub-optimal lighting conditions, the system comprising a processor configured to: input (402) the first image into a pre-processing neural network trained with training images taken under normalized lighting conditions, wherein, the pre-processing neural network is a parametric normalization network trained to output one or more normalization lighting parameters that, when applied to the first image, normalize the lighting conditions of the first image, and wherein the normalization lighting parameters are applied to the first image, and wherein the pre-processing neural network is configured to output a prediction for generating a second image containing the content of the first image under normalized lighting conditions; and wherein the pre-processing neural network is configured to output a prediction for generating a second image containing content of the first image under normalized lighting conditions; and performing (404) object detection on the second image, wherein: when performing object detection on an image taken under sub-optimal lighting conditions, resulting object detection confidence scores do not satisfy a confidence threshold, and when performing object detection on an image taken under normalized lighting conditions, resulting object detection confidence scores satisfy the confidence threshold.

11. The system of claim 10, wherein, the pre-processing neural network is an illumination normalization network trained to output the second image.

12. The system of claim 11, wherein, the illumination normalization network is further trained to output a performance score for the second image.

13. The method of claim 10, wherein, the pre-processing neural network is a parameter normalization network trained to output one or more normalized lighting parameters that, when applied to the first image, normalize lighting conditions of the first image, and the processor is further configured to apply the normalized lighting parameters to the first image to generate the second image.

14. The system of claim 13, wherein, the parameter normalization network is trained to output one or more of: a normalized image brightness; a normalized image contrast; and a normalized gamma correction.

15. The system of claim 13 or 14, wherein, the processor is further configured to generate user feedback in response to one or more of the normalized lighting parameters exceeding a first predetermined threshold.

16. The system of claim 13 or 14, wherein, the processor is further configured to determine a difference between the first image and the second image, and generate user feedback in response to the difference exceeding a second predetermined threshold.

17. The system of claims 10-16, wherein, the processor is configured to perform object detection by applying a body keypoint algorithm configured to output key points of a human body in the second image.

Images