Method for determining an illumination color, image processing device and recording system

Abstract

Computer-implemented method for determining an illumination color (F) for improving and / or optimizing the contrast of an image to be created using the illumination color (F), wherein the method comprises the following steps: - Receiving an input image (E) of an object (30) to be captured in greyscale, in which the object (30) is illuminated in white; - Generating a colored selection image (A) based on the input image (E); - Determining a color distribution (V) of the selected image (A); - Determining the illumination color (F) based on the color distribution (V).

Description

[0001] The invention relates to a computer-implemented method for determining an illumination color for improving and / or optimizing the contrast of an image to be captured using the illumination color. Furthermore, the invention relates to a recording system for implementing the method.

[0002] Cameras are used for a variety of purposes in industrial manufacturing facilities, particularly in quality assurance. Here, cameras are used to take photographs (also referred to as images or recordings) of the objects to be inspected, which are then processed and examined using special image processing techniques. In this way, for example, in the automotive industry, the presence of plastic rivets and seals can be automatically detected, weld nuts can be identified, and the alignment of body parts can be checked before welding.

[0003] Black-and-white (BW) cameras are often used in the above-mentioned applications because they are cheaper to purchase. Furthermore, black-and-white cameras offer the advantage of producing images that require a comparatively small amount of data.

[0004] In general, however, capturing a colored object in black and white is lossy because, as with the transformation of a color image into grayscale, different colors can correspond to one gray value.

[0005] For example, a simple method for transforming a color image defined in the RGB model into grayscale is based on assigning each color c = (r,g,b) a gray value x according to the following calculation rule: x=0.21r+0.72g+0.07b .

[0006] In other words, the gray value is the weighted average of the intensity values ​​of the individual color components. With such a transformation, for example, the colors c1 = (1,0,1) and c2 = (0,0.39,0) are mapped to approximately equal gray values ​​x1 = 0.28 and x2 = 0.2808. If an object to be captured has two adjacent areas with the colors c1 and c2, these areas become virtually indistinguishable in a black-and-white image created using the above calculation method. The contour running along the boundary between the two areas on the object is therefore no longer recognizable in the black-and-white image.

[0007] However, for the usability of a black-and-white image in the applications described above, it is often crucial that such contours are recognizable in the black-and-white image. To highlight relevant areas or contours of the captured object in a black-and-white image or to better distinguish them from one another, the object can be appropriately illuminated with colored light sources during the capture process.

[0008] For example, red areas of an object reflect red light when illuminated with red light, making the red areas appear very bright in a black-and-white image. At the same time, blue and green areas absorb a large portion of the red light, making such areas appear very dark in a black-and-white image.

[0009] In addition to suitable lighting, a multitude of other parameters are important when creating a picture, such as the camera's field of view, focus, exposure time, etc., which determine the quality of the image. The success of subsequent processing steps therefore depends crucially on the selected parameter values. Often, a suitable set of parameter values ​​must be determined individually for the respective application, which is also referred to as camera teaching. Various approaches to simplifying this process are known in the state of the art, particularly for cases where the teaching must be performed (partially) by a user.

[0010] DE 10 2018 119 691 A1 discloses a camera system for capturing codes. According to the invention, the positions of contour points of the codes are determined and analyzed in relation to the edges of a search zone of the camera system. If, for example, a large proportion of the contour points fall within a boundary area of ​​the camera system's search zone, this indicates that the search zone should be selected differently. This can then be done manually or automatically. This ensures greater reading reliability.

[0011] DE 10 2019 113 647 A1 describes a camera system for detecting objects and / or object properties. In addition to the object being detected, it also captures additional image features and information about misdetections. For example, if it is determined that detections of images with an increased mean gray value are frequently incorrect, appropriate feedback can be provided to the user, who can then correct the settings.

[0012] DE 10 2015 216 446 A1 improves a method for checking the completeness of a container containing a plurality of objects using a 3D camera by assisting the user in selecting suitable parameters. The parameters are, in particular, threshold values ​​for height values ​​that determine whether a space in a container is classified as occupied or unoccupied. A central aspect of the method is to provide the user with two simultaneous displays: the first displays a two-dimensional view of the container, the second displays the recorded height values ​​with adjustable threshold values. This provides the user with an improved overview of the effect of the set threshold values.

[0013] DE 10 2019 103 397 A1 describes a method for operating a camera to capture codes. A modular camera operation allows the learning phase, which is necessary for reliable recognition and usually has to be performed manually, to be performed by the camera itself.

[0014] DE 10 2011 010 315 A1 shows a method for the automatic real-time recognition of predetermined objects within a digital image stored in a computer, which image consists of a large number of individual pixels.

[0015] In another method, a template is created from a mathematical model that adapts to the search image to overcome challenges such as scaling and rotation and to determine the position of the screw head with high accuracy, so that the angular velocity of a rotating part can be determined from a video, e.g. from a smartphone (DAGA, A.; GARIBALDI, L.: GA-Adaptive Template Matching for Offline Shape Motion Tracking Based on Edge Detection: IAS Estimation from the SURVISHNO 2019 Challenge Video for Machine Diagnostics Purposes. Algorithms, 2020, 13th vol., no. 2, p. 33).

[0016] To efficiently improve contrast and brightness in low-contrast color images, the methods "Range-Limited Bi-Histogram Equalization" and "Adaptive Gamma Correction" are combined (GAUTAM, C.; TIWARI, N.: Efficient Color Image Contrast Enhancement Using Range-Limited Bi-Histogram Equalization With Adaptive Gamma Correction. In: 2015 International Conference on Industrial Instrumentation and Control (ICIC). IEEE, 2015. pp. 175-180). The state-of-the-art methods still require improvement with regard to the performance of both the training process and the subsequent image processing. Furthermore, most state-of-the-art solutions require a high degree of user interaction during the training process, which is also perceived as a disadvantage.

[0017] Against the background outlined above, the invention is based on the object of determining an improved method for adjusting the lighting for photographic images. In particular, the method should enable efficient and simple adjustment of an illumination color, thereby optimizing the contrast of a black-and-white image to be created.

[0018] This object is achieved by a computer-implemented method according to claim 1, a computer-readable storage medium according to claim 8, an image processing device according to claim 9, and a recording system according to claim 10. Further advantageous embodiments and developments of the invention emerge from the subclaims.

[0019] In particular, the problem is solved by a computer-implemented method for determining an illumination color for improving and / or optimizing the contrast of an image to be captured using the illumination color. The method comprises the following steps: - Receiving an input image of an object to be detected in grayscale, in which the object is illuminated in white; - Generating a color selection image based on the input image; - Determining a color distribution of the selection image; - Determine the lighting color based on the color distribution.

[0020] In the described process, the initial grayscale image of the object to be captured forms the basis for all subsequent process steps. A key aspect of the process is that the object to be captured is illuminated in white during the initial image. Since white light contains the colors red, green, and blue in equal intensity, this prevents certain areas from brightening or darkening compared to differently colored areas in the initial image. At the same time, the white illumination provides a sufficiently bright initial image. In this sense, the initial image corresponds to an unadulterated, authentic image of the object in neutral lighting.

[0021] A core concept of the invention is to generate a color selection image based on the input image. The selection image serves as a reference for the subsequent process steps regarding the color information a color image of the object to be captured would contain. In a sense, the selection image compensates for the loss of information due to the fact that the input image of the object to be captured is only available in grayscale.

[0022] A further method step involves determining a color distribution of the selected image. In the context of this description, a color distribution can be understood as one or more frequency distributions of the color values ​​in an image. A color distribution can be a frequency distribution across the entire color spectrum or a set of multiple frequency distributions, each projected onto a single color channel.

[0023] The described method determines the illumination color to be determined based on the color distribution of the selection image. As already described above, the determined color distribution serves as a reference for the color information of the object to be detected. In particular, the effects of colored illumination can be simulated using the colored selection image.

[0024] In one embodiment of the method, the illumination color can be determined further based on an image section in the selection image and a section color distribution of the image section. The image section can be defined by - an algorithm for object recognition and / or - a user, wherein the setting comprises in particular the use of a geometric selection tool and / or a brush tool.

[0025] In the context of this invention, an image section can be understood as a single image region or the combination of any number of image regions. In particular, an image section can comprise several unconnected image regions.

[0026] Defining an image section in the selection image offers the advantage that the selection image can be restricted to a relevant part, namely the object to be captured or a section of it. In particular, this can prevent irrelevant image elements from influencing further process steps. Without restricting the image section, for example, large, dark image areas in the background or surroundings of the object to be captured could significantly influence the determined color distribution, even though the object to be captured itself is located in a bright image area. In this case, the image section should include the object to be captured without the background in order to obtain a meaningful color distribution.

[0027] In a preferred embodiment, the selection image is generated and / or the illumination color is determined using artificial intelligence, in particular using a trained artificial neural network (ANN). The ANN can be implemented, for example, as a single- or multi-layer feed-forward network or as a recurrent network.

[0028] Training the ANN can, for example, be done using a supervised learning process, as described below. To generate a training dataset, color images y1,...,y n created that show one or more objects in different color schemes that can be expected in the intended application area. Such a targeted selection of training data can increase the quality of the subsequent output of the ANN. For each color image y i with 1 ≤ i ≤ n, a black and white image x iby transforming the color values ​​into grayscale. Such a pair (x i ,y i ) with 1 ≤ i ≤ n is called a training example, where x i as input and y i is considered the expected output. Using a learning algorithm, the weights of the ANN can now be optimized to minimize deviations from the expected output.

[0029] Similarly, an ANN can be trained to determine an illumination color. A pair of a black-and-white image and illumination color can serve as a training example. The illumination color should be chosen to optimize the contrast under appropriate colored illumination in a black-and-white image.

[0030] In another embodiment, the selection image can be generated by transforming grayscale values ​​of the input image into a predetermined color spectrum. This embodiment is based on the idea that, in some areas of application of this invention, the (approximate) color spectrum of the scene depicted in the input image is known. If such a color spectrum is given, each gray value can be assigned to a color from the color spectrum using a calculation rule. In this way, a color selection image can be calculated pixel by pixel for the input image. Assuming knowledge of a suitable color spectrum, this embodiment offers a simple and efficient way of generating the selection image.

[0031] A further embodiment of the method provides for determining the illumination color from the color distribution as a function of a first color and a second color. If a distance value between the first color and the second color, in particular a distance between gray values ​​of the first and second colors, falls below a threshold value, this embodiment is based on the idea of ​​quantifying the distinguishability of colors in a black-and-white image as a distance value. The distance value can be selected as the difference between the respectively assigned gray values, as described above. A low distance value between a first and a second color can then indicate that correspondingly colored areas of the object to be captured are difficult to distinguish in the input image. A distance value is considered low if it falls below a predetermined threshold value.By illuminating the object with the first color, the correspondingly colored areas may appear brighter than differently colored areas in a black and white image.

[0032] In another embodiment, the method further comprises sending a control signal to a color illumination unit, wherein the control signal corresponds to the determined illumination color and the color illumination unit illuminates the object to be detected.

[0033] Preferably, a method according to the embodiment described above further comprises the following steps: - receiving a second input image of the object to be detected in grayscale, wherein the object to be detected was illuminated in the second input image with the determined illumination color, in particular with a color illumination unit; - Detecting areas and / or contours in the second input image and / or in an image section of the second input image.

[0034] As previously described, illuminating the object to be captured with a specific illumination color can result in object regions of that color appearing brighter in the second input image, while object regions of the complementary color (with respect to the illumination color) appearing darker. This results in more pronounced contours in the second input image, thus optimizing the detection of regions and / or contours, which can be performed in a subsequent processing step.

[0035] The object is further achieved in particular by a computer-readable storage medium which contains instructions which cause at least one processor to implement a method as described above when the instructions are executed by the at least one processor.

[0036] Furthermore, the object is achieved in particular by an image processing device comprising a computer-readable storage medium as described above and a computing unit. The computing unit is configured to execute stored instructions of the computer-readable storage medium.

[0037] In particular, the task is further solved by a recording system that has the following components: - an image processing device as described above, - a black and white camera and - a color lighting unit.

[0038] The monochrome camera is designed to capture an image of an object to be captured and provide it to the image processing device. The color illumination unit is designed to illuminate the object to be captured in color.

[0039] According to one embodiment, the above-described components of the recording system, ie in particular the image processing device, the SW camera and the color illumination unit, can be installed in a single device.

[0040] In an alternative embodiment, only the image processing device and the SW camera are installed together in one device, while the color illumination unit is connected to the device as an external device.

[0041] In a further embodiment, a recording system as described above can further comprise a transmitting unit and a receiving unit. The transmitting unit can be configured to transmit a captured image to the receiving unit. The receiving unit can be configured to provide the received image to the image processing device. Thus, the image processing device does not have to be located in the immediate vicinity of the SW camera, but can be housed, for example, in a remote data center.

[0042] A preferred embodiment may provide that a recording system as described above further comprises a unit for interacting with a user. The interaction unit may be designed to use input and output devices - to display the input image and / or the selection image, preferably in reduced image resolution on an overview page, and / or - to capture an image section defined by a user, in particular by means of geometric selection tools and / or a brush tool.

[0043] A recording system according to this embodiment can provide a user with an improved overview of the progress of the computer-implemented method by allowing the user to view the color selection image generated for the initial recording.

[0044] The definition of an image section can be carried out conveniently by the user in a suitable graphical user interface. For this purpose, geometric selection tools and / or a brush tool are preferably provided, which allow the image section to be marked in the display of the selection image. A simple geometric selection tool can, for example, create a selection area in the form of a triangle, a quadrilateral, or an ellipse. To create polygon-shaped selection areas, a polygon tool can also be provided, with which the sections of a polygon line can be defined step by step. In addition, a brush tool can be provided, which creates an image section from the total set of all pixels selected with the brush tool. This is particularly advantageous if the area to be marked cannot be easily marked using a geometric selection tool.

[0045] Further embodiments of the invention emerge from the subclaims.

[0046] The invention is described below using exemplary embodiments, which are explained in more detail with reference to the figures. Herein: Fig. 1 an exemplary recording system; Fig. 2 a schematic representation of a method for determining an illumination color; Fig. 3 an exemplary user interface for defining an image section; Fig. 4 shows determining an illumination color based on a color distribution according to an embodiment;

[0047] In the following description, the same reference numbers are used for identical and equivalent parts.

[0048] Fig. 1 shows a recording system 70 according to the invention. In this embodiment, the recording system 70 is used in the context of a production control for which a photographic image of the object 30 is to be taken.

[0049] Object 30 is in Fig. It is a cuboid-shaped component with five screws arranged one below the other, the correct positioning of which is to be checked as part of the production control.

[0050] For this purpose, a black and white camera 10 is aimed at the object 30, whose surface is illuminated by a color lighting unit 20. The color lighting unit 20 is an RGBW lighting unit that has light sources in the colors red, green, blue, and white. By switching the individual colored light sources on and off and adjusting their intensity, the color lighting unit 20 allows the object 30 to be illuminated in a variety of colors.

[0051] Further components of the recording system 70 are an image processing device having a computer-readable storage medium 71 containing instructions for carrying out a method according to the invention, and a computing unit 72 which carries out the method according to the instructions stored on the storage medium 71.

[0052] To generate the selection image, the computing device 72 is connected to an AI module 74, which has a trained artificial neural network and offers corresponding interfaces that enable communication with the neural network.

[0053] Furthermore, the computing device 72 is connected to an interaction unit 73. The interaction unit 73 has input and output devices through which a user can view the input image from the SW camera 10 as well as the selection image generated by the AI ​​module 74. Furthermore, the interaction unit 73 allows a user to define an image section in the selection image.

[0054] In Fig. 2, a method for determining an illumination color is schematically shown, wherein the input image E represents a part of the object 30 from the embodiment according to Fig. 1 shows.

[0055] In the embodiment shown, the input image E shows several hexagonal screw heads 31, 32, 33 in a top view, whereby the screw heads 31 and 33 are located at the edge of the image and are not completely shown on the input image E. Since the input image E is only available in grayscale, the contours of the screw heads 31, 32, 33 are only weakly pronounced (in Fig. 2 shown as dashed lines). In addition, the surfaces of the screw heads 31, 32, 33 and the surrounding area appear in the same color.

[0056] In the next step, the process creates a color selection image A based on the input image E. This can be done using artificial intelligence, as is already the case in connection with Fig. 1. On the generated colored selection image A it can be seen that the surfaces of the screw heads 31 and 33 have a different color than the surface of the screw head 32 and all three surfaces have a different color than the background of the workpiece (in Fig. 2, different colors are represented by different hatching. The presence of different colors also makes the contours of the screw heads 31, 32, and 33 more easily visible.

[0057] A color distribution V is determined for the generated color selection image A. In Fig. 2 shows the color distribution two-dimensionally, with the colors on the horizontal axis and their relative frequencies in the selection image A on the vertical axis.

[0058] In the next step, a square image section A1 is defined in the selection image A. This definition can be performed either by a user or algorithmically. The image section A1 is selected such that it exclusively shows the screw head 32 on the surface of the object 30.

[0059] A color distribution V1 is determined for the specified image section A1. From the color distribution V1, it can be seen that there are two colors or color ranges that occur with a comparatively high relative frequency in the image section A1.

[0060] In a final step, the illumination color F is determined based on the image section A1 and the section color distribution V1. In the illustrated embodiment, the illumination color F is the color of the surface of the screw head 32 in the selected image A.

[0061] Following the described procedure, a second input image can be created in grayscale (not shown in Fig. 2). In the second input image, the detected object is colored with the previously determined illumination color F, instead of with white light, as in the (first) input image E. It is expected that the second input image will have improved contrast, and in particular that the surface of the screw head 32 will be clearly distinguished from its surroundings.

[0062] Fig. 3 shows an embodiment of a graphical user interface 90 for marking the image section A1 in the selection image A. At the top right edge of the user interface 90 there is a toolbar 91 which provides the user with various selection tools: a triangle tool 92, a rectangle tool 93, an ellipse tool 94, a polygon tool 95 and a brush tool 96. In addition, Fig. 3 that the user uses the rectangle tool 93 to define a square image section that encloses the central screw head.

[0063] Based on Fig. 4, the determination of an illumination color based on a color distribution V can be explained according to one embodiment. The color distribution V is represented as a histogram, with a continuous spectrum of all colors occurring in the selection image A on the horizontal axis and their relative frequency in the selection image A on the vertical axis. It can be seen from the histogram that colors in the range of the colors F1 = (1, 0, 1) and F2 = (0, 0.39, 0) occur in high proportions in the selection image A, whereby the notation of the colors F1 and F2 here refers to the RGB color model.

[0064] According to the method mentioned in the introduction to this description, the colors F1 and F2 can be assigned the gray values ​​x1 = 0.28 and x2 = 0.2808, which could occur upon transformation into a grayscale image. If the difference between the gray values ​​is interpreted as a distance value and a threshold of δ = 0.01 is set, the distance value of 0.0008 falls below this threshold. It can therefore be assumed that image areas of the colors F1 and F2, or neighboring colors, are not distinguishable or difficult to distinguish in a grayscale image.

[0065] Accordingly, it is expected that colored lighting with F1 as the illumination color will optimize the contrast of a black-and-white image, as the F1-colored areas are brightened. Alternatively, F2 can be selected as the illumination color.

[0066] At this point it should be noted that all of the parts described above are to be regarded individually - even without additional features described in the respective context, even if these have not been explicitly identified as optional features in the respective context, e.g. by using: in particular, preferably, for example, e.g., if necessary, round brackets, etc. - and in combination or any sub-combination as independent embodiments or further developments of the invention, as defined in particular in the introduction to the description and the claims. Deviations from this are possible. Specifically, it should be noted that the word in particular or round brackets do not identify any features that are mandatory in the respective context.

[0067] Finally, it should be explicitly pointed out that all aspects described with regard to the method according to the invention can be combined with the recording system according to the invention and vice versa. List of reference symbols 10 black and white camera 20 color lighting unit 30 objects 70 recording system 71 Storage medium 72 computing unit 73 Interaction Unit 74 AI module 90 User interface for marking an image section 91 Toolbar 92 Triangle tool 93 Rectangle tool 94 Ellipse tool 95 Polygon tool 96 Brush tool A selection image A1 image section E Input recording F Lighting color F1 first color F2 second color V Color distribution V1 cutout color distribution

Claims

[1] Computer-implemented method for determining an illumination color (F) for improving and / or optimizing the contrast of an image to be recorded using the illumination color (F), the method comprising the following steps: - receiving an input image (E) of an object (30) to be detected in grayscale, in which the object (30) is illuminated in white; - generating a color selection image (A) based on the input image (E); - determining a color distribution (V) of the selection image (A); - Determining the illumination color (F) based on the color distribution (V). [2] Method according to claim 1, wherein the determination of the illumination color (F) is further carried out on the basis of an image section (A1) in the selection image (A) and a section color distribution (V1) of the image section (A1), wherein the image section (A1) is determined by - an algorithm for object recognition and / or - a user, wherein the setting comprises in particular the use of a geometric selection tool (91, 92, 93, 94, 95) and / or a brush tool (96). [3] Method according to one of the preceding claims, wherein the generation of the selection image (A) and / or the determination of the illumination colour (F) is carried out using artificial intelligence, in particular using a trained artificial neural network. [4] Method according to one of the preceding claims, wherein the selection image (A) is generated by transforming grayscale values ​​of the input image (E) into a predetermined color spectrum. [5] Method according to one of the preceding claims, wherein the illumination color (F) is determined from the color distribution (V) as a function of a first color (F1) and a second color (F2) and a distance value between the first color (F1) and the second color (F2), in particular a distance between gray values ​​of the first and second colors, falls below a threshold value. [6] A method according to any one of the preceding claims, wherein the method further comprises the following step: - Sending a control signal to a color illumination unit (20), wherein the control signal corresponds to the determined illumination color (F) and the color illumination unit (20) illuminates the object (30) to be detected. [7] Method according to one of the preceding claims, in particular according to claim 6, wherein the method further comprises the following steps: - receiving a second input image of the object to be detected (30) in grayscale, wherein the object to be detected (30) was illuminated on the second input image with the determined illumination color (F), in particular with a color illumination unit (20); - Detecting areas and / or contours in the second input image and / or in an image section of the second input image. [8] A computer-readable storage medium containing instructions that cause at least one processor to implement the method of any one of the preceding claims when the instructions are executed by the at least one processor. [9] Image processing device comprising: - a computer-readable storage medium (71) according to claim 8 and - a computing unit (72) configured to execute stored instructions of the computer-readable storage medium. [10] Recording system (70) comprising: - an image processing device according to claim 9, - a SW camera (10) and - a colour lighting unit (20), wherein the SW camera (10) is designed to create a picture of an object (30) to be detected and to provide it to the image processing device, wherein the color illumination unit (20) is designed to illuminate the object (30) to be detected in color. [11] A recording system according to claim 10, wherein the image processing device, the SW camera (10) and the color illumination unit (20) are installed in a single device. [12] A recording system according to claim 10, wherein the image processing device and the SW camera (10) are installed in a single device and the color illumination unit (20) is an external device connected to the device. [13] Recording system according to one of claims 10 to 12, further comprising: - a transmitting unit and - a receiving unit, where - the transmitting unit is designed to transmit the created recording to the receiving unit and - the receiving unit is designed to provide the received image to the image processing device. [14] Recording system according to one of claims 10 to 13, further comprising an interaction unit (73) which is designed to use input and output devices - to display the input image and / or the selection image, preferably in reduced image resolution on an overview page, and / or - to capture an image section which a user defines, in particular by means of geometric selection tools (91, 92, 94, 95) and / or by means of a brush tool (96).

Images