A method for calibrating a spray tool mounted on a mobile marking robot

Abstract

The present invention relates to a method for calibrating a spray tool mounted on a mobile marking robot. The method comprises taking a photo with a handheld camera device of a preset pattern of painted lines marked by a spray tool of a mobile marking robot, with complementary IMU metadata of the handheld computing device; using said IMU metadata to create an image by rectifying the captured photo to get a projection that matches the plane of the surface on which the lines are painted; using machine vision to recognize and measure the painted lines on the rectified image; and using the measurements to calibrate the spray tool of the mobile marking robot.

Description

[0001] A method for calibrating a spray tool mounted on a mobile marking robot

[0002] Technical field of the invention

[0003] The present invention relates to a method for calibrating a spray tool mounted on a mobile marking robot.

[0004] Background of the invention

[0005] Today, robots have been developed that provide solutions for e.g., land surveying, sports field marking, and road marking. Some robots are e.g., developed to paint sports fields autonomously with minimal user interaction. This solution reduces the time it takes to paint a sports field compared to the traditional manual marking operation. These robots rely on GPS and IMU (inertial measurement unit) sensors to achieve millimeter-level accuracy. During calibration, these sensors generally do not require much attention, as they are calibrated by the hardware provider. However, each robot's painting release and attack times have an engineering tolerance of a few milliseconds, meaning that they can vary slightly from robot to robot. To compensate for these imperfections, the software of the robot is adjusted to account for these differences. While the calibration process itself has been solved, the current method of calibration is not efficient in terms of time, as it requires manual labor.

[0006] The calibration process of the attack and release time starts with a set of lines being painted by the uncalibrated robot at varying speeds. This will result in differently stretched lines, as the painting attack and release times are not calibrated. Measuring the differences between these lines allows the calibration to take place, by adjusting the attack and release times.

[0007] The calibration is done correctly when the length of all the lines is the same. Taking all the measurements of the painted lines by hand is not time effective and requires manual labor. US2020222929 (A1 ) discloses a smart nozzle assembly that includes a nozzle, a nozzle control mechanism, and camera rigidly attached to the nozzle for use with a mobile robot in an autonomous spray painting system. The nozzle control mechanism is configured to control flowrate, control the shape of the spray pattern, mix two or more colors, and clean dried paint at the nozzle tip. The nozzle assembly further includes a process for running software to manage or initiate the nozzle control mechanism's functionality and to provide the nozzle calibration. The calibration method for the nozzle uses an algorithm that measures the spray pattern, the distribution of paint within the spray pattern, and the relative position of the nozzle and camera. The distribution of paint within the spray pattern is measured in terms of physical quantity of delivered paint per unit area.

[0008] Summary of the invention

[0009] It is one object of the present invention to provide a faster and easier calibration process of the attack and release time, provide an alternative solution to the above methods that solves some of the above-mentioned problems.

[0010] To drastically increase the time efficiency of the calibration method, the manual labor can be replaced by technology. Using a handheld computing device to take a photo of painted lines by the spray tool of the mobile marking robot, while capturing complementary IMU (inertial measurement unit) metadata to get an understanding of the orientation of the handheld computing device, can automate the measurement process by processing the captured data with computer vision algorithms.

[0011] The basic steps of the method of the present invention, are as follows:

[0012] 1 . Take a photo with a handheld computing device of a preset pattern of painted lines marked by a spray tool of a mobile marking robot, with complementary IMU metadata of the handheld computing device;

[0013] 2. Use said IMU metadata to create an image by rectifying the captured photo to get a projection that matches the plane of the surface on which the lines are painted;

[0014] 3. Use machine vision to recognize and measure the painted lines on the rectified image; and

[0015] 4. Use said measurements to calibrate the spray tool of the mobile marking robot.

[0016] A first aspect relates to a method for calibrating a spray tool mounted on a mobile marking robot, the method comprising:

[0017] - instructing the mobile marking robot to mark a predetermined pattern of lines on a predefined surface area, preferably having a known dimension;

[0018] - taking a photo of said marked pattern of lines with a camera device configured for providing inertial measurement unit metadata about said camera device;

[0019] - using said inertial measurement unit metadata to create an image by rectifying the captured photo to get a projection that matches the plane of the surface on which the lines are painted;

[0020] - using machine vision to recognize and measure the painted lines on the rectified image; and

[0021] - using said measurements to calibrate the spray tool of the mobile marking robot.

[0022] In one or more embodiments, the step of taking a photo comprises guiding with the camera device a user to position the camera device in a wanted position relative to said predefined surface area.

[0023] In one or more embodiments, the guidance comprises showing one or more guidelines in the camera view of said camera device, said guideline(s) used to define at least a part of the predefined surface area with the marked pattern of lines. The guidance may also comprise showing three guidelines in the camera view of said camera device, one of said guidelines used to define at least a bottom or top part of the predefined surface area with the marked pattern of lines, and the two other guidelines used to define the sides edges / borders of the predefined surface area with the marked pattern of lines. The guidance may further comprise an interactive guidance showing the user to roll and / or pitch the camera device to align the camera device to an optimized orientation relative to the preset surface area with the marked pattern of lines.

[0024] In one or more embodiments, the rectification step is followed by a preprocessing step prior to initiating the machine vision step; wherein the preprocessing step comprises:

[0025] - Segmenting the rectified image to separate the predefined surface area from the ground surface;

[0026] - Optionally, cropping the rectified image to the predefined surface area; and

[0027] - Segmenting each of the lines defining the marked pattern of lines from the background surface.

[0028] In one or more embodiments, the step of segmenting the rectified image to separate the predefined surface area from the ground surface comprises one or more, preferably all, of the following steps:

[0029] 1 . Apply non-local means denoising to filter out small graininess from the rectified image;

[0030] 2. Apply gaussian blur to filter out spatially larger graininess from the rectified image;

[0031] 3. Apply Otsu’s method to threshold the rectified image;

[0032] 4. Apply a morphology opening operation to remove non-coherent blobs and smooth the edges of the segmented regions;

[0033] 5. Apply canny edge detection to get the outlines of the segmented regions;

[0034] 6. Obtain the expected position of the predefined surface area’s outlines;

[0035] 7. Define an error margin for the expected outlines;

[0036] 8. Apply hough line detection to get the possible, straight geometric lines in the rectified image; 9. Consider the most probable four outlines, which fall within the error margin; and

[0037] 10. Adjust the outlines to mark out a rectangle.

[0038] Non-local means denoising is described by Antoni Buades, Bartomeu Coll, and Jean-Michel Morel in Image Processing On Line 1 (2011 ), pp. 208-212, “Non- Local Means Denoising”.

[0039] Gaussian blur is described by Estevao S. Gedraite and Murielle Hadad in “Investigation on the effect of a Gaussian Blur in image filtering and segmentation” (2011 ), pp. 393-396.

[0040] The Otsu method is described by Jamileh Yousefi in “Image Binarization using Otsu Thresholding Algorithm” (2011 ).

[0041] The process of canny edge detection is described by Zhao Xu, Xu Baojie, and Wu Guoxin in “Canny edge detection based on Open CV” (2017), pp. 53-56.

[0042] The process of hough line detection is described by Stanley R. Deans in: IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-3.2 (1981 ), pp. 185-188, “Hough Transform from the Radon Transform”.

[0043] In one or more embodiments, the step of segmenting each of the lines defining the marked pattern of lines from the background surface comprises one or more, preferably all, of the following steps:

[0044] 1 . Apply non-local means denoising to filter out small graininess from the rectified image;

[0045] 2. Apply gaussian blur to filter out spatially larger graininess from the rectified image;

[0046] 3. Apply adaptive thresholding on the rectified image; and

[0047] 4. Apply a morphology opening operation to get rid of non-coherent blobs and to smooth the edges of the segmented regions.

[0048] The process of adaptive thresholding is described by Payel Roy et al. “Adaptive thresholding: A comparative study” (2014), pp. 1182-1186.

[0049] As an example of a morphology opening operation, morphology kernel size is responsible for the clear separation of the black blobs. If two neighboring painted lines ’melt’ together or there are other blobs in the picture, then the kernel size may be increased where the painted lines are positioned. Similarly, if a painted line disappears from the image, then it may be solved by decreasing the kernel size.

[0050] In one or more embodiments, the machine vision step comprises isolating and identifying the marked pattern of lines from the background surface.

[0051] In order to read off the measurements, the problem can be further simplified with the ‘divide and conquer’ method, where the painted lines are processed one by one. The first step is to find all the contours in the image and split it into smaller partitions. Then each painted line is processed individually, one by one.

[0052] In one or more embodiments, the machine vision step comprises isolating and identifying each line individually from the background surface.

[0053] In one or more embodiments, the machine vision step comprises one or more, preferably all, of the following steps on each individual line of the marked pattern of lines:

[0054] 1 . Apply canny edge detection to get the outlines of the painted line;

[0055] 2. Define a padding for the expected outlines;

[0056] 3. Apply hough line detection to get the possible, straight geometric lines in the image; and

[0057] 4. Consider the most probable four outlines, which fall within the padding. In some embodiments, the partitions are then re-assembled, and measurements may be performed to convert the pixel values into real life units, e.g., by use of known constants, such as line width and distance between individual lines. If a calibration mat is used, constants such as the height and width of the calibration mat may also be used.

[0058] In one or more embodiments, the predefined surface area is a calibration mat of known shape and dimensions.

[0059] The painting attack and release times have tolerances as there is always a propagation time between when a voltage is applied to the pump and when the paint actually leaves the nozzle. Similarly, there is also a propagation time between when the pump stops, and when the paint stops coming out of the nozzle. These propagation times are really sensitive to multiple factors, such as the precise length of the tube coming from the pump to the nozzle, the wear of the pump, atmospheric pressure, nozzle wear and many more. Thus, it is impractical (if not impossible) to produce robots with very tight attack and release time tolerances. The calibration method according to the present invention seeks to solve these problems.

[0060] It should be noted that embodiments and features described in the context of one of the aspects of the present invention also apply to the other aspects of the invention.

[0061] As used in the specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" or "approximately" one particular value and / or to "about" or "approximately" another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations, by use of the antecedent "about", it will be understood that the particular value forms another embodiment. Detailed description of the invention

[0062] Calibration of the spray tool of a mobile line marking robot

[0063] Generally, the calibration process of the spray tool starts with a predetermined number of lines being painted by the mobile robot. Each second line is painted from top-down (or from a first side to the other second side) and each second line is painted from bottom-up (or from the second side to the first side). Each neighboring line forms a pair as the two lines are painted by the mobile robot at a constant speed, but at different directions, as explained above.

[0064] The different pairs of lines are painted at different speeds, where the robot assumes that the attack and release times are 0 (are not calibrated). This results in different line lengths and offsets. As if the attack time of a robot is not zero, the robot will cover more distance during the attack time when going faster, resulting in an offset start position of the painted line. And as if the release time of a robot is not zero, the robot will cover more distance during the release time when going faster, resulting in an offset end position of the painted line.

[0065] The different pairs of lines are painted at different speeds but with the spray tool, resulting in longer or shorter pairs of lines. Measuring the vertical offset between the lines of each pair and the pairs itself, and the distance between each line, gives enough information to calculate the calibration values. The calibration is done correctly if the length of all of the lines in a pair is the same (there are no horizontal offsets). Performing all the measurements of the offsets manually is not time effective. The calibration has to be confirmed, thus requiring the painting process to be done at least twice. This on average takes 20 minutes per mobile robot. The new method

[0066] The new method for calibrating a spray tool mounted on a mobile marking robot comprises the following steps:

[0067] - instructing the mobile marking robot to mark a predetermined pattern of lines on a predefined surface area having a known dimension;

[0068] - taking a photo of said marked pattern of lines with a camera device configured for providing inertial measurement unit metadata about said camera device;

[0069] - using said inertial measurement unit metadata to create an image by rectifying the captured photo to get a projection that matches the plane of the surface on which the lines are painted;

[0070] - using machine vision to recognize and measure the painted lines on the rectified image; and

[0071] - using said measurements to calibrate the spray tool of the mobile marking robot.

[0072] A preferred process is shown in Figure 1 .

[0073] The calibration value of a mobile marking robot is a sum or a group of values that affect the robot’s behavior. When these values are correct, the robot can ideally achieve millimeter precision while marking / painting. These values are calculated from the differences in the lengths of the painted lines. One example of a line pattern painted during the calibration process may be nine lines, yielding eight measurements from the differences between neighboring lines. These eight measurements can be converted into a calibration value and associated with the calibrated robot. The lines may be painted on the ground, optionally on a sheet called a calibration mat to improve the visibility of the painted lines.

[0074] In the following, preferred examples of a selection of the individual method steps will be disclosed.

[0075] Taking the photo To be able to correct for the perspective distortion of the taken photo, it is necessary to record complimentary IMU (inertial measurement unit) data when taking the photo. The accelerometer can be used to get the relative orientation of the device, as the gravity vector conveniently matches the normal vector of the ground surface’s plane, i.e., the ROI’s plane (region of interest). These measurements will also aid the user with the positioning of the device while taking the photo. Guidelines may be shown drawn over the camera view, which instructs the user to approximately line up the borders of the ROI. This will provide valuable information about the expected position of the ROI later on. The three guidelines could e.g., be lined up with the edges of the used calibration mat, where the horizontal line is aligned with the bottom edge of the mat, and the two angled lines are aligned with the two side edges. The bottom of the mat should be the side where the painted lines start. The camera view could also include text on the top of the screen to help the user to align the device to the correct orientation. The text can e.g., show the following 5 messages:

[0076] • Pitch the device backward!

[0077] • Pitch the device forward!

[0078] • Roll the device right!

[0079] • Roll the device left!

[0080] • Perfect!

[0081] The roll and pitch axes are meant to be applied in landscape orientation of the device as illustrated in Figure 2. When the roll and the pitch are both aligned, the text could e.g., say “Perfect!”.

[0082] When taking a photo, the user should try to hold the device as steady as possible. If the device is shaking or has a sudden change in orientation while taking the photo, an additional text message may be generated to pop up stating “Photo not taken, device is shaking too much!”. Then the photo must be retaken.

[0083] The rectification step

[0084] Figure 3 illustrates why it is necessary to correct for perspective distortion, in order to accurately read off measurements from the taken image. One shown plane represents the camera sensor’s plane, and the other shown plane represents the plane of the Region of Interest (ROI) with the painted calibration lines. The normal vectors of the planes are labeled C and G for “camera” and “gravity” respectively, where gamma y represents the angle between them. To correct for perspective distortion, it is required to create an orthographic projection of the ROI’s plane. In an ideal scenario where the camera is stationary and y = 0, the taken image would be already equivalent to the orthographic projection. However, since the method is particularly suitable for a handheld camera device, this is not realistic. Instead, a transformation matrix can be used to reverse the effects of distortion by warping the image and virtually moving the camera to y = 0, thereby creating an orthographic projection.

[0085] The user should preferably inspect the rectified image, either on the same camera device or on another computer device with a screen for viewing the rectified image and evaluate if the calibration mat has any distortions. Having no distortions means that the calibration mat is a perfect rectangle, where the edges are perpendicular. If there is substantial distortion, the user is advised to retake the photo.

[0086] After visual inspection, the user may interactively, e.g., via a touch screen, be presented with guide points for use at defining the four comers of the calibration mat, and e.g., be instructed to correct their position, e.g., by dragging them in place above the image. This step can be skipped if the corners are already aligned, as an application, preferably operating on the camera device, may be configured to do this automatically.

[0087] An example of a photo (A) and a rectified image (B) may be seen in Figure 4.

[0088] Pre-processing

[0089] Pre-processing may be necessary prior to using machine vision to recognize and measure the painted lines on the rectified image. Preprocessing the rectified image simplifies the problem, to be able to read off precise measurements. It may comprise the following three main steps:

[0090] - Segment rectified image to separate ROI from the ground.

[0091] - Crop to ROI.

[0092] - Segment ROI to separate the painted lines from the ground.

[0093] The user may e.g., be asked to confirm that the calibration mat is a coherent white color, and the painted lines are visible as clearly separated black forms. The user may be provided with access to various parameters that can be adjusted to improve the segmentation.

[0094] In order to separate the ROI from the ground, all or a selection of the following steps may be performed:

[0095] 1 . Apply non-local means denoising to filter out small graininess from the image.

[0096] 2. Apply gaussian blur to filter out spatially larger graininess from the image.

[0097] 3. Apply Otsu’s method to threshold the image. [See Figure 5A]

[0098] 4. Apply morphology opening operation to get rid of non-coherent blobs and smooth the edges of the segmented regions. [See Figure 5B]

[0099] 5. Apply canny edge detection to get the outlines of the segmented regions. [See Figure 5C]

[0100] 6. Get the expected position of the ROI outlines.

[0101] 7. Define an error margin for the expected outlines. [See Figure 5D]

[0102] 8. Apply hough line detection to get the possible, straight geometric lines in the image.

[0103] 9. Consider the most probable 4 outlines, which fall within the error margin. [See Figure 5E]

[0104] 10. Adjust the outlines to mark out a rectangle. [See Figure 5F]

[0105] In order to separate the painted lines, the following steps may be performed:

[0106] 1 . Apply non-local means denoising to filter out small graininess from the image.

[0107] 2. Apply gaussian blur to filter out spatially larger graininess from the image.

[0108] 3. Apply adaptive thresholding the image. This is required, as global thresholding is proved to be insufficient to precisely segment the painted lines, as the background can have local intensity variations. [See Figure 6A]

[0109] 4. Apply morphology opening operation to get rid of non-coherent blobs and smooth the edges of the segmented regions. [See Figure 6B]

[0110] Using machine vision

[0111] After the ROI has been cropped and segmented, the next step is to divide it into smaller, individual ROIs. This operation allows for a simpler task, where further processing and analysis is performed for each individual line / bar. To accurately determine the position of each painted line / bar, the contours are extracted of the segmented image. This can be done using the contour detection algorithms, e.g., provided by OpenCV, which are optimized for efficiently identifying the outline of objects in images. By applying these algorithms to the segmented image, a set of points can be obtained that define the contour of each line / bar, which can then be used to calculate the position of each line / bar in the image. To accurately capture the details of each painted line / bar, the bounding rectangles are identified of each contour with added padding [See Figure 7A], This ensures that the surrounding details of each line / bar are included in the analysis, thereby improving the accuracy of the measurements.

[0112] Each individual line may then be processed as follows:

[0113] 1 . Apply canny edge detection to get the outlines of the painted line.

[0114] 2. Define a padding for the expected outlines. The same angle tolerance filtering is used, but instead of expected lines and regions around them, the padding of the ROI is used to separate the top, bottom, left, and right outlines [See Figure 7B]

[0115] 3. Apply hough line detection to get the possible, straight geometric lines in the image. [See Figure 7C]

[0116] 4. Consider the most probable four outlines, which fall within the padding.

[0117] The partitions can then be re-assembled, as seen in Figure 8A. As some constant properties are known about the painted lines, e.g., the fact that they are painted parallelly, makes it possible to adjust the outlines, as shown in Figure 8B. The final step is to read off the measurements and convert the pixel values into real life units. The conversion can be done in different ways. The most reliable method is to use the distance between every other line, as it is a known constant. Another method is to use the width of the painted lines, as it is also a known constant. A third method is to use the height of the calibration mat, if such has been used. Any of the three methods can result in a mm / px conversion value, which can be used to get the required measurements in real life units.

[0118] Figures 9 and 10 show which of the method steps are performed by the user and which are primarily performed by an application running on e.g., the camera device. The camera device could e.g., be a tablet computer.

Claims

Claims1 . A method for calibrating a spray tool mounted on a mobile marking robot, the method comprising:- instructing the mobile marking robot to mark lines on a predefined surface area;- taking a photo of said marked pattern of lines with a camera device; characterized in that the instruction to the mobile robot includes that said lines must be a predetermined pattern of lines, wherein the camera device is configured for providing inertial measurement unit metadata about said camera device; wherein the method further comprises:- using said inertial measurement unit metadata to create an image by rectifying the captured photo to get a projection that matches the plane of the surface on which the lines are painted;- using machine vision to recognize and measure the painted lines on the rectified image; and- using said measurements to calibrate the spray tool of the mobile marking robot.

2. The method according to claim 1 , wherein the step of taking a photo comprises guiding with the camera device a user to position the camera device in a wanted position relative to said predefined surface area.

3. The method according to claim 2, wherein said guidance comprises showing one or more guidelines in the camera view of said camera device, said guideline(s) used to define at least a part of the predefined surface area with the marked pattern of lines.

4. The method according to claim 2, wherein said guidance comprises showing three guidelines in the camera view of said camera device, one of said guidelines used to define at least a bottom or top part of the predefined surface area with the marked pattern of lines, and the two other guidelines used to define the sides edges / borders of the predefined surface area with the marked patternof lines.

5. The method according to claim 4, wherein said guidance further comprises an interactive guidance showing the user to roll and / or pitch the camera device to align the camera device to an optimized orientation relative to the preset surface area with the marked pattern of lines.

6. The method according to any one of the claims 1 -5, wherein the rectification step is followed by a preprocessing step prior to initiating the machine vision step; wherein the preprocessing step comprises:- Segmenting the rectified image to separate the predefined surface area from the ground surface;- Optionally, cropping the rectified image to the predefined surface area; and- Segmenting each of the lines defining the marked pattern of lines from the background surface.

7. The method according to claim 6, wherein the step of segmenting the rectified image to separate the predefined surface area from the ground surface comprises one or more of the following steps:1 . Apply non-local means denoising to filter out small graininess from the rectified image;2. Apply gaussian blur to filter out spatially larger graininess from the rectified image;3. Apply Otsu’s method to threshold the rectified image;4. Apply a morphology opening operation to remove non-coherent blobs and smooth the edges of the segmented regions;5. Apply canny edge detection to get the outlines of the segmented regions;6. Obtain the expected position of the predefined surface area’s outlines;7. Define an error margin for the expected outlines;8. Apply hough line detection to get the possible, straight geometric lines in the rectified image;9. Consider the most probable four outlines, which fall within the error margin; and10. Adjust the outlines to mark out a rectangle.

8. The method according to any one of the claims 6-7, wherein the step of segmenting each of the lines defining the marked pattern of lines from the background surface comprises one or more, of the following steps:1 . Apply non-local means denoising to filter out small graininess from the rectified image;2. Apply gaussian blur to filter out spatially larger graininess from the rectified image;3. Apply adaptive thresholding on the rectified image; and4. Apply a morphology opening operation to get rid of non-coherent blobs and to smooth the edges of the segmented regions.

9. The method according to any one of the claims 1 -8, wherein the machine vision step comprises isolating and identifying the marked pattern of lines from the background surface.

10. The method according to any one of the claims 1 -9, wherein the predefined surface area is a calibration mat of known shape and dimensions.

Images