Vision system for mobile robot

Abstract

A vision system for a mobile robot, the vision system comprising: a lens module; an illumination system comprising a plurality of light sources; and a lens module holder. The lens module holder is positioned around the lens module and includes a reflective surface for reflecting light emitted from the lighting system away from the lens module.

Description

[0001] This application is a divisional application of the invention patent application with application date of May 8, 2019, application number 201980035349.X, and invention name “Visual system for mobile robot”. Technical Field

[0002] The invention relates to a vision system for a mobile robot. Background Art

[0003] Mobile robots, such as vacuum cleaner robots, are increasingly employing vision systems to help them navigate and understand their environment. Using a vision system has many benefits, such as allowing the robot to better understand its environment and allowing for organized paths of travel. However, one disadvantage of using vision systems is that they do not perform well in low light conditions. As a result, mobile robots using vision systems often require lighting systems or headlights to improve the quality of the images captured by the vision system.

[0004] GB2529846 describes an example of a robot with a lighting system. In this example, a mobile robot includes a light in a handle located on the side of the robot.

[0005] Because of the many benefits of vision systems for mobile robots, improvements in vision systems as well as lighting systems for mobile robotic applications are continually sought. Summary of the invention

[0006] A first aspect of the present invention provides a vision system for a mobile robot, the vision system comprising: a lens module; an illumination system comprising a plurality of light sources; and a lens module holder. The lens module holder is positioned around the lens module and comprises a reflective surface to reflect light emitted from the illumination system away from the lens module.

[0007] As a result, the lens holder acts to direct light from the illumination away from the lens module, and thus illumination provided by the illumination system cannot directly enter the lens module, which could impair the proper functioning of the vision system.

[0008] The vision system may further include a vision system mount for mounting the vision system to the mobile robot, and the vision system mount may include a plurality of holes through which the plurality of light sources protrude.

[0009] The vision system mount may include a heat sink to draw heat away from the plurality of light sources. This may help extend the life of the light sources by reducing the risk of overheating the light sources, and also prevent heat from the light sources from potentially damaging the lens module.

[0010] The plurality of light sources may be arranged in a ring around the lens module. This is a particularly space-saving arrangement, which helps to reduce the size of the vision system.

[0011] The lens module may protrude above the lens module holder, and each of the plurality of light sources may be located below the reflective surface of the lens holder module. As a result, the lens module is not restricted by the lens module holder to obtain an optimal field of view, and a certain degree of protection is provided for the plurality of light sources to prevent the lens holder module from damaging the light sources.

[0012] The reflective surface may be a semi-parabolic toroidal mirror.

[0013] Fins may be provided between each of the plurality of light sources. The fins act to split the light generated by the plurality of light sources and thereby make the light more directional. Each fin may be provided on the vision system mount previously described.

[0014] The visual system may further include a light-transmitting cover that covers the lighting system and defines an aperture through which the top of the lens module protrudes. As a result, the light-transmitting cover protects the lighting system without covering or blocking the lighting generated by the lighting system.

[0015] The lens module holder may protrude above the upper surface of the body of the mobile robot. As a result, the lens module obtains an optimal field of view without being restricted by the body of the robot. In addition, light reflected from the reflective surface may be emitted above the upper surface of the body of the robot.

[0016] Each of the plurality of light sources may be located below the upper surface of the body of the mobile robot. This provides strong protection for the light source, preventing it from being damaged.

[0017] The lens module may include a plurality of lenses for focusing light onto the image sensor.

[0018] There are eight light sources, and each light source is independently controllable. The eight light sources allow a high level of directional control of the lighting without making the control too complex.

[0019] A second aspect of the present invention provides a mobile robot comprising the above-mentioned visual system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more easily understand the present invention, embodiments of the present invention will now be described by way of example with reference to the following drawings, in which:

[0021] Figure 1 A mobile robot is shown;

[0022] Figure 2 Shows Figure 1 Exploded view of the visual system of the mobile robot in ;

[0023] Figure 3 Shown through including Figure 2 The visual system Figure 1 A cross-sectional view of a portion of a mobile robot in FIG.

[0024] Figure 4 shows a cross-sectional view through a lens module for a vision system;

[0025] Figure 5 Shown through Figure 2 and 3 A cross-sectional view of a portion of the visual system in;

[0026] Figure 6 Shown from Figure 2 and 3 Vision system mounting parts for the vision system in;

[0027] Fig. 7A Is Figure 2 and Figure 3 A representation of a spherical image captured by a visual system, Figure 7B yes Fig. 7A The representation of the image after it is opened;

[0028] Figure 8 shows the sensor mounting on the PCB;

[0029] Fig. 9 Shows Figure 8 A lower side view of a sensor mounting member in FIG.

[0030] Fig.10 Shows passing through Fig. 9 A cross-sectional view of a sensor mounting member in FIG.

[0031] Fig.11 shows a schematic diagram of a mobile robot;

[0032] Fig.12 is a flow chart showing the input and output of a lighting algorithm for a mobile robot;

[0033] Fig.13 It shows Fig.12 A flowchart of the steps in the lighting algorithm in; and

[0034] Fig.14 It shows Fig.13 Flowchart of alternative steps in a portion of the lighting algorithm in . DETAILED DESCRIPTION

[0035] Figure 1A robot vacuum cleaner 1 is shown, comprising a main body 2 and a separation device 4. The main body 2 comprises a traction unit 5 in the form of a continuous groove track, and a cleaner head 6, which houses a brush bar, through which a dirty air flow is sucked into the robot vacuum cleaner 1 and into the separation device 4. Once the air has been cleaned of dirt in the separation device 4, the air flows out of the separation device 21 and passes through the main body 2 which houses the motor and fan that generate the air flow. The air is then discharged from the robot 1 through the air outlet. A vision system 8 protrudes from the top surface 3 of the main body 2 of the mobile robot 1. The vision system 8 comprises a camera 12 and a lighting system 14. The camera 12 is an omnidirectional camera capable of capturing a 360° view of the environment around the robot 1. The camera 12 may be, for example, a fisheye lens camera or a panoramic annular lens (PAL) camera.

[0036] The robot 1 includes a control system embodied in the software and electronics of the robot 1, which can use simultaneous localization and mapping (SLAM) technology to process images captured by the vision system 8, and allows the robot 1 to understand, learn and autonomously navigate the local environment. The control system also uses information collected from multiple other sensors (such as collision sensors) provided on the robot 1, and also includes multiple position sensors, such as position sensitive devices (PSD) and / or time of flight (ToF) sensors. The PSD and / or ToF sensors are housed in a sensor box 10 located on the side of the separation device 4. The sensor box is provided with a transparent cover, and the PSD and ToF sensors can emit and receive light (such as infrared (IR)) through the transparent cover. The sensor box 10 contains a sensor array, which is guided in different directions so that obstacles can be detected not only in front of the robot but also towards the side. Downward-facing sensors can also detect drops in the floor surface, commonly known as cliff sensors, so that the robot can stop and / or adjust its direction of travel before leaving a drop such as stairs. While some robots use a movable bumper portion as a physical contact sensor, this robot 1 has a collision sensor that detects relative movement between a separate chassis and the body of the main body 2 to register physical contact with an obstacle.

[0037] Figure 2An exploded view of a vision system 8 of a mobile robot is shown. The various parts of the vision system 8 have been separated from each other in an axial direction along an imaginary axis marked as X. The vision system 8 includes a plurality of components that form both the camera 12 and the lighting system 14, including a lens module 20, a lens module holder 22, a vision system mount 24, a printed circuit board (PCB) 30 on which an image sensor 31 is disposed, a sensor mount 32, a second PCB 28 on which a plurality of light sources (in the form of light emitting diodes (LEDs) 27) are disposed, and a light-transmitting cover 34. The lens module 20 is a fisheye lens module, and it is mounted to the image sensor 31 by a sensor mount 32. The sensor mount 32 is mounted to the PCB 30 above the image sensor 31 by screws 33. The sensor mount 32 is a mechanical component that is mounted between the image sensor 31 and the lens module 20, and its purpose is to align the lens module 20 and the image sensor 31. The image sensor 31 may be any sensor commonly used in cameras, for example, the image sensor 31 may be a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. Alternative embodiments of the sensor mounting to the sensor mount 32 will be discussed later. Figure 8 , 9 and 10 are described.

[0038] Eight LEDs 27 are arranged in a ring on a PCB 28. The PCB 28 defines an aperture through which the lens module 20 can extend so that the LEDs 27 are arranged in a ring around the lens module 20. Of course, it should be understood that the number of LEDs can be different, and the vision system can include two or more LEDs 27, depending on the lighting requirements of the vision system. The LEDs generate light in the visible portion of the electromagnetic spectrum, but in alternative embodiments, the LEDs can generate light in other regions of the spectrum, such as infrared or near infrared light.

[0039] The vision system mount 24 is used to mount the assembled vision system 8 to the mobile robot 1. The vision system mount 24 includes eight holes 26 through which eight LEDs 27 can protrude. It should be understood that the number of holes 26 can be different to match the number of LEDs 26. The vision system mount 24 also includes a plurality of fins 25, and is positioned so that the fins 25 are located between each pair of adjacent holes 26. These fins 25 act to divide the light generated by the LEDs 27, thereby making the light more directional. The vision system mount 24 is formed of a metal (such as magnesium) and acts as a heat sink to draw heat away from the LEDs 27. This helps to extend the life of the LEDs 27 by reducing the risk of overheating the LEDs 27, and also prevents heat from the LEDs 27 from damaging the lens module 20. In order to improve the efficiency of the vision system mount 24 as a heat sink, a heat sink 29 is fixed to the vision system mount 24 by a heat sink mount 24a.

[0040] The lens module holder 22 is assembled around the lens module 20 and functions to mount the lens module 20 to the vision system mount 24. A reflective surface 23 is provided below a flange that protrudes radially outward from the outside of the lens module holder 22. The reflective surface 23 forms a semi-parabolic toroidal mirror that functions to reflect light emitted from the LED 27 outwardly, as will be described in more detail below.

[0041] The light-transmissive cover 34 defines an aperture 35 through which the lens module holder 22 and the top of the lens module 20 may protrude. The light-transmissive cover 34 thus covers and protects the illumination system 14 portion of the vision system 8, but leaves the camera 12 exposed to capture the highest quality images possible.

[0042] Figure 3 A cross-section through a portion of the vision system 8 and the mobile robot is shown. The vision system 8 partially protrudes above the top surface 3 of the mobile robot 1. The image sensor 31 is positioned below the top surface 3 of the robot, and the lens module extends from the top surface of the image sensor 31 to the very top of the vision system 8. A cross-section through an example of a fisheye lens module is shown in FIG. Figure 4 A plurality of lenses 42, 44, 46, 48 are held within the lens module body 40. Figure 3 As can be seen in FIG. 2 , LED 27 is positioned below the top surface 3 of the mobile robot, which helps protect the LED and PCB from damage. However, the light emitted from LED 27 is directed upward and outward through the light-transmitting cover 35 to illuminate the environment around the mobile robot, as shown in the area marked as Y.

[0043] Figure 5 It is shown how the reflective surface 23 on the lens module holder 22 is a semi-annular parabolic reflector that acts to direct light from the LED 27 in a direction away from the lens module. As a result, light from the LED 27 cannot propagate directly into the lens module, which would effectively blind the vision system with excessive brightness. The reflective surface 23 is disposed below a flange that protrudes radially outward from the outside of the lens module holder 22, and the flange extends radially outward beyond the LED 27 so that no light from the LED 27 can propagate straight upward. If the light from the LED 27 could propagate directly upward, a bright spot would result (where the illumination is concentrated on the ceiling above the mobile robot), which would again effectively blind the vision system with excessive brightness. Alternatively, exemplary illumination rays U, V and W show how the reflective surface 23 reflects light radially outward into the environment surrounding the mobile robot 1.

[0044] Moreover, if Figure 3As shown in , since the reflective surface 23 extends slightly above the top surface 3 of the mobile robot, the light from the LED 27 can extend in a horizontal direction parallel to the top surface 3 of the robot 1. As a result, the lighting system can even illuminate areas to the side of the mobile robot.

[0045] Figure 6 A top view of the vision system mount 24 is shown, with an LED 27 protruding through each hole 26. Each LED produces light that acts to illuminate a specific lighting area AH. For example, LED 27A illuminates lighting area A, LED 27B illuminates the lighting area marked B, and so on. This segmentation effect is emphasized by the fin 25. The top of the fin 25 is shaped so that when the vision system 8 is assembled, the top of the fin follows the curved shape of the reflective surface, thereby reducing any light leakage from any specific LED outside its lighting area. Each LED 27A-H is individually controllable, so lighting can be provided in various parts of the environment only when necessary. This helps to minimize wasted power, thereby extending the battery life of the product. Extending battery life allows the mobile robot to have more operating time and cover more ground area before needing to return to the charging station for charging. The lighting areas AH together add up to a complete illuminable area that extends a full 360° around the mobile robot 1.

[0046] In some embodiments, the LEDs can be grouped together to simplify control. For example, LEDs 27A and 27B can be a first group, 27C and 27D a second group, 27E and 27F a third group, and then 27G and 27H a fourth group. These groups can then be independently controlled. This provides a quad lighting control for the robot compared to an octant lighting control (all eight LEDs 27 are individually controllable). It should be understood that other groupings and numbers of LED groups can be selected according to the requirements of the vision system and the mobile robot.

[0047] Fig. 7AA representation of a spherical image 70 captured by the vision system 8 is shown. The image 70 can be divided into a plurality of segments 72 corresponding to the LED illumination areas AH. During processing, the spherical image 70 is split and expanded as indicated by arrows Q and R, resulting in a rectangular image 74 showing the expansion of the entire 360° around the mobile robot 1. The segments 72 in the spherical image 70 then correspond to the segments 76 in the rectangular image 74. When the robot 1 is operating, the lighting system 14 can be used to illuminate any area of ​​the environment captured by the camera corresponding to the segments marked AH. For example, if the area in front of the robot is darker, such as when the robot enters a darker room from a well-lit room, the forward facing segments A and B will appear darker in the images 70 and 74. Therefore, the robot's control system can increase the power to the LEDs 27A and 27B so that they provide more illumination in the A and B segments, and the robot can continue to operate. More details about the control of the lighting system will be described below with respect to FIG. Figure 11-14 describe.

[0048] Figure 8 An alternative embodiment of a sensor mount 82 is shown which is mounted to an image sensor on a PCB 80. Figure 2 The sensor mount 32 is depicted as being mounted to the image sensor 31 and PCB 30 by screws 33. The screws help maintain the image sensor 31 and the sensor mount 32, and thus the lens module, in good alignment. Figure 8 In an alternative embodiment, no screws are used to mount the sensor mount 82 to the image sensor. As such, the sensor mount 82 of the alternative embodiment may be referred to as a passive alignment sensor mount 82. The passive alignment sensor mount 82 is provided with an engagement portion 83 that may engage with a lens or lens module. In this embodiment, the engagement portion 83 is provided with threads that are complementary to similar threads on the lens or lens module so that the lens or lens module may be screwed onto the passive alignment sensor mount 82.

[0049] The passive alignment sensor mount 82 is a mechanical component that is mounted between the image sensor and the lens or lens module. As with the previously described sensor mount 32, the purpose of the passive alignment sensor mount 82 is to align the lens or lens module with the image sensor, but in this case, the passive alignment sensor mount 82 must produce a centered focused image on the image sensor by mechanical means only and without external input (e.g., by screws). The sensor mount 82 must also allow for the relative motion required for focusing.

[0050] Fig. 9 shows a lower side view of the passive alignment sensor mount 82, and Fig.10 Shown along Fig. 9Line PP is shown passing through a cross-sectional view of the passive alignment sensor mount. An x-axis alignment is achieved due to a first deformable plastic element 84 acting on the side of the image sensor. This creates a force, indicated by arrow M, which presses the image sensor against a first vertical reference surface 86, which is incorporated into the body of the passive alignment sensor mount 82. Similarly, a y-axis alignment is achieved due to a second deformable plastic element 88 acting on the other side of the image sensor. This creates a force, indicated by arrow N, which presses the image sensor against a second vertical reference surface 89, which is also incorporated into the body of the passive alignment sensor mount 82. By pressing the image sensor against the two vertical reference surfaces 86, 89, the sensor axis is aligned with the axis of the passive sensor alignment mount 82.

[0051] Regarding the Z axis, a passive alignment sensor mount 82 is located on top of the image sensor package glass, which forms the top layer of the image sensor, which is used as a third reference plane and allows the optical axis of the lens to be aligned with the normal vector of the glass, which corresponds to the "pitch" and "yaw" axes of the lens. The "roll" axis can rotate freely for focusing. This set of reactions produces an alignment with tolerances minimized to the manufacturing tolerances of the lens, sensor mount, and image sensor package.

[0052] As previously described, the passive alignment sensor mount 82 allows the lens and image sensor to be aligned along both the x and y axes and also along the pitch and roll directions by passive mechanical means only. Another benefit of the passive alignment sensor mount 82 is that due to the elastic properties of the first and second deformable plastic elements, the passive alignment sensor mount can be attached to and detached from the image sensor without causing any damage to either component.

[0053] Fig.11 Schematic diagram of an embodiment of a mobile robot 1 is shown. The lighting control algorithm is executed in a control system 90 of the mobile robot 1. The control system 90 includes a processor 91, an image processing unit 92 and a navigation unit 93. The control system may include Fig.11 Other units not shown in the figure are used to control other systems of the mobile robot 1. For example, the control system may also include a task control unit to control the tasks performed by the mobile robot, such as a vacuum cleaning operation.

[0054] The vision system 8 includes a camera 12 and a lighting system 14. As already described, the camera 12 is capable of capturing images of the area surrounding the mobile robot 1. For example, the camera 12 is an upwardly directed camera for capturing images of the ceiling, a forward facing camera for capturing images in the forward travel direction of the robot 1, a panoramic annular lens (PAL) camera that captures a 360° field of view of the area surrounding the robot 1, or a fisheye lens camera (which is also capable of capturing a 360° field of view of the area surrounding the robot 1).

[0055] The lighting system 14 can improve the quality of the image captured by the camera 12 when the robot 1 is located in a poorly lit environment or when the image captured by the camera 12 has poor contrast. The lighting system 14 shown includes a plurality of LEDs 27A-H corresponding to the LEDs shown in the previous figures. Each of the LEDs 27A-H has a corresponding LED driver 94A-H that is used to drive power to the LED. The LEDs 27A-H emit light of any bandwidth that the camera's sensor can detect in order to improve the quality of the image captured by the camera 12. For example, the light emitted by the LEDs can be in the visible, near infrared (NIR), or infrared (IR) portion of the electromagnetic spectrum. Visible light has been found to be particularly advantageous, particularly because the robot owner can more easily determine that the lighting system is functioning correctly and effectively.

[0056] The control system 90 is also responsible for controlling the movement of the mobile robot 1 within its environment. The images and other data collected by the vision system 8 are fed into the navigation unit 93 of the control system 8. For example, the navigation unit 93 is a simultaneous localization and mapping (SLAM) unit. The SLAM unit can be used to analyze the image to find landmark features in the area around the robot 1 shown in the image. Landmark features are high contrast features that can be easily detected in the image, such as the edge of a table or the corner of a picture frame. Part or all of the analysis of the image can be performed by the image processing unit 92. The landmark features can then be used by the navigation unit 93 to triangulate and determine the position or posture of the robot in the environment. The navigation unit 93 can use information from the image and data captured from other sensors in the mobile robot 1 to create a map of the environment, which the robot 1 uses to understand and navigate the environment.

[0057] Instructions are sent from the control system 8 to the drive system 98, which causes the robot 1 to move. The drive system 98 Figure 11 is shown as including a left hand side (LHS) traction unit 99A and a right hand side (RHS) traction unit 99B. Each traction unit 99A, 99B can be independently controlled so that the robot 1 can be maneuvered. For example, if the RHS traction unit 99B is driven faster in the forward direction than the LHS traction unit 99A, the robot will turn to the left while moving forward, or as another example, if the LHS and RHS traction units 99A, 99B are driven at the same speed but in opposite directions, the robot 1 will turn in place.

[0058] The mobile robot 1 also includes a sensor system 95 that provides information to the robot 1 about the surrounding environment and the robot's posture within the environment. The sensor system 95 includes collision sensors 96 and a gyroscope system 97. The gyroscope system 97 includes an inertial measurement unit (IMU) 97A and an odometer 97B. The odometer 97B receives data from the drive system 98 to provide an indication of the distance traveled by the traction unit (for example, by using the number of rotations of the wheels). The collision sensor 96 allows the robot 1 to know when physical contact has been made with an obstacle. In response to signals from the collision sensor 96, the robot can, for example, stop and / or adjust its position and trajectory. This prevents the robot 1 from causing any damage to itself or to obstacles with which it comes into contact. The sensor system 95 may include Fig.11 Other sensors not shown in the figure, for example one or more proximity sensors, such as PSD and / or ToF sensors. The proximity sensors can give an indication of any obstacles close to the robot 1. This allows the robot 1 to avoid obstacles without coming into contact with them.

[0059] The details of how the lighting system is controlled are now described.

[0060] Fig.12 is a flow chart showing the inputs and outputs of an algorithm 100 for controlling the lighting system 14 on a mobile robot 1. The lighting algorithm has three inputs (which are used to provide LED control values ​​108): exposure time 102, an image 104 from a camera 12 and information 106 about the robot's rotation. Fig.12 A single LED control value 108 is shown as the output of the lighting algorithm 100, but it is understood that more than one LED control value 108 may be generated if desired to control multiple LEDs.

[0061] The lighting algorithm 100 controls the lighting system 14 to provide balanced light, which enables the navigation unit 93 to perform reliably in dark areas. The algorithm 100 analyzes each single frame captured by the camera 12 and evaluates the need to employ enhanced lighting from the lighting system 14. The frequency of algorithm execution is the same as the frequency of frames captured by the camera 12.

[0062] The exposure time 102 is calculated and passed to the lighting algorithm. The lighting system is activated only when the calculated exposure time 102 exceeds a certain threshold. This means that under normal lighting conditions, the lighting system is usually kept off to prevent unnecessary power waste, which would otherwise reduce the robot's operating time. The exposure time 102 is calculated by an exposure control algorithm, the details of which are outside the scope of the present invention. However, it should be understood that there are many well-known methods to calculate the exposure time, such as measurement algorithms for calculating exposure values, so for this document it is not necessary to explain in detail the exact method used.

[0063] If the robot 1 rotates, information about the robot's rotation 106 is used by the algorithm 100 to offset the light. This information is obtained from the gyroscope system 97. Without using information about the robot's rotation, the algorithm would be able to adjust the lighting accordingly, but it would take time for the algorithm to evaluate the new image frame and make the necessary adjustments. During this adjustment time, the image would appear saturated or very dark pixels and would negatively impact the robot's navigation. However, by using information about the robot's rotation, the algorithm is able to offset the light in the direction opposite to the robot's rotation to more quickly maintain the lighting in the necessary area of ​​the robot's 1 surroundings.

[0064] Fig.13 1 shows a flow chart of the steps taken during execution of the algorithm 100. The algorithm starts at 110, and the first action at 112 is to obtain a new frame 102 from the camera 12 and divide it into segments. The frame 102 is a rectangular image that is expanded, such as Figure 7B , which is expanded by the image processing unit 92. This section corresponds to the previous Figure 6 , 7A and 7B discussed segments and lighting zones AH.

[0065] In step 114, if the time period T X long time, calculated exposure time T EXP 102 is less than the threshold exposure time T THRESH , it is determined that the lighting system is not needed, and the LED power LP is reduced in step 115. The reduction of LP will cause the LED to dim to zero over time when analyzing each frame, and T EXP Keep less than T THRESH This has been chosen to give a smooth and gradual effect of lighting changes. However, an alternative approach is to immediately switch off power to the LEDs.

[0066] If the time period is longer than the predetermined time period T X long time, calculated exposure time T EXP 102 is not less than the threshold exposure time T THRESH(ie, it is greater than the threshold), then it is determined that a lighting system is necessary and the algorithm proceeds to step 116 where the next segment AH is selected as the area of ​​interest for the next step.

[0067] A threshold exposure time T of 150 milliseconds (0.15 seconds) has been found THRESH and a predetermined time period T of 3 seconds X provides the best balance. However, T in the range of 50-300 milliseconds (0.05 to 0.3 seconds) THRESH Can be selected, and T in the range of 1-5 seconds X Can be selected. If the robot requirements require, these ranges are used for T THRESH and T X Other values ​​of may be chosen.

[0068] The average brightness of the selected segment is then calculated in step 118. The average brightness is calculated using the median average of the pixel intensities in the segment.

[0069] The average intensity or brightness for each segment is calculated using the following formula 1, where sw is the segment width and sh is the segment height. For example, in a 1024×128 image with eight segments, sw=128 and sh=128, or in a different embodiment, in a 1024×128 image with four segments, sw=256 and sh=128. This formula calculates the sum of the intensities of all pixels and then divides it by the total number of pixels in the segment.

[0070]

[0071] (Formula 1)

[0072] The average intensity calculated in Formula 1 is then classified as average brightness using Formula 2 below. Two predetermined parameters a and b are used to determine the brightness of an image. If the average intensity of a segment is between a and b, then the brightness of that segment is considered normal. If the average intensity of a segment is above b, that segment is considered bright. Finally, if the average intensity of a segment is less than a, that segment is considered dark.

[0073]

[0074] (Formula 2)

[0075] In step 120, if the classified average brightness of the selected segment is "normal", the algorithm proceeds directly to step 122, where a check is made to determine if there are any remaining segments in the frame to be processed. If there are still remaining segments, the algorithm returns to step 116 and the next segment in the frame is selected as the region of interest and the brightness calculation and classification steps are repeated.

[0076] Alternatively, if the categorized average brightness of the selected zone is "dark" in step 120, the algorithm instead proceeds to step 121A where the LED power LP of the LEDs used to illuminate that zone is increased. This is done by sending an increased LP value to the LED driver 94 of that LED 27. The algorithm then proceeds to step 122 as previously described.

[0077] Alternatively, if the categorized average brightness of the selected zone is "bright" in step 120, the algorithm instead proceeds to step 121B, in which the LED power LP of the LED used to illuminate that zone is reduced. This is implemented by sending a reduced LP value to the LED driver 94 of that LED 27. The algorithm then proceeds to step 122 as previously described.

[0078] Once all segments in the frame have been processed, at step 122 the algorithm then proceeds to step 124 which determines whether the robot 1 has rotated using the robot rotation information 106 obtained from the gyroscope system 97. If the robot has not rotated, the algorithm returns to step 112, obtains a new frame 104 from the camera and repeats the process. However, if the robot has rotated, the algorithm proceeds to step 126 and offsets the LED power (LP) value in the direction opposite to the robot rotation by the number of segments appropriate to the detected robot rotation angle. For example, if there are eight LED segments, then for every 45° rotation opposite to the direction of rotation, the algorithm offsets the associated LP value by 1 segment. Alternatively, if there are four LED segments, then for every 90° rotation opposite to the direction of rotation, the algorithm offsets the associated LP value by 1 segment.

[0079] For example, take Fig. 7A and 7B The image segments shown in Figure 6 The first row of Table 1 below shows the LP value for each LED A-H in a first robot pose, with reference to the corresponding LEDs 27A-H in the robot 1. In this pose, there is a dark area in front of the robot 1, so the LEDs 27A and 27B directed toward the front of the robot 1 are fully illuminated, and the LEDs 27H and 27C directed toward the right and left peripheries of the front of the robot are slightly illuminated. Then, the second row of Table 1 shows the LP value for each LED A-H in a second robot pose, where the robot has been rotated 90° in a clockwise direction, while all other factors remain unchanged. After rotating 90° clockwise, the LEDs 27C and 27D directed toward the left-hand side of the robot are now facing the dark area, and are therefore illuminated because the algorithm offsets the LP value by two segments counterclockwise.

[0080] led 27A 27B 27C 27D 27E 27F 27G 27H Posture 1 0.3 0.3 0.1 0 0 0 0 0.1 Posture 2 0 0.1 0.3 0.3 0.1 0 0 0

[0081] Table 1

[0082] Of course, if the robot rotates slowly enough, the algorithm does not need to offset the light, as the lighting can be adequately adjusted by the brightness calculation. In this way, rotation is only considered when the rotation speed is faster than a predetermined threshold.

[0083] Once the LP value has been offset by the desired amount by the algorithm in step 126, the algorithm returns to step 112, a new frame 104 is obtained from the camera and the process is repeated.

[0084] use Fig.13 The algorithm of the steps listed in the flowchart in may result in some slight "oscillations" in the light generated by the lighting system. Oscillations may occur when the illumination produced by the lighting system changes rapidly. For example, when entering a dark area from a bright environment, a rapid increase in illumination may cause the image to appear too bright. To address this, the algorithm will reduce the illumination, which in turn will cause the image to appear too dark, and so on. To prevent this oscillation effect, there is a slight delay before the illumination is changed underwater to ensure that the detected change in ambient light is continuous and continuous. Additionally, the lighting transition should be slightly slowed down and smoothed. However, it is necessary for the robot to react quickly enough to changes in ambient light so that its navigation system can continue to operate unimpeded.

[0085] Fig.14 Shows Fig.13 Flowchart of alternative steps in a portion of the lighting algorithm that solves the oscillation problem. Steps 118 and 120 are Fig.13 The method is the same as in , but the subsequent steps are different. Light increase request (LIR) and light decrease request (LDR) are introduced into this method.

[0086] In the event that the average brightness is classified as "normal" in step 120, the algorithm proceeds to step 130, where the values ​​of LIR and LDR are reset to zero and the LED power (LP) remains unchanged. Fig.13 The algorithm is repeated for each image segment in the same manner as shown in Fig.14 This is indicated as step 160 in order to simplify the figure.

[0087] In case the average brightness is classified as "dark" in step 120, the algorithm proceeds to step 140 where the LIR value for the currently selected zone is increased. In this example, the LIR value is increased by 1. Then, in step 142, if the condition is met that the brightness is "dark" and the current LIR value is above a predetermined threshold value thr LIR, the LED power LP for this segment is increased (LP=LP+1) and the LIR value is reset to zero (LIR=0). The algorithm is then repeated in step 160 for each image segment.

[0088] In case the average brightness is classified as "bright" in step 120, the algorithm proceeds to step 150, in which the LDR value for the currently selected segment is increased. In this example, the LDR value is increased by 1. Then, in step 152, if the condition is met that the brightness is "bright" and the current LDR value is above a predetermined threshold value thr LDR , the LED power LP for this segment is reduced (LP=LP-1) and the LDR value is reset to zero (LIR=0). The algorithm is then repeated in step 160 for each image segment.

[0089] Once all segments of the current frame or image have been processed, the algorithm proceeds to step 162 where the segments are processed in accordance with the Fig.13 The LP values ​​are adjusted for any rotation of the robot in the same manner as described in steps 124 and 126 of FIG. 1. Once the LP values ​​have been adjusted for robot rotation, the algorithm proceeds to step 164 where all LP values ​​are sent to each corresponding LED driver. The algorithm then returns again to step 112 of the algorithm and a new frame is acquired.

[0090] So, in summary, the following formula 3 shows how to understand the brightness in the image. If the image is dark, then LIR will be incremented. If the brightness is normal, both light requests LIR and LDR will be reset. If the image is bright, then LDR will be incremented.

[0091]

[0092] (Formula 3)

[0093] By having light change requests, the transition between changes in light levels will be smooth. Additionally, the requirement for the LIR and LDR values ​​to exceed a threshold before a lighting change occurs causes a delay in applying the change. By using this approach in the algorithm, the light will not oscillate.

[0094] Another benefit of this approach is that the threshold thr for increasing / decreasing the number of requests can be changed LIR and thr LDR Easily adjust the rate at which the lighting changes. A higher threshold will decrease the rate of change, while a lower threshold will increase the rate of change.

[0095] It is worth noting that in the above algorithm, changes in LED power are described as +1 or -1. This corresponds to a step in the light change. A step in the light change can be defined as the smallest amount of change that can be applied to the LED driver. It will be easy to understand that this depends on the LED driver and the electrical characteristics. For example, if the LED light is in the range of 0%-100%, a step can be defined as a 1% change. Step changes can be defined differently depending on the requirements of the robot or the needs of the user. The light level step is a balance between the required light output and the smoothness of the transition. The light step should be small enough so that the user does not notice the change in light, but should also be fast enough so that the lighting can reach the maximum light level in an acceptable time for image processing.

[0096] Alternatively, instead of incremental changes in LED power, if a larger change in LP is to be calculated, it may be sent to the LED driver in discrete levels to give the user the impression of a smooth increase.

[0097] Although particular embodiments have been described, it will be appreciated that various modifications, including those described above, may be made without departing from the scope of the invention as defined by the claims.

Claims

1. A visual system for a mobile robot, the visual system comprising: Lens module; A lighting system comprising a plurality of light sources; as well as A lens module holder; wherein a lens module holder is positioned around the lens module and includes a reflective surface to reflect light emitted from the lighting system away from the lens module, Wherein, the fin is arranged between each of the plurality of light sources.

2. The vision system of claim 1, further comprising a vision system mount for mounting the vision system to a mobile robot, the vision system mount comprising a plurality of holes through which the plurality of light sources protrude.

3. The visual system according to claim 2, wherein: The vision system mount includes a heat sink to draw heat away from the plurality of light sources.

4. A visual system according to any one of the preceding claims, wherein: The plurality of light sources are arranged in a ring shape around the lens module.

5. A visual system according to any one of the preceding claims, wherein: The lens module protrudes above the lens module holder, and each of the plurality of light sources is located below a reflective surface of the lens holder module.

6. A visual system according to any one of the preceding claims, wherein: The reflecting surface is a semi-parabolic toroidal mirror.

7. The visual system according to claim 1, wherein: Each fin is disposed on the vision system mount of claim 2.

8. The visual system according to any one of the preceding claims, further comprising a light-transmissive cover.

9. The visual system according to claim 8, wherein: The light-transmissive cover covers the lighting system and defines an aperture through which a top portion of the lens module protrudes.

10. A visual system according to any one of the preceding claims, wherein: The lens module holder protrudes above an upper surface of a body of the mobile robot.

11. A visual system according to any one of the preceding claims, wherein: Each of the plurality of light sources is located below an upper surface of a body of the mobile robot.

12. A visual system according to any one of the preceding claims, wherein: The lens module includes a plurality of lenses for focusing light onto an image sensor.

13. A visual system according to any one of the preceding claims, wherein: There are eight light sources, and each is independently controllable.

14. A mobile robot comprising a vision system according to any one of the preceding claims.

Images