Mobile Robots

Abstract

A mobile robot performs obstacle avoidance, positioning and object recognition based on image frames acquired by the same optical sensor. The mobile robot comprises an optical sensor, a light emitting diode, a laser diode and a processor. The processor determines obstacles and measures distance based on image frames acquired by the optical sensor when the laser diode is lit. The processor also performs positioning and object recognition based on image frames acquired by the optical sensor when the light emitting diode is lit.

Description

Technical Field

[0001] The present invention relates to a mobile robot, in particular to a mobile robot capable of performing obstacle avoidance, positioning and object recognition according to image frames acquired when the same light sensor is illuminated relative to different light sources. Background Art

[0002] Smart home is one of the links in the development of smart city, and cleaning robot has almost become one of the essential electronic products of smart home. Generally speaking, cleaning robot is equipped with multiple functions to enhance user experience, including map construction of operation range, obstacle detection and avoidance during operation, etc. Current cleaning robots also include multiple detectors to perform these different detection functions.

[0003] For example, the cleaning robot includes a sensor configured on the top surface, which realizes visual localization and mapping (VSLAM) by acquiring the upper image of the path of the cleaning robot. In addition, the cleaning robot also includes a front sensor, which realizes obstacle detection and avoidance functions by acquiring the front image of the direction of travel of the cleaning robot.

[0004] That is, previous cleaning robots needed to include multiple sensors to achieve different detection functions.

[0005] In view of this, the present invention provides a mobile robot that performs obstacle avoidance, positioning, and object recognition based on image frames acquired when the same light sensor is illuminated relative to different light sources. Summary of the invention

[0006] The present invention provides a mobile robot, which avoids obstacles according to image frames acquired by a light sensor when a laser diode emits light, and performs visual positioning and map construction according to image frames acquired by the light sensor when a light emitting diode emits light.

[0007] The present invention also provides a mobile robot, which determines a key area based on an image frame obtained by a light sensor when a laser diode emits light, and performs object recognition in the key area of ​​the image frame obtained by the light sensor when the light-emitting diode emits light, so as to reduce the amount of calculation, reduce energy consumption and improve the recognition rate.

[0008] The present invention provides a mobile robot comprising a first light source, a second light source, a third light source, a light sensor and a processor. The first light source is used to project a horizontal light segment toward a traveling direction during a first period. The second light source is used to project a vertical light segment toward the traveling direction during a second period. The third light source is used to illuminate a front area in the traveling direction during a third period. The light sensor is used to acquire a first image frame, a second image frame and a third image frame during the first period, the second period and the third period, respectively. The processor is electrically coupled to the first light source, the second light source, the third light source and the light sensor. The processor is used to perform distance measurement according to the first image frame and the second image frame and to perform visual positioning and map construction according to the third image frame.

[0009] The present invention also provides a mobile robot comprising a first light source, a second light source, a pixel array and a processor. The first light source is used to project a horizontal light segment toward the direction of travel during a first period. The second light source is used to project a vertical light segment toward the direction of travel during a second period. The pixel array comprises a plurality of first pixels and a plurality of second pixels, wherein the plurality of first pixels receive incident light through an infrared filter, but the plurality of second pixels do not receive incident light through any filter. The pixel array is used to acquire a first image frame, a second image frame and a third image frame during the first period, the second period and a third period between the first period and the second period, respectively. The processor is electrically coupled to the first light source, the second light source and the pixel array, and is used to perform distance measurement according to the first image frame and the second image frame, and to perform visual positioning and map construction according to the pixel data related to the plurality of second pixels in the third image frame.

[0010] The present invention also provides a mobile robot comprising a first light source, a second light source, a third light source, a light sensor and a processor. The first light source is used to project a horizontal light segment toward the direction of travel during a first period. The second light source is used to project a vertical light segment toward the direction of travel during a second period. The third light source is used to illuminate the area in front of the direction of travel. The light sensor is used to acquire a first image frame and a second image frame during the first period and the second period, respectively. The processor is electrically coupled to the first light source, the second light source, the third light source and the light sensor, and is used to determine an obstacle based on the first image frame and the second image frame. When the obstacle is determined, the processor controls the third light source to light up during a third period and controls the light sensor to acquire a third image frame during the third period, determines a key area in the third image frame based on the position of the obstacle, and uses a learning model to identify the object type of the obstacle in the key area.

[0011] The present invention also provides a mobile robot comprising a first laser light source, a second laser light source, a light emitting diode light source and a light sensor. The first laser light source is used to project a horizontal light segment toward a traveling direction during a first period. The second laser light source is used to project a vertical light segment toward the traveling direction during a second period. The light emitting diode light source is used to illuminate a front area in the traveling direction during a third period. The light sensor is used to acquire a first image frame, a second image frame and a third image frame during the first period, the second period and the third period, respectively.

[0012] In the embodiment of the present invention, the mobile robot can realize multiple detection functions by using only a single optical sensor in combination with different light sources for time-sharing operation.

[0013] In order to make the above and other purposes, features and advantages of the present invention more obvious, the following will be described in detail with reference to the accompanying drawings. In addition, in the description of the present invention, the same components are represented by the same symbols, which are hereby described together. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A is a schematic diagram of a mobile robot according to an embodiment of the present invention;

[0015] Figure 1B is a block diagram of components of a mobile robot according to an embodiment of the present invention;

[0016] Figure 2 is a running timing diagram of the mobile robot according to the first embodiment of the present invention;

[0017] Figure 3 is a schematic diagram of a pixel array of a mobile robot according to an embodiment of the present invention;

[0018] Figure 4 is an operating timing diagram of a mobile robot according to a second embodiment of the present invention;

[0019] Figure 5 is a flow chart of an operating method of a mobile robot according to a second embodiment of the present invention;

[0020] Fig. 6A is a schematic diagram of an image frame related to a first light source acquired by a light sensor of a mobile robot according to an embodiment of the present invention;

[0021] Figure 6B Schematic diagram of an image frame related to a second light source acquired by the optical sensor of the mobile robot according to an embodiment of the present invention.

[0022] Description of reference numerals:

[0023] 100 Mobile Robot

[0024] 11 Light Sensor

[0025] 13 Processor

[0026] LS1 First Light Source

[0027] LS21, LS22 Second light source

[0028] LS3 Third Light Source

[0029] 15 Filters DETAILED DESCRIPTION

[0030] The mobile robot of the embodiment of the present invention uses a single light sensor with different light sources to operate. The line light source is used to find obstacles and measure the distance of obstacles as a basis for the robot to turn. The lighting light source is used to illuminate the area ahead of the travel direction for visual positioning, map construction and object recognition.

[0031] Please refer to Figure 1A As shown, it is a schematic diagram of a mobile robot 100 according to an embodiment of the present invention. Figure 1A The mobile robot 100 is shown to be a cleaning robot, but the present invention is not limited thereto. The mobile robot 100 can be any electronic robot that moves according to the image acquisition result to carry, communicate, guide, etc.

[0032] Please also refer to Figure 1B As shown, it is a block diagram of a mobile robot 100 according to an embodiment of the present invention. The mobile robot 100 comprises a first light source LS1, a second light source LS21 and LS22, a third light source LS3, a light sensor 11, and a processor 13. The processor 13 is, for example, an application specific integrated circuit (ASIC) or a microprocessor (MCU), which implements its functions using software, hardware, and / or firmware. Figure 1B Two second light sources are shown for illustration only and not for limiting the present invention. The mobile robot 100 may include only a single second light source.

[0033] The first light source LS1, for example, includes a laser light source and a diffractive optical component, and the diffractive optical component is used to generate horizontal projection light after the emitted light of the laser light source passes through, so that the first light source LS1 projects a horizontal light segment toward a traveling direction. The traveling direction is, for example, toward a side where the first light source LS1, the second light sources LS21 and LS22, the third light source LS3, and the optical sensor 11 are arranged.

[0034] The second light sources LS21 and LS22, for example, respectively include a laser light source and a diffractive optical component, which is used to generate vertical projection light when the emitted light of the laser light source passes through, so that the second light sources LS21 and LS22 respectively project vertical light segments toward the traveling direction.

[0035] In the present invention, the laser light source is, for example, an infrared laser diode (IR LD).

[0036] The third light source LS3 is, for example, an infrared light emitting diode (LED), which is used to illuminate the area ahead of the traveling direction. The illumination range of the third light source LS3 is preferably greater than or equal to the field of view of the light sensor 11. In the present invention, when the third light source point LS3 is lit, the first light source LS1 and the second light sources LS21 and LS22 are extinguished.

[0037] Please refer to Figure 2 , which is an operation timing diagram of the mobile robot 100 according to the first embodiment of the present invention. The first light source LS1 projects a horizontal light segment toward the traveling direction during the first period T1. The second light sources LS21 and LS22 project a vertical light segment toward the traveling direction during the second period T2. The third light source point LS3 illuminates the front area in the traveling direction during the third period T3.

[0038] The optical sensor 11 is, for example, a CCD image sensor or a CMOS image sensor, which acquires the first image frame, the second image frame, and the third image frame at a sampling frequency in the first period T1, the second period T2, and the third period T3, respectively. When the first image frame contains an obstacle, the first image frame will appear as follows: Fig. 6A When the first image frame does not contain an obstacle, the first image frame only contains continuous (no broken lines) horizontal straight lines. When the second image frame contains an obstacle, the second image frame will appear as shown in Figure 6B At least one broken line shown, wherein the angle of the broken line depends on the shape of the obstacle and is not limited to Figure 6B However, when the second image frame does not contain an obstacle, the second image frame only contains two continuous (no broken lines) inclined straight lines. It can be understood that Fig. 6A and 6B It is only used for illustration and is not intended to limit the present invention.

[0039] It can be understood that, since the second light sources LS21 and LS22 project two parallel light segments on the traveling surface, in the second image frame acquired by the light sensor 11, the two parallel light segments appear as inclined straight lines. Figure 6B Only the projected light segments on the traveling surface detected by the optical sensor 11 are displayed. When there is a wall in front of the mobile robot 100, two vertical light segments projected by the second light sources LS21 and LS22 will appear at the top in the second image frame.

[0040] The position of the broken line in the image frame reflects the position of the obstacle in front of the mobile robot 100. As long as the relationship between the position of the broken line in the image frame and the actual distance of the obstacle is recorded in advance, when an image frame containing the broken line is obtained, the distance between the mobile robot 100 and the obstacle can be obtained.

[0041] like Fig. 6A As shown, the processor 13 knows that the first light source LS1 projects a horizontal light segment at a predetermined distance in front of the mobile robot 100. According to triangulation, when the image of the horizontal light segment is broken, the processor 13 can calculate the distance and width of the obstacle.

[0042] like Figure 6B As shown, the processor 13 knows that the second light sources LS21 and LS22 project vertical light segments in front of the mobile robot 100. According to triangulation, when at least one broken line appears in the image of the vertical light segment, the processor 13 can calculate the distance and height of the obstacle based on the position and length of the broken line in the image of the vertical light segment (i.e., the inclined straight line).

[0043] The processor 13 is electrically coupled to the first light source LS1, the second light sources LS21 and LS22, the third light source LS3 and the light sensor 11, and is used to control the light source on and off and image acquisition. The processor 13 also generates a signal based on the first image frame (eg Fig. 6A ) and the second image frame (eg Figure 6B ) and perform visual localization and mapping (VSLAM) based on the third image frame (including the actual acquired object image), wherein the detailed implementation of VSLAM is known and will not be described here. The present invention is that the processor 13 performs different detections based on the image frames acquired when the same light sensor 11 is illuminated relative to different light sources.

[0044] Please refer to Figure 2 As shown, the light sensor 11 also acquires a first dark image frame during a first light source off period Td1 after the first period T1, which is used to perform a differential with the first image frame. The light sensor 11 also acquires a second dark image frame during a second light source off period Td2 after the second period T2, which is used to perform a differential with the second image frame. For example, the processor 13 subtracts the first dark image frame from the first image frame and subtracts the second dark image frame from the second dark image frame to eliminate background noise.

[0045] Although Figure 2The first light source off-period Td1 is shown after the first period T1 and the second light source off-period Td2 is shown after the second period T2, but the present invention is not limited to this. In other embodiments, the first light source off-period Td1 is configured before the first period T1 and the second light source off-period Td2 is configured before the second period T2. In another embodiment, the light sensor 11 only captures one dark image frame (for example, before T1, between T1 and T2, or after T2) in each cycle (the period during which each light source is sequentially lit). The processor 13 subtracts the dark image frame from the first image frame and subtracts the dark image frame (the same dark image frame) from the second image frame, which can also eliminate background noise and improve the overall frame rate.

[0046] In one embodiment, the light sensor 11 includes a pixel array, and all pixels of the pixel array receive incident light through an infrared filter. Figure 1B An infrared filter 15 is also arranged in front of the display light sensor 11. The infrared filter 15 may be an optical element located in front of the pixel array (eg coated on a lens), or directly arranged on each pixel of the pixel array.

[0047] In another embodiment, the pixel array of the light sensor 11 includes a plurality of first pixels P IR and a plurality of second pixels P mono ,like Figure 3 As shown. The first pixel P IR is an infrared pixel, that is, the pixel receives incident light through an infrared filter or film. The second pixel P mono The second pixel P receives the incident light without passing through the infrared filter or film. mono It is preferred that the incident light is received without passing through any filter element. The incident light is the reflected light of the floor, wall, object, etc. in front of the mobile robot 100.

[0048] In an embodiment including two types of pixels, the first image frame and the second image frame are composed of a plurality of first pixels P IR That is, the processor 11 only generates pixel data based on the plurality of first pixels P IR The third image frame is composed of a plurality of first pixels P IR and a plurality of second pixels P mono The pixel data generated together constitutes that when the third light source LS3 is turned on, the first pixel P IR and the second pixel P mono The processor 13 is configured to process corresponding pixel data relative to the lighting of different light sources.

[0049] In one embodiment, a plurality of first pixels P of the pixel array IRand a plurality of second pixels P mono Yes Figure 3 In other embodiments, the first pixel P IR and the second pixel P mono It can be arranged in other ways, for example, the left half or the upper half of the pixel array is the first pixel P IR The right half or lower half of the pixel array is the second pixel P mono , but the present invention is not limited thereto.

[0050] At the first pixel P IR and the second pixel P mono In the embodiment of the chessboard arrangement, the processor 11 performs pixel interpolation operation on the first image frame and the second image frame before calculating the object distance, so as to obtain the pixel P of the first image frame and the second image frame relative to the second pixel P. mono After the interpolation data is filled in the position, the distance measurement operation is performed.

[0051] When the pixel array of the optical sensor 11 is arranged in a chessboard pattern, the mobile robot 100 of the embodiment of the present invention may also operate in other ways to increase the frame rate of ranging and positioning (using VSLAM). Figure 2 In the embodiment of the present invention, the frame rates of the ranging and positioning are both 1 / 5 of the sampling frequency of the optical sensor 11 .

[0052] For example, refer to Figure 4 As shown, it is an operation timing diagram of the mobile robot 100 according to the second embodiment of the present invention. The first light source LS1 projects a horizontal light segment toward the traveling direction during the first period T1. The second light sources LS21 and LS22 project vertical light segments toward the traveling direction during the second period T2.

[0053] The pixel array of the light sensor 11 acquires the first image frame, the second image frame and the third image frame in the first period T1, the second period T2 and the third period T3 between the first period T1 and the second period T2, respectively. That is, when the pixel array of the light sensor 11 acquires the third image frame, all light sources are turned off. Figure 4 In FIG. 1 , the third period T3 is shown as a diagonal rectangular area.

[0054] The processor 13 performs distance measurement (including finding obstacles and calculating distance) according to the first image frame and the second image frame, wherein the first image frame and the second image frame are composed of a plurality of first pixels P IR That is, when the first light source LS1 and the second light sources LS21 and LS22 are turned on, the first pixel P IR The relevant pixel data will not be affected by other color lights, so the processor 13 only processes the first pixels PIR The generated pixel data is used to measure distance.

[0055] At this time, the third image frame is composed of a plurality of second pixels P mono The generated pixel data.

[0056] Similarly, the processor 11 also generates a pixel P corresponding to the first pixel P in the first image frame and the third image frame according to the pixel P corresponding to the first pixel P in the first image frame and the third image frame. IR The pixel data related to the first pixel P is differentially calculated, and the pixel data related to the first pixel P is differentially calculated according to the pixel data related to the second pixel P and the third pixel P. IR The relevant pixel data are differentiated to eliminate background noise.

[0057] Similarly, when the first pixel P IR and the second pixel P mono When the pixels are arranged in a chessboard pattern, the processor 11 performs pixel interpolation operation on the first image frame and the second image frame before performing distance measurement, so as to obtain the pixel P of the first image frame and the second image frame. mono After the interpolation data is filled in the position, the distance measurement operation is performed.

[0058] In the second embodiment, the processor 13 generates a pixel P corresponding to the second pixel P in the third image frame. mono In this embodiment, the third light source LS3 is not turned on (the third light source LS3 may not be included in this case), and due to the plurality of first pixels P IR The generated pixel data has excluded components other than infrared light, so the third image frame of this embodiment is only composed of a plurality of second pixels P mono In addition, before performing VSLAM according to the third image frame, the processor 13 may also perform pixel interpolation operation on the third image frame to obtain pixel data corresponding to the first pixel P in the third image frame. IR The interpolated data is filled in the position.

[0059] according to Figure 4 It can be seen that the frame rate of ranging is increased to 1 / 4 of the sampling frequency of the optical sensor 11 (eg, the frame period includes T1+T2+2×T3), while the frame rate of VSLAM is increased to 1 / 2 of the sampling frequency of the optical sensor 11 .

[0060] However, when the ambient light is insufficient, not lighting the third light source LS3 may cause the processor 13 to be unable to correctly perform VSLAM based on the third image frame. To solve this problem, the processor 11 also identifies the ambient light intensity based on the third image frame, for example, by comparing it with a brightness threshold. When the ambient light is identified as a low-light environment based on the third image frame, the processor 11 also changes the lighting sequence of the first light source LS1 and the second light sources LS21 and LS22. For example, the processor 11 controls the light source lighting and image acquisition to be changed as follows: Figure 2 That is, when in a strong light environment (for example, the average brightness of the third image frame is greater than the brightness threshold), the mobile robot 100 Figure 4 When in a low light environment (for example, the average brightness of the third image frame is less than the brightness threshold), the mobile robot 100 operates according to Figure 2 Timing operation.

[0061] The present invention also provides a mobile robot that performs distance measurement and obstacle recognition based on the image obtained by the same optical sensor 11. When the obstacle is determined to be a specific object, such as a wire, a sock, etc., the mobile robot 100 directly passes over the obstacle; and when the obstacle is determined to be an electronic device, such as a mobile phone, etc., the mobile robot 100 dodges and does not pass over the obstacle. Whether the obstacle can be directly passed over can be determined in advance according to different applications.

[0062] The mobile robot 100 of this embodiment is also Figure 1A and Figure 1B As shown, it includes a first light source LS1, a second light source LS21 and LS22, a third light source LS3, a light sensor 11 and a processor 13. For example, referring to Figure 4 As shown, the first light source LS1 projects horizontal light segments toward the traveling direction during the first period T1, and the second light sources LS21 and LS22 project vertical light segments toward the traveling direction during the second period T2. The third light source LS3 is used to illuminate the front area in the traveling direction.

[0063] As mentioned above, in order to eliminate the influence of ambient light, the light sensor 11 is also used during the first light source off period (eg, Figure 4 During the T3 period of the second period, the first dark image frame is obtained for differentiation with the first image frame; and during the second light source off period before or after the second period T2 (for example Figure 4 The light sensor 11 obtains the first image frame and the second image frame in the first period T1 and the second period T2 respectively.

[0064] In this embodiment, the pixel array of the light sensor 11 receives incident light through, for example, a filter 15 .

[0065] The processor 13 determines the obstacle according to the first image frame and the second image frame. The method of determining the obstacle has been described above and will not be repeated here. After finding the obstacle, the processor 13 controls the third light source LS3 to be in the third period (for example Figure 2 During T3 of the third period, the optical sensor 11 is turned on and controlled to acquire a third image frame during the third period.

[0066] In this embodiment, before the processor 13 determines that an obstacle appears, the third light source LS3 is not lit, so the running sequence of the mobile robot 100 is as follows: Figure 4 As shown. When the processor 13 determines that an obstacle appears, it controls the third light source LS3 to light up and controls the light sensor 11 to obtain a third image frame during the period when the third light source LS3 is lit. In other embodiments, multiple third image frames can be obtained. The present invention uses one third image frame as an example for explanation. In this embodiment, the third image frame is mainly used for object recognition by a pre-trained learning model.

[0067] After receiving the third image frame from the optical sensor 11, the processor 13 determines the key area ROI in the third image frame according to the position of the obstacle (eg, the disconnection position), such as Fig. 6A and Figure 6B As shown in the figure, since the present invention uses a single optical sensor, after the processor 13 determines the position of the obstacle and determines the key area ROI according to the first image frame and the second image frame, the key area ROI can directly correspond to the relative area in the third image frame.

[0068] In a non-limiting embodiment, the key region ROI has a predetermined image size. That is, when the position (such as the center or center of gravity, but not limited to) of an obstacle is determined, the processor 13 determines the key region ROI with a predetermined size at the position.

[0069] In another embodiment, the size of the key region ROI is determined by the processor 11 according to the first image frame and the second image frame. In this case, the larger the obstacle, the larger the key region ROI; otherwise, the smaller the key region ROI.

[0070] The processor 13 then uses a pre-trained learning model (e.g., built into the processor 11 by ASIC or firmware) to identify the object type of the obstacle in the key region ROI. Since the learning model does not identify (e.g., does not calculate convolution) the area outside the key region ROI in the third image frame, the amount of calculation, time spent, and power consumption during identification can be effectively reduced. At the same time, since the key region ROI contains a small number of object images, it is less likely to be interfered by other objects during identification, which can improve the recognition accuracy.

[0071] In addition, in order to further improve the recognition rate, the processor 11 also determines the height of the obstacle according to the second image frame, for example, Figure 6B The longitudinal length H of the interruption line is used as the height of the obstacle. The learning model also identifies the type of the object based on the height.

[0072] In one embodiment, the object height is used as training material together with the ground truth image during the training phase, and the learning model is generated using a data network architecture (for example, including a neural network learning algorithm, a deep learning algorithm, but not limited to this).

[0073] In another embodiment, during the training phase, the data network architecture uses only real images as training material to generate the learning model. In operation, when the learning model calculates the probability of several objects based on the third image frame, it is filtered by the height. For example, if the height of the object type classified by the learning model exceeds the height determined based on the second image frame, the learning model will exclude it even if the object type has the highest probability.

[0074] The use of learning models to classify objects in images is known, so it will not be repeated here. At the same time, the method of matching the learning model with the height of the object to identify obstacles is not limited to the one cited in the present invention.

[0075] In one embodiment, since the image capturing frequency of the optical sensor 11 is higher than the moving speed of the mobile robot 100, the processor 11 further controls the first light source LS1, the second light source LS21 and LS22, and the third light source LS3 to be turned off for a predetermined period after the third period T3 (i.e., obtaining a third image frame) until the obstacle leaves the projection range of the first light source LS1, so as to avoid repeated identification of the same obstacle. The predetermined period can be determined, for example, according to the moving speed of the mobile robot 100 and the height determined according to the second image frame.

[0076] Please refer to Figure 5 As shown, it is a flow chart of the operation method of the mobile robot 100 of an embodiment of the present invention, which includes the following steps: turning on the line light source to detect obstacles (step S51); judging whether the obstacle exists (step S52); when the obstacle does not exist, returning to step S51 to continue detection; and when the obstacle exists, turning on the illumination light source and acquiring the third image frame (step S53); determining the key area in the third image frame (step S54); and identifying the type of object with the learning model (steps S55 to S56). This embodiment also selectively includes detecting the height of the object to assist in identifying the type of the object (step S57).

[0077] In this embodiment, the line light source includes, for example, the first light source LS1 and the second light sources LS21 and LS22. The illumination light source includes, for example, the third light source SL3. It can be understood that: Figure 1A The positions of the light sources shown are merely examples and are not intended to limit the present invention.

[0078] Step S51: The processor 13 controls the first light source LS1 and the second light sources LS21 and LS22 to Figure 4 The processor 13 controls the light sensor 11 to obtain the first image frame and the second image frame in the first period T1 and the second period T2 respectively.

[0079] Step S52: When the processor 13 determines that the first image frame contains Fig. 6A The broken line or second image frame shown contains Figure 6B If the line shown is broken, it is determined that there is an obstacle ahead. The program then enters step S53. On the contrary, when the processor 13 determines that both the first image frame and the second image frame do not contain a broken line, it returns to step S51 to continue detecting obstacles.

[0080] When it is determined that the first image frame or the second image frame includes a broken line, the processor 13 further records (for example, in a memory) the position of the broken line as the position of the object.

[0081] Step S53: The processor 13 then controls the third light source SL3 to light up, for example Figure 2 The processor 13 controls the optical sensor 11 to obtain a third image frame during the third period T3, which includes at least one object image. In an embodiment in which the processor 13 recognizes an object based on a single image, the processor 13 only controls the third light source SL3 to turn on for a third period T3. In one embodiment, after the third period T3, the processor 13 controls the first light source LS1 and the second light sources LS21 and LS22 to return to the state of being on. Figure 4 In another embodiment, after the third period T3, the processor 13 controls all light sources to stop emitting light for a predetermined period to avoid repeated detection of the same obstacle, and then returns to the state of being illuminated. Figure 4 Timing operation.

[0082] Step S54: The processor 13 then determines a key region ROI in the third image frame, where the key region is located at the object position determined in step S52. As mentioned above, the size of the key region ROI can be preset or based on the line width W of the first image frame (refer to Fig. 6A As shown) and the line break height H of the second image frame (refer to Figure 6B as shown) decision.

[0083] Steps S55 to S56: Finally, the processor 13 recognizes the object image in the key area according to the learning model trained before leaving the factory to determine the type of the object.

[0084] Step S57: In order to improve the recognition rate, when it is determined in step S52 that there is an obstacle, the processor 13 further determines the height of the object according to the second image frame, for example, according to Figure 6B The determined object height can assist the learning model classification and identify the object type. Step S57 can be implemented selectively.

[0085] After identifying the type of the object, the processor 13 can avoid certain obstacles or directly cross certain obstacles according to the pre-set. The operation after identifying the type of the object can be set according to different applications and there is no specific restriction. If the type of some objects cannot be identified, the processor 13 can avoid or directly cross these unknown obstacles according to the pre-set.

[0086] It should be noted that although the second light sources LS21 and LS22 are shown to be lit and extinguished at the same time in the above embodiment, the present invention is not limited thereto. In other embodiments, LS21 and LS22 may be lit in sequence (and the light sensor acquires images accordingly), as long as they can project vertical light segments in front of the traveling direction.

[0087] In addition, the number of the first light source, the second light source, and the third light source is not limited to Figure 1A As shown, multiple identical light sources may be turned on or off at the same time.

[0088] In the present invention, the horizontal refers to being substantially parallel to a travel surface (eg, the ground), and the vertical refers to being substantially perpendicular to the travel surface. Objects located in the travel path are called obstacles.

[0089] In addition, before performing visual localization and mapping (VSLAM) based on the third image frame acquired by the optical sensor 11, the mobile robot 100 of the present invention may choose to first identify moving objects in the third image frame, such as human bodies, pets, etc., and remove the moving objects from the third image frame. This is because the moving objects may not always exist in the operating space of the mobile robot 100, so as to avoid the invalidation of the feature points calculated by the processor 13.

[0090] In one embodiment, after the optical sensor 11 acquires a third image frame, the processor 13 first identifies the object type of the obstacle in the third image frame (at this time, there is no need to determine the key area in the third image frame) according to the pre-trained learning model to distinguish the moving object. For example, certain object types are defined and recorded in advance as moving objects. If the third image frame does not contain classified moving objects, the processor 13 directly performs visual positioning and map construction based on the third image frame. If the third image frame contains classified moving objects, the processor 13 performs visual positioning and map construction based on the third image frame after removing the moving objects. In this way, the accuracy of visual positioning and map construction can be improved.

[0091] In another embodiment, the processor 13 calculates the pixels whose movement exceeds the threshold value by using optical flow or correlation according to the third image frame acquired by the optical sensor 11, and identifies a plurality of pixel regions containing high movement (i.e., greater than or equal to the movement threshold value) as moving objects and removes them from the third image frame. Then, the processor 13 performs visual positioning and map construction according to the third image frame after removing the moving objects. In this way, the accuracy of visual positioning and map construction can be improved. The method of calculating the movement of pixels by using optical flow or correlation is known, so it will not be repeated here.

[0092] In another embodiment, the processor 13 calculates the depth map of each pixel based on the third image frame acquired by the optical sensor 11, and regards the pixels whose depth changes are inconsistent with the depths of other pixels in the continuous third image frames as pixels related to the moving object. For example, when the mobile robot moves straight toward the wall, the depth of the wall will gradually become closer during continuous calculations. If there are other moving objects near the wall, the depth change of the moving object will be significantly different from the depth change of the wall. Therefore, the processor 13 can identify the moving object based on the continuous depth map and remove it from the third image frame. Then, the processor 13 performs visual positioning and map construction based on the third image frame after removing the moving object. In this way, the accuracy of visual positioning and map construction can be improved. The method of calculating the depth map of each pixel in the image frame is known, so it will not be repeated here.

[0093] In the present invention, the processor 13 is not limited to the above three ways to identify the moving object. The processor 13 can also identify the moving object in the third image frame in other ways to remove it. Fixed objects serve as obstacles or range limitations in the operating space of the mobile robot in visual positioning and map construction.

[0094] As mentioned above, the third image frame refers to the image frame acquired by the light sensor 11 when the illumination light source is turned on.

[0095] In summary, conventional cleaning robots use a variety of sensors to achieve different detection functions, and have the problems of large amount of computation, long time, high power consumption and low recognition rate required for obstacle recognition. Therefore, the present invention also provides a mobile robot suitable for smart home (e.g., FIG. 1 to FIG. 2 ). Figure 2 ) and its operation method (for example Figure 5 ), which simultaneously achieves the goals of obstacle avoidance, positioning and object recognition based on the detection results of a single image sensor.

[0096] Although the present invention has been disclosed through the above embodiments, it is not intended to limit the present invention. Any person skilled in the art with ordinary knowledge in the technical field to which the present invention belongs can make various changes and modifications without departing from the concept and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A mobile robot, comprising: a first light source, the first light source being used to project a horizontal light segment toward the traveling direction during a first period; a second light source, the second light source being used to project a vertical light segment toward the traveling direction during a second period; a third light source, the third light source being used to illuminate a front area in the traveling direction during a third period; A light sensor, the light sensor being used to acquire a first image frame, a second image frame and a third image frame during the first period, the second period and the third period respectively; as well as a processor, the processor being electrically coupled to the first light source, the second light source, the third light source, and the light sensor, and being configured to perform distance measurement according to the first image frame and the second image frame and to perform visual positioning and map construction according to the third image frame, The light sensor includes a pixel array, the pixel array includes a plurality of first pixels and a plurality of second pixels, the plurality of first pixels receive incident light through an infrared light filter, and the plurality of second pixels receive incident light without passing through any filter, wherein The first image frame and the second image frame are formed by pixel data generated by the plurality of first pixels; and The third image frame is composed of pixel data generated by the first pixels and the second pixels.

2. The mobile robot according to claim 1, wherein: The first light source and the second light source each include an infrared laser diode; and The third light source includes an infrared light emitting diode.

3. The mobile robot according to claim 1, wherein: The light sensor is also used Acquire a first dark image frame during a period in which the first light source is turned off before or after the first period, and the first dark image frame is used to perform a difference with the first image frame, and A second dark image frame is acquired during a period in which the second light source is turned off before or after the second period, and the second dark image frame is used for performing a difference with the second image frame.

4. The mobile robot according to claim 1, wherein: The plurality of first pixels and the plurality of second pixels are arranged in a chessboard shape.

5. The mobile robot according to claim 4, wherein the processor is further configured to performing pixel interpolation operations on the first image frame and the second image frame, and Before performing the visual positioning and map construction according to the third image frame, moving objects in the third image frame are removed.

6. A mobile robot, comprising: a first light source, the first light source being used to project a horizontal light segment toward the traveling direction during a first period; a second light source, the second light source being used to project a vertical light segment toward the traveling direction during a second period; A pixel array, the pixel array comprising a plurality of first pixels and a plurality of second pixels, the plurality of first pixels receiving incident light through infrared light filters, but the plurality of second pixels not receiving incident light through any filters, wherein: The pixel array is used to respectively acquire a first image frame, a second image frame, and a third image frame during the first period, the second period, and a third period between the first period and the second period; as well as A processor is electrically coupled to the first light source, the second light source and the pixel array, and is used to perform distance measurement based on the first image frame and the second image frame, and to perform visual positioning and map construction based on pixel data related to the plurality of second pixels in the third image frame.

7. The mobile robot according to claim 6, wherein the processor is further configured to performing a differential operation based on pixel data related to the plurality of first pixels in the first image frame and the third image frame to eliminate background noise, and A differential operation is performed based on the pixel data related to the plurality of first pixels in the second image frame and the third image frame to eliminate background noise. 8 . The mobile robot according to claim 6 , wherein the plurality of first pixels and the plurality of second pixels are arranged in a chessboard shape.

9. The mobile robot according to claim 8, wherein The first image frame and the second image frame are composed of pixel data generated by the plurality of first pixels; and The third image frame is formed by pixel data generated by the plurality of second pixels.

10. The mobile robot according to claim 9, wherein the processor is further configured to Before performing the ranging, pixel interpolation operation is performed on the first image frame and the second image frame, and A pixel interpolation operation is performed on the third image frame before performing the visual positioning and map construction.

11. The mobile robot according to claim 6, wherein the processor is further configured to determining the ambient light intensity according to the third image frame, and Before performing the visual positioning and map construction according to the third image frame, moving objects in the third image frame are removed.

12. The mobile robot according to claim 11, wherein when it is determined according to the third image frame that the ambient light intensity is less than a threshold, the processor is further configured to change a lighting sequence of the first light source and the second light source.

13. A mobile robot, comprising: a first light source, the first light source being used to project a horizontal light segment toward the traveling direction during a first period; a second light source, the second light source being used to project a vertical light segment toward the traveling direction during a second period; a third light source, the third light source being used to illuminate the area ahead in the direction of travel; A light sensor, the light sensor being used to acquire a first image frame and a second image frame during the first period and the second period, respectively; as well as a processor electrically coupled to the first light source, the second light source, the third light source and the light sensor, and configured to: Determine obstacles according to the first image frame and the second image frame, When the obstacle is determined, the third light source is controlled to light up during a third period and the light sensor is controlled to acquire a third image frame during the third period. determining a key area in the third image frame according to the image position of the obstacle, and The object type of the obstacle in the critical area is identified using a learning model.

14. The mobile robot according to claim 13, wherein: The learning model does not identify areas outside the key area in the third image frame.

15. The mobile robot according to claim 13, wherein: The processor further determines the height of the obstacle according to the second image frame, and The learning model also identifies the type of object based on the height.

16. The mobile robot according to claim 13, wherein: The processor further controls the first light source, the second light source, and the third light source to be turned off for a predetermined period after the third period.

17. The mobile robot according to claim 13, wherein The key area has a predetermined image size; or The image size of the key area is determined by the processor according to the first image frame and the second image frame.

18. The mobile robot according to claim 13, wherein the optical sensor is further used for Acquire a first dark image frame during a period in which the first light source is turned off before or after the first period, and the first dark image frame is used to perform a difference with the first image frame, and A second dark image frame is acquired during a period in which the second light source is turned off before or after the second period, and the second dark image frame is used for performing a difference with the second image frame.

19. A mobile robot, comprising: A first laser light source, the first laser light source being used to project a horizontal light segment toward the traveling direction during a first period; a second laser light source, the second laser light source being used to project a vertical light segment toward the traveling direction during a second period; a light emitting diode light source, the light emitting diode light source being used to illuminate the front area in the direction of travel during the third period; a light sensor, the light sensor being used to acquire a first image frame, a second image frame and a third image frame during the first period, the second period and the third period, respectively; The light sensor includes a pixel array, the pixel array includes a plurality of first pixels and a plurality of second pixels, the plurality of first pixels receive incident light through an infrared light filter, and the plurality of second pixels receive incident light without passing through any filter, wherein The first image frame and the second image frame are formed by pixel data generated by the plurality of first pixels; and The third image frame is composed of pixel data generated by the first pixels and the second pixels.

Images