Electronic robot

Abstract

A mobile robot performs obstacle avoidance, positioning and object recognition according to image frames acquired by the same optical sensor. The mobile robot comprises a light sensor, a light emitting diode, a laser diode and a processor. And the processor judges an obstacle and measures distance according to an image frame acquired by the light sensor when the laser diode is lightened. The processor carries out positioning and object identification according to an image frame obtained by the light sensor when the light emitting diode is lightened.

Description

[0001] This application is a divisional application of a Chinese invention patent application with the application number 202110075965.7, the application date of January 20, 2021, and the title of "Mobile Robot". Technical Field

[0002] The present invention relates to a mobile robot, and particularly to a mobile robot that can perform obstacle avoidance, positioning, and object recognition based on image frames obtained when the same optical sensor is illuminated by different light sources. Background Art

[0003] Smart home is an integral part of the development of smart cities, and cleaning robots have almost become one of the essential electronic products in smart homes. Generally, cleaning robots are configured with multiple functions to enhance the user experience, including map construction in the operating area, obstacle detection and avoidance during operation, etc. Currently, cleaning robots include multiple detectors to perform these different detection functions.

[0004] For example, a cleaning robot includes sensors configured on the top surface to achieve visual positioning and map construction (VSLAM) by acquiring images above the path traveled by the cleaning robot. In addition, the cleaning robot also includes front sensors to achieve functions such as obstacle detection and avoidance by acquiring images in the front direction of the traveling direction of the cleaning robot.

[0005] That is, past cleaning robots need to include multiple sensors to achieve different detection functions.

[0006] In view of this, the present invention provides a mobile robot that performs obstacle avoidance, positioning, and object recognition based on image frames obtained when the same optical sensor is illuminated by different light sources. Summary of the Invention

[0007] The present invention provides a mobile robot that performs obstacle avoidance based on an image frame obtained by an optical sensor when a laser diode emits light, and performs visual positioning and map construction based on an image frame obtained by the optical sensor when a light-emitting diode emits light.

[0008] The present invention also provides a mobile robot that determines a key area based on an image frame obtained by an optical sensor when a laser diode emits light, and performs object recognition within the key area of the image frame obtained by the optical sensor when a light-emitting diode emits light, so as to reduce the computational amount, reduce energy consumption, and improve the recognition rate.

[0009] The present invention provides an electronic robot comprising a line light source, an illumination light source, a light sensor, and a processor. The line light source is configured to project a light segment towards the traveling direction during a first period. The illumination light source is configured to illuminate a front area of the traveling direction during a second period. The light sensor is configured to acquire a first image frame and a second image frame during the first period and the second period respectively. The processor is electrically coupled to the line light source, the illumination light source, and the light sensor, and is configured to perform distance measurement based on the first image frame, and perform visual positioning and map construction based on the second image frame.

[0010] The present invention further provides an electronic robot comprising a line light source, a pixel array, and a processor. The line light source is configured to project a light segment towards the traveling direction during a first period. The pixel array includes a plurality of first pixels and a plurality of second pixels. The plurality of first pixels receive incident light through an infrared light filter, but the plurality of second pixels do not receive incident light through any filter. Wherein, the pixel array is configured to acquire a first image frame and a second image frame during the first period and the second period respectively, and the line light source is turned off during the second period. The processor is electrically coupled to the line light source and the pixel array, and is configured to perform distance measurement based on the first image frame, and perform visual positioning and map construction based on pixel data related to the plurality of second pixels in the second image frame.

[0011] The present invention further provides an electronic robot comprising a line light source, an illumination light source, a light sensor, and a processor. The line light source is configured to project a light segment towards the traveling direction during a first period. The illumination light source is configured to illuminate a front area of the traveling direction. The light sensor is configured to acquire a first image frame during the first period. The processor is electrically coupled to the line light source, the illumination light source, and the light sensor, and is configured to judge an obstacle based on a broken line in the first image frame. When the obstacle is judged, the illumination light source is controlled to be lit during a second period and the light sensor is controlled to acquire a second image frame during the second period. A key area is determined in the second image frame according to the image position of the obstacle, and a type of the obstacle in the key area is identified using a learning model.

[0012] In an embodiment of the present invention, the mobile robot can achieve multiple detection functions by only using a single light sensor and time-division operation of different light sources.

[0013] In order to make the above and other objects, features, and advantages of the present invention more obvious, the following will be described in detail in conjunction with the accompanying drawings. In addition, in the description of the present invention, the same components are denoted by the same reference numerals, which are stated here in advance. Description of the Drawings

[0014] Figure 1AIt is a schematic diagram of the mobile robot according to an embodiment of the present invention;

[0015] Figure 1B It is a block schematic diagram of the components of the mobile robot according to an embodiment of the present invention;

[0016] Figure 2 It is a running timing diagram of the mobile robot according to the first embodiment of the present invention;

[0017] Figure 3 It is a schematic diagram of the pixel array of the mobile robot according to an embodiment of the present invention;

[0018] Figure 4 It is a running timing diagram of the mobile robot according to the second embodiment of the present invention;

[0019] Figure 5 It is a flowchart of the running method of the mobile robot according to the second embodiment of the present invention;

[0020] Figure 6A It is a schematic diagram of the image frame related to the first light source acquired by the light sensor of the mobile robot according to an embodiment of the present invention;

[0021] Figure 6B It is a schematic diagram of the image frame related to the second light source acquired by the light sensor of the mobile robot according to an embodiment of the present invention.

[0022] Explanation of reference numerals:

[0023] 100 Mobile robot

[0024] 11 Light sensor

[0025] 13 Processor

[0026] LS1 First light source

[0027] LS21, LS22 Second light source

[0028] LS3 Third light source

[0029] 15 Filter Detailed implementation manners

[0030] The mobile robot according to the embodiment of the present invention operates by using a single light sensor in combination with different light sources. The line light source is used to find obstacles and measure the distance to the obstacles as a basis for the robot to turn. The illumination light source is used to illuminate the front area in the traveling direction for visual positioning, map building, and object recognition.

[0031] Please refer to Figure 1A as shown, which is a schematic diagram of the mobile robot 100 according to the embodiment of the present invention. Figure 1AThe display mobile robot 100 is a cleaning robot, but the present invention is not limited thereto. The mobile robot 100 can be any of various electronic robots that move according to imaging results for carrying, communicating, guiding, etc.

[0032] Please refer to Figure 1B as shown, which is a block diagram of the mobile robot 100 according to an embodiment of the present invention. The mobile robot 100 includes a first light source LS1, second light sources LS21 and LS22, a third light source LS3, a light sensor 11, and a processor 13. The processor 13 is, for example, an application specific integrated circuit (ASIC) or a microcontroller unit (MCU), which implements its functions using software, hardware, and / or firmware. Although Figure 1B two second light sources are shown, which is only for illustration and not for limiting the present invention. The mobile robot 100 may include only a single second light source.

[0033] The first light source LS1 includes, for example, a laser light source and a diffractive optical component, and the diffractive optical component is used to generate horizontal projection light after the emitted light of the laser light source passes through, so that the first light source LS1 projects a horizontal light segment in the traveling direction. The traveling direction is, for example, the side where the first light source LS1, the second light sources LS21 and LS22, the third light source LS3, and the light sensor 11 are arranged.

[0034] The second light sources LS21 and LS22 include, for example, a laser light source and a diffractive optical component respectively, and the diffractive optical component is used to generate vertical projection light when the emitted light of the laser light source passes through, so that the second light sources LS21 and LS22 project vertical light segments in the traveling direction respectively.

[0035] In the present invention, the laser light source is, for example, an infrared laser diode (IR LD).

[0036] The third light source LS3 is, for example, an infrared light emitting diode (LED), which is used to illuminate the front area of the traveling direction. The illumination range of the third light source LS3 is preferably greater than or equal to the field of view of the light sensor 11. In the present invention, when the third light source point LS3 is lit, the first light source LS1 and the second light sources LS21 and LS22 are extinguished.

[0037] Please refer to Figure 2 , which is the operation timing diagram of the mobile robot 100 according to the first embodiment of the present invention. The first light source LS1 projects a horizontal light segment in the traveling direction during the first period T1. The second light sources LS21 and LS22 project vertical light segments in the traveling direction during the second period T2. The third light source point LS3 illuminates the front area of the traveling direction during the third period T3.

[0038] The optical sensor 11 is, for example, a CCD image sensor or a CMOS image sensor, which acquires a first image frame, a second image frame, and a third image frame at a sampling frequency during a first period T1, a second period T2, and a third period T3, respectively. When an obstacle is included in the first image frame, the first image frame will have a broken line as shown in Figure 6A ; when no obstacle is included in the first image frame, the first image frame only includes continuous (no broken line) horizontal straight lines. When an obstacle is included in the second image frame, the second image frame will have at least one broken line as shown in Figure 6B , wherein the angle of the broken line depends on the shape of the obstacle and is not limited to that shown in Figure 6B ; however, when no obstacle is included in the second image frame, the second image frame only includes two continuous (no broken line) inclined straight lines. It can be understood that Figure 6A and 6B are only for illustration and not for limiting the present invention.

[0039] It can be understood that since the second light sources LS21 and LS22 project two parallel light segments on the traveling surface, in the second image frame acquired by the optical sensor 11, the two parallel light segments appear as inclined straight lines. In addition, Figure 6B only shows the projected light segments on the traveling surface detected by the optical sensor 11. When there is a wall in front of the mobile robot 100, two vertical light segments projected by the second light sources LS21 and LS22 will appear at the upper part of the second image frame.

[0040] The position where the broken line appears in the image frame reflects the position of the obstacle in front of the mobile robot 100. As long as the relationship between the position of the broken line in the image frame and the actual distance of the obstacle is recorded in advance, when an image frame containing a broken line is acquired, the distance between the mobile robot 100 and the obstacle can be obtained accordingly.

[0041] As shown in Figure 6A , the processor 13 knows that the first light source LS1 projects a horizontal light segment at a predetermined distance in front of the mobile robot 100. According to triangulation ranging, when a broken line appears in the image of the horizontal light segment, the processor 13 can calculate the distance and width of the obstacle.

[0042] As shown in Figure 6B , the processor 13 knows that the second light sources LS21 and LS22 project vertical light segments in front of the mobile robot 100. According to triangulation ranging, when at least one broken line appears in the image of the vertical light segment, the processor 13 can calculate the distance and height of the obstacle according to the position and length of the broken line in the image of the vertical light segment (i.e., the inclined straight line).

[0043] The processor 13 is electrically coupled to the first light source LS1, the second light sources LS21 and LS22, the third light source LS3, and the optical sensor 11, and is used to control the turning on and off of the light sources and image acquisition. The processor 13 also performs distance measurement based on the first image frame (e.g., Figure 6A ) and the second image frame (e.g., Figure 6B ), and performs visual positioning and map construction (VSLAM) based on the third image frame (including the image of the actually acquired object). Among them, the detailed implementation of VSLAM is known, so it will not be elaborated here. The present invention lies in that the processor 13 performs different detections according to the image frames acquired by the same optical sensor 11 when different light sources are lit.

[0044] Please refer to Figure 2 again. The optical sensor 11 also acquires a first dark image frame during the first light source off period Td1 after the first period T1, which is used for differencing with the first image frame. The optical sensor 11 also acquires a second dark image frame during the second light source off period Td2 after the second period T2, which is used for differencing with the second image frame. For example, the processor 13 subtracts the first dark image frame from the first image frame and subtracts the second dark image frame from the second dark image frame to eliminate background noise.

[0045] Although Figure 2 shows that the first light source off period Td1 is after the first period T1 and the second light source off period Td2 is after the second period T2, the present invention is not limited thereto. In other embodiments, the first light source off period Td1 is configured before the first period T1 and the second light source off period Td2 is configured before the second period T2. In another embodiment, the optical sensor 11 only captures one dark image frame in each cycle (the period when each light source is lit in sequence) (e.g., before T1, between T1 and T2, or after T2). The processor 13 then subtracts the dark image frame from the first image frame and subtracts the dark image frame (the same dark image frame) from the second image frame. In this way, background noise can be eliminated and the overall frame rate can be improved.

[0046] In one embodiment, the optical sensor 11 includes a pixel array, and all pixels of the pixel array receive incident light through an infrared light filter. For example, Figure 1B shows that an infrared light filter 15 is further disposed in front of the optical sensor 11. The infrared light filter 15 can be an optical element (e.g., coated on a lens) in front of the pixel array, or directly disposed on each pixel of the pixel array.

[0047] In another embodiment, the pixel array of the optical sensor 11 includes a plurality of first pixels P IR and a plurality of second pixels P mono , as Figure 3 shown. The first pixel P IRis an infrared light pixel, that is, the pixel receives incident light through an infrared light filter or thin film. The second pixel P mono does not receive incident light through an infrared light filter or thin film. The second pixel P mono preferably receives incident light without passing through any light filtering element. The incident light is the reflected light of the floor, wall, object, etc. in front of the mobile robot 100.

[0048] In an embodiment including two types of pixels, the above-mentioned first image frame and second image frame are composed of pixel data generated by a plurality of first pixels P IR . That is, the processor 11 performs ranging only based on the pixel data generated by a plurality of first pixels P IR . The above-mentioned third image frame is jointly composed of pixel data generated by a plurality of first pixels P IR and a plurality of second pixels P mono . When the third light source LS3 is lit, both the first pixel P IR and the second pixel P mono can detect infrared light. The processor 13 is configured to process corresponding pixel data for the lighting of different light sources.

[0049] In one embodiment, a plurality of first pixels P IR and a plurality of second pixels P mono of the pixel array are arranged in a checkerboard pattern as shown in Figure 3 . In other embodiments, the first pixel P IR and the second pixel P mono can be arranged in other ways. For example, the left half or the upper half of the pixel array is the first pixel P IR and the right half or the lower half of the pixel array is the second pixel P mono , but the present invention is not limited thereto.

[0050] In the embodiment where the first pixel P IR and the second pixel P mono are arranged in a checkerboard pattern, before calculating the distance of the object, the processor 11 also performs pixel interpolation operations on the first image frame and the second image frame to fill interpolation data at the positions of the second pixel P mono in the first image frame and the second image frame, and then performs ranging operations.

[0051] When the pixel array of the light sensor 11 is arranged in a checkerboard pattern, the mobile robot 100 of the embodiment of the present invention can also operate in other ways to increase the frame rate of ranging and positioning (using VSLAM). Figure 2 In the embodiment of

[0052] for example, referring toFigure 4 As shown, it is the operation timing diagram of the mobile robot 100 according to the second embodiment of the present invention. The first light source LS1 projects a horizontal light segment in the first period T1 towards the traveling direction. The second light sources LS21 and LS22 project vertical light segments in the second period T2 towards the traveling direction.

[0053] The pixel array of the light sensor 11 acquires a first image frame, a second image frame, and a third image frame in the first period T1, the second period T2, and the third period T3 between the first period T1 and the second period T2, respectively. That is, when the pixel array of the light sensor 11 acquires the third image frame, all light sources are not turned on. Figure 4 In the figure, the third period T3 is shown as a slanted rectangular area.

[0054] The processor 13 performs ranging (including finding obstacles and calculating distances) based on the first image frame and the second image frame, where the first image frame and the second image frame are composed of pixel data generated by a plurality of first pixels P IR That is, when the first light source LS1 and the second light sources LS21 and LS22 are lit, the pixel data related to the first pixel P IR is not affected by other colored lights. Therefore, the processor 13 performs ranging only based on the pixel data generated by a plurality of first pixels P IR at this time.

[0055] At this time, the third image frame is composed of pixel data generated by a plurality of second pixels P mono at this time.

[0056] Similarly, the processor 11 also performs a differential operation based on the pixel data related to the first pixel P IR in the first image frame and the third image frame, and performs a differential operation based on the pixel data related to the first pixel P IR in the second image frame and the third image frame to eliminate background noise.

[0057] Similarly, when the first pixel P IR and the second pixel P mono are arranged in a checkerboard pattern, before performing ranging, the processor 11 also performs an interpolation operation on the first image frame and the second image frame to fill interpolation data at the positions relative to the second pixel P mono in the first image frame and the second image frame, and then performs a ranging operation.

[0058] In the second embodiment, the processor 13 based on the second pixel P monoPerform visual positioning and mapping (VSLAM) on the relevant pixel data. In this embodiment, the third light source LS3 is not lit (the third light source LS3 may not be included at this time), and since the pixel data generated by multiple first pixels P IR has excluded components other than infrared light, the third image frame in this embodiment is only composed of the pixel data generated by multiple second pixels P mono . In addition, before performing VSLAM based on the third image frame, the processor 13 may also first perform pixel interpolation operation on the third image frame to fill interpolation data at the positions of the first pixels P IR in the third image frame.

[0059] According to Figure 4 , it can be seen that the frame rate of ranging is increased to 1 / 4 of the sampling frequency of the optical sensor 11 (for example, the frame period includes T1 + T2 + 2×T3), and the frame rate of VSLAM is increased to 1 / 2 of the sampling frequency of the optical sensor 11.

[0060] However, when the ambient light is insufficient, not lighting the third light source LS3 may cause the processor 13 to be unable to correctly perform VSLAM based on the third image frame. To solve this problem, the processor 11 also identifies the ambient light intensity according to the third image frame, for example, by comparing with a brightness threshold. When it is identified according to the third image frame that the ambient light belongs to a low-light environment, the processor 11 also changes the lighting timing of the first light source LS1 and the second light sources LS21 and LS22. For example, the processor 11 controls the lighting of the light source and the acquisition of the image to change as Figure 2 shown. That is, when in a strong light environment (for example, the average brightness of the third image frame is greater than the brightness threshold), the mobile robot 100 operates according to Figure 4 's timing; while when in a low-light environment (for example, the average brightness of the third image frame is less than the brightness threshold), the mobile robot 100 operates according to Figure 2 's timing.

[0061] The present invention also provides a mobile robot for ranging and obstacle recognition based on images acquired by the same optical sensor 11. When it is determined that the obstacle is a specific object, such as a wire, a sock, etc., the mobile robot 100 directly crosses over the obstacle; while when it is determined that the obstacle is an electronic device, such as a mobile phone, etc., the mobile robot 100 dodges and does not cross over the obstacle. Whether it can directly cross over the obstacle can be determined in advance according to different applications.

[0062] The mobile robot 100 in this embodiment is also as Figure 1A and Figure 1B shown, including a first light source LS1, second light sources LS21 and LS22, a third light source LS3, an optical sensor 11, and a processor 13. For example, referring to Figure 4As shown, the first light source LS1 projects a horizontal light segment in the traveling direction during the first period T1; the second light sources LS21 and LS22 project vertical light segments in the traveling direction during the second period T2. The third light source LS3 is used to illuminate the front area in the traveling direction.

[0063] As described above, in order to eliminate the influence of ambient light, the optical sensor 11 also acquires a first dark image frame during the first light source off period before or after the first period T1 (for example Figure 4 during T3) for differential with the first image frame; and acquires a second dark image frame during the second light source off period before or after the second period T2 (for example Figure 4 during T3) for differential with the second image frame. The optical sensor 11 respectively acquires the first image frame and the second image frame during the first period T1 and the second period T2.

[0064] In this embodiment, the pixel array of the optical sensor 11 receives incident light through a filter 15, for example.

[0065] The processor 13 determines an obstacle based on the first image frame and the second image frame. The method of determining an obstacle has been described above, so it will not be elaborated here. After finding the obstacle, the processor 13 controls the third light source LS3 to be lit during the third period (for example Figure 2 during T3) and controls the optical sensor 11 to acquire a third image frame during the third period.

[0066] In this embodiment, before the processor 13 determines the appearance of an obstacle, the third light source LS3 is not lit. Therefore, the operation timing of the mobile robot 100 is as Figure 4 shown. When the processor 13 determines the appearance of an obstacle, it controls the third light source LS3 to be lit and controls the optical sensor 11 to acquire a third image frame during the period when the third light source LS3 is lit. In other embodiments, multiple third image frames can be acquired. In the present invention, the use of one third image frame is taken as an example for illustration. In this embodiment, the third image frame is mainly used for object recognition by a pre-trained learning model.

[0067] After receiving the third image frame from the optical sensor 11, the processor 13 determines a region of interest ROI in the third image frame according to the position of the obstacle (such as the break line position), as Figure 6A and Figure 6B shown. Since the present invention uses a single optical sensor, after the processor 13 determines the position of the obstacle based on the first image frame and the second image frame and determines the region of interest ROI, the region of interest ROI can directly correspond to the relative region in the third image frame.

[0068] In a non-limiting embodiment, the key region ROI has a predetermined image size. That is, when the position of an obstacle (such as the center or the center of gravity, but not limited thereto) is determined, the processor 13 determines the key region ROI with a predetermined size at that position.

[0069] In another embodiment, the size of the key region ROI is determined by the processor 11 according to the first image frame and the second image frame. At this time, the larger the obstacle, the larger the key region ROI; conversely, the smaller the key region ROI.

[0070] The processor 13 then uses a pre-trained learning model (such as built into the processor 11 in the form of ASIC or firmware) to identify the type of the obstacle in the key region ROI. Since the learning model does not identify (such as does not calculate convolution) the regions outside the key region ROI in the third image frame, the computational amount, time consumption, and power consumption during identification can be effectively reduced. At the same time, since there are few object images in the key region ROI, it is less affected by other objects during identification, and the identification accuracy can be improved.

[0071] In addition, in order to further improve the recognition rate, the processor 11 also judges the height of the obstacle according to the second image frame, for example, Figure 6B using the longitudinal length H of the interruption line as the height of the obstacle. The learning model also identifies the type of the object according to the height.

[0072] In one embodiment, the object height is used as training material together with the ground truth image during the training phase, and a data network architecture (such as including a neural network learning algorithm, a deep learning algorithm, but not limited thereto) is used to generate the learning model.

[0073] In another embodiment, during the training phase, the data network architecture only uses the ground truth image as the training material to generate the learning model. During operation, when the learning model calculates the probabilities of several objects according to the third image frame, it is filtered by the height. For example, if the height of the object type classified by the learning model exceeds the height judged according to the second image frame, even if the object type has the highest probability, the learning model will exclude it.

[0074] It is known to use a learning model to classify objects in an image, so it will not be elaborated here. At the same time, the method of using the learning model in combination with the object height to identify obstacles is not limited to those exemplified in the present invention.

[0075] In one embodiment, since the imaging frequency of the optical sensor 11 is relatively high with respect to the moving speed of the mobile robot 100, the processor 11 also controls the first light source LS1, the second light sources LS21 and LS22, and the third light source LS3 to turn off for a predetermined period after the third period T3 (i.e., obtaining a third frame), until the obstacle moves out of the projection range of the first light source LS1, so as to avoid repeatedly identifying the same obstacle. The predetermined period can be determined, for example, according to the moving speed of the mobile robot 100 and the height judged from the second image frame.

[0076] Please refer to Figure 5 As shown, it is a flowchart of the operation method of the mobile robot 100 according to an embodiment of the present invention, including the following steps: turning on the line light source to detect obstacles (step S51); determining whether an obstacle exists (step S52); when no obstacle exists, returning to step S51 for continuous detection; and when an obstacle exists, turning on the illumination light source and obtaining a third image frame (step S53); determining a key area in the third image frame (step S54); and identifying the object type with a learning model (steps S55 to S56). This embodiment also selectively includes detecting the height of the object to assist in identifying the object type (step S57).

[0077] In this embodiment, the line light source includes, for example, the above-mentioned first light source LS1 and the second light sources LS21 and LS22. The illumination light source includes, for example, the above-mentioned third light source SL3. It can be understood that Figure 1A The positions of the light sources shown are only examples and are not intended to limit the present invention.

[0078] Step S51: The processor 13 controls the first light source LS1 and the second light sources LS21 and LS22 to emit light in sequence in the first period T1 and the second period T2 as shown in Figure 4 respectively. The processor 13 simultaneously controls the optical sensor 11 to obtain a first image frame and a second image frame in the first period T1 and the second period T2 respectively.

[0079] Step S52: When the processor 13 determines that the first image frame contains a broken line as shown in Figure 6A or the second image frame contains a broken line as shown in Figure 6B it is determined that there is an obstacle in the forward direction of travel. The program then proceeds to step S53. Conversely, when the processor 13 determines that neither the first image frame nor the second image frame contains a broken line, it returns to step S51 to continuously detect obstacles.

[0080] When it is determined that the first image frame or the second image frame contains a broken line, the processor 13 also records (for example, in the memory) the position of the broken line as the object position.

[0081] Step S53: The processor 13 then controls the third light source SL3 to turn on, for example, inFigure 2 The third period T3 shown. The processor 13 simultaneously controls the optical sensor 11 to obtain a third image frame during the third period T3, which includes at least one object image. In an embodiment where the processor 13 recognizes an object based on a single image, the processor 13 only controls the third light source SL3 to turn on for one third period T3. In one implementation, after the third period T3, the processor 13 then controls the first light source LS1 and the second light sources LS21 and LS22 to return to Figure 4 the timing operation. In another implementation, after the third period T3, the processor 13 then controls all light sources to pause emitting light for a predetermined period to avoid repeatedly detecting the same obstacle, and then returns to Figure 4 the timing operation.

[0082] Step S54: The processor 13 then determines a region of interest ROI in the third image frame, and this region of interest is located at the object position determined in step S52. As mentioned above, the size of the region of interest ROI can be preset or determined according to the broken line width W of the first image frame (refer to Figure 6A shown) and the broken line height H of the second image frame (refer to Figure 6B shown).

[0083] Steps S55 to S56: Finally, the processor 13 identifies the object image in the region of interest according to the learning model obtained through pre-factory training to determine the object type.

[0084] Step S57: In order to improve the recognition rate, when it is determined in step S52 that there is an obstacle, the processor 13 further determines the object height according to the second image frame, for example, according to Figure 6B the H shown. The determined object height can assist the learning model in classifying and identifying the object type. Step S57 can be selectively implemented.

[0085] After identifying the object type, the processor 13 can avoid specific obstacles or directly cross some obstacles according to prior settings. The operation after identifying the object type can be set according to different applications and there is no specific limitation. If the type of some objects cannot be identified, the processor 13 can avoid or directly cross these unknown obstacles according to prior settings.

[0086] It must be noted that although in the above embodiment it is shown that the second light sources LS21 and LS22 are lit and extinguished simultaneously, the present invention is not limited to this. In other implementations, LS21 and LS22 can be lit sequentially (and the optical sensor correspondingly acquires images), as long as vertical light segments can be projected respectively in front of the traveling direction.

[0087] In addition, the respective numbers of the first light source, the second light source, and the third light source are not limited to Figure 1A shown, and can respectively include the same multiple light sources being lit or extinguished simultaneously.

[0088] In the present invention, "horizontal" means substantially parallel to the traveling surface (such as the ground), and "vertical" means substantially perpendicular to the traveling surface. An object located on the traveling path is called an obstacle.

[0089] In addition, before performing visual positioning and mapping (VSLAM) based on the third image frame acquired by the optical sensor 11, the mobile robot 100 of the present invention may first identify moving objects in the third image frame, such as humans, pets, etc., and remove the moving objects from the third image frame. Since the moving objects may not always exist in the operating space of the mobile robot 100, the feature points calculated by the processor 13 can be prevented from becoming invalid.

[0090] In one embodiment, after the optical sensor 11 acquires a third image frame, the processor 13 first identifies the object types of the obstacles in the third image frame (without having to determine the key regions in the third image frame) according to a pre-trained learning model to distinguish moving objects. For example, certain object types are defined and recorded in advance as moving objects. If the third image frame does not contain classified moving objects, the processor 13 directly performs visual positioning and mapping based on the third image frame. If the third image frame contains classified moving objects, the processor 13 performs visual positioning and mapping based on the third image frame after removing the moving objects. Thereby, the accuracy of visual positioning and mapping can be improved.

[0091] In another embodiment, the processor 13 calculates the pixels with a movement amount exceeding a threshold based on the third image frame acquired by the optical sensor 11 using optical flow or correlation, and identifies the pixel regions with a high movement amount (i.e., greater than or equal to the movement threshold) as moving objects and removes them from the third image frame. Then, the processor 13 performs visual positioning and mapping based on the third image frame after removing the moving objects. Thereby, the accuracy of visual positioning and mapping can be improved. The method of calculating the movement amount of pixels using optical flow or correlation is known, so it will not be elaborated here.

[0092] In another embodiment, the processor 13 calculates a depth map of each pixel based on the third image frame obtained by the optical sensor 11, and in consecutive third image frames, pixels with depth changes inconsistent with the depths of other pixels are regarded as pixels related to moving objects. For example, when the mobile robot moves straight towards a wall, the depth of the wall will gradually become closer during consecutive calculations. If there are other moving objects beside the wall, the depth changes of these moving objects will be significantly different from the depth changes of the wall. Therefore, the processor 13 can identify the moving objects based on the consecutive depth maps and remove them from the third image frame. Then, the processor 13 performs visual positioning and map construction based on the third image frame after removing the moving objects. Thereby, the accuracy of visual positioning and map construction can be improved. The method of calculating the depth map of each pixel in the image frame is known, so it will not be elaborated here.

[0093] In the present invention, the method by which the processor 13 identifies moving objects is not limited to the above three. The processor 13 can also identify the moving objects in the third image frame according to other methods for removal. Fixed objects serve as obstacles or range limitations in the operating space of the mobile robot during visual positioning and map construction.

[0094] As mentioned above, the third image frame refers to the image frame obtained by the optical sensor 11 when the illumination light source is lit.

[0095] In summary, it is known that cleaning robots use multiple sensors to achieve different detection functions respectively, and have problems such as large computing amount, long time, high power consumption, and low recognition rate during obstacle recognition. Therefore, the present invention also provides a mobile robot applicable to a smart home (such as FIGS. 1 to Figure 2 ) and its operating method (such as Figure 5 ), which simultaneously achieves the purposes of obstacle avoidance, positioning, and object recognition based on the detection results of a single image sensor.

[0096] Although the present invention has been disclosed through the foregoing embodiments, it is not intended to limit the present invention. Any person with ordinary knowledge and skills in the technical field to which the present invention pertains can make various changes and modifications without departing from the concept and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. An electronic robot, comprising: A line light source for projecting a light segment in the traveling direction during a first period; An illumination light source for illuminating the front area in the traveling direction during a second period; A light sensor for acquiring a first image frame and a second image frame respectively during the first period and the second period; And A processor electrically coupled to the line light source, the illumination light source and the light sensor, and configured to Perform ranging based on the first image frame, and Perform visual positioning and map construction based on the second image frame.

2. The electronic robot according to claim 1, wherein The line light source includes an infrared laser diode; and The illumination light source includes an infrared light emitting diode.

3. The electronic robot according to claim 1, wherein, The light sensor is further configured to Acquire a first dark image frame during a first light source extinguishing period before or after the first period, and the first dark image frame is used for differencing with the first image frame.

4. The electronic robot according to claim 1, wherein, The light sensor includes a pixel array, and all pixels of the pixel array receive incident light through an infrared light filter.

5. The electronic robot according to claim 1, wherein, The light sensor includes a pixel array, the pixel array includes a plurality of first pixels and a plurality of second pixels, the plurality of first pixels receive incident light through an infrared light filter, but the plurality of second pixels do not receive incident light through any filter, wherein The first image frame is composed of pixel data generated by the plurality of first pixels; and The second image frame is jointly composed of pixel data generated by the plurality of first pixels and the plurality of second pixels.

6. The electronic robot according to claim 5, wherein, The plurality of first pixels and the plurality of second pixels are arranged in a checkerboard pattern.

7. The electronic robot according to claim 6, wherein the processor is further configured to perform in-pixel interpolation operation on the first image frame.

8. An electronic robot, comprising: A line light source for projecting a light segment in the traveling direction during a first period; A pixel array, the pixel array including a plurality of first pixels and a plurality of second pixels, the plurality of first pixels receiving incident light through an infrared light filter, but the plurality of second pixels receiving incident light without passing through any filter, wherein, The pixel array is configured to acquire a first image frame and a second image frame respectively during the first period and the second period, and the line light source is turned off during the second period; And A processor electrically coupled to the line light source and the pixel array, and configured to Perform ranging based on the first image frame, and Perform visual positioning and map construction based on the pixel data related to the plurality of second pixels in the second image frame.

9. The electronic robot according to claim 8, wherein the processor is further configured to Perform differencing operation based on the pixel data related to the plurality of first pixels in the first image frame and the second image frame to eliminate background noise.

10. The electronic robot according to claim 8, wherein the plurality of first pixels and the plurality of second pixels are arranged in a checkerboard pattern.

11. The electronic robot according to claim 10, wherein The first image frame is composed of pixel data generated by the plurality of first pixels; and The second image frame is composed of pixel data generated by the plurality of second pixels.

12. The electronic robot according to claim 11, wherein the processor is further configured to Performing an in-pixel interpolation operation on the first image frame before performing the ranging, and Performing an in-pixel interpolation operation on the second image frame before performing the visual positioning and mapping construction.

13. The electronic robot according to claim 8, wherein the processor is further configured to determine the ambient light intensity according to the second image frame.

14. The electronic robot according to claim 13, wherein when it is determined that the ambient light intensity is less than a threshold, the processor is further configured to change the lighting timing of the line light source.

15. An electronic robot, the electronic robot comprising: A line light source for projecting a light segment in the traveling direction during a first period; An illumination light source for illuminating the front area in the traveling direction; A light sensor for acquiring a first image frame during the first period; And A processor electrically coupled to the line light source, the illumination light source, and the light sensor, and configured to Judge an obstacle according to the broken line in the first image frame, When the obstacle is judged, control the illumination light source to be lit during a second period and control the light sensor to acquire a second image frame during the second period, Determine a key area in the second image frame according to the image position of the obstacle, and Use a learning model to identify the object type of the obstacle in the key area.

16. The electronic robot according to claim 15, wherein, The learning model does not identify areas outside the key area in the second image frame.

17. The electronic robot according to claim 15, wherein, The light segment is a longitudinal light segment, The processor further judges the height of the obstacle according to the second image frame, and The learning model further identifies the object type according to the height.

18. The electronic robot according to claim 15, wherein, The processor further controls the line light source and the illumination light source to be turned off for a predetermined period after the second period.

19. The electronic robot according to claim 15, wherein The key area has a predetermined image size, or The image size of the key area is determined by the processor according to the first image frame.

20. The electronic robot according to claim 15, wherein the light sensor is further configured to Acquire a first dark image frame during a first light source extinguishing period before or after the first period, and the first dark image frame is used for differential with the first image frame.

Images