Moving robot and method for controlling moving robot

Abstract

A moving robot disclosed herein comprises a main body, a plurality of wheels provided on the main body, a blade provided on the main body to cut grass, an optical sensor having a field of view directed toward the front of the main body to generate an image, a user interface for acquiring a user input and providing information on the operation of the moving robot, and at least one processor. The at least one processor: identifies a plurality of objects in the image acquired while the main body is moving; determines an object detected for a predetermined duration of time, among the plurality of identified objects, to be a contaminant; determines a contamination level of the optical sensor on the basis of the characteristics of the contaminant and the location of the contaminant in the image; provides information corresponding to the contamination level of the optical sensor via the user interface; and determines an operation mode of the moving robot on the basis of the contamination level of the optical sensor.

Description

Mobile robot and method for controlling mobile robot

[0001] The disclosed invention relates to a mobile robot and a method for controlling the mobile robot that cuts grass in a work area while moving around the work area and docks to a charging station when the grass management is completed.

[0002] Robots were developed for industrial use and have played a role in factory automation. Recently, the scope of robot application has expanded, with the development of medical robots, aerospace robots, service robots, and even domestic robots for general use. Among these robots, those capable of autonomous navigation are called mobile robots.

[0003] A prime example of a mobile robot is a lawnmower robot, which can mow and trim grass in outdoor settings, golf courses, and sports fields. Mobile robots feature wheels and blades, and they rotate the wheels to move around the work area, while the blades rotate to cut the top layer of grass.

[0004] Unlike mobile robots used indoors, mobile robots used outdoors are vulnerable to contamination and physical damage from external environmental factors. For example, optical sensors embedded in mobile robots can easily become contaminated. Contamination of these sensors can cause the mobile robot to misidentify objects ahead or fail to identify obstacles, potentially leading to malfunctions in the robot's operation.

[0005] The disclosed invention provides a mobile robot, a control method and a system for the mobile robot, which can detect contamination of an optical sensor exposed to an external environment and take appropriate measures in response to contamination of the optical sensor.

[0006] According to one embodiment, a mobile robot includes a main body, a plurality of wheels provided on the main body, a blade provided on the main body for cutting grass, an optical sensor having a field of view facing forward of the main body and generating an image, a user interface for obtaining a user input and providing information regarding the operation of the mobile robot, and at least one processor. The at least one processor can identify a plurality of objects in the image obtained while the main body moves. The control unit can determine an object detected for a predetermined period of time among the identified plurality of objects as a contaminant. The control unit can determine a contamination level of the optical sensor based on a characteristic of the contaminant and a location of the contaminant within the image. The control unit can provide information corresponding to the contamination level of the optical sensor through the user interface. The control unit can determine an operation mode of the mobile robot based on the contamination level of the optical sensor.

[0007] A method for controlling a mobile robot according to one embodiment includes detecting a plurality of objects in an image acquired by an optical sensor while the mobile robot moves; determining an object continuously detected for a predetermined period of time among the plurality of objects as a contaminant; determining a contamination level of the optical sensor based on a characteristic of the contaminant and a location of the contaminant in the image; providing information corresponding to the contamination level of the optical sensor through the user interface; and determining an operation mode of the mobile robot based on the contamination level of the optical sensor.

[0008] According to one embodiment, a system (0) may include a mobile robot (1), a server (3), and a charging station (4). The system (0) may further include a user device (2).

[0009] The disclosed mobile robot, mobile robot control method, and system can detect contamination of an optical sensor exposed to the external environment and take appropriate measures to address the contamination. Accordingly, incorrect operation and accidents of the mobile robot due to contamination of the optical sensor can be prevented.

[0010] FIG. 1 illustrates a system including a mobile robot according to one embodiment.

[0011] FIG. 2 illustrates a mobile robot, a charging station, and a beacon included in a system according to one embodiment.

[0012] Figure 3 is a perspective view of a mobile robot according to one embodiment.

[0013] Figure 4 is a plan view from below of a mobile robot according to one embodiment.

[0014] Figure 5 shows an example of a cross-section taken vertically of the mobile robot illustrated in Figure 3.

[0015] Figure 6 illustrates a charging station in more detail according to one embodiment.

[0016] Figure 7 is a control block diagram of a charging station according to one embodiment.

[0017] Figure 8 is a control block diagram of a mobile robot according to one embodiment.

[0018] FIG. 9 shows an image acquired by a first optical sensor according to one embodiment.

[0019] FIG. 10 shows an image acquired by a second optical sensor according to one embodiment.

[0020] Figure 11 shows an example of dividing an image generated by an optical sensor into multiple regions.

[0021] FIG. 12 illustrates an example of information provided through a user interface or user device of a mobile robot according to one embodiment.

[0022] FIG. 13 illustrates another example of information provided through a user interface or user device of a mobile robot according to one embodiment.

[0023] Fig. 14 is a flowchart illustrating a control method of a mobile robot according to one embodiment.

[0024] FIG. 15 is a flowchart illustrating in more detail the method for determining the contamination level of the optical sensor described in FIG. 14.

[0025] Fig. 16 is a flowchart explaining in more detail the operation of the mobile robot according to the contamination level of the optical sensor described in Fig. 14.

[0026] FIG. 17 illustrates a further embodiment of a method for estimating contaminants in an image as described in FIG. 14.

[0027] It should be understood that the various embodiments and terms used in this document are not intended to limit the technical features described in this document to specific embodiments, but rather to include various modifications, equivalents, or substitutes of the embodiments.

[0028] In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0029] The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise.

[0030] In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof.

[0031] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0032] When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0033] The terms “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in this document, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0034] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0035] When we say that a component is "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0036] The term “and / or” includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0037] The operating principle and embodiments of the present invention will be described with reference to the attached drawings below.

[0038] FIG. 1 illustrates a system including a mobile robot according to one embodiment.

[0039] Referring to FIG. 1, a system (0) according to one embodiment may include a mobile robot (1), a user device (2), a server (3), and a charging station (4).

[0040] A mobile robot (1) may include a communication module capable of communicating with a user device (2), a server (3) or a charging station (4), a user interface for receiving user input or outputting information to a user, at least one processor for controlling the operation of the mobile robot (1), and at least one memory storing a program for controlling the operation of the mobile robot (1).

[0041] The server (3) may include a communication module capable of communicating with a mobile robot (1), another server, a user device (2), or a charging station (4). The server (3) may include at least one processor capable of processing data received from the mobile robot (1), another server, a user device (2), or a charging station (4), and at least one memory capable of storing a program for processing data or processed data. The server (3) may be implemented as various computing devices such as a workstation, a cloud, a data drive, or a data station. The server (3) may be implemented as one or more servers that are physically or logically separated based on functions, detailed configurations of functions, or data, and may transmit and receive data and process the transmitted and received data through communication between each server.

[0042] The server (3) can perform functions such as managing user accounts, registering a mobile robot (1) by linking it to a user account, and managing or controlling the registered mobile robot (1). For example, a user can access the server (3) through a user device (2) and create a user account. The user account can be identified by an ID and password set by the user. The server (3) can register the mobile robot (1) to the user account according to a set procedure. For example, the server (3) can link identification information (e.g., serial number or MAC address) of the mobile robot (1) to the user account, thereby registering, managing, and controlling the mobile robot (1).

[0043] The user device (2) may include a communication module capable of communicating with a mobile robot (1), a server (3), or a charging station (4). The user device (2) may include a user interface for receiving user input or outputting information to the user. The user device (2) may include at least one processor for controlling the operation of the user device (2) and at least one memory for storing a program for controlling the operation of the user device (2).

[0044] The user device (2) may be carried by the user or placed in the user's home or office, etc. The user device (2) may include, but is not limited to, a personal computer, a terminal, a portable telephone, a smart phone, a handheld device, a wearable device, a display device, etc.

[0045] A program for controlling a mobile robot (1), i.e., an application, may be stored in the memory of the user device (2). The application may be sold installed on the user device (2) or downloaded and installed from an external server.

[0046] A user can access a server (3) by executing an application installed on a user device (2), create a user account, and communicate with the server (3) based on the logged-in user account to register a mobile robot (1).

[0047] For example, if the mobile robot (1) is operated so that the mobile robot (1) can connect to the server (3) according to the procedure guided by the application installed on the user device (2), the mobile robot (1) can be registered in the user account by registering the identification information (e.g., serial number or MAC address) of the mobile robot (1) in the corresponding user account on the server (3).

[0048] A user can control a mobile robot (1) using an application installed on a user device (2). For example, when a user logs into a user account using an application installed on a user device (2), a mobile robot (1) registered to the user account appears, and when a control command for the mobile robot (1) is input, the control command can be transmitted to the mobile robot (1) via a server (3).

[0049] A network can include both wired and wireless networks. Wired networks include cable networks or telephone networks, while wireless networks can include any network that transmits and receives signals via radio waves. Wired and wireless networks can be interconnected.

[0050] A network may include a wide area network (WAN) such as the Internet, a local area network (LAN) formed around an access point (AP), and a short-range wireless network that does not use an access point (AP). Short-range wireless networks may include, but are not limited to, Bluetooth (IEEE 802.15.1), Zigbee (IEEE 802.15.4), Wi-Fi Direct, Near Field Communication (NFC), Z-Wave, etc.

[0051] An access point (AP) can connect a mobile robot (1), a user device (2), or a charging station (4) to a wide area network (WAN) to which a server (3) is connected. The mobile robot (1), the user device (2), or the charging station (4) can be connected to the server (3) via the wide area network (WAN).

[0052] The access point (AP) communicates with a mobile robot (1), a user device (2), or a charging station (4) using wireless communication such as Wi-Fi (IEEE 802.11), Bluetooth (IEEE 802.15.1), or Zigbee (IEEE 802.15.4), and can connect to a wide area network (WAN) using wired communication, but is not limited thereto.

[0053] According to various embodiments, the mobile robot (1) may be directly connected to a user device (2), a server (3), or a charging station (4) without going through an access point (AP). The mobile robot (1) may be connected to the user device (2), the server (3), or the charging station (4) through a long-range wireless network or a short-range wireless network.

[0054] For example, the mobile robot (1) may be connected to a user device (2) via a short-range wireless network (e.g., Wi-Fi Direct). As another example, the mobile robot (1) may be connected to a user device (2), a server (3), or a charging station (4) via a wide area network (WAN) using a long-range wireless network (e.g., a cellular communication module).

[0055] As another example, a mobile robot (1) can access a wide area network (WAN) using wired communication and be connected to a user device (2), a server (3), or a charging station (4) via the wide area network (WAN). If the mobile robot (1) can access a wide area network (WAN) using wired communication, it can also function as a connection relay.

[0056] A mobile robot (1) can transmit information about its operation or status to a user device (2), a server (3), or a charging station (4) via a network. For example, the mobile robot (1) can transmit information about its operation or status to the user device (2), the server (3), or the charging station (4) when a request is received from the server (3), when a specific event occurs in the mobile robot (1), or periodically or in real time.

[0057] When information about the operation or status is received from the mobile robot (1), the server (3) can update the stored information about the operation or status of the mobile robot (1) and transmit the updated information about the operation and status of the mobile robot (1) to the user device (2) via the network. Here, the update of information can include various operations that change existing information, such as an operation of adding new information to existing information or an operation of replacing existing information with new information.

[0058] A mobile robot (1) can obtain various information from a user device (2), a server (3), or a charging station (4), and provide the obtained information to a user. For example, the mobile robot (1) can obtain information related to the function of the mobile robot (1) and various environmental information (e.g., weather, temperature, humidity, etc.) from the server (3), and output the obtained information through a user interface.

[0059] The mobile robot (1) can operate according to control commands received from a user device (2), a server (3), or a charging station (4). For example, if the mobile robot (1) has obtained prior approval from a user so that it can operate according to control commands from the server (3) even without user input, the mobile robot (1) can operate according to control commands received from the server (3). Here, the control commands received from the server (3) may include, but are not limited to, control commands input by the user through the user device (2) or control commands based on preset conditions.

[0060] The user device (2) can transmit information about the user to the mobile robot (1), the server (3), or the charging station (4) via the communication module. For example, the user device (2) can transmit information about the user's location, the user's health status, the user's preferences, the user's schedule, etc. to the server (3). The user device (2) can transmit information about the user to the server (3) with the user's prior consent.

[0061] The mobile robot (1), user device (2), server (3), or charging station (4) can determine control commands using technologies such as artificial intelligence. For example, the server (3) can process information regarding the operation or status of the mobile robot (1) and information regarding the user of the user device (2) using artificial intelligence technology, and transmit the processing result or control command to the mobile robot (1) or user device (2) based on the processing result.

[0062] The mobile robot (1) may include a robot capable of autonomous driving. Hereinafter, the mobile robot (1) is exemplified as a robot for cutting grass.

[0063] FIG. 2 illustrates a mobile robot, a charging station, and a beacon included in a system according to one embodiment.

[0064] Referring to FIG. 2, a mobile robot (1) can move on a lawn. The lawn can be considered a work area (P) of the mobile robot (1). A beacon (5) and a charging station (4) can be placed at each boundary area of ​​the work area (P). The system (0) according to one embodiment can further include a beacon (5).

[0065] The work area can be formed into a polygonal shape by a plurality of beacons (5) and charging stations (4). For example, as illustrated, the work area (P) can be formed into a square shape by three beacons (5) and charging stations (4). As another example, the work area (P) can be formed into a pentagonal shape by four beacons (5) and charging stations (4). In this way, the work area can have various shapes, and the shape of the work area can vary depending on the number of beacons.

[0066] A mobile robot (1) can communicate with a plurality of beacons (5) and a charging station (4). The mobile robot (1) can receive a signal from at least one of the plurality of beacons (5) and transmit a signal to at least one of the plurality of beacons (5). The mobile robot (1) can receive a signal from the charging station (4) and transmit a signal to the charging station (4).

[0067] The signal generated by at least one beacon (5) may be an ultra-wideband (UWB) communication signal, and the signal transmitted by the at least one beacon (5) to the mobile robot (1) may include location information of the at least one beacon (5). The signal generated by the charging station (4) may be an ultra-wideband (UWB) communication signal. The signal transmitted by the charging station (4) to the mobile robot (1) may include location information of the charging station (4).

[0068] A plurality of beacons (5) can communicate with each other and also with a charging station (4). Each beacon (5) and charging station (4) can obtain relative coordinates between each other and measure the UWB communication distance with the mobile robot (1). Each beacon (5) and charging station (4) can obtain the position of the mobile robot (1) within the relative coordinate system through triangulation.

[0069] The mobile robot (1) can measure the direction of a signal triggered from a beacon (5) placed at each of a plurality of points and the transmission distance of the signal, and can obtain current location information of the mobile robot (1) within the work area (P) based on the direction and distance of the measured signal. In addition, the mobile robot (1) can measure the direction of a signal triggered from a charging station (4) and the transmission distance of the signal, and can also obtain current location information of the mobile robot (1) within the work area (P) based on the direction and distance of the measured signal.

[0070] The mobile robot (1) may include a separate position sensor that obtains current position information of the mobile robot (1). For example, the position sensor may include a GPS (Global Positioning System) receiver.

[0071] The mobile robot (1) can set the boundaries of the work area (P) based on signals transmitted from the charging station (4) and the beacon (5). The charging station (4) can also set the boundaries of the work area (P). The mobile robot (1) can drive within the set work area (P).

[0072] Fig. 3 is a perspective view of a mobile robot according to one embodiment. Fig. 4 is a plan view of the mobile robot as viewed from below according to one embodiment. Fig. 5 illustrates an example of a cross-section taken vertically of the mobile robot illustrated in Fig. 3.

[0073] Terms such as forward, backward, upward, downward, left, and right are defined based on the forward movement direction of the mobile robot (1), and the shape and position of each component are not limited by these terms. In addition, expressions indicating directions are used to ensure a clear understanding of the present invention, and each direction may be defined differently.

[0074] Referring to FIGS. 3, 4, and 5, a mobile robot (1) may include a main body (10) and a plurality of wheels (20a, 20b, 20c, 20d) that are rotatable about an axis that is horizontal to the ground and move the main body (10). The main body (10) may include a case that forms an outer shape. The plurality of wheels (20a, 20b, 20c, 20d) may include a first driving wheel (20a), a second driving wheel (20b), a first auxiliary wheel (20c), and a second auxiliary wheel (20d).

[0075] The first driving wheel (20a) may be provided at the right rear of the main body (10). The second driving wheel (20b) may be provided at the left rear of the main body (10). The first auxiliary wheel (20c) may be provided at the right front of the main body (10). The second auxiliary wheel (20d) may be provided at the right front of the main body (10).

[0076] The first driving wheel (20a) is connected to the first wheel motor (30a). The first wheel motor (30a) can generate a rotational force for rotating the first driving wheel (20a). The second driving wheel (20b) is connected to the second wheel motor (30b). The second wheel motor (30b) can generate a rotational force for rotating the second driving wheel (20b). The rotational speeds of the first driving wheel (20a) and the second driving wheel (20b) can be adjusted according to the rotational speeds of the first wheel motor (30a) and the second wheel motor (30b).

[0077] An optical sensor (S) may be provided at the front of the main body (10). The optical sensor (S) has a field of view (FOV) facing the front of the main body (10) and can generate an image. The position of the optical sensor (S) is not limited to the front of the main body (10). The optical sensor (S) may be provided at various positions that can capture the surroundings of the mobile robot (1). The optical sensor (S) may also be provided at the side and / or rear of the main body (10).

[0078] The optical sensor (S) may include various sensors that collect light incident from the outside and generate image data. For example, the optical sensor (S) may include at least one of a first optical sensor (S1) that generates a color image and a second optical sensor (S2) that generates a depth image including distance information to an object.

[0079] The first optical sensor (S1) may include an RGB camera that collects visible light to generate a color image. The second optical sensor (S2) may include a binocular camera or a stereo camera. The second optical sensor (S2) may obtain depth information to an object by utilizing the disparity between the two eyes. The second optical sensor (S2) may include an infrared camera with binocular vision. The first field of view (FOV) of the first optical sensor (S1) and the second field of view (FOV) of the second optical sensor (S2) may be different.

[0080] The optical sensor (S) is not limited to the examples provided. For example, the optical sensor (S) may include various sensors such as a Light Detection and Ranging (Lidar) sensor, a Time-of-Flight (ToF) sensor, and an RGB-D camera.

[0081] A user interface (70) may be provided on a portion of the upper surface of the main body (10). In FIG. 2, the user interface (70) is illustrated as being provided on the rear upper surface of the main body (10), but is not limited thereto. The user interface (70) can obtain user input. The user interface (70) can provide various information regarding the operation of the mobile robot (1). As described below, the user interface (70) may include a display (71), a speaker (72), an indicator (73), and an input unit (74).

[0082] A mobile robot (1) may include a blade (40) for cutting grass. The blade (40) may be provided in a circular fan shape. The blade (40) may be provided at the lower portion of the main body (10) and may rotate around an axis that is perpendicular to the ground. The blade (40) is connected to a blade motor (40a). The blade motor (40a) may generate a rotational force for rotating the blade (40). The rotational axis of the blade motor (40a) may be perpendicular to the rotational axis of the wheel motor (30).

[0083] The blade (40) can move in the up-and-down direction of the main body (10). The blade (40) can be spaced from the ground (GND). As the blade (40) moves in the up-and-down direction, the gap between the ground (GND) and the blade (40) can be adjusted. The height of the grass to be cut can be adjusted according to the up-and-down movement of the blade (40).

[0084] A charging terminal (11) used for charging a battery (50) may be provided on the front of the main body (10). The charging terminal (11) of the mobile robot (1) may be connected to a charger (440) of a charging station (4). When the mobile robot (1) is docked to the charging station (4), the charging terminal (11) may be electrically connected to the charger (440) of the charging station (4), and the battery (50) of the mobile robot (1) may be charged by power supplied from the charger (440). The battery (50) may supply power to various electronic components included in the mobile robot (1).

[0085] Figure 6 illustrates a charging station in more detail according to one embodiment. Figure 7 is a control block diagram of a charging station according to one embodiment.

[0086] Referring to FIGS. 6 and 7, the charging station (4) may include a communication module (4c) and a charger (440). In addition, the charging station (4) may further include at least one of a brush (410), a fan (420), and a heater (430). The control unit (450) of the charging station (4) may include a processor (451) and a memory (452).

[0087] The charging station (4) can communicate with the mobile robot (1), the user device (2), and / or the beacon (5) via the communication module (4c). The communication module (4c) may include a communication circuit for performing various wireless communications. For example, the communication module (4c) may support ultra-wideband communications.

[0088] The charging station (4) may include a housing (4a) and a support (4b) supporting the housing (4a). The support (4b) may provide a movement path for the mobile robot (1) when the mobile robot (1) docks with the charging station (4). The mobile robot (1) may reach an accurate charging position by moving along the support (4b).

[0089] The charging station (4) can supply power for charging the battery (50) of the mobile robot (1). The charging station (4) can be connected to a commercial power source, receive power from the commercial power source, convert the supplied power, and provide the converted power to the mobile robot (1). When the mobile robot (1) is docked to the charging station (4), the charging station (4) can convert the commercial power into power for charging the battery (50) of the mobile robot (1).

[0090] The charging station (4) may include a structure for removing contaminants attached to the optical sensor (S) of the mobile robot (1). For example, the charging station (4) may include a brush (410) that comes into contact with the optical sensor (S) of the mobile robot (1) as docking of the mobile robot (1) is completed. The charging station (4) may rotate the brush (410). The rotating brush (410) provides frictional force to a portion that comes into contact with the optical sensor (S), and contaminants attached to the optical sensor (S) may be removed by the frictional force.

[0091] Additionally, the charging station (4) may include a fan (420) that generates wind and a heater (430) that generates heat. The wind generated by the rotation of the fan (420) may remove contaminants attached to the optical sensor (S). The heat generated by the heater (430) may also help remove contaminants attached to the optical sensor (S). When the fan (420) and the heater (430) operate together, hot air may be supplied to the optical sensor (S) of the mobile robot (1).

[0092] Although not shown, a marker used for docking of a mobile robot (1) may be provided in the housing (4a) of the charging station (4). The marker may also be used to determine contamination of the optical sensor (S) of the mobile robot (1). The marker may have various patterns (e.g., a grid pattern, a chessboard pattern). The optical sensor (S) of the mobile robot (1) may capture the marker of the charging station (4). When the marker comes into the field of view of the optical sensor (S), the mobile robot (1) may move toward the marker.

[0093] If a marker is incompletely identified in an image generated by an optical sensor (S), the mobile robot (1) can determine that the optical sensor (S) is contaminated. In addition, the mobile robot (1) can also determine the contamination level of the optical sensor (S) based on the size and shape of the marker identified in the image generated by the optical sensor (S). In determining the contamination level of the optical sensor (S), the contamination level can be determined more accurately by using the marker of the charging station (4) together with the method described below.

[0094] The control unit (450) of the charging station (4) can control the operation of the charging station (4). The processor (451) of the charging station (4) can perform control related to the operation of the charging station (4) using data stored in the memory (452). The control unit (450) can control the communication module (4c) for docking with the mobile robot (1). The control unit (450) can control the charger (440) to charge the battery (50) included in the mobile robot (1). The control unit (450) can process signals transmitted from the mobile robot (1) and the beacon (5) to set the boundary of the work area (P). In addition, the control unit (450) can operate at least one of the brush (410), the fan (420), and the heater (430) to remove contaminants present in the optical sensor (S) of the mobile robot (1).

[0095] The components of the charging station (4) are not limited to those illustrated. Some of the described components may be omitted or other components may be added. Furthermore, the arrangement structure of the components included in the charging station (4) is not limited to that illustrated in the drawing.

[0096] Figure 8 is a control block diagram of a mobile robot according to one embodiment.

[0097] Referring to FIG. 8, the mobile robot (1) may include a wheel motor (30), a blade motor (40a), an optical sensor (S), a battery (50), a user interface (70), and a control unit (200). In addition, the mobile robot (1) may further include a communication unit (80). The mobile robot (1) may additionally include a motion sensor (90) and a position sensor (100).

[0098] The control unit (200) can control components of the mobile robot (1). The control unit (200) can include a processor (210) and a memory (220). The processor (210) is hardware and can include logic circuits and arithmetic circuits. The processor (210) can control components of the mobile robot (1) that are electrically connected using programs, instructions, and / or data stored in the memory (220) for the operation of the mobile robot (1). The control unit (200) can be implemented as a control circuit including circuit elements such as capacitors, inductors, and resistors. The processor (210) and the memory (220) can be implemented as separate chips or as a single chip. In addition, the control unit (200) can include a plurality of processors and a plurality of memories.

[0099] The memory (220) can store programs, applications, and / or data for the operation of the mobile robot (1), and can store data generated by the processor (210). The memory (220) can include non-volatile memory such as ROM (Read Only Memory) and flash memory for long-term storage of data. The memory (220) can include volatile memory such as S-RAM (Static Random Access Memory) and D-RAM (Dynamic Random Access Memory) for temporarily storing data.

[0100] The wheel motor (30) can rotate the driving wheels (20a, 20b). The wheel motor (30) can include a first wheel motor (30a) and a second wheel motor (30b). The first wheel motor (30a) is connected to the first driving wheel (20a), and the first driving wheel (20a) can rotate according to the rotation of the first wheel motor (30a). The second wheel motor (30b) is connected to the second driving wheel (20b), and the second driving wheel (20b) can rotate according to the rotation of the second wheel motor (30b). The mobile robot (1) can drive by rotating the first driving wheel (20a) and the second driving wheel (20b).

[0101] The blade motor (40a) can be connected to the blade (40). The blade (40) can rotate according to the rotation of the blade motor (40a). The rotation of the blade (40) can cut grass that has grown from the ground. The drive shaft of the blade motor (40a) can move up and down. By moving the drive shaft of the blade motor (40a) up and down, the height of the blade (40) from the ground can be adjusted, and the height of the grass to be cut can be varied.

[0102] The optical sensor (S) can generate an image with a field of view (FOV) facing the front of the main body (10). The image can include multiple image frames acquired at predetermined intervals. That is, a video can be generated by sequentially arranging multiple image frames. The optical sensor (S) can generate an image by collecting light incident from the outside. The optical sensor (S) can transmit image data and / or image signals corresponding to the generated image to the control unit (200).

[0103] For example, the optical sensor (S) may include at least one of a first optical sensor (S1) and a second optical sensor (S2). The first optical sensor (S1) may collect visible light to generate a color image. The first optical sensor (S1) may include a lens and an image sensor. The second optical sensor (S2) may generate a depth image including distance information to an object. The second optical sensor (S2) may include an infrared camera having binoculars. The second optical sensor (S2) may include two lenses and two image sensors. In other words, the second optical sensor (S2) may include a left-eye camera and a right-eye camera. The second optical sensor (S2) may generate a black-and-white image.

[0104] When both the first optical sensor (S1) and the second optical sensor (S2) are provided in the main body (10), the first optical sensor (S1) and the second optical sensor (S2) may be provided as one module or as separate modules.

[0105] The control unit (200) can control the driving of the mobile robot (1) based on the image acquired by the optical sensor (S). The control unit (200) can analyze the image acquired by the optical sensor (S) to identify various objects included in the image. The control unit (200) can identify an object from the image and identify the characteristics of the object using an artificial intelligence model acquired from the memory (220) or the server (3). For example, the control unit (200) can identify the size, shape, brightness, clarity, transparency, location, and / or color of the object from the image. In addition, the control unit (200) can estimate the distance to the object and the height of the object using the depth information included in the image. The control unit (200) can identify an obstacle existing on the driving path of the mobile robot (1) through image processing and control the wheel motor (30) to avoid the obstacle.

[0106] The battery (50) can supply power to various electronic components included in the mobile robot (1). For example, the battery (50) can supply power to each of the wheel motor (30), the blade motor (40a), the optical sensor (S), the user interface (70), the communication unit (80), the motion sensor (90), the position sensor (100), and the control unit (200). The power supplied from the battery (50) may be converted by the control unit (200) and then supplied to each electronic component. When the mobile robot (1) docks to the charging station (4), the battery (50) can be charged by the charging power provided from the charger (440) of the charging station (4).

[0107] The user interface (70) can obtain user input. The user interface (70) can provide various information regarding the operation of the mobile robot (1). The user interface (70) can include a display (71), a speaker (72), an indicator (73), and an input unit (74).

[0108] The display (71) can display information related to the operation of the mobile robot (1). The display (71) can display information input by the user or information provided to the user on various screens. The display (71) can display information related to the operation of the mobile robot (1) in the form of at least one image or text. For example, the display (71) can display at least one image or text regarding contamination of the optical sensor (S). The display (71) can display battery information.

[0109] Additionally, the display (71) can display a graphic user interface (GUI) that enables control of the mobile robot (1). That is, the display (71) can display a UI element (User Interface Element) such as an icon.

[0110] The display (71) may be provided as various types of display panels. For example, the display (71) may include a liquid crystal display panel (LCD Panel), a light emitting diode panel (LED Panel), an organic light emitting diode panel (OLED Panel), or a micro LED panel. The display (71) may include a touch display that also functions as an input device.

[0111] The speaker (72) can output information input by the user or information provided to the user in various sounds. For example, the control unit (200) can control the speaker (130) to output various sounds related to the operating status of the mobile robot (1) and contamination of the optical sensor (S). If the optical sensor (S) is contaminated, a voice message, notification sound, or warning sound notifying the contamination of the optical sensor (S) can be output through the speaker (72).

[0112] The indicator (73) is a status indicator for indicating various states of the mobile robot (1). The indicator (73) may include LEDs of various colors. When a specific event related to the operation of the mobile robot (1) occurs, at least one indicator (73) corresponding to the specific event may light up. For example, when contamination of the optical sensor (S) occurs, the indicator (73) may light up to indicate contamination of the optical sensor (S). The user can easily check the current state of the mobile robot (1) through the light-emitting indicator (73).

[0113] The input unit (74) can acquire user input. The user input can include various commands. For example, the input unit (74) can acquire a power-on command, a power-off command, an operation start command, an operation stop command, an operation continue command, a sensor cleaning command, and a charging command. The user input can also be acquired from the user device (2). The input unit (74) can transmit an electrical signal (voltage or current) corresponding to the user input to the control unit (200).

[0114] The input unit (74) may include various buttons and / or dials. For example, the input unit (74) may include at least one of a power button for turning the mobile robot (1) on or off, a start / stop button for starting or stopping a lawn mowing operation, and a docking button for docking the mobile robot (1) to the charging station (4). The buttons may be physical buttons or touch buttons. The display (71) and the input unit (74) may be provided as separate devices or as a single device (e.g., a touch display).

[0115] The control unit (200) can control the operation of the mobile robot (1) by processing a command received through at least one of the display (71), the input unit (74), or the user device (2).

[0116] The communication unit (80) can perform a connection with at least one of the user device (2) or the server (3) through a network. In addition, the communication unit (80) can communicate with the beacon (5). The control unit (200) can obtain various information, various signals, and / or various data from the user device (2) or the server (3) through the communication unit (80). For example, the communication unit (80) can receive a remote control signal from the user device (2). The control unit (200) can obtain an artificial intelligence model used to analyze an image from the server (3) through the communication unit (80). In addition, the control unit (200) can transmit information corresponding to the contamination level of the optical sensor (S) to the user device (2) through the communication unit (80) so that the information is displayed on the user device (2).

[0117] The communication unit (80) may include various communication circuits. The communication unit (80) may include wireless communication circuits and / or wired communication circuits. The wireless communication circuits may support various wireless communications, such as wireless local area network (LAN), home radio frequency (RF), infrared communication, ultra-wide band (UWB) communication, Wi-Fi, Bluetooth, Zigbee, and long-distance wireless networks (e.g., cellular communication).

[0118] The control unit (200) can identify the current location of the mobile robot (1) using SLAM (Simultaneous localization and mapping) based on ultra-wideband communication. That is, the control unit (200) can generate a map of the work area (P) using the SLAM (Simultaneous Localization And Mapping) algorithm while the mobile robot (1) is driving in the work area (P), and can identify the location of the mobile robot (1) within the work area (P).

[0119] The movement sensor (90) may include at least one of a wheel sensor for detecting the rotational speed of the wheel, a gyro sensor for detecting a change in the posture of the main body (10), and an inertial sensor (IMU: Inertial Measurement Unit) for detecting the acceleration, speed, and direction of the mobile robot (1). In addition, various sensors capable of detecting the movement of the mobile robot (1) may be provided. The control unit (200) may identify the movement of the mobile robot (1) based on a signal transmitted from the movement sensor (90).

[0120] The position sensor (100) can obtain position information of the mobile robot (1). For example, the position sensor (100) can include a GPS (Global Positioning System) receiver. The position sensor (100) can transmit an electrical signal containing position information of the mobile robot (1) to the control unit (200). The control unit (200) can also identify the movement of the mobile robot (1) based on changes in the position information obtained by the position sensor (100).

[0121] The components of the mobile robot (1) are not limited to those described above. The mobile robot (1) may include various other components in addition to the components described above, and some of the components described above may be omitted.

[0122] The optical sensor (S) of the aforementioned mobile robot (1) is exposed to the outside of the mobile robot (1), and thus may be contaminated by external objects. If the optical sensor (S) is contaminated, the mobile robot (1) may incorrectly identify objects in front or may not be able to identify obstacles, which may result in the mobile robot (1) operating incorrectly or colliding with an obstacle. Since the lawn mowing blade (40) is sharp, if an accident occurs with the mobile robot (1), the user may also be harmed. Therefore, it is necessary to detect contamination of the optical sensor (S) and take appropriate measures in response to the contamination.

[0123] The control unit (200) can acquire images, image data, and / or image signals from the optical sensor (S). The control unit (200) can identify objects in the images acquired by the optical sensor (S). The control unit (200) can identify multiple objects in the images acquired while the main body (10) moves. The control unit (200) can determine an object detected for a predetermined time (e.g., 10 seconds) among the multiple identified objects as a contaminant. At least one of the multiple objects identified in the images can be determined as a contaminant. The predetermined time for determining an object as a contaminant can be set to various values ​​depending on the design.

[0124] In other words, the control unit (200) can determine an object that is continuously detected for a predetermined period of time in an image acquired while the main body (10) is moving as a contaminant. That is, an object whose features change relatively little in the image despite the movement of the main body (10) can be determined as a contaminant. Since the contaminant moves together with the mobile robot (1) while attached to the optical sensor (S), the features of the object corresponding to the contaminant in the image hardly change even if the mobile robot (1) moves. When the mobile robot (1) moves, the feature change of an object that actually exists in the surrounding environment in the image becomes relatively large, whereas the feature change of the contaminant in the image becomes almost non-existent or relatively small. That is, if at least a part of the field of view of the optical sensor (S) is blocked by a contaminant, the change in at least a part of the image becomes relatively small compared to the change in other areas of the image.

[0125] For example, the control unit (200) may determine an object as a contaminant based on whether the size or shape of an object detected in the image remains constant for a predetermined period of time while the main body (10) is moving. In other words, even if the mobile robot (1) moves, the size or shape of an object corresponding to a contaminant in the image may remain substantially constant. If the size or shape of at least a portion of the image remains constant despite the movement of the main body (10), it may be determined that a contaminant is located in at least the portion of the image.

[0126] The control unit (200) can determine an object as a contaminant based on the brightness of the object in the image remaining lower than a predetermined threshold brightness for a predetermined period of time while the main body (10) is moving. Since the contaminant at least partially obscures the lens of the optical sensor (S), the brightness of at least some areas in the image where the contaminant is located can be detected to be lower than the brightness of other areas.

[0127] The control unit (200) can determine an object as a contaminant based on the fact that the sharpness of the object in the image remains lower than a predetermined threshold sharpness for a predetermined period of time while the main body (10) is moving. If a contaminant is attached to the optical sensor (S), the area where the contaminant is located among the entire area of ​​the image generated by the optical sensor (S) may become out of focus. Accordingly, the sharpness of the contaminant portion in the image may be detected to be relatively low.

[0128] Additionally, the control unit (200) can determine an object as a contaminant based on the distance from the current position of the mobile robot (1) to the object remaining less than a predetermined threshold distance for a predetermined period of time while the main body (10) is moving. The distance from the current position of the mobile robot (1) to the object can be obtained from depth information included in the image.

[0129] When the optical sensor (S) is a binocular camera, if there is a contaminant in either the left or right eye lens, the binocular difference matching may fail and the distance to the object may be output as a random value. The control unit (200) may determine the object as a contaminant based on the fact that the distance to the object in the image acquired by the binocular camera is repeatedly output as a random value while the main body (10) moves.

[0130] Meanwhile, the mobile robot (1) may be provided with two or more different optical sensors (S). For example, as described above, the mobile robot (1) may include a first optical sensor (S1) that generates a color image (first image) and a second optical sensor (S2) that generates a depth image (second image) that includes distance information to an object. The control unit (200) can align the first image of the first optical sensor (S1) with the second image of the second optical sensor (S2), and compare an object included in the first image with an object included in the second image. The control unit (200) can determine whether an object (first object) detected in the first image and an object (second object) detected in the second image are the same or different.

[0131] The control unit (200) can determine an object as a foreign object and not a contaminant based on the fact that the object is detected identically in both the first image of the first optical sensor (S1) and the second image of the second optical sensor (S2). For example, even if an object included in the first image of the first optical sensor (S1) is determined to be a contaminant, if the object is also detected identically in the second image of the second optical sensor (S2), there is a high probability that the object is not a contaminant but another foreign object. In this way, the accuracy of determining a contaminant can be improved by comparing images acquired by different optical sensors.

[0132] The control unit (200) can determine the contamination level of the optical sensor (S) based on the characteristics of the contaminant and the location of the contaminant within the image. The control unit (200) can divide the image into multiple regions based on a predetermined reference line. The control unit (200) can determine the contamination level of the optical sensor (S) differently depending on the region in which the object determined to be a contaminant is located among the multiple regions of the image. The multiple regions can be set in various ways depending on the design.

[0133] For example, the control unit (200) can divide the entire area of ​​the image acquired by the optical sensor (S) into a central area, a left area, a right area, an upper area, and a lower area. The control unit (200) can apply the largest first weight to the central area of ​​the image. The control unit (200) can apply a second weight, which is smaller than the first weight, to the left and right areas of the image. The control unit (200) can apply a third weight, which is smaller than the second weight, to the upper and lower areas of the image. The control unit (200) can determine that the contamination level of the optical sensor (S) increases as the weight of the area where the object is located increases.

[0134] In general, the field of view (FOV) of the optical sensor (S) is designed to view an area larger than the size of the mobile robot (1). The area of ​​the central area of ​​the image can be set to correspond to the size of the mobile robot (1). The optical sensor (S) captures the front of the mobile robot (1), and the mobile robot (1) moves forward along a driving path. Among the areas of the image acquired by the optical sensor (S), the area that has a great influence on the driving of the mobile robot (1) may be the central area. Since an object detected in the central area of ​​the image acquired by the optical sensor (S) corresponds to an object on the driving path of the mobile robot (1), the operation of the mobile robot (1) changes depending on what object is detected in the central area of ​​the image. Therefore, among the multiple areas of the image, the central area should be processed most importantly, and thus the greatest weight can be assigned to the central area.

[0135] In other words, among the multiple regions of the image, the region that has the greatest influence on the straight driving of the mobile robot (1) may be the central region. In addition, when the mobile robot (1) needs to change the driving direction to the left or right, the operation of the mobile robot (1) varies depending on which object is detected in the left or right region of the image. On the other hand, an object detected in the upper region of the image may actually exist above the mobile robot (1), and an object detected in the lower region of the image may exist below the lower surface of the mobile robot (1). In other words, an object detected in the upper region of the image or the lower region of the image may not affect the driving of the mobile robot (1). Therefore, the left and right regions of the image need to be processed more importantly than the upper and lower regions of the image.

[0136] The control unit (200) can determine the contamination level of the optical sensor (S) by considering not only the area where the contamination is located in the image but also the characteristics of the contamination. For example, the characteristics of the contamination may include at least one of the size of the contamination or the transparency of the contamination. The control unit (200) can determine the contamination level of the optical sensor (S) to be higher as the size of the contamination increases. As the size of the contamination increases, the field of view (FOV) of the optical sensor (S) is more obscured, and thus the contamination level can be determined to be higher. The control unit (200) can determine the contamination level of the optical sensor (S) to be higher as the transparency of the contamination decreases. As the contamination becomes more opaque, light cannot penetrate the contamination, and therefore at least a portion of the image where the contamination is located becomes dark.

[0137] In addition, the control unit (200) can determine the contamination level of the optical sensor (S) to be higher as the brightness of the contaminant is lower. The control unit (200) can determine the contamination level of the optical sensor (S) to be higher as the clarity of the contaminant is lower.

[0138] There may be multiple objects determined to be contaminants in the image. The control unit (200) can determine the contamination level of the optical sensor (S) by considering the weights of the areas where each of the multiple contaminants is located in the image and the characteristics of each of the multiple contaminants.

[0139] The control unit (200) of the mobile robot (1) can provide information corresponding to the contamination level of the optical sensor (S) through at least one of the user interface (70) or the user device (2). The information corresponding to the contamination level of the optical sensor (S) can include at least one of visual information and auditory information. The visual information can be provided through the display (71) and indicator (73) of the user interface (70). The auditory information can be provided through the speaker (72) of the user interface (70).

[0140] The control unit (200) can determine the operation mode of the mobile robot (1) based on the contamination level of the optical sensor (S). The operation mode of the mobile robot (1) can be determined as a normal mode, a standby mode, or a cleaning mode. The normal mode can be set so that the mobile robot (1) performs the grass cutting task normally. The standby mode can be set so that the mobile robot (1) stops the grass cutting task and waits. The cleaning mode can be set so that the mobile robot (1) stops the grass cutting task and returns to the charging station (4).

[0141] For example, the control unit (200) may provide a contamination notification to notify of contamination of the optical sensor (S) through at least one of the user interface (70) or the user device (2) based on the fact that the contamination level of the optical sensor (S) is lower than a threshold level, and may determine the operation mode of the mobile robot (1) to be a normal mode. The threshold level may be set in various ways depending on the design. If the contamination level of the optical sensor (S) is lower than the threshold level, there may be no problem in the mobile robot (1) driving normally. Therefore, the mobile robot (1) may notify the user that contamination of the optical sensor (S) has occurred and continue driving to cut the grass.

[0142] When the contamination level of the optical sensor (S) is lower than a critical level, the control unit (200) can control the wheel motor (30) to adjust the driving speed of the mobile robot (1) according to the contamination level of the optical sensor (S). The control unit (200) can reduce the driving speed of the mobile robot (1) as the contamination level of the optical sensor (S) increases.

[0143] If the contamination level of the optical sensor (S) is higher than or equal to a threshold level, the mobile robot (1) may not be able to identify other objects located in front, and may abnormally deviate from the driving path or collide with other objects. Based on whether the contamination level of the optical sensor (S) is higher than or equal to a threshold level, the control unit (200) may provide a risk notification to notify of the risk of driving through at least one of the user interface (70) or the user device (2), and may determine the operation mode of the mobile robot (1) to be a standby mode or a cleaning mode.

[0144] The mobile robot (1) can stop driving and enter standby mode by stopping the operation of the wheel motor (30) and the blade motor (40a). When the mobile robot (1) enters standby mode, the control unit (200) can maintain the standby mode until the contaminants are removed from the optical sensor (S). The user can directly remove the contaminants from the optical sensor (S) or dock the mobile robot (1) to the charging station (4). The charging station (4) can clean the optical sensor (S) when the mobile robot (1) is docked.

[0145] The control unit (200) of the mobile robot (1) may determine to return to the charging station (4) upon entering the cleaning mode. That is, upon entering the cleaning mode, the control unit (200) may stop the operation of the blade motor (40a) and control the wheel motor (30) to move the mobile robot (1) to the charging station (4). An accident may occur while the mobile robot (1) is returning to the charging station (4) due to contamination of the optical sensor (S). The control unit (200) may provide a notification regarding the return of the mobile robot (1) through at least one of the user interface (70) or the user device (2) until docking of the mobile robot (1) and the charging station (4) is completed.

[0146] The control unit (200) can analyze the image acquired by the optical sensor (S) at predetermined intervals to determine whether the optical sensor (S) is contaminated. If an object determined to be a contaminant is not identified in the image, the control unit (200) can determine that the contamination of the optical sensor (S) has been removed. Based on the fact that an object determined to be a contaminant is not identified in the image, the control unit (200) can provide a contamination removal notification through at least one of the user interface (70) or the user device (2) and determine the operation mode of the mobile robot to be a normal mode.

[0147] In this way, the disclosed mobile robot (1) can prevent incorrect operation and accidents of the mobile robot (1) due to contamination of the optical sensor (S) by taking appropriate measures to respond to contamination of the optical sensor (S).

[0148] Meanwhile, despite the contamination of the optical sensor (S), the user may want the mobile robot (1) to continue performing the lawn cutting task. After the mobile robot (1) enters the standby mode or cleaning mode, a command to continue operation may be input through the user interface (70) or the user device (2). Based on obtaining the command to continue operation, the control unit (200) may change the operation mode of the mobile robot (1) from the standby mode or cleaning mode to the normal mode. In addition, according to the command to continue operation, the control unit (200) may change the threshold level regarding the contamination level to be higher than the current contamination level of the optical sensor (S). Therefore, even if the contamination level of the optical sensor (S) corresponds to the current contamination level in the future, a danger alert may not be provided, and the mobile robot (1) may not enter the standby mode or cleaning mode.

[0149] Fig. 9 shows an image acquired by a first optical sensor according to one embodiment. Fig. 10 shows an image acquired by a second optical sensor according to one embodiment.

[0150] Referring to FIG. 9, a first optical sensor (S1) can acquire a first image (IM1). Depending on a predetermined first field of view (FOV) of the first optical sensor (S1) and the size of a first image sensor included in the first optical sensor (S1), the first image (IM1) can have a first width (W1) and a first height (H1). For example, the first image (IM1) can have a vertically elongated rectangular frame.

[0151] Referring to FIG. 10, the second optical sensor (S2) can acquire a second image (IM2). Depending on a predetermined second field of view (FOV) of the second optical sensor (S2) and the size of the second image sensor included in the second optical sensor (S2), the second image (IM2) can have a second width (W2) and a second height (H2). For example, the second image (IM2) can have a rectangular frame with a long horizontal length.

[0152] The control unit (200) of the mobile robot (1) can align the first image (IM1) of the first optical sensor (S1) and the second image (IM2) of the second optical sensor (S2), and compare an object included in the first image (IM1) with an object included in the second image (IM2). In order to align the first image (IM1) and the second image (IM2), the control unit (200) can perform at least one of a cropping process that cuts out a portion of the image and a rescaling process of the image. For example, the second width (W2) of the second image (IM2) can be larger than the first width (W1) of the first image (IM1), and the second height (H2) of the second image (IM2) can be smaller than the first height (H1) of the first image (IM1). The control unit (200) can crop a portion of the second image (IM2) corresponding to the first horizontal width (W1) of the first image (IM1) and align it with the first image (IM1). The control unit (200) can determine whether an object (first object) detected in the first image (IM1) and an object (second object) detected in the second image (IM2) are the same or different.

[0153] The control unit (200) can detect objects with no feature change or relatively small feature change in each of the first image (IM1) and the second image (IM2) while the mobile robot (1) is moving. For example, despite the movement of the mobile robot (1), it can be detected that there is no feature change in the objects indicated as OB1 and O1 in the first image (IM1). Accordingly, the control unit (200) can determine that both OB1 and O1 detected in the first image (IM1) are contaminants.

[0154] However, the control unit (200) can detect an object corresponding to O1 of the first image (IM1) in the second image (IM2). In this case, the control unit (200) can determine the object indicated by O1 as a foreign object rather than a contaminant. An object located far from the mobile robot (1) (e.g., an object located near the vanishing point of the image) may show relatively little change in the image even when the mobile robot (1) moves. If such an object is determined to be a contaminant, the reliability of the contaminant determination decreases. The accuracy of the contaminant determination can be improved by comparing images acquired by different optical sensors.

[0155] Figure 11 shows an example of dividing an image generated by an optical sensor into multiple regions.

[0156] Referring to Fig. 11, the control unit (200) of the mobile robot (1) can divide the image (IM) acquired by the optical sensor (S) into a plurality of regions (RC, RL, RR, RU, RL). The image (IM) of Fig. 11 can correspond to the second image (IM2) of Fig. 10.

[0157] The control unit (200) can divide the image (IM) into a plurality of regions based on predetermined reference lines (L1, L2, L3, L4). For example, the image (IM) can be divided into a central region (RC), a left region (RL), a right region (RR), an upper region (RU), and a lower region (RL) by a left vertical line (L1), a right vertical line (L2), an upper horizontal line (L3), and a lower horizontal line (L4).

[0158] As illustrated in FIG. 11, an object OB2 located in an upper region (RU) of an image (IM) may be determined as a contaminant, and an object OB3 located in a central region (RC) of the image (IM) may be determined as a contaminant. The control unit (200) may determine the contamination level of the optical sensor (S) based on the weight of the central region (RC), the weight of the upper region (RU), the characteristics of the object OB2, and the characteristics of the object OB3. Since the weight of the upper region (RU) is relatively small, and the size of the object OB2 is also relatively small, the object OB2 may be ignored in determining the contamination level of the optical sensor (S).

[0159] However, since the weight of the central region (RC) is the greatest and the size of object OB2 located in the central region (RC) is also relatively large, the contamination level of the optical sensor (S) may be determined to be higher than the threshold level. In this case, the mobile robot (1) may provide a hazard notification to notify of the danger of driving through at least one of the user interface (70) or the user device (2). In addition, the mobile robot (1) may change its operation mode to a standby mode or a cleaning mode.

[0160] Fig. 11 is merely one example illustrating a method for determining the contamination level of an optical sensor (S). The contamination level of an optical sensor (S) can be determined in various ways depending on various situations.

[0161] FIG. 12 illustrates an example of information provided through a user interface or user device of a mobile robot according to one embodiment. FIG. 13 illustrates another example of information provided through a user interface or user device of a mobile robot according to one embodiment.

[0162] The first screen (1200) illustrated in FIG. 12 and the second screen (1300) illustrated in FIG. 13 may be displayed through the display (71) of the mobile robot (1) or the user device (2). Referring to FIGS. 12 and 13, the first screen (1200) and the second screen (1300) may include a notification box (N1) that displays contamination information of the optical sensor (S) and corresponding work information, and an image box (N2) that displays an image acquired by the optical sensor (S). An image including a contaminant may be displayed in the image box (N2).

[0163] When the contamination level of the optical sensor (S) is lower than a threshold level, a contamination notification may be displayed in the notification box (N1) as shown in the first screen (1200) to notify of contamination of the optical sensor (S) and normal operation of the mobile robot (1). For example, the contamination notification may be provided as a text message such as "Contamination of the sensor has been detected. Operation will continue, but sensor cleaning may be required."

[0164] If the contamination level of the optical sensor (S) is higher than or equal to a critical level, a hazard notification may be displayed in the notification box (N1) as shown in the second screen (1300) to indicate a risk of driving and that the mobile robot (1) has stopped operating and entered standby mode. For example, the hazard notification may be provided as a text message such as "Contamination of the sensor has been detected. Operation has been urgently stopped as a hazardous situation may occur."

[0165] In addition, the first screen (1200) and the second screen (1300) may include a work continuation button (B1) for inputting a command to continue operation of the mobile robot (1), a work stop button (B2) for inputting a command to stop operation of the mobile robot (1), and a sensor cleaning button (B3) for inputting a command to clean the sensor.

[0166] When a user input (e.g., touch input) is obtained through the work continuation button (B1) after the mobile robot (1) enters the standby mode or cleaning mode, the control unit (200) of the mobile robot (1) can change the operation mode of the mobile robot (1) to the normal mode. When the operation mode of the mobile robot (1) is forcibly changed to the normal mode according to the user input through the work continuation button (B1), a warning regarding a collision or abnormal driving can be displayed on the work continuation button (B1).

[0167] When a user input is obtained through a work stop button (B2) while the mobile robot (1) is operating in normal mode, the control unit (200) can determine the operation mode of the mobile robot (1) to be a standby mode. When a user input is obtained through a sensor cleaning button (B3), the control unit (200) can determine the operation mode of the mobile robot (1) to be a cleaning mode. When the mobile robot (1) enters the cleaning mode, a notification regarding the return of the mobile robot (1) can be displayed on the sensor cleaning button (B3).

[0168] In addition to what is exemplified in FIGS. 12 and 13, various information may be provided through the user interface (70) of the mobile robot (1) and the user device (2). For example, when contamination of the optical sensor (S) is removed, a contamination removal notification may be displayed in the notification box (N1).

[0169] Fig. 14 is a flowchart illustrating a control method of a mobile robot according to one embodiment.

[0170] Referring to FIG. 14, the control unit (200) of the mobile robot (1) can acquire an image from the optical sensor (S) while the mobile robot (1) is moving (1401). The control unit (200) can identify multiple objects in the image, and determine an object detected for a predetermined period of time among the identified multiple objects as a contaminant (1402). The control unit (200) can determine the contamination level of the optical sensor (S) based on the characteristics of the contaminant and the location of the contaminant in the image (1403).

[0171] The control unit (200) can provide information corresponding to the contamination level of the optical sensor (S) through the user interface (70) (1404). The information corresponding to the contamination level of the optical sensor (S) can include at least one of visual information and auditory information. The visual information can include at least one of an image, text, and graphic element displayed through the display (71). In addition, the visual information can include the color of the indicator (73). The auditory information can include at least one of a sound and a voice provided through the speaker (72). The control unit (200) can transmit the information to the user device (2) so that the information corresponding to the contamination level of the optical sensor (S) is output from the user device (2).

[0172] The control unit (200) can determine the operating mode of the mobile robot (1) based on the contamination level of the optical sensor (S) (1405). The operating mode of the mobile robot (1) can be determined as a normal mode, a standby mode, or a cleaning mode. The control unit (200) can control the mobile robot (1) according to the determined operating mode (1406).

[0173] FIG. 15 is a flowchart illustrating in more detail the method for determining the contamination level of the optical sensor described in FIG. 14.

[0174] Referring to FIG. 15, in order to determine the contamination level of the optical sensor (S) according to the location of an object within an image, the control unit (200) of the mobile robot (1) can divide the image into a plurality of regions based on a predetermined reference line (1501). For example, the control unit (200) can divide the entire region of the image acquired by the optical sensor (S) into a central region, a left region, a right region, an upper region, and a lower region. The control unit (200) can determine the contamination level of the optical sensor (S) differently depending on the region in which the object determined to be a contaminant is located among the plurality of regions of the image.

[0175] The control unit (200) can apply weights to each of multiple regions of the image (1502). The control unit (200) can apply the largest first weight to the central region of the image. The control unit (200) can apply a second weight, which is smaller than the first weight, to the left and right regions of the image. The control unit (200) can apply a third weight, which is smaller than the second weight, to the upper and lower regions of the image.

[0176] The control unit (200) can determine the contamination level of the optical sensor (S) based on the weight of the area where the contaminant is located and the characteristics of the contaminant (1503). The control unit (200) can determine the contamination level of the optical sensor (S) to be higher as the weight of the area where the contaminant is located increases. The characteristics of the contaminant considered to determine the contamination level of the optical sensor (S) may include at least one of the size of the object or the transparency of the object. The control unit (200) can determine the contamination level of the optical sensor (S) to be higher as the size of the contaminant increases. The control unit (200) can determine the contamination level of the optical sensor (S) to be higher as the transparency of the contaminant decreases. In addition, the control unit (200) can determine the contamination level of the optical sensor (S) to be higher as the brightness of the contaminant decreases. The control unit (200) can determine the contamination level of the optical sensor (S) to be higher as the clarity of the contaminant decreases.

[0177] There may be multiple objects determined to be contaminants in the image. The control unit (200) can determine the contamination level of the optical sensor (S) by considering the weights of the areas where each of the multiple contaminants is located in the image and the characteristics of each of the multiple contaminants.

[0178] Fig. 16 is a flowchart explaining in more detail the operation of the mobile robot according to the contamination level of the optical sensor described in Fig. 14.

[0179] Referring to FIG. 16, the control unit (200) of the mobile robot (1) can determine whether the contamination level of the optical sensor (S) is lower than the threshold level, higher than the threshold level, or equal to the threshold level (1601).

[0180] The control unit (200) may provide a contamination notification to notify of contamination of the optical sensor (S) through at least one of the user interface (70) or the user device (2) based on the contamination level of the optical sensor (S) being lower than a threshold level (1602), and may determine the operation mode of the mobile robot (1) to be a normal mode (1603).

[0181] The control unit (200) may provide a risk notification to notify of a driving risk through at least one of the user interface (70) or the user device (2) based on whether the contamination level of the optical sensor (S) is higher than or equal to a threshold level (1604), and may determine the operation mode of the mobile robot (1) to be a standby mode or a cleaning mode (1605).

[0182] After the mobile robot (1) enters standby mode or cleaning mode, the control unit (200) may change the operation mode of the mobile robot (1) to normal mode (1607) upon receiving an operation continuation command through the user interface (70) or the user device (2). In addition, the control unit (200) may update the threshold level regarding the contamination level (1608). That is, the control unit (200) may change the threshold level regarding the contamination level to be higher than the current contamination level of the optical sensor (S).

[0183] The control unit (200) can identify whether a contaminant has been removed from the image of the optical sensor (S) (1609). If an object determined to be a contaminant is not identified in the image, the control unit (200) can determine that the contamination of the optical sensor (S) has been removed. Based on the fact that a contaminant is not identified in the image, the control unit (200) can provide a contamination removal notification through at least one of the user interface (70) or the user device (2) (1610) and determine the operation mode of the mobile robot to be the normal mode. Accordingly, the mobile robot (1) can operate in the normal mode (1611).

[0184] FIG. 17 illustrates a further embodiment of a method for determining contaminants in an image as described in FIG. 14.

[0185] As described above, the mobile robot (1) may be provided with two or more different optical sensors (S). For example, the mobile robot (1) may include a first optical sensor (S1) that generates a first image and a second optical sensor (S2) that generates a second image.

[0186] Referring to FIG. 17, the control unit (200) of the mobile robot (1) can obtain a first image from the first optical sensor (S1) while the mobile robot (1) moves (1701), and can obtain a second image from the second optical sensor (S2) (1702). The control unit (200) can determine an object detected in the first image of the first optical sensor (S1) for a predetermined period of time as a contaminant (1703).

[0187] The control unit (200) can align the first image of the first optical sensor (S1) with the second image of the second optical sensor (S2), and compare an object included in the first image with an object included in the second image. The control unit (200) can determine whether an object determined to be a contaminant in the first image is also detected in the same manner in the second image of the second optical sensor (S2) (1704).

[0188] The control unit (200) can determine that an object determined as a contaminant in the first image is a foreign object rather than a contaminant based on the fact that the object determined as a contaminant in the first image is also detected in the second image (1705). If the object determined as a contaminant in the first image is not detected in the second image, the control unit (200) can determine that contamination has occurred in the first optical sensor (S1) and can determine the contamination level of the first optical sensor (S1) (1403).

[0189] According to one embodiment, a mobile robot includes a main body (10), a plurality of wheels (20a, 20b, 20c, 20d) provided on the main body, a blade (40) provided on the main body for cutting grass, an optical sensor (S) having a field of view facing forward of the main body and generating an image, a user interface (70) obtaining a user input and providing information regarding the operation of the mobile robot, and at least one processor (210). The at least one processor can identify a plurality of objects in the image obtained while the main body moves. The at least one processor can determine an object detected for a predetermined period of time among the identified plurality of objects as a contaminant. The at least one processor can determine a contamination level of the optical sensor based on a characteristic of the contaminant and a location of the contaminant within the image. The at least one processor can provide information corresponding to the contamination level of the optical sensor through the user interface. The at least one processor can determine an operation mode of the mobile robot based on the contamination level of the optical sensor.

[0190] The at least one processor may provide a contamination notification to notify of contamination of the optical sensor through the user interface based on the contamination level of the optical sensor being lower than a threshold level, and determine the operation mode of the mobile robot to be a normal mode.

[0191] The at least one processor may provide a hazard notification to notify of a driving hazard through the user interface based on whether the contamination level of the optical sensor is higher than or equal to a threshold level, and may determine the operation mode of the mobile robot to be a standby mode or a cleaning mode.

[0192] The at least one processor can change the operation mode of the mobile robot from the standby mode or the cleaning mode to the normal mode based on an operation continuation command input through the user interface, and change the threshold level to be higher than the determined contamination level of the optical sensor.

[0193] The at least one processor may provide a contamination removal notification through the user interface based on the object determined as the contamination not being identified in the image, and determine the operation mode of the mobile robot to be a normal mode.

[0194] The at least one processor may divide the image into a plurality of regions based on a predetermined reference line, and determine the contamination level of the optical sensor differently depending on the region where the contaminant is located among the plurality of regions.

[0195] The at least one processor may apply a first weight, which is the largest, to a central region of the image, a second weight, which is smaller than the first weight, to a left region and a right region of the image, and a third weight, which is smaller than the second weight, to an upper region and a lower region of the image, and the greater the weight of the region where the contaminant is located, the higher the contamination level of the optical sensor may be determined to be.

[0196] The characteristics of the contaminant include at least one of a size of the contaminant and a transparency of the contaminant, and the at least one processor can determine a higher level of contamination of the optical sensor as the size of the contaminant increases or the transparency of the contaminant decreases.

[0197] The optical sensor may include a first optical sensor that generates a color image; and a second optical sensor that generates a depth image including distance information to an object. The at least one processor may determine the object as a foreign object rather than a contaminant based on the object being detected identically in both the color image of the first optical sensor and the depth image of the second optical sensor.

[0198] The at least one processor may provide at least one of visual information or auditory information corresponding to the contamination level of the optical sensor through the user interface.

[0199] The above mobile robot may further include a communication unit for communicating with an external device. The at least one processor may transmit the information corresponding to the contamination level of the optical sensor to the user device through the communication unit so that the information is displayed on the user device.

[0200] A method for controlling a mobile robot according to one embodiment includes detecting a plurality of objects in an image acquired by an optical sensor while the mobile robot moves; determining an object detected for a predetermined period of time among the plurality of objects as a contaminant; determining a contamination level of the optical sensor based on a characteristic of the contaminant and a location of the contaminant within the image; providing information corresponding to the contamination level of the optical sensor through the user interface; and determining an operation mode of the mobile robot based on the contamination level of the optical sensor.

[0201] Based on the contamination level of the optical sensor being lower than a threshold level, information corresponding to the contamination level of the optical sensor is provided as a contamination notification to notify of contamination occurrence of the optical sensor, and the operation mode of the mobile robot can be determined as a normal mode.

[0202] Based on whether the contamination level of the optical sensor is higher than or equal to a threshold level, information corresponding to the contamination level of the optical sensor is provided as a risk notification to indicate a risk of driving, and the operation mode of the mobile robot can be determined as a standby mode or a cleaning mode.

[0203] The method for controlling the mobile robot may further include changing the operation mode of the mobile robot from the standby mode or the cleaning mode to the normal mode based on a motion continuation command input through the user interface, and changing the threshold level to be higher than the determined contamination level of the optical sensor.

[0204] Based on the fact that the object determined to be the contaminant is not identified in the image, information corresponding to the contamination level of the optical sensor is provided as a contamination removal notification, and the operation mode of the mobile robot can be determined as a normal mode.

[0205] Determining the contamination level of the optical sensor may include dividing the image into a plurality of regions based on a predetermined reference line; and determining the contamination level of the optical sensor differently depending on the region in which the object is located among the plurality of regions.

[0206] Determining the contamination level of the optical sensor may include: applying a first weight, which is the largest, to a central area of ​​the image; applying a second weight, which is smaller than the first weight, to a left area and a right area of ​​the image; applying a third weight, which is smaller than the second weight, to an upper area and a lower area of ​​the image; and determining a higher contamination level of the optical sensor as the weight of the area where the contamination is located is larger.

[0207] The characteristics of the contaminant include at least one of the size of the contaminant and the transparency of the contaminant, and determining the degree of contamination of the optical sensor may include determining a higher degree of contamination of the optical sensor as the size of the contaminant increases or the transparency of the contaminant decreases.

[0208] The optical sensor may include a first optical sensor that generates a color image; and a second optical sensor that generates a depth image including distance information to an object. The control method of the mobile robot may further include determining that the object is a foreign object rather than a contaminant based on the object being detected identically in both the color image of the first optical sensor and the depth image of the second optical sensor.

[0209] As described above, the disclosed mobile robot, mobile robot control method, and system can detect contamination of an optical sensor exposed to the external environment and take appropriate measures to address the contamination. Accordingly, incorrect operation and accidents of the mobile robot due to contamination of the optical sensor can be prevented.

[0210] The disclosed embodiments may be implemented in the form of a storage medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments.

[0211] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0212] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0213] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present invention can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. Main body (10); A plurality of wheels (20a, 20b, 20c, 20d) provided on the above body; A blade (40) provided on the above body for cutting grass; An optical sensor (S) having a field of view facing the front of the main body and generating an image; A user interface (70) that obtains user input and provides information about the movement of the mobile robot; At least one processor (210) controlling the wheel, the blade, the optical sensor and the user interface; At least one of the above processors Detecting multiple objects in the image acquired while the above main body moves, Among the above detected multiple objects, an object detected for a predetermined period of time is determined as a contaminant, Determining the contamination level of the optical sensor based on the characteristics of the contaminant and the location of the contaminant within the image; Providing information corresponding to the contamination level of the optical sensor through the user interface, A mobile robot that determines the operation mode of the mobile robot based on the contamination level of the optical sensor.

2. In paragraph 1, At least one of the above processors A mobile robot that provides a contamination notification to notify of contamination of the optical sensor through the user interface based on the contamination level of the optical sensor being lower than a threshold level, and determines the operation mode of the mobile robot to be the normal mode.

3. In paragraph 1, At least one of the above processors A mobile robot that provides a hazard notification to notify of a driving hazard through the user interface based on whether the contamination level of the optical sensor is higher than or equal to a threshold level, and determines the operation mode of the mobile robot to be a standby mode or a cleaning mode.

4. In paragraph 3, At least one of the above processors A mobile robot that changes the operation mode of the mobile robot from the standby mode or the cleaning mode to the normal mode based on a motion continuation command input through the user interface, and changes the threshold level to be higher than the determined contamination level of the optical sensor.

5. In paragraph 1, At least one of the above processors A mobile robot that provides a contamination removal notification through the user interface based on the object determined as the contaminant in the image being not identified, and determines the operation mode of the mobile robot to be the normal mode.

6. In paragraph 1, At least one of the above processors Divide the image into multiple regions based on pre-determined baselines, A mobile robot that determines the contamination level of the optical sensor differently depending on the area where the contaminant is located among the above multiple areas.

7. In paragraph 6, At least one of the above processors Apply the largest first weight to the central area of ​​the above image, Applying a second weight smaller than the first weight to the left and right areas of the image, A third weight smaller than the second weight is applied to the upper and lower areas of the image, A mobile robot that determines the contamination level of the optical sensor as the weight of the area where the contaminant is located is greater.

8. In paragraph 1, The characteristics of the above contaminants are Containing at least one of the size of the contaminant and the transparency of the contaminant, The above control unit A mobile robot that determines a higher contamination level of the optical sensor as the size of the contaminant increases or the transparency of the contaminant decreases.

9. In paragraph 1, The above optical sensor a first optical sensor for generating a color image; and a second optical sensor for generating a depth image including distance information to the target; At least one of the above processors A mobile robot that determines that the object is a foreign object and not a contaminant based on the object being detected identically in both the color image of the first optical sensor and the depth image of the second optical sensor.

10. In paragraph 1, At least one of the above processors A mobile robot providing at least one of visual information or auditory information corresponding to the contamination level of the optical sensor through the user interface.

11. In paragraph 1, Further comprising a communication unit for communicating with an external device; At least one of the above processors A method for controlling a mobile robot, wherein the information corresponding to the contamination level of the optical sensor is displayed on the user device through the communication unit, thereby transmitting the information to the user device.

12. Detecting multiple objects in an image acquired by an optical sensor having a field of view facing the front of the main body while the mobile robot moves; An object detected for a predetermined period of time among the above multiple objects is determined as a contaminant; Determining the contamination level of the optical sensor based on the characteristics of the contaminant and the location of the contaminant within the image; Providing information corresponding to the contamination level of the optical sensor through a user interface; A method for controlling a mobile robot, comprising: determining an operation mode of the mobile robot based on the contamination level of the optical sensor.

13. In paragraph 12, Based on the contamination level of the optical sensor being lower than the critical level, A control method for a mobile robot, wherein information corresponding to the contamination level of the optical sensor is provided as a contamination notification for notifying the occurrence of contamination of the optical sensor, and the operation mode of the mobile robot is determined as a normal mode.

14. In paragraph 12, Based on the contamination level of the optical sensor being higher than or equal to the threshold level, A method for controlling a mobile robot, wherein information corresponding to the contamination level of the optical sensor is provided as a hazard notification for notifying a hazard of driving, and the operation mode of the mobile robot is determined as a standby mode or a cleaning mode.

15. In paragraph 14, A method for controlling a mobile robot, further comprising: changing the operation mode of the mobile robot from the standby mode or the cleaning mode to the normal mode based on a motion continuation command input through the user interface; and changing the threshold level to be higher than the determined contamination level of the optical sensor.

Images