Information processing device, robot, and program

Abstract

To provide a robot for relieving the burden of work to determine an area of map data to be performed by a user.SOLUTION: An information processing apparatus determines an area to which the robot is moved, based on first map data which a moving robot acquired by a laser scanner, an execution result of semantic segmentation on an image acquired by a camera, and second map data acquired by the camera.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to technology for robots that perform autonomous travel. [Background Art]

[0002] Technology that causes a robot itself to create map data is known. For example, Patent Document 1 describes an autonomous traveling work device that recognizes markers present on a work surface from captured images obtained by imaging the work surface, creates a movement trajectory of the marker from time-series consecutive captured images, and sets an area inside the movement trajectory of the marker as a work area. [Prior Art Literature] [Patent Literature]

[0003] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2022-26429 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, the above technique requires manual placement of markers to be recognized by the robot in order to determine the area of the map data created by the robot.

[0005] In view of this, the present invention provides a system that creates map data (hereinafter referred to as "exploration") by causing a robot to perform autonomous travel while the robot itself determines the area of map data created by a user, in a state where no map data is stored inside the robot. [Means for Solving the Problem]

[0006] The present invention provides, in its first embodiment, an information processing device that determines an area in which a moving robot moves, based on first map data acquired by a laser scanner, the results of semantic segmentation performed on an image acquired by a camera, and second map data acquired by the camera. According to the information processing device of the first embodiment, the user does not need to place markers or the like to allow the robot to recognize the area of ​​the map data.

[0007] In the first embodiment of the information processing device, a second embodiment may be adopted in which the execution result includes information indicating areas where it is not necessary to acquire first map data and second map data. According to the second embodiment of the information processing device, the robot can recognize areas where it is not necessary to acquire map data.

[0008] A third embodiment may be adopted in which, in the information processing device of the first embodiment, the first map data and the second map data include information indicating the area in which the robot can move. According to the information processing device of the third embodiment, the robot can recognize the area in which it can move.

[0009] The present invention provides, in a fourth embodiment, a robot having a laser scanner and a camera, which moves within an area determined by an information processing device according to any one of claims 1 to 3. According to the robot of the fourth embodiment, the burden on the user in determining the area of ​​map data is reduced.

[0010] In the fourth embodiment of the robot, a fifth embodiment may be adopted in which the laser scanner is a two-dimensional laser scanner or a three-dimensional laser scanner, and the camera is a monocular camera or a stereo camera.

[0011] In the robot of the fourth embodiment, a sixth embodiment may be adopted in which the laser scanner and the camera are activated simultaneously, and the position of the robot at the time of activation is set as the origin of the first map data and the second map data.

[0012] The present invention provides, in a seventh aspect, a program for causing a computer to perform the following steps: acquiring first map data acquired by a moving robot with a laser scanner; acquiring images acquired by the robot with a camera; performing semantic segmentation on the images; acquiring second map data acquired by the camera; and determining an area in which the robot will move based on the first map data, the second map data, and the results of the semantic segmentation. According to the program in the seventh aspect, the burden on the user in determining the area of ​​the map data is reduced. [Effects of the Invention]

[0013] According to the present invention, the burden of work performed by the user to determine the area of ​​map data can be reduced. [Brief explanation of the drawing]

[0014] [Figure 1] A diagram showing the configuration of robot 1. [Figure 2] A diagram showing the external appearance of robot 1. [Figure 3] A diagram showing the configuration of the laser scanner 16. [Figure 4] A flowchart illustrating an example of the operation flow of robot 1. [Figure 5] A diagram showing an example of LiDAR map data. [Figure 6] A diagram showing an example of image data. [Figure 7] A diagram showing an example of classification data. [Figure 8] A diagram showing an example of image map data that reflects the results of classification data. [Figure 9] A diagram showing an example of integrated map data. [Figure 10] A flowchart illustrating the movement control of robot 1. [Modes for carrying out the invention]

[0015] [1] Example The robot 1 in the present example is a robot mainly used in buildings. The robot 1 is used, for example, to acquire various types of data such as map data in a building. The robot 1 stores the acquired data and performs cleaning while autonomously traveling within the building, for example.

[0016] Figure 1 is a diagram showing the configuration of the robot 1. The robot 1 shown in Figure 1 includes a processor 11, a memory 12, a communication unit 13, an operation unit 14, a display unit 15, a laser scanner 16, a camera 17, an image processing unit 18, a determination unit 19, and a moving unit 20. These components are communicably connected to each other via, for example, a bus.

[0017] The processor 11 controls each part of the robot 1 by reading and executing a computer program (hereinafter simply referred to as a program) stored in the memory 12. The processor 11 is, for example, a CPU (Central Processing Unit).

[0018] The memory 12 stores an operating system, various programs, data and the like to be read by the processor 11. The memory 12 includes a RAM (Random Access Memory) and a ROM (Read Only Memory). Note that the memory 12 may also include a storage device such as a solid-state drive or a hard disk drive.

[0019] Memory 12 also stores LiDAR (Light Detection and Ranging) map data (hereinafter also referred to as first map data), image data, classification data, image map data (hereinafter also referred to as second map data), and integrated map data. LiDAR map data is two-dimensional map data or three-dimensional map data sampled using LiDAR SLAM (Simultaneous Localization and Mapping) technology using LiDAR. Image data is data showing images taken by the robot using a camera during exploration operations. Classification data is data obtained by the robot classifying what is displayed in each pixel of the image data by performing semantic segmentation on the image data. Image map data is three-dimensional map data sampled using Visual SLAM technology using a camera. Integrated map data is data showing a two-dimensional map or three-dimensional map that integrates the LiDAR map data, image map data, and classification data.

[0020] The communication unit 13 is a communication circuit that connects the robot 1 to other devices via wired or wireless means, enabling communication between them.

[0021] The operation unit 14 transmits signals to the processor 11 in response to user operations. The operation unit 14 has control elements such as operation buttons and a touch panel for receiving various instructions. The robot 1 does not necessarily have to have an operation unit 14. The robot 1 may be operated from an external device via the communication unit 13.

[0022] The display unit 15 displays an image under the control of the processor 11. The display unit 15 has a display screen such as a liquid crystal display. A transparent touch panel of the operation unit 14 may be placed on top of this display screen. The robot 1 does not necessarily have to have a display unit 15. The robot 1 may present information to an external device via the communication unit 13.

[0023] The laser scanner 16 is a two-dimensional or three-dimensional laser scanner that acquires LiDAR map data. The laser scanner 16 has a light-emitting element, an optical system, and a photodetector. The light-emitting element generates laser light. The laser light generated by the light-emitting element is, for example, a pulsed laser in the near-infrared wavelength band. The optical system changes the direction of propagation of the generated laser light to scan the space in which an object exists with this laser light. The optical system includes, for example, an electromagnetically driven MEMS (Micro Electro Mechanical System). This optical system periodically performs Lissajous scanning of the laser light in the up and down and left and right directions. The photodetector has a photodetector that receives reflected light that returns when the laser light scanning the space is reflected off an object. The photodetector is, for example, a photodiode.

[0024] Furthermore, the scanning method of the optical system is not limited to Lissajous scanning. The optical system may perform so-called raster scanning, in which information corresponding to a line is obtained by scanning in one dimension in the vertical or horizontal direction, and then the scanning point is shifted in a direction perpendicular to the line and the scanning is repeated in another dimension.

[0025] The optical system may have either a separate optical system or a coaxial optical system structure, but a coaxial optical system is preferable from the viewpoint of resistance to ambient light.

[0026] The laser scanner 16 uses a method known as time-of-flight (ToF), which measures distance from the round-trip time it takes for the emitted laser light to reflect off the object and return. The laser scanner 16 measures the distance at each coordinate from the combination of the time it passes through the coordinate on the Lissajous figure and the received light intensity corresponding to that coordinate, and generates a distance image. Alternatively, the laser scanner 16 may measure the distance to the object using a frequency-modulated continuous wave (FMCW) method instead of the ToF method described above.

[0027] Camera 17 is an imaging device that captures images of the building surrounding Robot 1 to generate and acquire image data, and also acquires three-dimensional map data using Visual SLAM technology. Camera 17 can be, for example, a monocular camera or a stereo camera. Visual SLAM can be implemented with a monocular camera by combining it with technologies such as wheel odometry.

[0028] The image processing unit 18 performs semantic segmentation on the image data to generate classification data. The image processing unit 18 also generates image map data based on the image data. Semantic segmentation is a type of algorithm in deep learning that classifies what is reflected in each pixel of an image by associating it with a label or category. Image map data is data that shows areas that are impassable due to obstacles, areas that have been deemed impassable by semantic segmentation, and areas that are passable.

[0029] Furthermore, the image processing unit 18 integrates the LiDAR map data, image map data, and classification data to generate integrated map data. Integrated map data is data obtained by integrating information on whether or not movement is possible, obtained by LiDAR SLAM, and information on "areas that do not need to be explored," obtained by semantic segmentation, for the area surrounding the robot 1, using information on the correspondence between images and maps obtained by Visual SLAM.

[0030] "Areas that do not need to be explored" include areas such as outside buildings, inside shops, stairs, escalators, elevators, and restricted areas. These "areas that do not need to be explored" are examples of areas where it is not necessary to acquire map data such as LiDAR map data, image map data, and integrated map data. "Areas that need to be explored" (hereinafter also referred to as unexplored areas) include areas such as customer passageways. Unexplored areas are areas where the laser beam of the laser scanner 16 has not scanned and which are also not "areas that do not need to be explored." These areas are predefined.

[0031] The determination unit 19 determines, based on the integrated map data, whether or not there are areas that need to be explored, i.e., unexplored areas. Note that the processor 11 may also function as the determination unit 19.

[0032] Figure 2 shows the external appearance of robot 1. This robot 1 is an example of a robot that senses reflected light from an emitted laser beam. In the following diagram, the space in which each component is arranged is represented as an xyz right-handed coordinate system. Among the coordinate symbols shown in the diagram, the symbol of a point drawn inside a circle represents an arrow pointing from the back of the paper to the front. In space, the direction along the x-axis is called the x-axis direction. Furthermore, within the x-axis direction, the direction in which the x component increases is called the +x direction, and the direction in which the x component decreases is called the -x direction. The y-axis direction, +y direction, -y direction, z-axis direction, +z direction, and -z direction are also defined according to the above definition for the y and z components.

[0033] The moving unit 20 is a tire located on the bottom surface of the robot 1. The moving unit 20 moves the robot 1 under the control of the processor 11. The moving unit 20 has tires, one on each side, an axle that rotatably supports the tires, and a drive device such as a motor (not shown) that rotates the axle. The moving unit 20 is not limited to these one tire on each side; for example, as shown in Figure 2(c), it may have another tire (auxiliary wheel) on the rear (-x direction) side. In this embodiment, since the moving unit 20 is a tire, "movement" of the robot 1 is synonymous with "driving". Note that the robot 1 may be moved by means other than driving. For example, the robot 1 may be an unmanned aerial vehicle or flying object that flies (moves) in the air using rotor blades or the like.

[0034] Furthermore, the -z direction is the direction of gravity, i.e., downwards, and the +z direction is upwards. The +x direction is the direction of movement when robot 1 moves forward, and the -x direction is the direction of movement when robot 1 moves backward. The y-axis direction is perpendicular to the direction of movement of robot 1, i.e., the width direction.

[0035] Figures 2(a), 2(b), and 2(c) show the top view, front view, and side view of robot 1, respectively. As shown in Figure 2(b), a laser scanner 16 and a camera 17 are provided on the front (+x direction) side of robot 1.

[0036] In the laser scanner 16, the optical system, for example, changes the direction of laser light propagation only in the horizontal direction and not in the vertical direction. The optical system used in the laser scanner 16 is, for example, a mirror that rotates around an axis parallel to the z-axis direction. The laser scanner 16 reflects the laser light generated by the light-emitting element using the rotating mirror and sequentially irradiates a predetermined range, such as a fan shape, in front of the robot 1. The laser scanner 16 receives the reflected light with a light-receiving element such as a photodiode and detects an object that reflects the laser light at a predetermined height in front of it.

[0037] Figure 3 shows the scanning range of the laser scanner 16. The laser scanner 16 shown in Figure 3(a) scans an area corresponding to at least the width W1 of the robot 1. The road width W2 indicates the width of the road that the robot 1 is traveling on. The laser scanner 16 shown in Figure 3(b) scans the space in front of it. Since the scanning range of the laser scanner 16 is the space in front of the robot 1, there may be a blind spot directly below the laser scanner 16.

[0038] Figure 4 is a flowchart showing an example of the operation flow of robot 1. The processor 11 of robot 1 determines whether or not it has received an input to start exploration from an external device via the communication unit 13 or from the operation unit 14 (step S101). The processor 11 continues this determination until it determines that it has not received an input to start exploration (step S101: NO).

[0039] If the processor determines that it has received input to start exploration (step S101: YES), it instructs the laser scanner 16 to acquire LiDAR map data. The processor 11 also instructs the camera 17 to acquire image data (step S102). Step S102 is an example of a step in which a moving robot acquires map data acquired by the laser scanner. Step S102 is also an example of a step in which a moving robot acquires images acquired by the camera. The processor 11 instructs the robot 1 to acquire LiDAR map data and image data while moving or rotating the robot 1 over a predetermined period of time without contacting any obstacles.

[0040] Figure 5 shows an example of LiDAR map data. Figure 5(a) shows the state at the start of the exploration. Robot 1, located at position O, scans its surroundings with laser light using a laser scanner 16. By detecting the reflection of the emitted laser light, Robot 1 identifies the presence or absence of obstacles such as walls, doors, and signs.

[0041] On the other hand, laser light attenuates as it travels further, so it may not reach objects that are a certain distance away. Also, even if the laser light reaches a distant object, the reflected light may attenuate and may not return to the irradiation position with detectable intensity. Robot 1 cannot identify what is in the direction from which the reflected laser light does not return. Furthermore, at this stage, since Robot 1 has only emitted laser light from position O, if the two adjacent obstacles that have been identified are not consecutive, it cannot identify what is between those two obstacles.

[0042] In other words, robot 1 uses LiDAR to explore the surrounding area and distinguish it into three types: areas where it can travel without obstacles (hereinafter referred to as the traversable area 501), areas where it cannot travel due to obstacles such as walls (hereinafter referred to as the non-traversable area 502), and areas that do not fall into any of the above categories and have not yet been explored (hereinafter referred to as the unexplored area 504).

[0043] However, this LiDAR map data only identifies the presence or absence of obstacles within the range of the laser beam, and does not indicate the type of surrounding objects, such as whether they are shops or elevators, which is relevant to determining whether an investigation is necessary. Therefore, robot 1 cannot determine whether an investigation is necessary based solely on this LiDAR map data. In other words, the unexplored area 504 and the traversable area 501 at this point may include not only the "areas that need to be explored" but also the "areas that do not need to be explored."

[0044] Figure 5(b) shows the area identified by robot 1 at position O through exploration. In Figure 5(b), the white area is the traversable area 501, the black area is the non-traversable area 502, and the shaded area is the unexplored area 504. The boundaries E1 to E5 shown in this figure are the boundaries between the traversable area 501 and the unexplored area 504. As mentioned above, the unexplored area 504 includes areas where reflected light of detectable intensity did not return because the obstacles were too far away, and areas between two non-contiguous obstacles that are in a blind spot due to the obstacle in front, making it impossible to know what is there. For example, boundary E3 shown in this figure is the former, while boundaries E1, E2, E4, and E5 are the latter.

[0045] Next, if there is an unexplored area 504 within the identified region, robot 1 moves to attempt to explore that unexplored area 504. That is, robot 1 moves to change the irradiation position and irradiation angle in an attempt to identify new obstacles that the laser light could not reach. At this time, since movement is limited to the traversable area 501, robot 1 moves within the traversable area 501 to approach the unexplored area 504. Thereupon, robot 1 identifies the boundaries E1 to E5 shown in Figure 5(b) as candidate destinations for movement.

[0046] However, as mentioned above, the unexplored area 504 and the drivable area 501 may include areas that do not need to be explored, such as the inside of a store. Therefore, robot 1 utilizes the results of semantic segmentation of the surrounding images captured by camera 17.

[0047] Figure 6 shows an example of image data. Image data is an image such as a photograph, but here it will be explained using a line diagram. The image data shown in Figure 6 is data showing the surroundings captured by camera 17. Figure 6 shows areas such as the passageway 601a, 601b, wall 602a, 602b, signboard 603a, 603b, 603c, and store entrance 604a, 604b.

[0048] Returning to Figure 4, the processor 11 instructs the image processing unit 18 to perform Visual SLAM using the acquired image data (step S103). The image processing unit 18 extracts singularities from the surrounding images shown in each of the acquired image data, creates a map of the surroundings using the changes in these singularities, and estimates the position of the robot 1 on that map. As a result, the image processing unit 18 generates image map data that includes the information of the surrounding map and the position of the robot 1 itself.

[0049] Furthermore, the processor 11 causes the image processing unit 18 to perform semantic segmentation on the acquired image data and generate classification data (step S104). Step S104 is an example of a step in which semantic segmentation is performed on the acquired image. Well-known techniques are used for semantic segmentation. As shown in Figure 4, this semantic segmentation may be performed in parallel with the Visual SLAM described above, or it may be performed sequentially before or after.

[0050] Figure 7 shows an example of classified data. The figure in Figure 7 shows how semantic segmentation has been performed on the image data shown in Figure 6, and how the displayed areas have been classified. The image processing unit 18 uses semantic segmentation to classify the areas displayed in the image data into categories that have been pre-assigned meaning by humans, such as corridors, walls, signs, and store entrances. This classification is performed using a pre-generated, trained model. This trained model can be generated, for example, by training a computer using photographic images of various types of objects as training data.

[0051] Furthermore, the above classifications are also categorized based on whether or not robot 1 can travel through them. For example, passageways are classified as areas where robot 1 can travel, while walls and signs are classified as areas where robot 1 cannot travel. Additionally, the above classifications are also categorized based on whether or not exploration is necessary. For example, the outside of buildings, stairs, escalators, elevators, restricted areas, and store entrances are all classified as areas that do not require exploration.

[0052] The image processing unit 18 classifies the areas displayed in the image data by type, and further classifies them into traversable areas 701, non-traversable areas 702, and non-traversable areas 703, as shown in Figure 7. The traversable areas 701 are areas corresponding to, for example, the passage 601a shown in Figure 6, and are areas where the robot 1 can travel. The non-traversable areas 702 are areas corresponding to, for example, the wall 602a shown in Figure 6, and are areas where the robot 1 cannot travel. The non-traversable areas 703 are areas corresponding to, for example, the store entrance 604a shown in Figure 6, and are areas where the robot 1 does not need to travel.

[0053] As mentioned above, the image map data and the classification data are generated from the same image data, and therefore there is a correspondence between them. Based on this correspondence, the image processing unit 18 reflects the results of the classification data in the image map data.

[0054] Figure 8 shows an example of image map data that reflects the results of the classification data. The image processing unit 18 obtains the correspondence between the image and the map using the image map data and applies the classification results described above to the map. As a result, the drivable area 701, the non-drivable area 702, and the non-drivable area 703 shown in Figure 7 are assigned locations in the image map data 800 shown in Figure 8, becoming the drivable area 801, the non-drivable area 802, and the non-drivable area 803, respectively.

[0055] Furthermore, the positional relationship between the camera 17 and the laser scanner 16 in robot 1 is known. Returning to Figure 4, the image processing unit 18 uses this known positional relationship to integrate the LiDAR map data, image map data, and classification data to generate integrated map data (step S105).

[0056] Figure 9 shows an example of integrated map data. Integrated map data is generated by reflecting classification data information in LiDAR map data via image map data. Robot 1 starts acquiring LiDAR map data and image data simultaneously from the moment it receives input to start exploration (steps S101, S102). Therefore, the position of the robot (Robot 1) estimated in the LiDAR map data at startup is also the position of the robot when the image data acquired simultaneously was taken. As described above, since the positional relationship between the laser scanner 16 and the camera 17 is known, the LiDAR map data and image map data can be integrated by using the estimated position of the robot at startup as a common origin. In other words, this robot 1 is an example of a robot that starts the laser scanner and camera simultaneously and uses the robot's position at startup as the origin of the map data. Note that Robot 1 does not have to start the laser scanner 16 and camera 17 at the same time. The laser scanner 16 and camera 17 can be started at different times, for example, as long as a common origin can be recognized. For example, the laser scanner 16 and camera 17 are activated at different times, and robot 1 simply needs to avoid moving during that time. Since the image map data is three-dimensional data (length, width, height), the image processing unit 18 converts the image map data into two-dimensional data (length, width) at the height where the camera 17 is mounted when generating the integrated map data.

[0057] Furthermore, as mentioned above, since image map data and classification data are generated from common image data, they can also be associated with each other. In other words, image map data associates LiDAR map data and classification data using location information and image information. While a map can be obtained using image map data alone, due to camera resolution, optical system distortion, etc., image map data generated by Visual SLAM generally tends to be less accurate than LiDAR map data generated by LiDAR SLAM. Therefore, in this embodiment, robot 1 improves map accuracy by integrating image map data with LiDAR map data.

[0058] Returning to Figure 4, the processor 11 instructs the determination unit 19 to determine whether or not unexplored areas exist based on the integrated map data generated by the image processing unit 18 (step S106). For this determination, well-known techniques, such as explore lite, a package of the ROS (Robot Operating System) library, are used. If it is determined that no unexplored areas exist (step S105: NO), the processor 11 terminates processing.

[0059] On the other hand, if it is determined that an unexplored area exists (step S106; YES), the processor 11 sets the destination to, for example, the point in the unexplored area closest to the robot 1, or a point on the boundary of the unexplored area (step S107). Step S107 is an example of a step in which the area to move the robot is determined based on map data and the results of semantic segmentation.

[0060] As a result of the integration described above, for example, the non-travel area 803 in Figure 8 is positioned as non-travel area 503 in the integrated map data shown in Figure 9. This shows that boundaries E1, E2, E4, and E5 shown in Figure 5(b) were each included in the non-travel area 503. Therefore, these boundaries are excluded from the list of potential destinations, and robot 1 recognizes only boundary E3 as a potential destination.

[0061] Furthermore, when the entire traversable area 501 where robot 1 is located is surrounded by either the non-traversable area 502 or the area where traversal is not required 503, the boundary between the unexplored area 504 and the traversable area 501 disappears. At this time, the robot 1's processor 11 determines that the remaining unexplored area 504 has all changed into the area where traversal is not required 503.

[0062] Once a destination is set, robot 1 begins movement control to move to that destination (step S200).

[0063] Figure 10 shows the flow of movement control for robot 1. After starting movement control in step S200, processor 11 determines whether or not it has reached the set destination (step S201). If it determines that the destination has been reached (step S201; YES), processor 11 returns processing to the input reception for starting exploration shown in Figure 4. As a result, processor 11 executes step S102 shown in Figure 4.

[0064] On the other hand, if it is determined that the destination has not been reached (step S201; NO), the processor 11 determines whether or not the robot has approached the destination (step S202). Approaching the destination means reaching a position where the width of the road to the destination can be measured. If it is determined that the robot has not approached the destination (step S202; NO), the processor 11 performs "high-speed movement," which involves moving the robot 1 at a constant speed for a certain period of time without reducing its speed (step S203), and then returns to step S201.

[0065] On the other hand, if it is determined that the robot is approaching its destination (step S202; YES), the processor 11 performs "slow-speed movement" to reduce the robot's movement speed (step S204).

[0066] Then, when the robot 1 moves at a low speed for a certain period of time, the processor 11 has the laser scanner 16 measure the width W2 of the approaching destination and determines whether the width W2 is wider than the width W1 of the robot 1 itself (step S205).

[0067] If the processor 11 determines that the measured path width W2 is wider than the robot 1's own device width W1 (step S205: YES), the processor 11 returns to step S201.

[0068] On the other hand, if it is determined that the measured road width W2 is not wider than the width W1 of the robot 1's own device (step S205: NO), the processor 11 classifies the destination as an area where travel is unnecessary (step S206) and returns processing to the input reception for starting exploration shown in Figure 4. As a result, the processor 11 executes step S102 shown in Figure 4.

[0069] By performing these actions, robot 1 classifies each region of the map data acquired by the laser scanner based on the results of semantic segmentation performed on the image acquired by the camera. Therefore, the user of robot 1 does not need to perform tasks to determine the regions of the map data, such as placing markers on the site, in order to decide whether or not to explore those areas.

[0070] The configurations, shapes, sizes, and arrangements described in the above embodiments are merely schematic representations to the extent that the present invention can be understood and implemented. Therefore, the present invention is not limited to the described embodiments and can be modified in various forms as long as it does not deviate from the scope of the technical idea set forth in the claims.

[0071] <Variation> The above describes the embodiment, but the contents of this embodiment can be modified as follows. Furthermore, the following modifications may be combined.

[0072] <1> In the embodiments described above, the processor 11 was a CPU, but it may have other configurations. For example, the processor 11 may be an FPGA (Field Programmable Gate Array) or may include an FPGA. Furthermore, the processor 11 may have an ASIC (Application Specific Integrated Circuit) or other programmable logic device, and control may be performed by these. Also, the processor 11 may include a GPU (Graphics Processing Unit).

[0073] <2> The program executed by the processor 11 described above may be provided stored on a computer-readable recording medium such as magnetic tape and magnetic disks, optical disks, magneto-optical recording media, or semiconductor memory. Alternatively, this program may be downloaded via a communication line such as the Internet.

[0074] <3> In the embodiments described above, the robot 1 had a laser scanner 16 and a camera 17, but it is not limited to these. For example, the robot 1 may have sensors such as scanners in addition to the laser scanner 16 and camera 17. Also, the number of laser scanners 16 and cameras 17, and other sensors, are not limited to those shown in the embodiments.

[0075] <4> In the embodiment described above, the robot 1 had an image processing unit 18, but the functions of the image processing unit 18 may be implemented by the processor 11. Furthermore, the robot 1 does not need to have an image processing unit 18. In this case, the robot 1 may be connected to an external information processing device (not shown) via a communication unit 13, and this information processing device may be made to perform the functions of the image processing unit 18 described above.

[0076] In this case, the information processing device acquires image data from the processor 11 in accordance with the instructions of the processor 11, performs semantic segmentation on this image data to generate classification data. The information processing device also generates image map data based on the acquired image data in accordance with the instructions of the processor 11. Furthermore, the information processing device integrates the LiDAR map data, the image map data, and the classification data in accordance with the instructions of the processor 11 to generate integrated map data.

[0077] In other words, this information processing device is an example of an information processing device that determines the area in which a moving robot will move based on map data acquired by a laser scanner and the results of semantic segmentation performed on images acquired by a camera. [Explanation of Symbols]

[0078] 1...Robot, 11...Processor, 12...Memory, 13...Communication unit, 14...Operation unit, 15...Display unit, 16...Laser scanner, 17...Camera, 18...Image processing unit, 19...Determination unit, 20...Movement unit

Claims

1. An information processing device that determines the area in which a moving robot will move, based on first map data acquired by a laser scanner, the results of semantic segmentation performed on images acquired by a camera, and second map data acquired by a camera.

2. The execution result includes information indicating areas where it is not necessary to acquire the first map data and the second map data. The information processing apparatus according to claim 1.

3. The first map data and the second map data include information indicating the area in which the robot can move. The information processing apparatus according to claim 1.

4. A robot having a laser scanner and a camera, which moves within an area determined by the information processing device according to any one of claims 1 to 3.

5. The laser scanner is a two-dimensional laser scanner or a three-dimensional laser scanner. The camera is either a monocular camera or a stereo camera. The robot according to claim 4.

6. The laser scanner and the camera are activated simultaneously, and the robot's position at the time of activation is set as the origin of the first map data and the second map data. The robot according to claim 4.

7. On the computer, The steps include: a moving robot acquiring first map data obtained with a laser scanner, and The robot acquires images from a camera, The steps include performing semantic segmentation on the aforementioned image, The steps include acquiring second map data obtained from the camera, A step of determining the area in which the robot will move based on the first map data, the second map data, and the result of the semantic segmentation; A program to execute.

Images