Information processing device, method, and program

Abstract

To improve setting of an approach inhibition area for an autonomous moving body.SOLUTION: An information processing device includes: means for acquiring an image obtained by capturing the environment in which a moving body moves; and means for inputting the acquired image into a semantic area division model, and acquiring at least three categories of an approach inhibition area, an approach permission area, and an area that is not determined to be the approach inhibition area or the approach permission area of the moving body in the acquired image as an output of the semantic area division model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, a method, and a program. [Background technology]

[0002] Conventionally, autonomous mobile objects such as autonomously traveling service robots are known. In such autonomous mobile objects, a movable area is defined. In general, in the case of a cleaning robot that performs cleaning work in a commercial facility or a security robot that performs patrol security, the area managed by the facility management company is limited to the common area of ​​the facility, and the management of the area occupied by the tenants may not be performed. In such a case, when the facility management company uses a cleaning robot to perform cleaning work, the area that the cleaning robot travels is limited to the common area of ​​the facility, and the range that the security robot patrols is also limited to the common area.

[0003] In this way, cleaning robots and security robots, which are autonomous mobile bodies, need to be able to distinguish between areas where entry is permitted and areas where entry is prohibited.

[0004] In order to set a no-entry area for an autonomous moving body, it is known to manually move the autonomous moving body and set the no-entry area based on the movement trajectory of the autonomous moving body (see Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2022-86464 A [Non-patent literature]

[0006] [Non-Patent Document 1] F.Dellaert,D.Fox,W.Burgard,S.Thrun, Monte Carlo localization for mobile robotS, Proc.of IEEE International Conference on RoboticS and Automation,1999 [Non-Patent Document 2] Dieter Fox, Adapting the Sample Size in particle filterS through KLD-Sampling, International Journal of RoboticS ReSearch,2003 [Non-Patent Document 3] Giorgio GriSetti,Cyrill StachniSS,Wolfram Burgard, Improved techniqueS for grid mapping with rao-blackwellized particle filterS, IEEE tranSactionS on RoboticS,2007 [Non-Patent Document 4] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam, Encoder-Decoder with AtrouS Separable Convolution for Semantic Image Segmentation, European Conference on Computer ViSion,2018 Summary of the Invention [Problem to be solved by the invention]

[0007] However, when setting the no-entry area based on the movement trajectory of the autonomous moving body, there is a problem that the setting work is complicated and the possibility of missing the setting increases when the no-entry area is wide or has a complex shape. In this case, the autonomous moving body cannot travel as desired. Thus, there is room for improvement in setting the no-entry area for the autonomous moving body.

[0008] In view of the above-mentioned problems, an object of the present invention is to provide an improvement in setting of no-entry areas for autonomous moving bodies. [Means for solving the problem]

[0009] An information processing device according to one embodiment of the present invention has a means for acquiring an image of an environment in which a moving object moves, and a means for inputting the acquired image into a semantic area segmentation model and acquiring, as output of the semantic area segmentation model, at least three categories in the acquired image: no-entry areas of the moving object, permitted areas, and areas which cannot be determined as either no-entry areas or permitted areas. Effect of the Invention

[0010] According to the present invention, it is possible to improve the setting of no-entry areas for autonomous moving bodies. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing a configuration of a mobile object management system 101 according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a mobile system. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a mobile system. [Figure 4] FIG. 2 is a diagram illustrating a work flow of a mobile system. [Diagram 5] FIG. 2 illustrates an example of a hardware configuration of a server. [Figure 6] FIG. 2 illustrates an example of a functional configuration of a server. [Figure 7]FIG. 4 is a diagram illustrating a flow of processing for creating a no-entry area according to the first embodiment. [Figure 8] 5A to 5C are diagrams illustrating an example of creating a no-entry area according to the first embodiment. [Figure 9] FIG. 11 is a diagram illustrating a flow of processing for creating a no-entry area according to the second embodiment. [Figure 10] 13A and 13B are diagrams illustrating an example of creating a no-entry area according to the second embodiment. [Figure 11] FIG. 13 is a diagram showing an example of an overhead view of a no-entry area and an entry-permitted area. [Figure 12] FIG. 11 is a diagram showing an example of an overhead view of a no-entry area according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicated descriptions are omitted.

[0013] In the following, as an example, a case where the present invention is applied to a mobile object management system in which a mobile object performs cleaning work will be described. However, the present invention is not limited to this, and can be applied to any system that analyzes sensor information and outputs predetermined information.

[0014] [First embodiment] <System configuration> 1 is a diagram showing a configuration of a mobile object management system 101 according to a first embodiment of the present invention. The mobile object management system 101 includes mobile object systems 110a, 11b, and 110c, a network 120, and a server 130. Note that, hereinafter, each of the mobile object systems 110a, 11b, and 110c may be referred to as a mobile object system 110.

[0015] The mobile body system 110 is a mobile body capable of autonomously traveling and equipped with functions necessary for cleaning work. In the first embodiment, the mobile body system 110 is assumed to have a built-in computing device capable of processing sensor information such as images, but the present invention is not limited to this. For example, there may be an external computer such as a PC connected to the mobile body system 110, and the combination of these may be treated as the mobile body system 110. PC is an abbreviation for personal computer.

[0016] The server 130 is a computer such as a PC, and has a function of analyzing and processing sensor information such as images, etc. Furthermore, the server 130 is a device that receives input from a user and outputs information to the user (for example, displays information).

[0017] The mobile system 110 and the server 130 are connected to each other via a network 120 so as to be able to communicate with each other. The network 120 includes a plurality of routers, switches, cables, etc. that meet a communication standard such as Ethernet (registered trademark). In this embodiment, the network 120 may be any network that enables communication between the mobile system 110 and the server 130, and may be constructed with any scale, configuration, and conforming communication standard. For example, the network 120 may be the Internet, a wired LAN, a wireless LAN, a WAN, etc. LAN is an abbreviation for Local Area Network. WAN is an abbreviation for Wide Area Network. The network 120 may be configured to enable communication using a communication protocol that complies with the ONVIF standard, for example. ONVIF is an abbreviation for Open Network Video Interface Forum. However, the present invention is not limited to this, and the network 120 may be configured to enable communication using another communication protocol such as a unique communication protocol, for example.

[0018] <Device configuration> (Hardware configuration of the mobile system 110) Next, the configuration of the mobile body system 110 will be described. Fig. 2 is a diagram showing an example of the hardware configuration of the mobile body system 110. The mobile body system 110 has a camera 201, a LiDAR 202, an information processing device 203, a communication unit 204, a control device 205, and a wheel unit 206. LiDAR is an abbreviation for Light Detection and Ranging. The camera 201 and the LiDAR 202 constitute a sensor unit 200.

[0019] The camera 201 is a distance image camera, for example, a TOF camera that performs distance measurement by a TOF method. TOF is an abbreviation for Time Of Flight. The camera 201 can acquire distance information in addition to RGB image information. The camera 201 includes a lens unit for forming an image of light, an image sensor that converts the imaged light into an analog signal, and a light source. The lens unit of the camera 201 has a zoom function for adjusting the angle of view and an aperture function for adjusting the amount of light. The image sensor of the camera 201 has a gain function for adjusting the sensitivity when converting light into an analog signal. The light source of the camera 201 emits pulses of light, and performs distance measurement by measuring the time from when the light is irradiated to when it hits an object and is returned in synchronization with the image sensor of the camera 201. These functions are adjusted based on setting values ​​notified by the information processing device 203. The analog signal acquired by the camera 201 is converted into a digital signal by an analog-digital conversion circuit of the camera 201, and is transferred to the information processing device 203 as an image signal.

[0020] The LiDAR 202 is also called a laser scanner or a laser range finder, and performs distance measurement using a TOF method like the camera 201, but can acquire distance information over a wide range by scanning the surroundings using a laser light as a light source. The LiDAR 202 includes a laser element and a light receiving element that converts light that hits an object and returns to an analog signal. There are various laser scanning methods, which are roughly divided into a rotation type (mechanical type) and a non-rotation type (solid-state type). In this embodiment, a rotation type two-dimensional LiDAR is used in which a laser beam scans a two-dimensional plane and distance information distributed on the plane is obtained. The functions of the LiDAR 202, such as the laser irradiation function and scanning function, and the gain function that adjusts the sensitivity when converting the light of the light receiving element into an analog signal, are adjusted based on the setting value notified from the information processing device 203. The analog signal acquired by the LiDAR 202 is converted to a digital signal by an analog-to-digital conversion circuit of the LiDAR 202 and transferred to the information processing device 203 as a distance signal.

[0021] The information processing device 203 has the functions of a general embedded PC device, and includes a CPU 207, a ROM 208, a RAM 209, a storage unit 210 such as an HDD or SSD, a general-purpose I / F 211 such as USB, and a system bus 212. CPU is an abbreviation for Central Processing Unit. ROM is an abbreviation for Read Only Memory. RAM is an abbreviation for Random Access Memory. HDD is an abbreviation for Hard Disk Drive. SSD is an abbreviation for Solid State Drive. USB is an abbreviation for Universal Serial Bus. I / F is an abbreviation for interface.

[0022] <Functional configuration of the mobile system 110> The functional configuration of the mobile system according to the first embodiment of the present invention will be described below. The processing of each unit described below is realized by software, in that a program is loaded from ROM 208 or the like onto RAM 209, and then the program is executed by CPU 207. A part or all of the functional configuration may be configured as hardware, such as ASIC or FPGA. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array.

[0023] 3 is a block diagram showing the functional configuration of the mobile system 110. The mobile system 110 includes an image capture control unit 301, a recognition unit 302, a storage unit 303, a self-position estimation unit 304, It includes a plan determination unit 305 , a network communication unit 306 , and a control unit 307 .

[0024] The imaging control unit 301 acquires sensor information from the sensor unit 200 through the general-purpose I / F 211. In this embodiment, the imaging control unit 301 acquires image information and distance information from the camera 201, and acquires distance information from the LiDAR 202. In this embodiment, the sensor information is the image information and distance information from the camera 201, and the distance information from the LiDAR 202.

[0025] The recognition unit 302 performs a predetermined recognition process based on the sensor information acquired by the imaging control unit 301 from the sensor unit 200 and an environmental map (described later) read from the storage unit 303. Examples of the recognition process include detecting obstacles and people, detecting landmarks, etc. while driving, and performing recognition processes necessary for driving safely without collisions even if the environment changes dynamically.

[0026] The self-position estimation unit 304 calculates the self-position of the mobile system 110 at the time of acquiring the sensor information, based on the sensor information acquired by the imaging control unit 301 from the sensor unit 200 and an environmental map (described later) read from the storage unit 303. The self-position of the mobile system 110 is the position and attitude of the mobile system 110.

[0027] In this embodiment, MCL is used as an example of a self-location estimation technique using a known particle filter. MCL is an abbreviation of Monte Carlo Localization. In this embodiment, the technique described in Non-Patent Document 1 and the technique described in Non-Patent Document 2, in which the number of particles is variable, are applied and used.

[0028] When there is no environmental map yet, SLAM technology is used as a method to create an environmental map using sensor information. SLAM is an abbreviation for Simultaneous Localization and Mapping. Since the creation of an environmental map and self-localization are interdependent, SLAM simultaneously performs self-localization and map creation. SLAM takes sensor data as input and outputs the movement trajectory and map information of the mobile system.

[0029] In this embodiment, as an example, a SLAM technique using distance information from a 2D LiDAR is used, for example, by applying the technique described in Non-Patent Document 3. This is one of the techniques called sequential SLAM, which builds an environmental map each time sensor information is obtained.

[0030] The output of the self-location estimation unit 304 is sent to the plan determination unit 305 and used for driving control of the mobile system 110, or the self-location information is sent to a display device (not shown) and used for displaying the current location of the mobile system 110 on the UI. The UI includes GUI. UI is an abbreviation for User Interface. GUI is an abbreviation for Graphical User Interface.

[0031] It is also possible to transmit sensor information acquired by the imaging control unit 301 to an external device via a network communication unit, and to have the position and orientation calculated by the external device be acquired by the self-position estimation unit 304 via the network communication unit 306.

[0032] The plan determination unit 305 plans and determines the next operation based on the self-position of the mobile system 110 acquired from the self-position estimation unit 304 and the state of the surrounding environment acquired from the recognition unit 302. For example, when an obstacle is detected, the plan determination unit 305 determines to plan and execute a local detour route. The plan determination unit 305 also makes a determination such as changing the cleaning mode (suction strength, movement speed, movement route, etc.) for each location. The output of the plan determination unit 305 is sent to the control unit 307. The control unit 307 receives the determination by the plan determination unit 305 and performs automatic driving control of the mobile system 110 based on the input.

[0033] <Environmental Map> The environmental map used in this embodiment will be described below. When performing self-location estimation using the LiDAR 202, for example, a point cloud map created based on distance information obtained by the LiDAR 202 can be used as the environmental map. Also, based on the point cloud map, a space is divided into cells in a lattice pattern, and an occupancy grid map formed in a format that indicates the probability of whether each cell has been observed and whether a point cloud is included can be used as the environmental map. In this embodiment, the occupancy grid map is used as the environmental map.

[0034] (How to obtain an environmental map) The environmental map is generated by moving the mobile system 110 equipped with a sensor within the environment. When there is no environmental map, a travel route for acquiring sensor data is provided, for example, by operating the mobile system 110 with a remote control or by hand. The environmental map based on the sensor information acquired by the sensor mounted on the mobile system 110 may be generated simultaneously with the operation, or may be generated offline after data collection. Alternatively, the mobile system 110 may autonomously plan a travel route exploratory by using maps generated up to that point and local maps around the current location, and travel within the environment.

[0035] <Workflow of mobile systems> The workflow of the mobile system 110 when performing the work will be described with reference to the flowchart of Fig. 4. Fig. 4 is a diagram illustrating the workflow of the mobile system 110.

[0036] First, in step S401, the self-position estimation unit 304 reads a previously created environmental map from the storage unit 303. In step S402, the self-position estimation unit 304 calculates the self-position of the mobile system 110 from the environmental map read in step S401 and local information about the current location from the imaging control unit 301.

[0037] Next, in step S403, the plan determination unit 305 performs a route plan from the self-position calculated in step S402 to the work start point. In step S404, the mobile system 110 moves to the work start point while estimating the self-position on the environmental map.

[0038] Next, in step S405, the plan determination unit 305 creates a route plan from the work start point to the work end point. If a task of the type that involves working evenly within an area, such as cleaning, is assigned, a route plan for that purpose is created. Alternatively, if a route plan has been created in advance and stored in the storage unit 303, the plan determination unit 305 reads out the route plan from the storage unit 303.

[0039] Next, in step S406, the mobile system 110 travels according to the route plan. At this time, it performs a predetermined operation as necessary. The predetermined operation is a suction or wiping operation for a cleaning robot, a grass cutting operation for a lawnmower, or a tilling operation for a cultivator.

[0040] When the work end point is reached, in step S407, the plan determination unit 305 performs a route plan for returning from the current value to a predetermined position such as the travel start point. In step S408, the mobile system 110 travels to the predetermined position according to the route plan, and transmits to the server 130 via the network communication unit 306 that the work has been completed.

[0041] If the mobile system 110 detects an obstacle not on the environmental map while traveling, it replans a local route to avoid the obstacle and continues traveling along that route. If the mobile system 110 cannot avoid the obstacle, it stops, indicates that an abnormality exists using the indicator light of the mobile system 110, notifies the user of a message via the network communication unit 306, and waits for an operation from the administrator.

[0042] The cleaning work time and travel route, or areas where work could not be performed due to obstacle avoidance, etc. are stored in the storage unit 303 as a report after the work is completed. Alternatively, the report after the work is completed is sent to the server 130 via the network communication unit 306.

[0043] (Server hardware configuration) 5 is a diagram showing an example of a hardware configuration of server 130. Server 130 is configured as a general computer such as a PC. Server 130 includes a CPU 501 which is a processor, RAM 502 and ROM 503 which are memories, a storage unit 504 such as an HDD, and a communication unit 505. Server 130 realizes various functions by CPU 501 executing programs stored in ROM 503, storage unit 504, or an external storage device.

[0044] (Server Functionality Configuration) 6 is a diagram showing an example of the functional configuration of the server 130. The server 130 includes a network communication unit 601, a control unit 602, a display unit 603, an operation unit 604, an analysis unit 605, a storage unit 606, and a setting processing unit 607.

[0045] The network communication unit 601 connects to, for example, the network 120 and executes communication with the mobile system 110 and the like via the network 120. Note that this is merely an example, and for example, the network communication unit 601 may be configured to establish a direct connection with the mobile system 110 and communicate with the mobile system 110 without going through the network 120 or another device.

[0046] The control unit 602 controls the network communication unit 601, the display unit 603, the operation unit 604, the analysis unit 605, the storage unit 606, and the setting processing unit 607 so that they each execute their respective processes.

[0047] The display unit 603 presents information to the user via, for example, a display. In this embodiment, the information is presented to the user by displaying the result of rendering by the browser on the display. Note that information may be presented by a method other than a screen display, such as sound or vibration.

[0048] The operation unit 604 accepts operations from a user. In this embodiment, the operation unit 604 is a mouse or a keyboard. The user operates the operation unit 604 to input a user operation to the browser. However, the present invention is not limited to this, and the operation unit 604 may be any device capable of detecting the intention of other users, such as a touch panel or a microphone.

[0049] The analysis unit 605 performs analysis work required for the mobile system 110 to perform a predetermined operation. A method for setting partial areas that are to be no-entry areas in the environmental map based on sensor information will be described later.

[0050] The storage unit 606 stores information necessary for the mobile system 110 to perform a predetermined operation. The setting processing unit 607 executes the setting process described below.

[0051] <Flow of setting no-entry areas for mobile systems> The flow of setting no-entry areas in the mobile system is described below. Each process below is executed by the information processing device 203 of the mobile system 110. However, this is merely an example, and some or all of the processes described below may be realized not only by the mobile system 110, but also by the server 130, a notebook PC or tablet terminal capable of communicating with the mobile system 110, or dedicated hardware.

[0052] A method in which the mobile body system 110 recognizes a no-entry area while traveling and reflects the recognition result in a route plan will be described with reference to the flowchart of Fig. 7. Fig. 7 is a diagram illustrating the no-entry area creation processing flow according to the first embodiment.

[0053] First, in step S701, the mobile system 110 starts autonomous traveling to execute a given task. The task here is, for example, transporting luggage or cleaning. In the case of transporting luggage, the mobile system 110 plans a route to reach the destination without coming into contact with obstacles in the shortest time possible and return to a predetermined position such as a waiting place, and travels according to the route plan. In the case of cleaning, the mobile system 110 plans a route to travel thoroughly through a cleaning area and return to a predetermined position such as a waiting place, and travels according to the route plan. At this time, the mobile system 110 acquires image information and distance information of the surrounding environment while traveling.

[0054] Next, in step S702, the mobile system 110 determines whether it has arrived at the destination (a predetermined position such as a waiting place). If it is determined that the mobile system 110 has arrived at the destination, it stops self-propelling and ends. If it is determined that the mobile system 110 has not arrived at the destination, the process of step S703 is executed.

[0055] In step S703, the mobile system 110 performs processing for segmenting the acquired image into no-entry areas. The no-entry area segmentation is performed using a semantic area segmentation model trained by a technique called machine learning, which finds patterns hidden in a large amount of data through repeated calculations.

[0056] Semantic segmentation is a task of dividing an image into sub-regions by inferring a category of an object for each pixel. Semantic segmentation is used, for example, to divide an acquired image into various categories so that a mobile system 110 operating in an indoor environment can understand the environment and act accordingly. The categories into which the acquired image is divided include, for example, people, walls, fences, floors, ceilings, pillars, chairs, desks, shelves, trash cans, signs, escalators, and elevators.

[0057] In particular, since models using deep neural networks have become capable of performing inference with high accuracy in recent years, the present embodiment also uses a deep neural network model, and applies the technology described in, for example, Non-Patent Document 4.

[0058] In this embodiment, the purpose is to set no-entry areas in a commercial facility. For this purpose, this area division model is used to divide the floor surface of the commercial facility from an image into no-entry areas mainly consisting of tenant areas and allowed areas mainly consisting of common areas such as corridors.

[0059] For convenience, the following description will be given assuming that the tenant area is a prohibited area and the common area is a permitted area. In reality, even if the area is managed by a building, it may be categorized as a prohibited area in the same way as the tenant area. Conversely, even if the area is a tenant area, it may be categorized as a permitted area in the same way as the common area managed by the building. For example, an area near an escalator entrance is not a tenant area, but it can be categorized as a prohibited area. This is determined by the annotation rules of the correct answer data used for learning.

[0060] In many cases, the tenant area and the common area are clearly indicated by the color and texture of the floor. Therefore, at first glance, it seems possible to separate the two areas if the area can be divided based on color and texture. However, in reality, the design of commercial facilities is diverse, so it is not possible to completely distinguish between the two areas based on color and texture alone.

[0061] FIG. 8 is a diagram showing an example of the creation of a forbidden area according to the first embodiment. FIG. 8 is an example of an image captured by the camera 201 of the mobile system 110. An intermediate area 803 with a different color and texture is included between a tenant area (forbidden area) 801 and a common area (permitted area) 802. This intermediate area 803 is highly context-dependent, and it is not possible to uniformly determine from the image alone whether it is included in the tenant area or the common area, depending on the design of the facility. If it is determined uniformly, it means that the determination is highly likely to be incorrect. In the example of FIG. 8, there is an area (intermediate area 803) between the tenant area (forbidden area) 801 and the common area (permitted area) 802, where it is difficult to determine whether it is included in the tenant area or the common area. However, areas where it is difficult to determine exist other than between the tenant area (forbidden area) and the common area (permitted area). The semantic area division model outputs an area that is difficult to determine whether it is a tenant area (prohibited area) or a shared area (permitted area) as an area where the determination is not definite. Hereinafter, for convenience, an area that is difficult to determine as a semantic area division model is described as an "intermediate area." However, as described above, an area that is difficult to determine may exist other than the area between the tenant area (prohibited area) and the shared area (permitted area).

[0062] Therefore, in this embodiment, the semantic area division model is trained to divide and output at least three categories: a no-entry area (tenant area), an entry-permitted area (shared area), and an intermediate area (ambiguous area). The intermediate area 803 is classified as an intermediate area (ambiguous area).

[0063] However, it is known that the ambiguity of the correct labels in the training data deteriorates the inference accuracy of the model. If we try to forcibly assign areas to either prohibited or permitted areas, the correct category will fluctuate for each image, and the consistency of the correct data will be lost. By assigning new labels to intermediate areas that could be either depending on the facility, we can reduce the ambiguity and improve the inference accuracy of the semantic area segmentation model.

[0064] Next, in step S704, the mobile system 110 integrates the area division information into a three-dimensional data structure and stores it in the storage unit 606. The mobile system 110 divides the three-dimensional space into voxels for environmental understanding, and stores in memory a data structure that allows each voxel to selectively have one of three states: occupied, unoccupied, and unmeasured. Also, for those voxels that are in an occupied state, the category of the area can be stored as an attribute of the voxel in addition to the color, normal, etc.

[0065] The mobile system 110 handles the surrounding environment as information from the distance imaging camera 201 as input, but updates information based on the recognition results for voxels that correspond to the viewing frustum determined by the distance measurement results and distance measurement range of the distance imaging camera 201. However, values ​​of voxels that are further back than the distance measurement results even if they are within the viewing frustum are not updated. The updating method may be to replace them with new ones each time a measurement is made, or to take a weighted average. In this way, the results of region division are added and updated to a data structure that handles information on three-dimensional space.

[0066] In this embodiment, the view frustum obtained from the image information and distance information is fitted to a position in a map coordinate system using the current position and orientation of the mobile system 110, and the data is stored in the data structure of the above-mentioned three-dimensional space. The map coordinate system is a world coordinate system.

[0067] Next, in step S705, the mobile system 110 sets the no-entry area in a format that can be used for route planning. In this embodiment, the mobile system 110 performs route planning based on an occupancy grid map (cost map) that divides space into a grid based on sensor information and indicates the probability of an obstacle existing in each grid. For this reason, the mobile system 110 projects the three-dimensional space data integrated in step S704 onto a two-dimensional plane, and sets the grids of the occupancy grid map (cost map) at positions corresponding to the no-entry area to an occupied state.

[0068] In addition, whether or not the grid of the occupancy grid map at the position corresponding to the intermediate area is to be in an occupied state is subject to the user's settings. In many cases, it is determined whether the intermediate area is included in the tenant area or the common area for each facility or for each floor. Therefore, for the intermediate area, it is possible to determine whether it is included in the no-entry area or the permitted-entry area according to the settings of the user who knows the design rules of the facility. In this way, it is possible to automatically set the no-entry area of ​​the traveling system 110 for each facility, each floor, each partial area, etc.

[0069] The mobile system 110 can create a route plan that does not pass through the no-entry areas by passing the occupancy grid map thus obtained to a module that creates a route plan. The no-entry areas that are automatically set in this way can also be modified by the user as appropriate using a GUI or the like. Once the no-entry areas have been set, the mobile system 110 continues traveling toward the destination.

[0070] [Second embodiment] The second embodiment of the present invention has the same hardware configuration, functional block diagram, and the like as the first embodiment except for the flow of setting no-entry areas in a mobile system, so only the flow of setting no-entry areas in a mobile system will be described.

[0071] <Flow of setting no-entry areas for mobile systems> A method for recognizing a no-entry area while the mobile body system 110 is traveling and for reflecting the recognition result in a route plan will be described below with reference to the flowchart of Fig. 9. Fig. 9 is a diagram for explaining the no-entry area creation processing flow according to the second embodiment.

[0072] The processes in steps S701 and S702 are the same as those in the first embodiment, and therefore the description thereof will be omitted.

[0073] In step S903, the mobile system 110 performs a process of dividing the acquired image into no-entry areas. Fig. 10 is a diagram showing an example of creating a no-entry area according to the second embodiment. Fig. 10 is an example of an image captured by the camera 201 of the mobile system 110.

[0074] Tenant area 1001 includes a hallway 1004. The color and texture of hallway 1004 may be the same as or different from the rest of tenant area 1001.

[0075] As described above, division is particularly difficult when the tenant area 1001 of a commercial facility includes additional structures such as a passageway, or when the color or texture of the floor of the common area is continuous and directly connected to the tenant area. In such cases, the method of the first embodiment alone may not be sufficient for dividing the area. Therefore, in the second embodiment, information about the hanging wall is used to divide the area.

[0076] That is, the semantic area division model used in the second embodiment is trained to divide a hanging wall area 1005 in addition to a no-entry area 1001 for tenants, an allowed area 1002 for a common area, and an intermediate area 1003 between these areas. A hanging wall is also called a hanging wall. A hanging wall is a wall that hangs down from the ceiling by several tens of centimeters to several meters, and has the function of dividing a space.

[0077] Next, in step S904, the mobile system 110 integrates the area division information into a three-dimensional data structure and stores it in the storage unit 606, similar to the process in step S704.

[0078] Next, in step S905, the mobile system 110 performs area division using information obtained by projecting the three-dimensional spatial information onto a two-dimensional plane.

[0079] Fig. 11 is a diagram showing an example of an overhead view of a forbidden area. Fig. 11 is a diagram showing an example of a forbidden area 1101, an allowed area 1102, an intermediate area 1103, and an unmeasured area 1104 projected onto a two-dimensional plane from three-dimensional spatial information. In the example of Fig. 11, because there is a passageway within the tenant, an erroneous division has been made in which the allowed area exists within the tenant.

[0080] Therefore, in this embodiment, the region is divided using the hanging wall region 1005 as shown in FIG. 10. FIG. 12 is a diagram showing an example of an overhead view of the entry-prohibited region according to the second embodiment. In this embodiment, a dividing line 1201 is drawn at a position projected from the three-dimensional position of the hanging wall region 1005 to the floor surface in the state of FIG. 11. In this embodiment, the entry-permitted region 1102 is divided by this dividing line 1201, and the region on the back side of the dividing line 1201 as viewed from the camera position is labeled as the entry-prohibited region 1202. For example, the user can set whether the region divided by the dividing line 1201, i.e., the hanging wall region 1005, is the entry-permitted region or the entry-prohibited region while checking on the display unit 603.

[0081] Next, in step S906, the mobile system 110 sets the no-entry areas in a format that can be used for route planning, similar to the process in step S705.

[0082] According to the second embodiment, even in a case where both the passage 1004 and the common area 1002 in the tenant area are mistakenly divided as entry-permitted areas, it is possible to correctly set the entry-prohibited area.

[0083] Whether or not the above-mentioned area division by the hanging wall is enabled can be determined by the user through settings.

[0084] [Another embodiment] Although the present invention has been described with an example in which images captured by the mobile system 110 are analyzed and processed by the mobile system 110, the images captured by the mobile system 110 may be analyzed and processed by the server 130.

[0085] (Other embodiments) The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. Also, the present invention can be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0086] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0087] The disclosure of the present embodiment includes the following configurations: a method and a program. (Configuration 1) A means for acquiring an image of an environment in which a moving object moves; a means for inputting the acquired image into a semantic region segmentation model and acquiring at least three categories of areas in the acquired image, the categories being a no-entry area of ​​the moving object, an entry-permitted area, and an area which is not determined to be either a no-entry area or an entry-permitted area, as an output of the semantic region segmentation model; 13. An information processing device comprising: (Configuration 2) 2. The information processing apparatus according to configuration 1, further comprising a setting unit for setting the area, which is not determined to be an entry prohibited area or an entry permitted area, as an entry prohibited area or an entry permitted area. (Configuration 3) The semantic area division model divides and outputs at least four categories: a no-entry area, an allowed area, an area in which it is not clear whether the area is a no-entry area or an allowed area, and a hanging wall area. 3. The information processing device according to configuration 1 or 2. (Configuration 4) 4. The information processing device according to configuration 3, further comprising a setting unit for setting an area partitioned by the hanging wall area as either an entry prohibited area or an entry permitted area. (Method 1) acquiring an image of an environment in which the moving object moves; inputting the acquired image into a semantic region segmentation model, and acquiring at least three categories of areas in the acquired image, including a no-entry area for the moving object, an entry-permitted area, and an area that is not determined to be a no-entry area or an entry-permitted area, as an output of the semantic region segmentation model; The method according to claim 1, further comprising: (Program 1) Computer, A means for acquiring an image of an environment in which the moving object moves; and a means for inputting the acquired image into a semantic region segmentation model, and acquiring at least three categories of areas in the acquired image, the categories being a no-entry area for the moving object, an entry-permitted area, and an area which is not determined to be either a no-entry area or an entry-permitted area, as an output of the semantic region segmentation model; A program characterized by causing the program to function as a [Explanation of symbols]

[0088] 101 Mobile Management System 110 Mobile Systems 120 Network 130 Server

Claims

1. A means for acquiring an image of an environment in which a moving object moves; a means for inputting the acquired image into a semantic region segmentation model and acquiring at least three categories of areas in the acquired image, the categories being a no-entry area of ​​the moving object, an entry-permitted area, and an area which is not determined to be either a no-entry area or an entry-permitted area, as an output of the semantic region segmentation model; 13. An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, further comprising a setting unit for setting the area, which is not determined to be an entry prohibited area or an entry permitted area, as an entry prohibited area or an entry permitted area.

3. The semantic area division model divides and outputs at least four categories, namely, a no-entry area, an entry permitted area, an area in which it is not clear whether the area is a no-entry area or an entry permitted area, and a hanging wall area.

2. The information processing apparatus according to claim 1,

4. 4. The information processing apparatus according to claim 3, further comprising a setting unit for setting the area partitioned by the hanging wall area as either an entry-prohibited area or an entry-permitted area.

5. acquiring an image of an environment in which the moving object moves; inputting the acquired image into a semantic region segmentation model, and acquiring at least three categories of areas in the acquired image, including a no-entry area of ​​the moving object, an entry-permitted area, and an area that is not determined to be a no-entry area or an entry-permitted area, as an output of the semantic region segmentation model; The method according to claim 1, further comprising:

6. Computer, A means for acquiring an image of an environment in which the moving object moves; and a means for inputting the acquired image into a semantic region segmentation model, and acquiring at least three categories of areas in the acquired image, the categories being a no-entry area for the moving object, an entry-permitted area, and an area which is not determined to be either a no-entry area or an entry-permitted area, as an output of the semantic region segmentation model; A program characterized by causing the program to function as a

Images