Information processing method, information processing system, and program

Abstract

To propose an information processing system and the like that can shorten a time for setting an entry-inhibited area of a moving object in an office, a commercial facility and the like.SOLUTION: An information processing method for setting an entry-inhibited area of a moving body 110, includes: a detection step of detecting an object 1101 from image information and determining its position; a search step of searching a surrounding according to a type of the object 1101 and its position; and a setting step of setting the entry-inhibited area from a position of at least one object 1101 obtained by repeating the detection step and the search step.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for setting no-entry areas for autonomously traveling mobile bodies. [Background technology]

[0002] Autonomous service robots (mobile objects) cannot travel anywhere they want, but must travel within a designated area. Until now, methods have been used to indicate no-entry areas, such as physical stoppers or special devices using infrared rays. Alternatively, people have manually designated no-entry areas on a map via a GUI (Graphical User Interface) on a tablet or other device.

[0003] Patent Document 1 describes a method in which a moving object is manually driven and a no-entry area is specified according to the trajectory at that time. Based on the map information obtained in this way, the moving object plans a route so that the driving route does not include the no-entry area, and drives while recognizing the environment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-86464 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, setting no-entry areas using physical stoppers or dedicated tools using infrared rays or the like requires cost and effort to install. In addition, since no-entry areas change depending on the situation, it is not always possible to prepare objects corresponding to no-entry areas in advance. Even if objects representing no-entry areas are installed, there are areas where certain people or moving objects are allowed to enter, and physically blocking such areas may not be suitable for the actual situation at the site. In addition, in offices and commercial facilities, no-entry areas may be temporarily set by objects indicating no entry, such as tape partitions, color cones (registered trademark), and signs. For this reason, it is not possible for a person to specify these areas in advance using the GUI or to teach them from the trajectory of the vehicle by actually driving through them. It is not practical for a user to set no-entry areas on a map using software every time such objects are installed.

[0006] The present invention has been made in consideration of the above problems, and has an object to easily set no-entry areas for moving objects in offices, commercial facilities, and the like. [Means for solving the problem]

[0007] In order to solve the above problems, an information processing method in an embodiment of the present invention is an information processing method for setting a no-entry area for a moving body, and is characterized in that it includes a detection process for detecting an object from image information and determining the position of the object, a search process for searching the surroundings according to the type of the object and its position, and a setting process for setting the no-entry area from the position of at least one of the objects obtained by repeating the detection process and the search process. Effect of the Invention

[0008] According to the present invention, it is possible to easily set no-entry areas for moving objects in offices, commercial facilities, and the like. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a configuration of a mobile object management system (information processing system) 101. [Diagram 2] FIG. 2 is a diagram showing a hardware configuration of a moving object 110. [Diagram 3] FIG. 2 is a block diagram showing the functional configuration of a moving object 110. [Figure 4] FIG. 2 is a flow chart for explaining the operation of the moving object 110. [Diagram 5]FIG. 12 is a diagram showing a hardware configuration of a server 130. [Figure 6] FIG. 12 is a diagram showing the functional configuration of a server 130. [Figure 7] 11 is a flowchart illustrating a process (information processing method) for creating a no-entry area for a moving object 110. FIG. [Figure 8] FIG. 13 is a diagram showing the relationship between the type of object, the search target at that time, and the setting method (location) of the no-entry area. [Figure 9] FIG. 2 is a diagram showing an example of creating a no-entry area for a moving object 110. [Figure 10] FIG. 2 is a diagram showing an example of creating a no-entry area for a moving object 110. [Figure 11] FIG. 2 is a diagram showing an example of creating a no-entry area for a moving object 110. [Figure 12] FIG. 2 is a diagram showing an example of creating a no-entry area for a moving object 110. [Figure 13] FIG. 2 is a diagram showing an example of creating a no-entry area for a moving object 110. [Figure 14] FIG. 2 is a diagram showing an example of creating a no-entry area for a moving object 110. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] In the following, an example will be described in which a system is intended for a moving object to perform cleaning work. However, the following technical idea can be applied to any system that analyzes sensor information and outputs predetermined information, not limited to cleaning work.

[0012] [First embodiment] <Configuration of the Mobile Management System 101> FIG. 1 is a diagram showing the configuration of a mobile object management system 101 according to this embodiment. The mobile object management system (information processing system) 101 includes a mobile object 110, a network 120, and a server 130. The mobile object 110 is, for example, three mobile objects 110a to 110c.

[0013] The mobile object 110 is an autonomously movable object equipped with functions necessary for cleaning work. In this embodiment, the moving object 110 is assumed to have a built-in computing device capable of processing sensor information such as images, but is not limited to this. For example, there may be an external computer such as a PC (personal computer) connected to the moving object 110, and the combination of these may be treated as the moving object 110.

[0014] The server 130 is a computer such as a PC, and has a function of analyzing and processing sensor information such as images, etc. Furthermore, the server 130 is a device that receives input from a user and outputs information to the user (for example, displays information).

[0015] The mobile object 110 and the server 130 are communicatively connected via a network 120. The network 120 includes a plurality of routers, switches, cables, etc. that meet a communication standard such as Ethernet (registered trademark). In this embodiment, the network 120 may be any network that enables communication between the mobile object 110 and the server 130, and may be constructed with any scale, configuration, and conformance to any communication standard. For example, the network 120 may be the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), etc. Furthermore, the network 120 may be configured to enable communication using a communication protocol that complies with the ONVIF (Open Network Video Interface Forum) standard, for example. However, the present invention is not limited to this, and the network 120 may be configured to enable communication using other communication protocols, such as a proprietary communication protocol.

[0016] <Hardware Configuration of Mobile Object 110> Next, the hardware configuration of the moving object 110 will be described. FIG. 2 is a diagram showing a hardware configuration of the moving object 110. As shown in FIG. The hardware configuration of the moving object 110 includes, for example, a sensor unit including a camera 201 and a LiDAR 202, an information processing unit 203, a communication unit 204, a control unit 205, and a wheel unit 206.

[0017] The camera 201 is a distance image camera, for example a TOF (Time Of Flight) camera that measures distances using a TOF method, and can obtain distance information in addition to RGB image information. The camera 201 includes a lens unit for focusing light, an image sensor for converting the focused light into an analog signal, and a light source. The lens section has a zoom function to adjust the angle of view and an aperture function to adjust the amount of light. The image sensor has a gain function to adjust the sensitivity when converting light into an analog signal. The light source emits pulses and measures distance by measuring the time from when the light is emitted to when it hits an object and is reflected back in synchronization with the image sensor. These functions are adjusted based on setting values ​​notified by the information processing unit 203. An analog signal acquired by the camera 201 is converted into a digital signal by an analog-digital conversion circuit and transferred to the information processing unit 203 as an image signal.

[0018] The LiDAR 202 is also called a laser scanner or laser range finder, and performs distance measurement using the TOF method, similar to the camera 201. By scanning the surroundings using laser light as a light source, it is possible to obtain distance information over a wide range. The LiDAR 202 includes a laser element and a light receiving element that converts the light that hits an object and returns to an analog signal. There are various laser scanning methods, which can be broadly divided into rotary (mechanical) and non-rotary (solid-state) types. In this example, we use a rotary 2D LiDAR in which a laser beam scans a two-dimensional plane and obtains distance information distributed on the plane. Functions such as the laser irradiation function, the scanning function, and the gain function for adjusting the sensitivity when converting the light of the light receiving element into an analog signal are adjusted based on the setting values ​​notified by the information processing unit 203. The analog signal acquired by the LiDAR 202 is converted into a digital signal by an analog-digital conversion circuit and transferred to the information processing unit 203 as a distance signal.

[0019] The information processing unit 203 has the functions of a general embedded PC device, and is composed of a CPU 207 , a ROM 208 , a RAM 209 , a storage unit 210 such as an HDD or SSD, a general-purpose I / F 211 such as a USB, and a system bus 212 .

[0020] <Functional configuration of the moving object 110> Next, the functional configuration of the moving object 110 will be described. The processing of each unit described below is implemented as software by loading a computer program from ROM 208 or the like onto RAM 209 and then executing the program by CPU 207. A part or all of the functional configuration may be configured as hardware such as ASIC or FPGA.

[0021] FIG. 3 is a block diagram showing the functional configuration of the moving object 110. As shown in FIG. The imaging control unit 301 acquires sensor information from the sensor unit through the general-purpose I / F 211. In this embodiment, the imaging control unit 301 acquires image information and distance information from the camera 201, and distance information from the LiDAR 202. The recognition section (detection means) 302 performs a predetermined recognition process based on the sensor information acquired by the imaging control section 301 from the sensor section and an environmental map (to be described later) read from the storage section 303 . Examples of recognition processing include detecting obstacles, people, and landmarks while driving, and performing the recognition processing necessary to drive safely without colliding even if the environment changes dynamically.

[0022] The self-position estimation unit 304 calculates the self-position (position and attitude) of the moving body 110 at the time the sensor information is acquired, based on the sensor information acquired by the imaging control unit 301 from the sensor unit and the environmental map described later read from the memory unit 303. In this embodiment, a self-location estimation technique (Monte Carlo Localization, MCL) using a particle filter (known) is used as an example. For example, the technique described in the following document 1 and the technique described in the following document 2, in which the number of MLC particles is variable, are applied and used.

[0023] (Reference 1) F.Dellaert, D.Fox, W.Burgard, S.Thrun, Monte Carlo localization for mobile robotS, Proc.of IEEE International Conference on RoboticS and Automation, 1999

[0024] (Reference 2)Dieter Fox, Adapting the Sample Size in particle filterS through KLD-Sampling, International Journal ofRoboticS ReSearch, 2003

[0025] When an environmental map is not yet available, SLAM (Simultaneous Localization and Mapping) technology is used as a method of creating an environmental map using sensor information. Since the creation of an environmental map and self-localization are interdependent, SLAM simultaneously performs self-localization and map creation. SLAM receives sensor data as input and outputs the movement trajectory of a moving object and map information. In this embodiment, as an example, a SLAM technique using distance information from a 2D LiDAR is used, for example, the technique described in the following document 3. This is one of the techniques called sequential SLAM, which builds an environmental map each time sensor information is obtained.

[0026] (Reference 3) Giorgio GriSetti, Cyrill StachniSS, Wolfram Burgard, Improved techniqueS for grid mapping with rao-blackwellized particle filterS, IEEE tranSactionS on RoboticS, 2007

[0027] The output of the self-location estimation unit 304 is sent to the plan judgment unit 305 and used for driving control of the moving body 110, or the self-location information is sent to a display device not shown and used to display the current location of the moving body 110 on the UI. It is also possible to transmit sensor information acquired by the imaging control unit 301 to an external device via a network communication unit, and to have the position and orientation calculated by the external device be acquired by the self-position estimation unit 304 via the network communication unit 306. The plan determination unit 305 plans and determines the next operation based on the self-position of the mobile object 110 acquired from the self-position estimation unit 304 and the state of the surrounding environment acquired from the recognition unit 302. For example, if an obstacle is detected, it determines to plan and execute a local detour route. Alternatively, it determines to change the cleaning mode (suction strength, movement speed, movement route, etc.) for each location. The output of the plan determination unit 305 is sent to the control unit 307, and automatic driving control of the moving body 110 is performed.

[0028] <Environmental Map> The environmental map used in this embodiment will be described below. When performing self-location estimation using the LiDAR 202 as the environmental map, for example, a point cloud map created based on distance information obtained by the LiDAR 202 is used. Alternatively, an occupancy grid map is used, which is obtained by dividing a space into cells in a grid pattern based on the point cloud map and formatting each cell to indicate the probability of whether it has been observed or not and whether it contains a point cloud or not. In this embodiment, an occupancy grid map is used.

[0029] (How to obtain an environmental map) The environmental map is generated by moving a mobile object 110 equipped with a sensor within the environment. When there is no environmental map, a movement route for acquiring sensor data is given by, for example, operating the moving object 110 by remote control or by hand. The generation of the environmental map based on the sensor information acquired by the sensor mounted on the mobile object 110 may be performed simultaneously with the operation, or may be performed offline after data collection. Alternatively, the mobile object 110 may travel autonomously within the environment, planning a travel route exploratory using the maps generated up to that point and local maps around the current location.

[0030] <Workflow of the Mobile Unit 110> FIG. 4 is a flow chart for explaining the operation of the moving object 110. As shown in FIG. First, in step S401, an environmental map that has been created in advance is read from the storage unit 303. Next, in step S 402 , the self-position estimation unit 304 calculates the self-position of the mobile object 110 from the environmental map and local information about the current location obtained by the image capture control unit 301 . Next, in step S403, the plan determination unit 305 performs a route plan from the calculated self-position to the work start point.

[0031] In step S404, the robot moves to the work start point while estimating its own position on the environmental map. Next, in step S405, a route plan from the work start point to the work end point is created. If a task of the type that involves working evenly within an area, such as cleaning, is assigned, a route plan for that purpose is created. Alternatively, if a route plan has been created in advance, it is read from the storage unit 303. Next, in step S406, the robot travels according to the route plan. At this time, a predetermined operation is performed as necessary. The predetermined operation is a suction or wiping operation for a cleaning robot, a grass cutting operation for a lawnmower, or a tilling operation for a cultivator.

[0032] When the task end point is reached, in step S407, a route plan is made for returning from the current position to a predetermined position such as the travel start point. In step S408, the vehicle travels to a predetermined location according to the route plan, and a message is sent to the server 130 via the network communication unit 306 indicating that the work has been completed. If an obstacle not on the environmental map is detected while traveling, the robot replans a local route to avoid it and continues traveling along that route. If the robot cannot avoid the obstacle, it stops, indicates that an abnormality exists using the mobile unit's indicator light, notifies the user of a message via the network communication unit, and waits for an operation from the administrator. The cleaning time and travel route, or areas where cleaning could not be performed due to obstacle avoidance, etc. are stored in the memory unit 303 as a report after the work is completed, or are sent to the server 130 via the network communication unit 306.

[0033] <Hardware configuration of server 130> FIG. 5 is a diagram showing a hardware configuration of the server 130. As shown in FIG. The server 130 is configured as a general computer such as a PC, etc. For example, as shown in Fig. 5, the server 130 includes a processor such as a CPU 501, memories such as a RAM 502 and a ROM 503, a storage unit 504 such as an HDD, and a communication unit 505. The server 130 can execute various functions by the CPU 501 executing programs stored in the memory or storage device.

[0034] <Functional configuration of server 130> FIG. 6 is a diagram showing the functional configuration of the server 130. As shown in FIG. The server 130 includes, as its functional configuration, a network communication unit 601, a control unit 602, a display unit 603, an operation unit 604, an analysis unit 605, a storage unit 606, and a setting processing unit 607, for example. The network communication unit 601 connects to, for example, the network 120 and executes communication with the mobile unit 110 and the like via the network 120. Note that this is merely an example, and for example, the network communication unit 601 may be configured to establish a direct connection with the mobile unit 110 and communicate with the mobile unit 110 without going through the network 120 or other devices.

[0035] The control unit 602 controls the network communication unit 601, the display unit 603, the operation unit 604, the analysis unit 605, the storage unit 606, and the setting processing unit 607 so that they execute their respective processes. The display unit 603 presents information to the user via, for example, a display. In this embodiment, the information is presented to the user by displaying the result of rendering by the browser on the display. Note that information may be presented by a method other than a screen display, such as sound or vibration. The operation unit 604 accepts operations from the user. In this embodiment, the operation unit 604 is a mouse or a keyboard, and the user operates these to input user operations to the browser. However, the present invention is not limited to this, and the operation unit 604 may be any device capable of detecting the intention of other users, such as a touch panel or a microphone.

[0036] The analysis unit (searching means) 605 performs analysis work required for the mobile unit 110 to perform a predetermined task. A method for setting partial areas that are to be no-entry areas in the environmental map based on sensor information will be described later. The storage unit 606 stores information necessary for the mobile unit 110 to perform a given task. A setting processing unit (setting means) 607 executes the setting process described below.

[0037] <Processing for Setting No-Entry Areas for Mobile Object 110 (Information Processing Method)> The following describes the process of setting the no-entry area of ​​the moving object 110. The following processes are executed by the information processing unit 203 of the moving object 110. However, this is merely one example, and some or all of the processing described below may be realized not only by the mobile object 110, but also by the server 130, a notebook PC or tablet terminal capable of communicating with the mobile object 110, or dedicated hardware.

[0038] FIG. 7 is a flow chart for explaining the process (information processing method) of creating a no-entry area for the moving object 110. In FIG. First, in step S701, the moving body 110 starts autonomous traveling to execute a given task. Examples of tasks include transporting luggage and cleaning. If the task is to transport luggage, the mobile object 110 plans a route to get from the current location to the destination without coming into contact with obstacles and return to a predetermined location such as a waiting area, and travels according to the route plan. If the task is to clean, the mobile object 110 plans a route to travel thoroughly through the cleaning area and return to a predetermined location such as a waiting area, and travels according to the route plan. At this time, the mobile object 110 acquires image information and distance information of the surrounding environment while moving.

[0039] Next, in step S702, it is determined whether the moving object 110 has arrived at the destination (a predetermined position such as a waiting place). If the moving object 110 has arrived at the destination, it stops self-propelled and ends the process. If the moving object 110 has not arrived at the destination, it proceeds to the next step. If the destination has not been reached, in step S703 (detection step), the moving body 110 performs a detection process for objects related to no-entry areas from the acquired image. Objects to be detected related to no-entry areas include traffic cones (registered trademark), tape partitions, chain partitions, signboards, barricades, fences, and warning tapes (traffic jambs). Object detection processing is carried out using an object detection model trained using a technique called machine learning, which finds patterns hidden in large amounts of data through repeated calculations. In this embodiment, as an example, a model for detecting an object using an image as input is used, and for example, the technology described in the following document 4 is applied and used.

[0040] (Reference 4) Mingxing, Tan, Ruoming Pang, Wuoc V. Le. EfficientDet: Scalable andEfficient ObjectDetection. CVPR2020

[0041] Of these, for objects such as signboards that are not self-evident as to whether they are related to no-entry, it is acceptable to increase the amount of data used to train the machine learning model if this does not cause any problems in practical use. However, if there are many false positives depending on the environment, it is also possible to read the character strings contained in the signboard and determine whether a specific character string is included. That is, the information processing unit 203 uses the OCR function to recognize whether the signboard contains character strings such as "No entry", "No entry", "Cleaning in progress", "Work in progress", "Caution", "caution", and "cleaning". If the signboard contains these character strings, it is treated as an object related to no-entry, and if not, it is treated as an object not related to no-entry.

[0042] Furthermore, objects related to these no-entry areas may be placed one on top of another, in which case they may be recognized as being different from objects placed alone and excluded from the objects related to the no-entry areas. If the reliability (certainty) of the object output by the object detection model is above a threshold (e.g., 70% or more), the object is processed as having been detected. However, if the reliability is somewhat low (e.g., 40% to 70%), the user is notified along with the image so that the user can make a judgment. If an object related to the no-entry area is detected, the process proceeds to the next step, and if not, the process returns to S701 and continues traveling to the destination.

[0043] If an object is detected, the position of the detected object in three-dimensional space is recorded in step S704. In this embodiment, since image information and distance information are acquired, the position of the object in the map coordinate system (world coordinate system) is calculated and recorded using the distance from the camera to the detected object and the current position and orientation of the moving body. Next, in step S705 (search step), the environment around the object is searched to determine a no-entry area according to the type of the identified object.

[0044] The search method differs depending on the type of object, and is described below. FIG. 8 is a diagram showing the relationship between the type of object, the search target at that time, and the setting method (location) of the no-entry area. Objects related to no-entry areas can be classified into those that make sense when there is only one, those that make sense when there are multiple, and those that make sense in either one or multiple. An example of a single meaningful object is a signboard. When such an object is detected, the system will prohibit entry to likely areas nearby, such as passageways and openings such as entrances and exits. In this embodiment, the vehicle is driven autonomously using LiDAR, so specific shapes and patterns (e.g., straight paths, paths with a constant width, etc.) can be treated as passageways from the occupancy grid map used by the LiDAR.

[0045] 9 to 12 are diagrams showing an example of creating a no-entry area. As shown in FIG. 9, when an object 901, such as a stand-alone item such as a signboard, is detected, a no-entry line (no-entry area) 902 is set to block the nearest passageway within a certain distance of the object 901.

[0046] As shown in FIG. 10, when an object 1001, which is a standalone object such as a signboard, is placed on a passageway, a no-entry line (no-entry area) 1002 is set by drawing a perpendicular line from the object 1001 to the parallel wall of the passageway. If a floor map containing semantic information about the building can be obtained, when a detected object indicates that it is being cleaned, it may be possible to check whether there is a toilet within a certain distance from the object, and if so, to set a no-entry line in the passageway leading to the toilet.

[0047] As shown in Fig. 11, if there is no passage within a certain distance of an object, or if the object is not on a passage, a check is made to see if there is an opening within a certain distance of the object 1101, and if there is, a no-entry line (no-entry area) 1102 is set at the opening. An opening here is a place that is not occupied for a certain distance in a linearly occupied space. This mainly corresponds to an entrance or exit. If a floor map containing semantic information about the building can be obtained, it is possible to check whether there is an opening (or entrance / exit) within a certain distance from the object, and if there is, to set a no-entry line at the opening (or entrance / exit). If the LiDAR measurement values ​​acquired by the mobile object management system 101 are significantly different from the occupancy grid map created in advance, the mobile object management system 101 moves around the detected object and updates the occupancy grid map based on the current sensor information. Then, the mobile object management system 101 may find passages and openings based on the occupancy grid map.

[0048] If neither a passage nor an opening is found, an image of the object and the location where the object is detected are transmitted to the user, allowing the user to decide whether or not to set a no-entry area. The above rules are just examples, and new rules may be used when an object is found. If the previously set rules cannot be used to make a decision, the user is notified and allowed to decide whether or not to make new settings.

[0049] Next, the term "multiple" means (used in plural) for example tape partitions and chain partitions. There is a high possibility that these will not fit in the field of view of one frame of image captured by the moving body 110. Therefore, when a pole (or tape or chain) that constitutes a partition is detected, a search is performed to see how far the partition continues around it.

[0050] The following description will be given assuming an environment as shown in FIG. First, when object 1201-1, which is one of multiple objects 1201 that make sense, is detected, the position of object 1201-1 is recorded. Then, the moving body 110 is moved left and right to check whether any similar objects are detected. When multiple objects are detected, the moving body 110 moves toward the detected object (object 1201-2 or object 1201-6) and records its position. The reason for moving closer here is to accurately determine the object's position due to the limitations of the distance measurement range and accuracy of the camera 201, but it is not necessarily necessary to get closer if sufficient accuracy can be obtained. If no object is detected (only object 1201-1 is detected), the mobile unit 110 moves and changes the viewpoint, and then checks whether a similar object is detected within a certain range around object 1201-1 (in this case, object 1201-2 or object 1201-6 should be detected). By repeating this process, the location of the detected object is determined. Of course, a search is also performed in the opposite direction to the initial movement direction (if the object initially moved toward object 1201-2, then a search is performed again toward object 1201-5).

[0051] Once the placement of the detected objects has been determined in this way, lines connecting the objects 1201-1 to 1201-6 are created using, for example, a nearest neighbor method, to create no-entry lines (no-entry areas) 1202. If the detected object is close to another occupied part (such as a wall), the no-entry lines are extended from the detected object to the other occupied part. If chains or tapes are detected, they may be used as clues to create lines connecting the detected objects. An example of an object that makes sense whether it is one or more is a traffic cone. When an object of this type is detected, a search is performed to see if there are any similar objects in the surrounding area, just as with objects that make sense when multiple are detected. If it is not found in the surrounding area, the same processing is performed as for objects that make sense when they are one, and if it is found in the surrounding area, the same processing is performed as for objects that make sense when multiple are detected.

[0052] Returning to FIG. 7, once it has been confirmed that no similar object exists within a certain distance of the detected object and the no-entry line has been set, a no-entry area is set in step S706 (setting step). In this embodiment, the autonomous driving system performs route planning based on an occupancy grid map (cost map) that divides space into a grid based on sensor information and indicates the probability of an obstacle existing in each grid. Therefore, it is possible to intervene in the route planning of the autonomous driving system by reflecting no-entry lines in the occupancy grid map (cost map). In other words, the grid on the occupancy grid map at the position corresponding to the no-entry line is set to an occupied state.

[0053] Alternatively, the mobile object 110 may be set to occupy the entire drivable area behind the no-entry line as viewed from the mobile object management system 101. This makes it possible to prohibit the mobile object 110 from going around to the other side of the no-entry line. FIG. 13 is a diagram showing an example of creating a no-entry area. 13, a no-entry line 1302 is created from two objects 1301-1 and 1301-2. Then, the travelable area on the far side of the no-entry line 1302 as viewed from the mobile object management system 101 is uniformly set as a no-entry area 1303.

[0054] In this way, an occupancy grid map is created from the no-entry lines, and once the no-entry areas have been set, the vehicle returns to the original route and resumes self-driving toward the destination in step S707.

[0055] In the above, we have described cases where a new object related to the no-entry rule is found in a place where it was not there before, or where an object related to the no-entry rule is found in a place visited for the first time. However, there are also cases where an object that was originally there disappears. Below, we will provide additional explanation on a method that enables processing when an object related to a previous no-entry rule no longer exists.

[0056] The mobile object 110 holds a data structure that divides the three-dimensional space into voxels for environmental recognition, and allows each voxel to have one of three states: occupied, unoccupied, and unmeasured. For those voxels that are in an occupied state, the object category can be held as an attribute of the voxel, in addition to the color, normal, etc. The mobile object 110 handles the surrounding environment as information from the camera 201 as input. For voxels corresponding to the viewing frustum determined by the distance measurement result and distance measurement range of the camera 201, information is updated based on the recognition result (values ​​of voxels inside the viewing frustum that are further back than the distance measurement result are not updated). The updating method may be a new replacement each time a measurement is performed, or a weighted average may be taken. By having such a data structure related to the three-dimensional space, it is possible to grasp the appearance and disappearance of objects related to the no-entry rule. When there is a change in the information of this three-dimensional voxel, a search is performed to see if there is any change in the surrounding environment not only when a new object appears, but also when an object that was originally there disappears. By resetting the no-entry line, it is possible to respond to changes (disappearance) of the no-entry area. In the above, we have also explained the mechanism for achieving the purpose of temporarily prohibiting entry during cleaning, construction, etc., by recognizing objects related to no-entry areas and reflecting the no-entry lines in the occupancy grid map.

[0057] FIG. 14 is a diagram showing an example of creating a no-entry area. However, in reality, as shown in FIG. 13, there may be cases where objects 1401 such as traffic cones or partitions are installed on the passageway for the purpose of regulating the traffic flow. In this embodiment, in such a case, the moving object 110 cannot cross a no-entry line (no-entry area) 1402 that is configured by a traffic cone, partition, or the like. Therefore, in order to get to the other side of the no-entry line 1402, the moving object 110 must make a detour to a place where the line is broken. In such a case, if it is desired to allow the moving object 110 to travel without making a detour, the user can use a GUI or the like to delete the no-entry line 1402 or conversely add another line. In addition, by using a machine learning model that uses images or 3D reconstructed 3D models as inputs to set no-entry areas, it may be possible to infer no-entry areas end-to-end. However, since there are many variations between objects related to no-entry and the environments in which they are located, a huge amount of data must be collected to achieve satisfactory accuracy when using a machine learning model. In addition, when the area division is not successful, the interpretability is low and correction is not easy. In addition, when trying to improve accuracy, the scale of the machine learning model tends to increase and the calculation cost tends to become higher, so it is not necessarily suitable for processing by an information processing device built into a moving object.

[0058] On the other hand, in this embodiment, object detection and creation of no-entry areas (no-entry lines) are configured as independent modules. Therefore, object detection and optimization of no-entry areas can be performed independently, and interpretability is high. Since what each module tries to achieve is relatively simple, calculation costs can be suppressed. Another advantage is that it is easy to change the application of rules for each location. Therefore, in this embodiment, instead of estimating the no-entry areas from an image end-to-end, a method that incorporates rules is used as a method for setting the no-entry areas more efficiently and reliably. This makes it possible to set the no-entry areas with high accuracy while suppressing the data collection costs and calculation costs.

[0059] [Second embodiment] In this embodiment, a case will be described in which the mobile unit 110 can share information with a surveillance camera. If the external parameters of the surveillance camera are appropriately given, when an object related to the no-entry zone is detected, the position of the object can be identified on the assumption that it is placed on the floor. The mobile unit 110 can shorten the time required to set the no-entry zone by searching for the position information of the object related to the no-entry zone, mainly in the blind spot of the surveillance camera, using the upward direction (detection result) from the surveillance camera.

[0060] [Other embodiments] Although the present invention has been described with an example in which the images captured by the mobile object 110 are analyzed by the mobile object 110, the images captured by the mobile object 110 may be analyzed by the server 130. The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

[0061] The disclosure of this embodiment includes the following information processing method. (Method 1) An information processing method for setting a no-entry area for a moving object, comprising: a detection step of detecting an object from the image information and determining a position of the object; a searching step of searching the surroundings according to the type and position of the object; a setting step of setting the no-entry area based on the position of at least one of the objects obtained by repeating the detection step and the search step; An information processing method comprising: (Method 2) The information processing method according to method 1, characterized in that, when the object is used alone, the no-entry area is set in a passage or opening closest to the object. (Method 3) The information processing method according to method 1 or 2, characterized in that, when the object is used alone and there is a registered passage or opening in the vicinity of the object, the no-entry area is set in the passage or opening. (Method 4) The information processing method according to any one of Methods 1 to 3, characterized in that, when the object is used in a plurality of instances, a certain range is searched around the object to detect the plurality of objects, and the no-entry area is set by determining line segments connecting the objects. (Method 5) An information processing method according to any one of methods 1 to 4, characterized in that, when the object is used singly or in a plurality of instances, a certain range around the object is searched, and when only one object is detected, a nearby passage or opening is set as the no-entry area, and when a plurality of objects are detected, a line segment connecting the detected objects is obtained to set the no-entry area. (Method 6) The information processing method according to any one of methods 1 to 5, characterized in that, when the detection result of a surveillance camera is available, a search is performed for locations that are blind spots of the surveillance camera. (Configuration 1) An information processing system for setting a no-entry area for a moving object, comprising: a detection means for detecting an object from image information and determining a position of the object; a search means for searching the surroundings according to the type and position of the object; a setting means for setting the no-entry area based on the position of the object obtained by repeatedly using the detection means and the search means; An information processing system comprising: (program) A program for causing the CPU of an information processing system to function as each of the means of component 1.

[0062] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0063] 101 Mobile management system (information processing system) 110 Mobile 120 Network 130 Server 201 Camera 202 LiDAR 203 Information Processing Department 205 Control Unit 207 CPU 301 Imaging control unit 302 Recognition unit (detection means) 304 Self-position estimation unit 305 Planning and Judgment Department 307 Control Unit 501 CPU 602 Control section 605 Analysis section (search means) 607 Setting processing unit (setting means) 901 Object 902 No entry line (no entry area) 1001 Object 1002 No entry line (no entry area) 1101 Object 1102 No entry line (no entry area) 1201 Object 1202 No entry line (no entry area) 1301-1 Object 1301-2 Object 1302 Entry prohibited ライン 1303 Entering the Impossible Area 1401 Object 1402 Entering the forbidden area

Claims

1. An information processing method for setting a no-entry area for a moving object, comprising: a detection step of detecting an object from the image information and determining a position of the object; a searching step of searching the surroundings according to the type and position of the object; a setting step of setting the no-entry area based on the position of at least one of the objects obtained by repeating the detection step and the search step; An information processing method comprising:

2. 2. The information processing method according to claim 1, wherein, when the object is used alone, the no-entry area is set in a passage or an opening that is closest to the object.

3. 2. The information processing method according to claim 1, wherein, when the object is used alone, if there is a registered passage or opening in the vicinity of the object, the no-entry area is set at the passage or opening.

4. 2. The information processing method according to claim 1, characterized in that, when the object is used in a plurality of instances, a certain range is searched around the object to detect the plurality of objects, and the no-entry area is set by determining line segments connecting the objects.

5. 2. The information processing method according to claim 1, characterized in that, when the object is used singly or in a plurality of instances, a certain range around the object is searched, and when only one object is detected, a nearby passage or opening is set as the no-entry area, and when a plurality of objects are detected, a line segment connecting the detected objects is obtained to set the no-entry area.

6. 2. The information processing method according to claim 1, further comprising the step of: searching for locations that are blind spots of a surveillance camera when a detection result of the surveillance camera is available.

7. An information processing system for setting a no-entry area for a moving object, comprising: a detection means for detecting an object from image information and determining a position of the object; a search means for searching the surroundings according to the type and position of the object; a setting means for setting the no-entry area based on the position of the object obtained by repeatedly using the detection means and the search means; An information processing system comprising:

8. A program for causing a CPU of an information processing system to function as each of the means recited in claim 7.

Images