Management system and map generation system

Abstract

This disclosure relates to a management system and a map generation system that efficiently ensure a good environment for recognition processing using recognition sensors. The management system manages the environment of a defined area. Within the defined area, there are recognition sensors that recognize the surrounding conditions and recognition objects that should be recognized by recognition sensors located at specific locations. The three-dimensional map data of the defined area includes specific spatial information representing specific spaces that are the space between specific locations and recognition objects. The management system acquires recognition result information representing the recognition results obtained by recognition sensors located at specific locations. Furthermore, the management system determines whether objects in specific spaces not included in the three-dimensional map data are included in the specific spaces in the recognition result information. Then, the management system senses objects in specific spaces not included in the three-dimensional map data but included in the specific spaces in the recognition result information as objects to be removed.

Description

Technical Field

[0001] This disclosure relates to techniques useful for managing the environment of a designated area. Furthermore, this disclosure relates to three-dimensional map data of the designated area and techniques for generating such three-dimensional map data. Background Technology

[0002] Patent Document 1 discloses a method for creating a three-dimensional digital map. An object configuration data holding unit holds object configuration data representing the absolute configuration of objects. A three-dimensional digital map creation unit sets a virtual field of view and retrieves object configuration data corresponding to the virtual field of view from the object configuration data holding unit. Then, the three-dimensional digital map creation unit models the objects represented by the retrieved object configuration data into three-dimensional shapes to create a three-dimensional digital map.

[0003] Patent Document 2 discloses a method for displaying a three-dimensional map in a navigation device. In this method, a three-dimensional map is displayed, and details of traffic signs on the guidance route are magnified for display.

[0004] Patent document 3 discloses a technique for displaying information based on images captured from a moving object. When an object such as a traffic light is obscured by an obstacle, the image is processed so that the object can be seen through the obstacle.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2005-044224

[0008] Patent Document 2: Japanese Patent Application Publication No. 2003-337040

[0009] Patent Document 3: Japanese Patent Application Publication No. 2012-208111

[0010] Consider recognition processing that uses recognition sensors (e.g., cameras) to identify objects by recognizing their surroundings. For example, autonomous vehicles use onboard recognition sensors to identify traffic lights, signs, pedestrian crossings, and other similar objects. In such recognition processing, the surrounding environment is crucial. For instance, if there are obstacles (e.g., protruding branches or leaves) in the space between the object and the recognition sensor, the recognition process will be hindered, and the accuracy of object recognition may decrease. To ensure a favorable environment for recognition processing, such obstacles need to be sensed and removed (eliminated).

[0011] However, generally speaking, determining which object is actually an obstacle hindering the recognition process is not easy. Removing (excluding) irrelevant objects along with the recognition process is inefficient. The goal is to more efficiently ensure a favorable environment for the recognition process. Summary of the Invention

[0012] One object of this disclosure is to provide an environment that can efficiently ensure good performance for identification processing using identification sensors.

[0013] The first point is related to the management system that manages the environment of a designated area.

[0014] The management system includes: one or more processors; and one or more storage devices for storing three-dimensional map data of a specified area.

[0015] Within a designated area, there are identification sensors that identify the surrounding conditions and identification objects that should be identified by identification sensors located in specific locations.

[0016] 3D map data includes specific spatial information representing the space between a specific location and an identified object.

[0017] One or more processors acquire identification result information representing the identification results obtained by identification sensors located at a specific location.

[0018] One or more processors determine whether an object in a specific space that is not included in the 3D map data is included in the specific space in the recognition result information.

[0019] One or more processors will sense objects in a specific space that are not included in the 3D map data but are included in the identification results information as objects to be removed.

[0020] The second point is related to the map generation system that generates 3D map data for a specified area.

[0021] The map generation system includes: one or more processors; and one or more storage devices for storing semantic model information representing attribute information for each constituent element constituting a defined area.

[0022] Within a designated area, there are identification sensors that identify the surrounding conditions and identification objects that should be identified by identification sensors located in specific locations.

[0023] One or more processors define the identified object as one of the constituent elements in the semantic model information.

[0024] One or more processors define a specific location as one of the constituent elements in the semantic model information.

[0025] One or more processors define a specific space, which is the space between the specific location and the identified object, as one of the constituent elements in the semantic model information based on the identified object and the specific location.

[0026] One or more processors generate 3D map data, including specific spatial information representing a specific space, based on semantic model information that defines a specific space.

[0027] The third point is related to the three-dimensional map data of the designated area.

[0028] Within a designated area, there are identification sensors that identify the surrounding conditions and identification objects that should be identified by identification sensors located in specific locations.

[0029] Three-dimensional map data has a data structure that includes specific spatial information representing the space between a specific location and an identified object.

[0030] The identification result information represents the identification result obtained through an identification sensor located at a specific location.

[0031] The 3D map data is read by a management system that manages the environment of the designated area.

[0032] Whether an object in a specific space not included in the 3D map data is included in the specific space in the recognition result information is determined by the management system.

[0033] Objects in a specific space that are not included in the 3D map data but are included in the identification results information are sensed by the management system as objects to be removed.

[0034] Invention Effects

[0035] According to the first viewpoint, three-dimensional map data including specific spatial information representing a specific space is utilized. A specific space is defined as the space between an object to be identified and a specific location where that object should be identified. By utilizing three-dimensional map data including specific spatial information representing such a specific space, obstacles hindering the identification process can be accurately sensed as objects to be removed. As a result, a favorable environment for the identification process can be efficiently ensured; that is, the environment of a designated area can be efficiently managed.

[0036] According to the second viewpoint, by utilizing the semantic model information of a defined area, three-dimensional map data including specific spatial information representing a specific space can be generated efficiently. By utilizing this three-dimensional map data, a favorable environment for recognition processing can be efficiently ensured.

[0037] According to the third perspective, 3D map data is provided for accurately sensing obstacles hindering the recognition process as objects to be removed. By utilizing 3D map data, a favorable environment for recognition processing can be efficiently ensured. Attached Figure Description

[0038] Figure 1 It is a concept map used to illustrate a specified area.

[0039] Figure 2 This is a conceptual diagram used to illustrate an example of identification processing that uses an identification sensor.

[0040] Figure 3 This is a conceptual diagram used to illustrate an example of a problem involving identification processing that uses identification sensors.

[0041] Figure 4 This is a conceptual diagram used to illustrate the identification processing using an identification sensor and its problems in embodiments of this disclosure.

[0042] Figure 5 This is a conceptual diagram used to illustrate a specific space for embodiments of this disclosure.

[0043] Figure 6 This is a conceptual diagram used to illustrate a specific surrounding space for embodiments of this disclosure.

[0044] Figure 7 This is a block diagram used to illustrate an overview of the three-dimensional map data, map generation system, and management system of embodiments of this disclosure.

[0045] Figure 8 This is a block diagram illustrating an example configuration of a map generation system according to an embodiment of the present disclosure.

[0046] Figure 9 This is a flowchart illustrating a first example of map generation processing performed by the map generation system according to an embodiment of the present disclosure.

[0047] Figure 10 This is a flowchart illustrating a second example of map generation processing performed by the map generation system according to an embodiment of the present disclosure.

[0048] Figure 11 This is a conceptual diagram illustrating the sensor field-of-view verification process implemented by the map generation system according to embodiments of the present disclosure.

[0049] Figure 12 This is a flowchart illustrating the process associated with the sensor field-of-view verification process implemented by the map generation system according to embodiments of the present disclosure.

[0050] Figure 13This is a block diagram illustrating an example of the configuration of a management system according to an embodiment of the present disclosure.

[0051] Figure 14 This is a flowchart illustrating a first example of management processing implemented by the management system of the embodiments of this disclosure.

[0052] Figure 15 This is a flowchart illustrating a second example of management processing implemented by the management system of the embodiments of this disclosure.

[0053] Figure 16 This is a block diagram illustrating the provision of information to a moving body in accordance with embodiments of the present disclosure.

[0054] Explanation of reference numerals in the attached figures:

[0055] 1: Moving body

[0056] 10: First object

[0057] 15: Identification Sensor

[0058] 20: Second object

[0059] 30: Obstacles

[0060] 100: 3D map data

[0061] 110: Specific spatial information

[0062] 120: Specific surrounding spatial information

[0063] 200: Map Generation System

[0064] 210: User Interface

[0065] 220: Communication device

[0066] 230: Processor

[0067] 240: Storage device

[0068] 250: Map Generation Program

[0069] 260: Semantic model information

[0070] 300: Management System

[0071] 310: User Interface

[0072] 320: Communication device

[0073] 330: Processor

[0074] 340: Storage device

[0075] 350: Management Procedures

[0076] 360: Management Information

[0077] 370: Recognition Result Information

[0078] AR: Designated Area

[0079] PX: Specific Location

[0080] SX: Specific Space

[0081] SY: Specific surrounding space. Detailed Implementation

[0082] The embodiments of this disclosure will be described with reference to the accompanying drawings.

[0083] 1. Summary

[0084] 1-1. Specified Area

[0085] Figure 1 This is a conceptual diagram used to illustrate a defined area (AR). For example, a defined area AR can be a street (e.g., a smart city). As another example, a defined area AR can also be facility land. As yet another example, a defined area AR can also be an area within a building.

[0086] Within a defined AR (Augmented Reality) area, there exist various objects. Typically, there is a Mobility 1 that moves within the AR area. Examples of Mobility 1 include vehicles, robots, and flying objects. Vehicles can be either autonomous or driven by a human. Examples of robots include logistics robots, cleaning robots, and planter robots. Examples of flying objects include airplanes and drones.

[0087] 1-2. Identification processing using an identification sensor.

[0088] The mobile body 1 possesses at least a recognition function for identifying its surroundings. More specifically, the mobile body 1 has a recognition sensor for identifying its surroundings. Examples of recognition sensors include cameras, LIDAR (Laser Imaging Detection and Ranging), and radar. The mobile body 1 performs recognition processing to identify its surroundings using the recognition sensor. In particular, the mobile body 1 uses the recognition sensor to identify objects around itself. Examples of objects around the mobile body 1 include pedestrians, other vehicles (vehicles ahead, parked vehicles, etc.), traffic lights, white lines, crosswalks, and signs. For example, objects can be identified and their relative positions calculated by analyzing an image (IMG) obtained from a camera. Furthermore, objects can also be identified and their relative positions and relative speeds obtained from point group information obtained from LIDAR can be acquired.

[0089] The mobile body 1 performs prescribed processing based on the identification results obtained through the identification sensor. For example, the mobile body 1 moves voluntarily based on the identification results obtained through the identification sensor. As another example, the mobile body 1 can also assist in operations performed by an operator based on the identification results obtained through the identification sensor. As yet another example, the mobile body 1 can also prompt the operator with the identification results obtained through the identification sensor.

[0090] Figure 2 This is a conceptual diagram used to illustrate an example of recognition processing that utilizes a recognition sensor. Figure 2 In the example shown, the moving body 1 is an autonomous vehicle. The autonomous vehicle is equipped with recognition sensors such as cameras. The cameras acquire image (IMG) images of the area surrounding the autonomous vehicle. The autonomous vehicle uses the IMG images obtained from the cameras to identify the traffic lights and their signal displays (green, yellow, red, etc.) ahead. Specifically, the autonomous vehicle needs to identify the traffic lights and their displays within a certain range in front of them. Then, the autonomous vehicle performs autonomous driving control based on the identification results of the traffic lights and their displays.

[0091] However, as Figure 3 As shown, if the branches and leaves of roadside trees extend outwards, at least a portion of the traffic light may be obscured when viewed from a certain distance in front of it. That is, at least a part of the traffic light may not be visible from the location where it should be recognized. This leads to a decrease in the recognition accuracy of the traffic light and its signal display. This decrease in the recognition accuracy of the traffic light and signal display contributes to a decrease in the accuracy of autonomous driving control. To ensure a favorable environment for signal recognition, it is necessary to prune the outward branches and leaves of roadside trees.

[0092] As a comparative example, consider uniformly and periodically pruning all street trees within a designated AR area. However, in this case, branches and leaves of street trees in locations that do not affect signal recognition at all may be unnecessarily pruned. Furthermore, even street trees in locations that might affect signal recognition may be unnecessarily pruned before their branches and leaves extend to the point of obstructing signal recognition. Conversely, branches and leaves that actually obstruct signal recognition may be left untouched because the scheduled pruning period has not yet arrived. Thus, the comparative example of uniformly and periodically pruning all street trees within a designated AR area is "inefficient." To efficiently ensure a good environment for signal recognition, branches and leaves should be pruned only in areas where they are truly necessary at appropriate intervals.

[0093] Reference Figure 4 This section provides a more general explanation of the recognition processing using recognition sensors and its associated problems. Within a defined AR region, there exists a first object 10 and a second object 20.

[0094] The first object 10 is equipped with an identification sensor 15 for recognizing the surrounding conditions. Examples of identification sensors 15 include cameras, LiDAR, and radar. For example, the first object 10 is the aforementioned moving body 1, and the identification sensor 15 is mounted on the moving body 1. As another example, the first object 10 may be a monitoring device that monitors the conditions within a specified area, and the identification sensor 15 may be included in the monitoring device.

[0095] The second object 20 is the "identified object" recognized by the identification sensor 15. Specifically, the second object 20 is an identified object that should be recognized by the identification sensor 15 located at a specific position PX within the designated area AR. For example, the second object 20 is a traffic light (see reference). Figure 2 , Figure 3 The traffic light needs to be identified by the recognition sensor 15 within a certain range in front of it, which corresponds to a specific location PX. The second object 20 is not limited to traffic lights. Other examples of the second object 20 include pedestrian crossings, signs, and pedestrians. It should be noted that the second object 20 does not need to be identified by the recognition sensor 15 located outside the specific location PX.

[0096] Obstacle 30 is an object that obstructs the identification processing performed by the identification sensor 15. That is, obstacle 30 is an object existing in the space between the identification sensor 15 at a specific location PX and the second object 20. Examples of obstacles 30 include tree branches and leaves, fallen objects, flying objects, garbage, illegally parked vehicles, etc.

[0097] If such an obstacle 30 exists, the recognition processing using the recognition sensor 15 will be hindered, potentially reducing the recognition accuracy of the second object 20. To ensure a favorable environment for recognition processing, it is necessary to sense and remove (eliminate) the obstacle 30. However, generally speaking, determining which object is actually the obstacle 30 hindering recognition processing is not easy. Removing (eliminating) irrelevant objects along with the obstacle 30 that do not actually affect recognition processing is inefficient. A more efficient approach to ensuring a favorable environment for recognition processing is desired.

[0098] Therefore, this embodiment proposes a technique that can accurately sense obstacles 30 (i.e., actual objects to be removed) that hinder the recognition process performed by the recognition sensor 15. By accurately sensing the actual objects to be removed, a favorable environment for the recognition process can be ensured more efficiently.

[0099] 1-3. Specific Space

[0100] Figure 5 This is a conceptual diagram used to illustrate a feature of this embodiment. According to this embodiment, in order to accurately sense obstacles 30 that obstruct the identification process performed by the identification sensor 15, the concept of a "specific space SX" is introduced. As described above, the second object 20 is an object to be identified by the identification sensor 15, which exists at a specific location PX. The specific space SX refers to the space between the specific location PX and the second object 20 (the object to be identified).

[0101] In order for the recognition sensor 15 to accurately identify the second object 20, good visibility is required in the specific space SX. If an object is present in the specific space SX, the object becomes an obstacle 30 that hinders the recognition process performed by the recognition sensor 15. Therefore, objects included in the specific space SX are sensed as obstacles 30 to be removed. On the other hand, objects existing outside the specific space SX do not affect the recognition process and therefore do not need to be removed. That is, objects existing outside the specific space SX are not sensed as objects to be removed, and only objects included in the specific space SX are sensed as objects to be removed.

[0102] Thus, by defining a specific space SX by considering the positional relationship between a specific location PX and the second object 20 (the object to be identified), obstacles 30 that hinder the identification process (i.e., the actual objects to be removed) can be sensed with high accuracy. As a result, a favorable environment for the identification process can be efficiently ensured. That is, the environment of the specified AR area can be efficiently managed.

[0103] This also takes into account situations where the branches and leaves of roadside trees have not yet invaded a specific space SX, but extend into the vicinity of that space SX. From the perspective of managing the environment of the designated AR area, it is also useful to pre-sensor such branches and leaves as "removal object candidates." For example, it is possible to pre-plan the order in which multiple removal object candidates will be removed. This, in turn, can efficiently ensure a favorable environment for recognition processing.

[0104] Figure 6 This is a conceptual diagram illustrating a "specific surrounding space SY" used for sensing and removing object candidates. The specific surrounding space SY is the finite space surrounding a specific space SX. For example, the specific surrounding space SY is the space within a certain distance from the outer surface of the specific space SX. Objects included in this specific surrounding space SY are sensed as removal object candidates. These removal object candidates may become removal objects in the near future.

[0105] Information related to the specific space SX and the specific surrounding space SY described above is provided as 3D map data.

[0106] 1-4. 3D map data, map generation system, and management system

[0107] Figure 7 This is a block diagram used to explain the outline of the three-dimensional map data 100, the map generation system 200, and the management system 300 in this embodiment.

[0108] 3D map data 100 is the 3D map data of the specified area AR. That is to say, 3D map data 100 represents the 3D configuration of structures (roads, road structures, buildings, etc.) within the specified area AR.

[0109] The 3D map data 100 also includes specific spatial information 110 representing the specific space SX mentioned above. Specific spatial information 110 represents the location of the specific space SX within the defined area AR. Furthermore, specific spatial information 110 may also include metadata indicating that "objects existing in that specific space SX are removed objects."

[0110] The 3D map data 100 may also include specific surrounding space information 120 representing the aforementioned specific surrounding space SY. Specific surrounding space information 120 represents the location of the specific surrounding space SY within the defined area AR. Furthermore, specific surrounding space information 120 may also include metadata indicating that "objects existing in the specific surrounding space SY are candidates for removal."

[0111] The map generation system 200 generates three-dimensional map data 100. Specifically, the map generation system 200 efficiently generates three-dimensional map data 100 including specific spatial information 110. The map generation system 200 is implemented, for example, through a cloud-based management server. The management server can consist of multiple servers performing distributed processing. As another example, at least a portion of the functionality of the map generation system 200 may also be included in the first object 10 (moving body 1). Details of the map generation system 200 will be described in detail in Section 2 below.

[0112] The management system 300 manages the environment of a designated AR area. Specifically, the management system 300 efficiently manages the environment of the designated AR area by utilizing the 3D map data 100 of the AR area. The management system 300 is implemented, for example, through a management server in the cloud. The management server can consist of multiple servers performing distributed processing. As another example, at least a portion of the functionality of the management system 300 may also be included in the first object 10 (moving body 1). Details of the management system 300 will be described in detail in Section 3 below.

[0113] The map generation system 200 and the management system 300 can be separate entities or at least partially shared. The management system 300 can also have the functions of the map generation system 200. The map generation system 200 can also have the functions of the management system 300.

[0114] The map generation system 200 and management system 300 of this embodiment will be described in detail below.

[0115] 2. Map Generation System

[0116] 2-1. Example of composition

[0117] Figure 8 This is a block diagram illustrating a configuration example of the map generation system 200 according to this embodiment. The map generation system 200 includes a user interface 210, a communication device 220, one or more processors 230 (hereinafter referred to as processors 230 only), and one or more storage devices 240 (hereinafter referred to as storage devices 240 only).

[0118] User interface 210 receives information input from the user (administrator) and provides the user with various information. User interface 210 includes input devices and output devices. Examples of input devices include keyboards, mice, and touch panels. Examples of output devices include display devices, touch panels, and speakers. User interface 210 may also be a GUI (Graphical User Interface).

[0119] The communication device 220 communicates with the outside world via a communication network.

[0120] Processor 230 performs various processes. For example, processor 230 includes a CPU (Central Processing Unit). Storage device 240 stores various information. Examples of storage devices 240 include volatile memory, non-volatile memory, HDD (Hard Disk Drive), SSD (Solid State Drive), etc.

[0121] Map generation program 250 is a computer program executed by processor 230. Processor 230 executes map generation program 250, thereby implementing the functions of map generation system 200. Map generation program 250 is stored in storage device 240. Alternatively, map generation program 250 may also be recorded on a computer-readable recording medium. Map generation program 250 may also be provided via a network.

[0122] Semantic model information 260 is information about the "semantic model" of the defined area AR. The semantic model is a 3D model based on concepts such as BIM (Building Information Modeling) and CIM (Construction Information Modeling). However, the semantic model is not only a 3D model, but also includes "attribute information" of each constituent element (object) that constitutes the defined area AR. Examples of the attribute information for constituent elements include their type, location, shape, size, and material. Semantic model information 260 is stored in storage device 240.

[0123] The processor 230 efficiently generates 3D map data 100 by utilizing semantic model information 260. The generated 3D map data 100 is stored in the storage device 240. The map generation process performed by the processor 230 will be described below.

[0124] 2-2. Map Generation and Processing

[0125] 2-2-1. The first example

[0126] Figure 9 This is a flowchart illustrating the first example of the map generation process in this embodiment.

[0127] In step S210, the processor 230 defines the object to be identified by the recognition sensor 15 as one of the constituent elements (targets) in the semantic model information 260. For example, if the existing semantic model information 260 includes a second object 20, the user can specify the second object 20 as the object to be identified using the user interface 210. Alternatively, the user can also input a new object to be identified and its attribute information using the user interface 210. The attribute information associated with the object to be identified includes the scope (location and shape) of the object. The processor 230 defines the object to be identified as one of the constituent elements in the semantic model information 260 according to the specification or input by the user.

[0128] Furthermore, the processor 230 defines the location of the identification sensor 15, which should identify the object, i.e., the specific location PX, as one of the constituent elements (targets) in the semantic model information 260. For example, in the case of a traffic light, the specific location PX is a certain range in front of the traffic light. The specific location PX can be calculated based on a combination of the location of the first object 10 that should be identified and the setting position of the identification sensor 15 in the first object 10. For example, the user inputs the attribute information of the specific location PX for each identified object using the user interface 210. The attribute information associated with the specific location PX includes the range (position and shape) of the specific location PX. As another example, the processor 230 can also automatically calculate the attribute information of the specific location PX for each identified object based on the type of each identified object. In this way, the processor 230 defines the specific location PX of each identified object as one of the constituent elements in the semantic model information 260.

[0129] In step S220, the processor 230 defines a specific space SX as one of the constituent elements (targets) in the semantic model information 260. The specific space SX is the space between a specific location PX and the identified object. Therefore, the processor 230 can automatically calculate the specific space SX based on the specific location PX and the identified object in the semantic model information 260. Then, the processor 230 appends the calculated specific space SX as one of the constituent elements in the semantic model information 260. Attribute information associated with the specific space SX includes the extent (location and shape) of the specific space SX. The attribute information associated with the specific space SX may also include metadata indicating that "the object existing in the specific space SX is the object to be removed."

[0130] In step S240, processor 230 generates 3D map data 100 based on semantic model information 260 defining a specific space SX. More specifically, processor 230 generates specific spatial information 110 representing the specific space SX based on the semantic model information 260 defining the specific space SX. Specific spatial information 110 represents the location of the specific space SX within a defined area AR. Furthermore, specific spatial information 110 may also include metadata indicating that "objects existing in the specific space SX are removed objects." Processor 230 may also append the generated specific spatial information 110 to the existing 3D map data 100.

[0131] In this way, 3D map data 100 is generated, including specific spatial information 110 representing a specific space SX. By utilizing semantic model information 260 to define a specific space SX, 3D map data 100 including specific spatial information 110 can be generated efficiently.

[0132] 2-2-2. Second example

[0133] Figure 10 This is a flowchart illustrating a second example of the map generation process in this embodiment. (Appropriate omissions and...) Figure 9 The first example shown is a repeat of the previous one. Steps S210 and S220 are the same as in the first example.

[0134] In step S230, the processor 230 defines a specific surrounding space SY as one of the constituent elements (targets) in the semantic model information 260. The specific surrounding space SY is the finite space surrounding the specific space SX (refer to...). Figure 6 For example, a specific surrounding space SY is the space within a certain distance from the outer surface of a specific space SX. The processor 230 can automatically calculate the specific surrounding space SY based on the specific space SX in the semantic model information 260. Alternatively, the user can specify the specific surrounding space SY using the user interface 210. The processor 230 then appends the specific surrounding space SY as one of the constituent elements in the semantic model information 260. Attribute information associated with the specific surrounding space SY includes the extent (location and shape) of the specific surrounding space SY. The attribute information associated with the specific surrounding space SY may also include metadata indicating that "objects existing in this specific surrounding space SY are candidates for removal."

[0135] In step S240A, processor 230 generates 3D map data 100 including specific spatial information 110 and specific surrounding spatial information 120. Regarding the specific spatial information 110, it is the same as in the first example described above. Processor 230 generates specific surrounding spatial information 120 representing the specific surrounding space SY based on semantic model information 260 that defines the specific surrounding space SY. Specific surrounding spatial information 120 represents the location of the specific surrounding space SY within the defined area AR. Furthermore, specific surrounding spatial information 120 may also include metadata indicating that "objects existing in the specific surrounding space SY are candidate objects to be removed." Processor 230 may also append the generated specific surrounding spatial information 120 to the existing 3D map data 100.

[0136] In this way, 3D map data 100 including specific spatial information 110 and specific surrounding spatial information 120 is generated. By utilizing semantic model information 260 to define specific space SX and specific surrounding space SY, 3D map data 100 including specific spatial information 110 and specific surrounding spatial information 120 can be generated efficiently.

[0137] 2-3. Map Update Processing

[0138] Sometimes the configuration of the identified object (second object 20) in the designated area AR is changed. For example, a new traffic light is installed. As another example, the location of an existing traffic light is changed. When the configuration of the identified object in the designated area AR is changed, the processor 230 performs map update processing to update the 3D map data 100. Specifically, the processor 230 updates the definitions of the identified object, specific location PX, specific space SX, and specific surrounding space SY in the semantic model information 260 based on the changed configuration of the identified object. Then, the 3D map data 100 is updated based on the updated semantic model information 260.

[0139] In this way, the 3D map data 100 can be easily updated by utilizing the semantic model information 260.

[0140] 2-4. Sensor Field of View Inspection and Processing

[0141] Figure 11An example of the field of view (FOV) of the recognition sensor 15 located at a specific position PX and the specific space SX is shown. When at least a portion of the specific space SX is detached from the FOV, the object to be recognized (the second object 20) cannot be adequately identified, resulting in decreased recognition accuracy. If such a field-of-view deficiency can be sensed in advance, the placement of the object to be recognized or the design of the recognition sensor 15 can be modified. That is, the design of the AR within the specified area can be improved. Therefore, it is useful to compare the FOV of the recognition sensor 15 located at the specific position PX with the specific space SX. This process will be referred to as "sensor field-of-view verification processing".

[0142] Figure 12 This is a flowchart illustrating the processes associated with the sensor field-of-view inspection process. Steps S210 and S220 are as described above.

[0143] In step S250, the processor 230 defines the field of view (FOV) of the recognition sensor 15 located at a specific location PX as one of the components (targets) in the semantic model information 260. For example, a user uses the user interface 210 to input the setting and performance information of the recognition sensor 15. Based on the specific location PX, the setting and performance information of the recognition sensor 15 in the semantic model information 260, the processor 230 calculates the FOV of the recognition sensor 15 located at the specific location PX. Then, the processor 230 defines the calculated FOV as one of the components in the semantic model information 260. The attribute information related to the FOV includes the range (position and shape) of the FOV.

[0144] In step S260, processor 230 compares a specific space SX defined in semantic model information 260 with the field of view (FOV). If at least a portion of the specific space SX is detached from the FOV (step S260: Yes), the process proceeds to step S270.

[0145] In step S270, processor 230 appends a warning message indicating insufficient field of view to specific spatial information 110. Processor 230 may also notify the user (administrator) of the insufficient field of view warning message via user interface 210. The user (administrator) can change the location of the identified object or modify the design of the identification sensor 15. That is, the user (administrator) can improve the design of the AR within a specified area.

[0146] 3. Management System

[0147] 3-1. Example of composition

[0148] Figure 13This is a block diagram illustrating an example configuration of the management system 300 of this embodiment. The management system 300 includes a user interface 310, a communication device 320, one or more processors 330 (hereinafter referred to as processors 330 only), and one or more storage devices 340 (hereinafter referred to as storage devices 340 only).

[0149] User interface 310 receives information input from the user (administrator) and provides the user with various information. User interface 310 includes input devices and output devices. Examples of input devices include a keyboard, mouse, and touch panel. Examples of output devices include a display device, touch panel, and speaker. User interface 310 can also be a GUI (Graphical User Interface).

[0150] The communication device 320 communicates with the outside world via a communication network.

[0151] Processor 330 performs various processes. For example, processor 330 includes a CPU (Central Processing Unit). Storage device 340 stores various information. Examples of storage device 340 include volatile memory, non-volatile memory, HDD (Hard Disk Drive), SSD (Solid State Drive), etc. At least a portion of processor 330 and storage device 340 may also be included in the first object 10 (mobile body 1). That is, at least a portion of the functions of management system 300 may also be included in the first object 10 (mobile body 1).

[0152] Management program 350 is a computer program executed by processor 330. Processor 330 executes management program 350, thereby implementing the functions of management system 300. Management program 350 is stored in storage device 340. Alternatively, management program 350 may be recorded on a computer-readable recording medium. Management program 350 may also be provided via a network.

[0153] Management information 360 is information used for managing the designated area AR. For example, management information 360 includes a "removal history database" showing the correspondence between the locations of obstacles 30 within the designated area AR and the removal history of obstacles 30. As another example, management information 360 may also include a "removal plan database" showing the correspondence between the locations of obstacles 30 within the designated area AR and the removal plans for obstacles 30. Management information 360 is stored in storage device 340.

[0154] The recognition result information 370 represents information indicating the result of the recognition process obtained by the recognition sensor 15 located at a specific location PX. In particular, the recognition result information 370 represents the position of the object recognized by the recognition sensor 15 located at the specific location PX.

[0155] It should be noted that the position of an object can be either an absolute position in an absolute coordinate system or a relative position with respect to the recognition sensor 15. The relative position of the object with respect to the recognition sensor 15 can be calculated. For example, the object can be identified and its relative position calculated by analyzing the image IMG obtained from the camera. Furthermore, the object can be identified and its relative position obtained based on point group information obtained through LIDAR. On the other hand, the first object 10 (moving body 1) has a position determination function to determine its own position. For example, the first object 10 has a GPS (Global Positioning System) sensor, and its own position is determined using the GPS sensor. The first object 10 can also determine its own position with high precision through localization processing based on the recognition result and map information. The setting position of the recognition sensor 15 in the first object 10 is known information. Therefore, conversion between the relative position and absolute position of the object can be performed. In the following description, the relative position and absolute position of the object are treated as equivalent positions. The same applies to a specific space SX.

[0156] The 3D map data 100 is provided by the map generation system 200 described above. For example, the processor 330 acquires the 3D map data 100 from the map generation system 200 via the communication device 320. As another example, if the functionality of the map generation system 200 is included in the management system 300, the processor 330 generates the 3D map data 100. The obtained 3D map data 100 is stored in the storage device 340. Furthermore, the processor 330 reads the 3D map data 100 from the storage device 340 and performs the "management processing" described below.

[0157] 3-2. Management and Processing

[0158] 3-2-1. The first example

[0159] Figure 14 This is a flowchart illustrating the first example of the management process in this embodiment.

[0160] In step S310, the processor 330 acquires recognition result information 370, which represents the result of the recognition processing obtained by the recognition sensor 15 located at a specific position PX.

[0161] In step S320, processor 330 reads 3D map data 100 from storage device 340. 3D map data 100 includes specific spatial information 110 representing a specific space SX. Processor 330 compares 3D map data 100 with recognition result information 370 to determine whether objects not included in the specific space SX in 3D map data 100 are included in the specific space SX in recognition result information 370. If objects not included in the specific space SX in 3D map data 100 are included in the specific space SX in recognition result information 370 (step S320: Yes), the process proceeds to step S330. Otherwise (step S320: No), Figure 14 The processing shown is now complete.

[0162] In step S330, the processor 330 senses the object that is not included in a specific space SX in the 3D map data 100 but is included in the specific space SX in the recognition result information 370 as a removal object (obstacle 30).

[0163] In step S340, the processor 330 notifies the administrator of recommendation information via the user interface 310. The recommendation information suggests removing the object (obstacle 30) sensed in step S330. The recommendation information may also include an image IMG obtained from a camera. The image IMG is included in the recognition result information 370. The recommendation information may also indicate the object (obstacle 30) to be removed in the image IMG. The recommendation information may also emphasize the object (obstacle 30) to be removed in the image IMG. The administrator who sees the recommendation information can then consider immediately removing the obstacle 30.

[0164] In step S350, processor 330 can create a plan to immediately remove obstacle 30. Processor 330 updates management information 360 (removal plan database) based on the created plan. The manager removes obstacle 30 according to the plan shown in the removal plan database. When the removal of obstacle 30 is complete, processor 330 updates management information 360 (removal history database).

[0165] As explained above, obstacles 30 that hinder the identification process can be sensed by utilizing three-dimensional map data 100, including specific spatial information 110 representing a specific space SX. Specifically, the specific space SX is defined considering the positional relationship between the object to be identified and a specific location PX where the object should be identified. Therefore, obstacles 30 that actually hinder the identification process are sensed as objects to be removed. Conversely, objects existing outside the specific space SX that do not affect the identification process are not sensed as objects to be removed. In other words, objects existing outside the specific space SX are prohibited from being sensed as objects to be removed. Thus, false sensing of objects to be removed is suppressed.

[0166] Furthermore, obstacles 30 are sensed by comparing the actual recognition result information 370 obtained through the recognition sensor 15 with the 3D map data 100. Therefore, obstacles 30 that hinder the recognition process can be sensed at appropriate timing. This also helps to suppress false object detection. In addition, the generation of unpredictable obstacles 30 (e.g., flying objects) can be sensed almost in real time.

[0167] Thus, according to this embodiment, obstacles 30 (i.e., actual objects to be removed) that hinder the recognition process can be accurately sensed. As a result, a favorable environment for the recognition process can be efficiently ensured. That is, the environment of the designated AR area can be efficiently managed.

[0168] 3-2-2. Second example

[0169] Figure 15 This is a flowchart illustrating a second example of the management process in this embodiment.

[0170] In step S310, the processor 330 acquires recognition result information 370, which represents the result of the recognition processing obtained by the recognition sensor 15 located at a specific position PX.

[0171] In step S360, processor 330 reads 3D map data 100 from storage device 340. 3D map data 100 includes specific surrounding space information 120 representing a specific surrounding space SY. Processor 330 compares 3D map data 100 with recognition result information 370 to determine whether an object not included in the specific surrounding space SY in 3D map data 100 is included in the specific surrounding space SY in recognition result information 370. If an object not included in the specific surrounding space SY in 3D map data 100 is included in the specific surrounding space SY in recognition result information 370 (step S360: Yes), the process proceeds to step S370. Otherwise (step S360: No), Figure 15 The processing shown is now complete.

[0172] In step S370, the processor 330 senses an object that is not included in the specific surrounding space SY of the 3D map data 100 but is included in the specific surrounding space SY of the recognition result information 370 as a candidate for removal. The candidate for removal may become a removal object in the near future.

[0173] In step S380, the processor 330 notifies the administrator of attention information via the user interface 310. The attention information informs the administrator of the existence of candidate objects to be removed. The attention information may also include an image IMG obtained from a camera. The image IMG is included in the recognition result information 370. The attention information may also indicate candidate objects to be removed in the image IMG. The attention information may also emphasize candidate objects to be removed in the image IMG. The attention information may also suggest to the administrator the creation of a plan to remove candidate objects. The administrator who sees the attention information can review the plan to remove candidate objects.

[0174] In step S390, processor 330 can create a plan for removing candidate objects. For example, processor 330 divides the designated area AR into multiple zones and counts the number of candidate objects to be removed in each zone. Then, processor 330 creates a plan that prioritizes removing objects from zones with more candidate objects. As another example, processor 330 refers to management information 360 (removal history database) to know the timing of the last removal operation. Then, processor 330 can also create a plan that prioritizes removing objects from zones with longer elapsed time since the last removal operation. Processor 330 updates management information 360 (removal plan database) based on the created plan. The manager removes the obstacle 30 according to the plan shown in the removal plan database. When the removal of obstacle 30 is completed, processor 330 updates management information 360 (removal history database).

[0175] As explained above, by utilizing 3D map data 100 including specific surrounding space information 120 representing a specific surrounding space SY, candidate objects for removal can be detected with high accuracy. Furthermore, the removal operation can be planned efficiently by detecting candidate objects for removal. Thus, a favorable environment for recognition processing can be efficiently ensured. That is, the environment of the designated AR area can be managed efficiently.

[0176] 3-2-3. The Third Case

[0177] exist Figure 14 The first example shown is in Figure 15 The second example shown can also be combined.

[0178] 4. Providing information to mobile entities

[0179] Figure 16This is a block diagram illustrating the provision of information to the mobile body 1. The management system 300 provides notification information INF to the mobile body 1. For example, the notification information INF includes the location of an obstacle 30 that obstructs the identification process. A target route can be created so that the mobile body 1 avoids the obstacle 30 until the obstacle 30 is removed. Thus, situations where the mobile body 1 is unable to accurately identify the object and becomes trapped can be prevented.

Claims

1. A management system for managing the environment of a designated area, said management system comprising: One or more processors; and One or more storage devices are used to store three-dimensional map data of the designated area. Within the designated area, there are identification sensors that recognize the surrounding conditions and identification objects that should be recognized by the identification sensors located in specific positions. The three-dimensional map data includes specific spatial information representing a specific space between the specific location and the identified object. The one or more processors are configured to: Obtain identification result information representing the identification result obtained by the identification sensor present at the specific location; Determine whether an object in a specific space not included in the 3D map data is included in the specific space in the recognition result information; and Objects in a specific space that are not included in the 3D map data but are included in the recognition result information are sensed as objects to be removed. The three-dimensional map data also includes specific surrounding space information representing the specific surrounding space around the specific space. The one or more processors are further configured to: Determine whether an object in the specific surrounding space not included in the 3D map data is included in the specific surrounding space in the recognition result information; and Objects in the specific surrounding space that are not included in the 3D map data but are included in the recognition result information are sensed as candidate objects to be removed.

2. The management system according to claim 1, wherein, The one or more processors also prevent objects existing outside the specific space from being sensed as the objects to be removed.

3. The management system according to claim 1 or 2, wherein, The one or more processors will also recommend that the administrator notify the administrator of the recommended information for removing the sensed objects.

4. The management system according to claim 3, wherein, The identification sensor includes a camera. The recognition result information includes the image obtained through the camera. The recommendation information includes the image.

5. The management system according to claim 1, wherein, The one or more processors also create a plan to remove the sensed candidate objects or recommend the creation of the plan to the administrator.

6. The management system according to any one of claims 1, 2, 4, and 5, wherein, The one or more storage devices also store semantic model information representing attribute information for each constituent element constituting the defined region. The one or more processors are further configured to: The identified object is defined as one of the constituent elements in the semantic model information; The specific location is defined as one of the constituent elements in the semantic model information; Based on the identified object and the specific location, the specific space is defined as one of the constituent elements in the semantic model information; and Based on the semantic model information that defines the specific space, the three-dimensional map data including the specific spatial information is generated.

7. The management system according to any one of claims 1, 2, 4, and 5, wherein, The identification sensor is mounted on a moving body that moves within the specified area.

8. A map generation system for generating three-dimensional map data of a specified area, the map generation system comprising: One or more processors; and One or more storage devices store semantic model information representing attribute information for each constituent element constituting the defined region. Within the designated area, there are identification sensors that recognize the surrounding conditions and identification objects that should be recognized by the identification sensors located in specific positions. The one or more processors are configured to: The identified object is defined as one of the constituent elements in the semantic model information; The specific location is defined as one of the constituent elements in the semantic model information; Based on the identified object and the specific location, the specific space, which is the space between the specific location and the identified object, is defined as one of the constituent elements in the semantic model information; and Based on the semantic model information that defines the specific space, the three-dimensional map data including specific spatial information representing the specific space is generated.

9. The map generation system according to claim 8, wherein, If the configuration of the identified object in the specified area is changed, the one or more processors update the definition of the identified object, the specific location, and the specific space in the semantic model information, thereby updating the three-dimensional map data.

10. The map generation system according to claim 8 or 9, wherein, The one or more processors are further configured to: The field of view of the recognition sensor located at the specific location is defined as one of the constituent elements in the semantic model information; as well as If at least a portion of the specific space is outside the field of view of the identification sensor located at the specific location, a warning message indicating insufficient field of view will be added to the specific space information or the warning message will be notified to the administrator.

Images