Managing Mobile Data Collection Agents

Abstract

The present invention relates to a method and apparatus for implementing and using techniques for generating a map of physical features in a physical environment. Feature datasets are received from a plurality of mobile agents. The feature datasets include data describing detected physical features, the geographic location of the features, and a timestamp representing the time of detection. The geographic location data of the various feature datasets are integrated into a map of the physical environment and displayed to a user. One or more regions of interest are identified on the map. The regions of interest include geographic areas for which no feature datasets have been collected or for which an insufficient number of feature datasets have been collected. When a region of interest is identified, instructions are given to one or more mobile agents to collect a supplemental feature dataset for the region of interest. The map is updated with the supplemental feature dataset for the identified region of interest.

Description

[Technical field]

[0001] The present invention relates to detecting features or anomalies in public and private infrastructure, and more particularly to a cognitive system and associated methods for orchestrating multiple mobile data collection agents to inspect, monitor and determine features or anomalies in various areas of interest. [Background technology]

[0002] Maintaining surveillance of infrastructure, whether public or private, often requires a lot of time, resources and costs. Addressing the problem properly and early can save a lot of money for private actors and society as a whole. For example, in Sweden, vehicle damage due to poor road quality costs private car owners the equivalent of about US$100 million per year. In many cities, citizen reports are the only source of road damage data available. Once a citizen reports it, inspectors go to inspect the damage, but in urban environments, the location of the damage may not be easily found, as the location in the citizen report may be inaccurate or may not be reported at all. Once damage is found, a manual inspection is performed and the road surface is subjectively assessed. As the reports received by the city are based on a limited number of people, there is no overview of the overall condition of the city's roads, and it was therefore difficult for those responsible for the road network to decide which locations to prioritize for repairs.

[0003] The problems mentioned above are not limited to road damage. There is an increasing need to address the maintenance of traffic signs throughout Europe. For example, in Germany, 33% of traffic signs are considered difficult to read, and in France, 40-50% of traffic signs have exceeded their expected useful life. The detection of traffic sign damage also often relies on citizen involvement. As with road damage reports, cities often receive a limited number of reports on traffic signs. Furthermore, it is essentially impossible for inspectors with limited resources to cover all roads in a city. Thus, in the current situation, it is difficult to get an overview of the location and condition of all traffic signs.

[0004] Further issues are related to road construction. Typically, road works inspectors travel around the city every week to inspect road works that are being carried out. The road works inspectors ensure that road works sites are safe, that road works are on schedule, and that there is no impact on traffic around the roads. However, road works sites are sometimes not found. This is because the agreement with the contractor on the implementation date is loosely determined (e.g., within three months). However, in many cases, the actual work is carried out in a shorter period. Therefore, the city does not have a clear idea of ​​where road works are being carried out on a given day, and as a result, road works inspectors sometimes visit sites where road works are not being carried out at that time. These unnecessary travels cost the city thousands of wasted man-hours per year, resulting in unnecessary costs. Furthermore, the lack of inspections risks non-compliance, which leads to increased traffic congestion and reduced safety. Summary of the Invention [Problem to be solved by the invention]

[0005] The above problems are only a few, and in each case they show the difficulty of obtaining a larger-scale overview of the events occurring in a city during a single day, and of dealing with these problems in the best possible way. Attempts have been made to mitigate these problems, for example by using images recorded by static surveillance cameras or stored in online databases such as Google Street View. However, such data is often outdated (i.e. not representative of the current situation) and therefore has very limited use for decision-making, and also carries the risk of causing inaccurate actions that can lead to the loss of time, money, life, etc. Therefore, there is a need for better methods and systems to collect current data on features and anomalies and to process the collected data in a way that allows them to be used efficiently in the decision-making process on how to deal with the identified problems. [Means for solving the problem]

[0006] In general, in one aspect, various embodiments of the present invention provide methods and apparatus, including computer program products, for generating a map of physical features in a physical environment. Feature datasets are received from a plurality of mobile agents. The feature datasets include data describing detected physical features, geographic locations of the physical features, and timestamps representing the time of detection. The geographic location data of the various feature datasets are integrated into a map of the physical environment and displayed to a user. One or more regions of interest are identified on the map. The regions of interest include geographic areas for which no feature datasets have been collected or for which an insufficient number of feature datasets have been collected. When a region of interest is identified, instructions are given to one or more mobile agents to collect a supplemental feature dataset for the region of interest. The map is updated with the supplemental feature dataset for the identified region of interest.

[0007] Various embodiments of the invention may include one or more of the features described below. The steps of integrating, identifying, indicating, and updating the supplemental data features may be repeated until a termination condition is reached. The map represents an urban environment and includes one or more roads, and the physical features may include road conditions, signs associated with the roads, and pedestrian or vehicular traffic in or around the roads. The feature dataset may further include images of the features. The mobile agent includes a cell phone camera or a dash camera and a neural network trained to detect one or more categories of physical features in images captured by the cell phone camera or dash camera. The one or more mobile agents may further be provided with wireless communication devices for communicating the feature dataset, the mobile collection agent's location, metadata to a cloud service and for receiving dispatch instructions.

[0008] It may be determined whether two or more feature datasets received from two or more mobile agents belong to the same physical feature, and if it is determined that the two or more feature datasets belong to the same physical feature, the two or more feature datasets may be merged into a single feature dataset. A history of the physical feature may be displayed to a user, the history being generated from feature datasets that share the same geographic location but have different timestamps. The step of instructing the one or more mobile agents to collect the supplemental feature dataset may include modifying one or more collection parameters on the one or more mobile agents before initiating collection of the supplemental feature dataset.

[0009] In the step of directing one or more mobile agents to the selected region of interest, an incentive may be provided to the mobile agents for visiting the identified region of interest to collect the supplemental feature dataset. In the step of directing one or more mobile agents to collect the supplemental feature dataset, a number of mobile agents in a given region of interest may be optimized to avoid collection of duplicate supplemental feature datasets for the region of interest. The mobile agents may be terrestrial mobile agents. The mobile agents may be autonomous vehicles. Based on the information in the feature dataset, issues with the detected physical features may be identified and remedial actions may be provided to address the identified issues. The map may further include third party data not collected by the mobile agents.

[0010] In general, in another aspect, various embodiments of the invention provide a system for generating a map of physical features in a physical environment, the system including one or more mobile agents configured to detect physical features in the physical environment, a data store that stores data collected by the one or more mobile agents, a map generator that generates a map of the physical environment including the data collected by the one or more mobile agents, a computing device that displays the generated map to a user and receives instructions from the user, an orchestrator component that receives high level user instructions and translates the received instructions into lower level instructions for the one or more agents, an agent configuration data repository that stores configuration data and settings used by the one or more agents, a processor, and a memory.

[0011] The memory stores instructions which, when executed by the processor, cause the processor to perform the following operations: receiving, from a plurality of mobile agents configured to detect physical features, a feature dataset describing the detected physical features, a geographic location of the physical features, and a timestamp representing a time when the physical features were detected; Integrating the geographic location data of the various feature datasets into a map of the physical environment and displaying the physical map together with the integrated geographic location data to a user; Identifying one or more regions of interest on the map, including geographic areas for which no feature dataset has been collected or for which an insufficient number of feature datasets have been collected; if one or more regions of interest are identified, instructing one or more mobile agents to collect a supplemental feature dataset for the identified regions of interest; Updating the map with a complementary feature dataset for the identified region of interest. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates a schematic diagram of a system for generating a map of physical features in a physical environment, according to one embodiment. [Diagram 2] 1 illustrates a flowchart of a method for generating a map of physical features in a physical environment, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The details of one or more embodiments of the invention are set forth in the accompanying drawings and the following description. Additional features and advantages of the invention will be set forth in the detailed description, drawings, and claims.

[0014] In the drawings, identical elements are given common reference symbols.

[0015] [overview] Various embodiments of the present invention relate to techniques for generating a map of physical features within a physical environment, such as a city or neighborhood, that provides current information and a high-level overview of various issues occurring within the physical environment. The map is used by users responsible for the physical environment as a tool to initiate corrective action for detected issues. For example, a map of a city may show where road damage such as potholes have been found, where damaged road signs are located, or where ongoing road work may cause traffic congestion. The locations of these issues may be accompanied on the map by an indicator of the "severity" of the issue, which may help users prioritize and allocate resources to address the issues.

[0016] In particular, multiple mobile agents (hereafter simply referred to as "agents") work together to collect data of the physical environment. The collected data is integrated (possibly with additional third-party data) into a map that is displayed to the user. Based on the information contained in the map, the user can decide the best course of action for the detected problem. To make the data collection by the agents efficient, an orchestrator component is used. When the orchestrator component (hereafter simply referred to as "orchestrator") receives a "high-level goal" from the user, such as "monitor damage to urban roads at least once a week," it interprets this high-level goal and creates low-level data collection tasks for the agents. For example, the orchestrator creates tasks and routes to enable individual agents to collect data according to lower-level objectives. The lower-level data collection tasks may be configured to provide routes and rewards to the agents, to cover a certain geographic area within a certain time, or to adapt sensing techniques in a certain agent to collect a certain type of data. All of these operations are highly automated using a variety of techniques, including 3D Positioning, 3D Mapping, 3D Localization, Spatial Deep Learning, Visual-inertial SLAM (Simultaneous Localization and Mapping), LIDAR (Light Detection And Ranging) SLAM, Visual Odometry, Object Detection, and Tracking. More details on these are provided below.

[0017] The above-described configuration of data collection and visualization to the user allows the user or the person in charge of the physical environment to better and more quickly determine how to address the problems found, thus providing a beneficial effect on the problems mentioned in the background of this specification.

[0018] [System Architecture] Figure 1 illustrates a schematic diagram of a system 100 for detecting features and anomalies in a physical environment, according to one implementation. As shown in Figure 1, the system 100 includes a number of agents 102 for collecting data of the physical environment, a data store 104 for storing data collected by the agents 102, an optional external data store 106 for storing external data, a map generator 108, a computing device 110, an orchestrator component 112, and an agent configuration data repository 114. All these components communicate over a network 116, which may be a combination of wired or wireless communication networks, using conventional communication protocols known to those skilled in the art.

[0019] The agent 102 is a vehicle equipped with imaging and motion sensors that move on the ground or in the air and record images and location data. The agent 102 may be a manned or unmanned vehicle, such as, for example, a car, an airplane, a bicycle, a drone, etc. The imaging sensor may be permanently or temporarily attached to the agent and may be various types of cameras, such as, for example, a cell phone camera or a camera located in an Internet of Things (loT) device. The location information of the agent 102 may be determined, for example, by sensors associated with a global navigation satellite system permanently or temporarily on board the agent 102 and may include inputs from various types of cameras, such as, for example, a cell phone camera or a location sensor in an Internet of Things (IoT) device.

[0020] In the illustrated embodiment, the agent 102 includes a computing device or processor and a data link, such as a 4G or 5G cellular connection, which allows the agent 102 to pre-process collected image and location data via the data link before transmitting the data to the data store 104. Pre-processing can significantly reduce the amount of data transmitted from the agent 102 to the data store 104, making the transmission more efficient and reducing the amount of data held in the data store 104. The agent 102 can also communicate with an agent configuration data repository 114 to obtain specific settings and software, such as image processing and neural network software, that are tailored to the specific type of data that the agent is expected to collect and pre-process. In general, the agent 102 pre-processes the captured images according to settings obtained from the agent configuration data repository 114 to extract "useful" information, refine the information, and transmit it to the data store 104 along with location information, as will be described in more detail below with reference to FIG. 2.

[0021] The data store 104 stores the data recorded and pre-processed by the agent 102. The data store 104 is implemented on a cloud platform, such as a private cloud platform, a public cloud platform, or a hybrid platform of these. In addition to storing the data in the data store 104, the cloud platform may also further process the data before it is stored, such as anonymizing, refining the data regarding positioning, clustering (i.e., determining if what is detected is actually new or has been detected previously), comparing newly received data with existing data for an area to update inventory or identify discrepancies between the data in the data store 104 and the real world.

[0022] In some implementations, an external data store 106 is provided. The external data store 106 is configured to store third-party data that can be combined with data in the data store 104 to increase the generality of the data in the data store 104. For example, the external data store 106 stores information about weather, road regulations (e.g., speed limits) for various road segments, an inventory of roadside assets (e.g., traffic signs), temporary changes (e.g., road works), and maps of parking spaces. Similar to the data store 104, the external data store 106 is implemented on a cloud platform, such as a private cloud platform, a public cloud platform, or a hybrid platform thereof.

[0023] The map generator 108 retrieves data from the data store 104 and the optional external data store 106 and overlays the data onto a map of the physical environment, which may be a commercially available map. The map generator 108 may be implemented locally on a computing device or as a cloud application.

[0024] The computing device 110 is used to display the generated map to a user, typically an administrator with a particular responsibility for the physical environment of a city or part of it. The map is displayed on a display built into the computing device 110 (e.g., if the computing device 110 is a mobile phone, tablet, laptop, or similar device) or on a display external to the computing device 110 (e.g., one or more separate computer monitors). The computing device 110 further includes inputs such as a keyboard, mouse, touch screen, voice input, etc., for receiving instructions from the user, as well as an interface (e.g., wireless or electrical / mechanical connection such as a USB port or CD-ROM) for allowing import of program instructions for changing or modifying the map and for providing instructions to the orchestrator 112.

[0025] The map may be displayed on the display in a web-based user interface or on the computing device 110 in a standalone application implemented locally. Typically, the map user interface is configured to allow the user to control zooming in, zooming out, panning, and the like, as well as select which "layers" to display. For example, the user may select to display information such as congestion, road types, speed limits, road construction, and parking availability. As the user reviews the displayed information, the computing device 110 is used to provide instructions to the orchestrator 112 as high-level objectives. In some implementations, the user may also have access to historical data, so that the user can see, for example, how long road construction is ongoing, or how the total number of potholes in a particular area of ​​a city has changed over the past few months.

[0026] When the orchestrator 112 receives a high-level goal from the user, it breaks it down into smaller tasks. The tasks are provided to the agent 102 along with the necessary configuration settings from the agent configuration data repository 114 (or instructions on how the agent 102 itself can obtain the configuration settings). For example, a high-level goal could be "provide information on certain new road works started in certain areas of the city on a weekly basis," or "monitor potholes daily during winter, but only weekly during summer." The orchestrator 112 sets up a schedule that explicitly directs a particular agent 102 to go to a certain section of the city to collect certain data during a certain period of time. While this configuration can work well for an autonomous agent 102, such as a self-driving vehicle, it may not work as well when the agent 102 is operated by a human, e.g., a taxi or bus driver. In such a situation, it is more preferable to instead provide some incentive to the driver, as will be described in more detail below with reference to FIG. 2.

[0027] The agent configuration repository 114 stores data to be sent to or retrieved by the agent 102. In general, the configuration data, as described above, relates to what data the agent 102 should collect and how the data should be processed on the agent 102 before being sent to the data store 104. The data in the configuration repository 114 may be, for example, various convolutional neural networks or image processing software to be used by the agent 102 based on the target data to be collected, the data to be sent to the data store 104, the frequency of sending data to the data store 104, etc.

[0028] [Example: Road damage] The present invention will now be described in more detail with reference to Figure 2, which illustrates a flow chart of a method 200 for generating a map of physical features in a physical environment, according to one embodiment. Specifically, the example illustrated in Figure 2 relates to generating a map displaying an overall assessment of road damage in a section of a city. However, this example is not intended to be limiting, and similar principles can be applied to other features, such as traffic signs, road construction, parking availability, traffic congestion, road obstructions, etc. Thus, many variations on the techniques described herein will be readily available to those skilled in the art.

[0029] 2, the method 200 begins when an agent 102 receives a collected feature dataset (step 200). As described above, the agent 102 is a vehicle equipped with imaging and motion sensors that travel on the ground or in the air and record images and position data. Depending on the implementation, the agent 102 travels throughout the city independently, i.e., more or less randomly, or, as described above, typically under control under instructions from an orchestrator 112. The ultimate objective of the agent 102 is to fulfill higher level goals defined by a user.

[0030] The type of data collected by the agent 102 varies from implementation to implementation. In this example, the user is interested in creating an overview of the road damage situation in the city, so the agent 102 uses different types of sensors, alone or in combination, to determine the location of the road damage. Suitable sensors for use here include sensors with global navigation satellite systems (e.g., GPS, GEONASS, Baidu, etc.), imaging sensors, inertial sensors, etc. Triangulation with the information obtained by these sensors can provide the precise location of the features (e.g., road damage) detected by the agent 102. An imaging sensor is used to capture images of the road damage, and image analysis software, such as neural networks, is used to assess the condition of the pothole. In some implementations, all of these operations are performed on a single device, such as a mobile phone that can be temporarily attached to the dashboard of a car or the handlebars of a motorcycle or bicycle.

[0031] Typically, software used by the agent 102 is downloaded to the agent 102 from the agent configuration data repository 114 before the agent 102 is active. For example, when using a smartphone as the agent 102, the user installs an app (which may be obtained from a standard online app store) that contains a configuration file that dictates the settings and CNNs to use to detect features of interest. The agent 102 may retrieve these settings and CNNs from the agent data repository 114 before collecting data. In an implementation in which the agent 102 is a connected car, the app may be customized for the particular type of car. Allowing software to be downloaded to the agent 102 allows the agent 102 to be reconfigured for different purposes. For example, the agent 102 may be configured to detect potholes in the morning and road signs in the afternoon of the same day. This configuration makes the system 100 much more flexible and efficient than if each agent 102 were configured with a single, dedicated purpose. The app's user interface can have varying degrees of configurability, but it is generally desirable to keep it as simple as possible so that it can be used by a wide variety of people without requiring special technical skills. For example, in one embodiment, the user interface consists of basic on-boarding instructions for the phone, controls for starting and stopping data collection, a display of detected features using graphics on the phone's screen, and controls for obtaining an overview of features detected during a data collection session.

[0032] Since the size of the image data can be quite large, software on the agent 102 is required to reduce the amount of data and create fewer feature datasets to send to the data store 104 rather than sending many images. There are multiple ways to configure this, each depending on the specific situation. For example, when collecting data on road damage, the agent captures location information and images as it travels along the roads of a city. In one implementation, the roads are divided into smaller sections, referred to herein as "bins." A "bin" is defined with respect to a global reference frame, such as WGS-84, or a local orthogonal projection, such as OSGB36, both techniques well known to those skilled in the art. Typically, a bin corresponds to a section of road of about 5-7 meters. Images captured by the agent 102 while it crosses a bin are analyzed on the agent 102 by a CNN retrieved from the agent configuration data repository 114. Specific instances of road damage are detected (e.g., potholes, cracks, patches, etc.) and the characteristics of the damage are evaluated (e.g., size, length, location context, etc.). The analysis is output using a "score" representing the quality of the road section within the bin, for example a score ranging from 0 to 5, with "0" representing a newly repaired road quality and "5" representing severe damage. The bins are assigned an identifier that, together with the score, bin location coordinates, and optionally one verification image, form a feature dataset and are transmitted to the data store 104 using a wireless link or other type of connection between the agent 102 and the data store 104, as described above.

[0033] As mentioned above, in some embodiments, the data store 104 is implemented on a cloud computing platform. The cloud computing platform may be configured to perform further processing on the feature set before storing it in the data store 104. For example, the feature dataset may be anonymized, overlapping feature datasets collected by other agents 102 may be identified and merged, the location (or the bin in which the feature is located) may be refined, previous detections may be compared to determine the state of change of the feature, multiple detections in a local area may be combined (e.g., averaged), etc. These are all different types of processing depending on the situation and are within the ability of one of ordinary skill in the art to implement.

[0034] Once the step of receiving the feature dataset is completed, the method proceeds to step 204 of creating and displaying a map to the user. The map can be created by utilizing a conventional geographical map in an electronic format, similar to the maps used by various map service providers such as Google Maps, Apple Maps, etc., storing the collected feature dataset in a georeferenced data type format (e.g., GeoJson open standard format), and adding the collected feature dataset to the map. In some implementations, various types of third-party data may be added to the map, such as information about weather, road regulations (e.g., speed limits) on various road segments, an inventory of assets (e.g., traffic signs) installed on the roadside, temporary changes (e.g., road works), a map of parking spaces, etc.

[0035] The created map is displayed to the user on a user interface of the computing device 110. In the illustrated embodiment, the user interface is a web-based user interface that allows the user to select the type of information to include in the map. For example, information about road damage, road signs, speed limits, etc., in any combination may be displayed to allow the agent 102 to draw relevant conclusions from the collected data.

[0036] Upon reviewing the map, the user (or, in some implementations, the computing device 110 itself) identifies areas of interest for which the agents 102 should be dispatched to collect supplemental data sets in step 206. For example, the user or computing device 110 may determine that certain areas of the map do not contain sufficient information to assess road conditions and determine repair strategies for damages such as potholes. For example, in a city where the agents 102 move more or less arbitrarily, a large amount of data may be collected on busy roads, but little data may be collected on side streets with less traffic. The user may use the computing device 110 to instruct the orchestrator 112 to collect supplemental data sets for these areas of interest.

[0037] Instructions to the orchestrator 112 may be, for example, "Collect additional data on road damage on Green Street by 3:00 PM," or more specific, such as "Update road damage data for at least 80% of major roads in Sector A of the city every two weeks." Upon receiving these instructions, the orchestrator 112 evaluates which agents 102 and configuration options are available and instructs the agents 102 to collect the supplemental data required.

[0038] As an example, consider a scenario where a city manager requires data from each road in the city at a certain frequency within a certain time period. For example, a maintenance manager may want to inspect each road in the city for potholes or other damage once a month. This is a goal created based on the manager's experience and corresponds to their knowledge of road damage progressing over time. The goal (what to detect) may vary as well as the frequency (once a day, twice a month). The goal may also vary by road type (once a day for major roads, once a week for smaller roads), location (once a day for urban roads, once a week for rural roads), and the purpose at the time (the city manager needs the data to receive and review reports of specific events).

[0039] The orchestrator 112 can observe both when each road in the city was last surveyed and what it has been surveyed by the agent 102. To achieve the goals set by the city manager, the orchestrator 112 generates routes (or generates rewards that influence the agent's route) and configures the agent's data collection device (e.g., a smartphone) based on location and time so that the correct data is collected as the agent 102 travels each road.

[0040] As the time since the last survey increases and the likelihood that the objectives set by the city manager will not be met, more emphasis is placed on resurveying a particular road. For example, in some embodiments, the increase in priority (e.g., via some reward system) may be linear with respect to time. Alternatively, the increase in priority may be non-linear, e.g., the agent 102 may reset the "clock" for that road by randomly traveling down the street and resurveying it. As the time since the last survey approaches a deadline set by the city manager, the need to "influence" the agent 102 to resurvey that particular road increases, as the opportunity for the agent 102 to randomly travel down the street decreases. Thus, the level of reward provided to the agent 102 effectively increases if no further surveys are performed in time, until the time since the last survey reaches a point in time where the high-level objectives set by the city manager are not met.

[0041] In this manner, the orchestrator 112 instructs the agent 102 to collect data according to low-level instructions that the agent 102 can interpret. Commands can be issued, while those commands are crafted so that the actions of individual agents generate data that achieves high-level goals set by the city manager.

[0042] As mentioned above, in an implementation where the agent 102 is an autonomous vehicle, the orchestrator 112 can create low-level instructions to re-dispatch the same agent 102 to collect supplemental data sets, possibly with modified configuration settings, or simply dispatch other agents 102 to collect supplemental data sets, if necessary. In other implementations, where the orchestrator 112 has less direct control over the agent 102, the orchestrator 112 can provide incentives to the agent. For example, if 80% of the roads in a particular area are covered, a taxi driver can be rewarded for preferentially driving on the remaining 20% ​​of roads. For example, a taxi driver can receive a notification (e.g., a list or map of roads) on his mobile phone informing him that if he passes one or more of these locations in the next minutes / hours / days, he will be eligible for a discount on the price of gas at a particular gas station. It is well known to those skilled in the art that there are a great many ways to incentivize people to perform certain tasks, and it is not possible to provide an exhaustive list of these methods here. Suffice it to say here that there are several ways to incentivize agents 102 to collect supplemental data sets.

[0043] Finally, in step 208, the map is updated with the supplemental data set, which concludes method 200. Updating the map can be done in a similar manner as described above for the initial data set in step 204. If the user determines that there is still not enough information available to determine how to address the road damage, steps 206 and 208 may be repeated until a sufficient number of supplemental data sets have been collected such that the user is satisfied with the information contained in the map. In this manner, the user can initiate actions to address the road damage issue. What these actions are and how they are carried out is beyond the scope of this invention.

[0044] Some embodiments may include a feedback loop whereby upon indication of a particular event, e.g., a pothole being repaired by a repair crew, the orchestrator 112 motivates the agent 102 to go to the location to ensure that the road is in a satisfactory condition, so that the task of repairing the road damage is automatically updated to a "completed" status.

[0045] Although the above exemplary embodiment has focused mainly on problems related to road conditions, similar techniques can be applied to other situations. For example, in cities, the management of on-street parking is often problematic. Occupancy rates are often higher than agreed targets. Existing methods for monitoring available spaces and associated occupancy rates are time-consuming, costly, inefficient, and may only be performed at certain times of the year. The data obtained quickly becomes outdated and only covers limited spaces. Thus, it is difficult to provide adequate levels of on-street parking, and for drivers, finding parking in cities is often difficult and stressful. Furthermore, a significant part of emissions from the transportation sector is due to the behavior of searching for a parking space. The lack of data on the current location of specially designated parking spaces (e.g., parking spaces reserved for loading and unloading luggage, or parking spaces for disabled people) can also lead to particular problems.

[0046] Thus, the technology described herein can be used to improve city management by mapping the recent availability of on-street parking spaces. Recent changes such as obstructions, temporary parking, road works, etc. can be more easily detected. This data can also be shared with drivers to guide them to the nearest available parking space, reducing the number of drivers searching for parking spaces. This can also have a beneficial impact on congestion and emissions, and generally contribute to a safer urban environment.

[0047] Although the above principles have been illustrated using an urban environment as an example, they can also be applied in various civilian environments. For example, in a large warehouse, forklifts can be equipped with cameras and act as mobile agents detecting people (safety hazards), damage, pallets, recording accidents, etc. These recorded features and events can be provided to the warehouse manager as a "warehouse map" that gives him an overview of what is happening in the warehouse at any given time and allows him to take appropriate actions to address the issues.

[0048] Another environment where the above principles can also be applied is an airport tarmac, where the consequences of an undetected event or danger are much more severe and need to be addressed much more quickly than in an urban environment. Various vehicles traversing the tarmac are equipped with cameras and act as agents, with the results displayed on a map and presented to a manager, who then makes a decision on the actions required to address the identified concerns. It will be appreciated by those skilled in the art that the two application cases mentioned above are merely illustrative and that there are many other areas where the above techniques can be applied, such as railway and subway stations.

[0049] In some implementations, the orchestration techniques described above can also be used on a smaller scale, such as a city performing periodic inspections of its water or sewer systems, or a building manager performing inspections of the air ducts of a large office building, etc. Agents 102 can be appropriately configured for these specialized environments, and the principles described above can be applied to how agents are traversed.

[0050] Additionally, while the embodiments described above use image sensors such as cameras, other types of sensors may be used, such as optical sensors, infrared imaging sensors, ultraviolet imaging sensors, light detection and ranging (LIDAR) sensors, synthetic aperture radar (SAR) sensors, electromagnetic (EM) and acoustic sensors, or other sensors capable of generating one or more images of physical features in the physical environment.

[0051] The present invention may be a system, method, or computer program product at any possible level of technical detail integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions for causing a processor to carry out aspects of the present invention.

[0052] A computer-readable storage medium may be a tangible device that holds and stores instructions for use by an instruction-executing device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, punch cards or mechanically encoded devices such as ridge structures in grooves with instructions recorded thereon, and suitable combinations thereof. As used herein, a computer-readable storage medium is not to be construed as being a transitory signal itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a wave guide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electric signal transmitted through a wire.

[0053] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to each computing and processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network. The network may be comprised of copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, edge servers. A network adapter card or network interface in each computing and processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in each computing and processing device.

[0054] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can utilize state information in computer readable program instructions to execute a computer readable program and individually optimize the electronic circuitry to carry out aspects of the invention.

[0055] Aspects of the present invention are described herein with reference to flowchart illustrations and block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. Each block of the flowchart illustrations and block diagrams, and combinations of blocks in the flowchart illustrations and block diagrams, are implemented by computer readable program instructions.

[0056] These computer readable program instructions can be provided to a processor of a computer or other programmable data processing apparatus to produce a machine such that the instructions, executed via the processor of the computer or other programmable data processing apparatus, create means for performing the functions or operations specified in the block or blocks of the flowcharts or block diagrams. These computer readable program instructions can also be stored on a computer readable storage medium that can instruct a computer, programmable data processing apparatus, or other device to function in a particular manner, such that the computer readable storage medium having instructions stored therein constitutes an article of manufacture including instructions for performing aspects of the functions or operations specified in the block or blocks of the flowcharts or block diagrams.

[0057] The computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable apparatus, or other device to execute a series of operational steps to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the function or operation specified in the flowchart or block diagram block or block.

[0058] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation that may be implemented for systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of instructions, comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be accomplished as a single step, or may be executed simultaneously, substantially simultaneously, partially or fully overlapping in time, or the blocks may be executed in reverse order depending on the functionality involved. Each block of the block diagrams or flowchart diagrams, as well as combinations of blocks in the block diagrams or flowchart diagrams, may be implemented by a system on dedicated hardware that performs the specified functions or acts, or a combination of dedicated hardware and computer instructions.

Claims

1. 1. A method for generating a map of physical features in a physical environment, comprising: receiving a feature dataset from a plurality of mobile agents configured to detect physical features, the feature dataset describing detected physical features, geographic locations of the physical features, and timestamps representing the times at which the physical features were detected; integrating geographic location data from the various feature datasets into the map and displaying the map together with the integrated geographic location data to a user; identifying one or more regions of interest on the map, including geographic areas where no feature datasets have been collected or where an insufficient number of feature datasets have been collected; If the one or more regions of interest are identified, instructing one or more of the mobile agents to collect a supplemental feature dataset for the identified regions of interest; updating the map with the supplemental feature data set for the identified region of interest; A method comprising:

2. repeating the steps of integrating, identifying, indicating, and updating the supplemental feature dataset until a termination condition is reached; The method of claim 1 further comprising:

3. the map represents an urban environment and includes one or more roads; the physical features include one or more of the following: the condition of the road, signs relating to the road, and pedestrian or vehicular traffic in or around the road; 2. The method of claim 1 .

4. the feature dataset further includes images of the physical features; 2. The method of claim 1 .

5. the one or more mobile agents include a cell phone camera or a dash camera and a neural network trained to detect one or more categories of physical features in images captured by the cell phone camera or the dash camera; 2. The method of claim 1 .

6. the one or more mobile agents are further provided with wireless communication devices for communicating one or more of the feature dataset, the location of the mobile agents, and metadata to a cloud service and for receiving dispatch instructions; 2. The method of claim 1 .

7. determining whether two or more of said feature data sets received from two or more of said mobile agents belong to the same physical feature; if it is determined that the two or more feature data sets belong to the same physical feature, merging the two or more feature data sets into a single feature data set; The method of claim 1 further comprising:

8. displaying to a user a history of physical features generated from feature datasets that share the same geographic location but have different timestamps; The method of claim 1 further comprising:

9. Instructing the one or more mobile agents to collect the supplemental feature data set comprises: modifying one or more collection parameters on the one or more mobile agents before initiating collection of the supplemental feature data set; The method of claim 1 ,

10. In the step of directing one or more of the mobile agents to the selected region of interest, providing incentives to the mobile agents for visiting the identified areas of interest and collecting the supplemental feature dataset; The method of claim 1 ,

11. Instructing the one or more mobile agents to collect the supplemental feature data set comprises: optimizing the number of mobile agents within a given region of interest to avoid collecting duplicate supplemental feature data sets for said region of interest; The method of claim 1 ,

12. the mobile agent is a terrestrial mobile agent; The method of claim 1 ,

13. the mobile agent is an autonomous vehicle; The method of claim 1 ,

14. identifying a problem with the detected physical feature based on information in the feature dataset; Providing remedial actions to address the identified problems; The method of claim 1 further comprising:

15. the map further includes third party data not collected by the mobile agent; The method of claim 1 ,

16. 1. A computer program product for generating a map of physical features in a physical environment, comprising: the computer program product comprising a computer-readable storage medium having program instructions embodied therein; The program instructions, when executed by a processor, cause the processor to: receiving, from a plurality of mobile agents configured to detect physical features, a feature dataset describing detected physical features, geographic locations of said physical features, and timestamps representing the times when said physical features were detected; integrating geographic location data from the various feature data sets into the map and displaying the map together with the integrated geographic location data to a user; identifying one or more areas of interest on the map, including geographic areas where no feature datasets have been collected or where an insufficient number of feature datasets have been collected; If the one or more regions of interest are identified, instructing one or more of the mobile agents to collect a supplemental feature dataset for the identified regions of interest; updating the map with the supplemental feature data set for the identified region of interest; A computer program product characterized by causing a computer to perform the following.

17. 1. A system for generating a map of physical features in a physical environment, comprising: one or more mobile agents configured to detect physical features in a physical environment; a data store for storing data collected by said one or more mobile agents; a map generator for generating the map including data collected by the one or more mobile agents; a computing device that displays the generated map to a user and receives instructions from the user; an orchestrator component that receives high-level user instructions and translates the received instructions into lower-level instructions for the one or more agents; an agent configuration data repository that stores configuration data and settings used by said one or more agents; a processor; Memory and Equipped with The memory stores instructions that, when executed by the processor, cause the processor to: receiving, from a plurality of mobile agents configured to detect physical features, a feature dataset describing detected physical features, geographic locations of said physical features, and timestamps representing the times when said physical features were detected; integrating geographic location data from the various feature data sets into the map and displaying the map together with the integrated geographic location data to a user; identifying one or more areas of interest on the map, including geographic areas where no feature datasets have been collected or where an insufficient number of feature datasets have been collected; If the one or more regions of interest are identified, instructing one or more of the mobile agents to collect a supplemental feature dataset for the identified regions of interest; updating the map with the supplemental feature data set for the identified region of interest; A system characterized by causing a user to:

Images