System and method for generating image map

Abstract

SOLUTION: A method can include examining image deviation data of an image tile of a set forming a set view of a volume-of-space and determining whether or not a tile is in revision time for renewing the tile. The present method can include making a schedule for a first vehicle (102) to capture a renewed image tile and / or capturing a renewed image tile by identifying a second vehicle (102). A system (150) can include one or a plurality of processors (132) in order to examine image deviation data of an image tile of a set forming a set view of a volume-of-space and determine whether or not one or a plurality of image tiles of the set is in revision time for renewing the view. The one or a plurality of processors (132) are to make a schedule for the first vehicle (102) to move in or nearby a volume-of-space and capture one or a plurality of renewed image tiles or to capture one or a plurality of renewed image tiles by identifying the second vehicle (102).SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the priority of U.S. Provisional Application No. 63 / 330,204, filed on April 12, 2022, the entire disclosure of which is incorporated herein by reference.

[0002] The disclosed subject matter described herein relates to systems and methods for generating and refreshing three - dimensional (3D) map tiles of a transportation network and assets within the transportation network.

Background Art

[0003] Transportation networks, such as rail freight networks, are increasingly utilizing cameras and other vision systems. Cameras and vision systems are mounted on moving assets such as trains and can collect information regarding encounters with both other moving objects such as infrastructure and other assets, or automobiles and people within the transportation network. New technologies such as virtual reality enable exploration of physical areas that have been previously observed and mapped, even when active cameras are not available within that area. Tools such as virtual reality can be used to train vehicle operators and can serve as a substitute for decision - making, investigation, or auditing. However, virtual reality requires a map to be created based on recent visual information. It is not always possible to have an active camera within the area where a remote operator needs visibility.

[0004] There may be a desire to have systems and methods that are different from currently available systems and methods.

Summary of the Invention

[0005] In one embodiment or example, the method may include examining image deviation data associated with one or more image tiles of a set of image tiles used to form a larger set view of a volume of space, and determining, based on the image deviation data being examined, whether it is time for revision of one or more image tiles of the set in order to update the larger set view of the volume of space. The method may include either or both of the following: (a) a first vehicle, accompanied by one or more sensors, moves in or near the volume of space to capture one or more updated image tiles; or (b) in response to the determination that it is time for revision of one or more image tiles of the set, a second vehicle, accompanied by one or more sensors, is moving in, near, or toward the volume of space to capture one or more updated image tiles.

[0006] In one embodiment or example, the system may include one or more processors. One or more processors may examine image deviation data associated with one or more image tiles of a set of image tiles used to form a larger set view of a spatial volume. One or more processors may determine, based on the image deviation data being examined, whether it is time for revision of one or more image tiles of the set in order to update the larger set view of the spatial volume. One or more processors may (a) schedule a first vehicle moving in or near the spatial volume with one or more sensors to capture one or more updated image tiles, and / or (b) in response to determining that it is time for revision of one or more image tiles of the set, identify a second vehicle moving in, near, or toward the spatial volume with one or more sensors to capture one or more updated image tiles.

[0007] In one embodiment or example, the system may include a controller. The controller may examine image deviation data associated with one or more image tiles of a set of image tiles used to form a larger set view of the spatial volume. Based on the image deviation data being examined, the controller may determine whether it is time for revision of one or more image tiles of the set to update the larger set view of the spatial volume. The controller may instruct a vehicle moving within, beside, or toward the spatial volume using one or more onboard sensors to capture partial data of one or more updated image tiles in response to determining that it is time for revision of one or more image tiles of the set. The controller may examine the partial data to determine whether the partial data indicates that one or more updated image tiles have changed. The controller may update the timestamp of one or more updated image tiles in response to determining that one or more updated image tiles have not changed, or instruct one or more sensors to capture full additional data of one or more updated image tiles in response to determining that one or more updated image tiles have changed. [Brief explanation of the drawing]

[0008] The subject matter can be understood by referring to the attached drawings and reading the following description of non-limiting embodiments. [Figure 1] This is a diagram showing an example of a vehicle system. [Figure 2] This figure shows an example of a map generation system. [Figure 3] This figure schematically illustrates a method according to one embodiment. [Figure 4] This figure schematically illustrates a method according to one embodiment. [Figure 5] This figure schematically illustrates a method according to one embodiment. [Figure 6] This figure schematically illustrates a method according to one embodiment. [Figure 7] This figure schematically illustrates a method according to one embodiment. [Modes for carrying out the invention]

[0009] The embodiments of the subject matter described herein relate to the creation and / or updating of virtual reality maps of transportation networks and assets including vehicle systems within the transportation networks. A tile map of a transportation network may be generated from image data collected by vehicle systems operating within the transportation network. The tile map may be a three-dimensional (3D) or two-dimensional (2D) map formed from several images (2D or 3D images). The tile map may be formed and / or created from collected image data. The tile map may be used in a virtual reality environment, for example, to train train operators of vehicle systems operating within the transportation network and to identify sections of the transportation network (e.g., routes, tunnels, bridges, roadside equipment, switches, gates, optical signals, signs, etc.) that require repair, replacement, or more thorough inspection. The map and virtual reality environment may be used for making decisions regarding traffic routing within the transportation network, conducting investigations (e.g., for incidents including accidents involving vehicle systems), auditing the operation of vehicle systems within the transportation network, and so on.

[0010] Maps may be maintained and updated to provide the map and virtual reality environment with recent information representing the transport network. Visual or image tiles may be updated and timestamped to create a representation of the latest or more recent information from the transport network. For example, a virtual reality map of a route corridor may be created by a first vehicle system having one or more cameras that record and map the transport route or transport corridor as a 3D rendering with photographic tiles of each part of the corridor observed by the cameras. Each visual tile may be associated with a date represented by the image associated with the tile, and / or a timestamp indicating the date and / or time. The map may then be made available for a virtual reality tour. When another asset or vehicle system travels along the transport route or transport corridor in the same or opposite direction, or travels along a different route, the viewpoints of this other asset or vehicle may differ, and the other asset or vehicle may acquire new images or tiles, which may be used to fill gaps between images or tiles from a preceding vehicle or asset, update images or tiles from a preceding vehicle, etc. A second vehicle can also refresh any existing tile (e.g., by obtaining an updated image of an existing tile) and replace the existing tile with the updated image. In a later audit, the map of a transport route or transport corridor may be identified as outdated (e.g., the most recently acquired image, or the average age of the images forming the tile is older than a specified age) or may have certain sections or tiles that require refreshing. If one or more vehicle systems are scheduled to pass through a corridor, this vehicle may be instructed to collect images of a portion of the transport route or transport corridor. Otherwise, a vehicle system may be dispatched to the transport route or transport corridor to collect images. These images may be used to replace or update the map of the corridor.Over time, entire transportation networks, such as freight, transit, rail, road, mining, or highway networks, can be mapped using images acquired by several vehicles or vehicle systems. The maps can be made available as a service to software applications and can be used in cases where it is necessary to remotely observe scalable and complex infrastructure and network representations.

[0011] The frequency at which a transport network map may be updated or refreshed may depend on several factors, including the number of vehicle systems within the transport network available to capture or collect image data, the availability of memory for storing image data on the vehicle systems within the transport network, and / or the availability of the system's processing power to generate, maintain, and update the map. The time between image data updates may be determined to maintain the accuracy of the map so that this time represents the most recent state or condition of one or more transport routes or one or more transport corridors within the transport network. Captured or collected image data may also be analyzed to determine whether the captured or collected image data differs from the map image data to the extent that an update or refresh is necessary.

[0012] A prioritization model can be established to help ensure the accuracy of the virtual reality map without requiring the entire map to be refreshed in real time. This prioritization model can be used to determine which tiles should be updated before others. Prioritizing the map refreshing process may include ensuring that all tiles are refreshed within a specified period (e.g., monthly). When a tile is refreshed, it can be compared to a previous version of the same tile to determine the difference calculation. If the difference between the images falls below a threshold, the time before the same tile is refreshed or updated again may be increased. If the difference between the tiles exceeds a threshold, the time before the same tile is refreshed or updated again may be shortened to allow for more frequent tile updates.

[0013] Within the map, the prioritization model can also recognize specific objects. For example, a vehicle such as a railcar on a route may appear in the image and be remembered as a type of railcar, and as an instance of that railcar. A gondola may be recognized as a railcar type with an underlying 3D model. A particular gondola railcar being observed can have its own 3D record each time the vehicle is observed from a different angle. A moving camera may recognize an obstacle in the environment (e.g., a railcar) as a known object, without reproducing that part of the ground topology, and may erroneously depict the railcar's features as a change in infrastructure. For example, if a map was previously created using an image in which this railcar does not appear, a subsequently acquired image used to update the map may show the railcar. Railcars can be identified using a data model that tells the system how railcars appear in the image, so the system can distinguish between railcars and static, non-mobile infrastructure. The system can update the tile map with information from the new image, but not the railcar itself (to avoid the system identifying the railcar as a change in static infrastructure shown on the map). The railcar itself can be accurately represented in the image even if the camera has never observed the railcar within its current placement. For example, if the dispatch system indicates a particular railcar in a specific location, virtual reality can have a tile model of that particular railcar at its current location, including details such as damage and / or graffiti, and allow the user to virtually walk around the railcar and observe each side of it.

[0014] One or more embodiments are described in relation to rail vehicle systems, but not all embodiments relate to rail vehicle systems. Furthermore, the embodiments described herein cover a wide variety of vehicle systems. Suitable vehicle systems may include rail vehicles, automobiles, trucks (with or without trailers), buses, ships, aircraft, mining vehicles, agricultural vehicles, and off-highway vehicles. Suitable vehicle systems described herein may consist of a single vehicle. In other embodiments, a vehicle system may include multiple vehicles moving in coordination. With respect to multiple vehicle systems, the vehicles may be mechanically coupled to one another (e.g., by couplings), or the vehicles may be virtually or logically coupled but not mechanically coupled. For example, when separate vehicles communicate with each other to coordinate their movements so that the vehicles travel together (e.g., as a convoy, platoon, swarm, or fleet), the vehicles may be communicatively coupled but not mechanically coupled. Suitable vehicle systems may be rail vehicle systems that travel on tracks, or vehicle systems that travel on roads or streets.

[0015] Referring to Figure 1, an image data acquisition and communication system 100 (also called an image map generation system) may be located on a vehicle system 102. The vehicle system can travel along a route 104 during a trip from a departure point or origin to a destination or arrival point. The route may be a road (e.g., a multi-lane highway or other road), a track, a railroad, airspace, a waterway, etc. The vehicle system may include a propulsion-generating vehicle 108 and optionally one or more non-propulsion-generating vehicles 110 interconnected to travel together along the route. The vehicle system may include at least one propulsion-generating vehicle and optionally one or more non-propulsion-generating vehicles.

[0016] One or more propulsion-generating vehicles can generate a traction force to propel (e.g., pull or push) one or more non-propulsion-generating vehicles along a path. The propulsion-generating vehicles include a propulsion subsystem 118 for driving an axle 122 connected to a wheel 120. According to one embodiment, the propulsion system includes one or more traction motors that generate a traction force to propel the vehicle system. According to one embodiment, one of the propulsion-generating vehicles may be the lead vehicle in a multi-vehicle system, and the other vehicles may be the remote vehicles in the multi-vehicle system. The remote vehicles may be propulsion-generating vehicles or non-propulsion-generating vehicles.

[0017] Vehicles within a vehicle system may be mechanically coupled to one another. For example, a propulsion-generating vehicle may be mechanically coupled to a non-propulsion-generating vehicle by a coupler 123. Alternatively, vehicles within a vehicle system may not be mechanically coupled to one another, but may be logically coupled to one another. For example, vehicles may be logically coupled to one another by communicating with each other and coordinating their movements so that they travel together as a vehicle system in a convoy or group.

[0018] According to one embodiment, the vehicle system may be a railway vehicle system, and the route may be a track formed by one or more rails. The propulsion-generating vehicle may be a locomotive, and the non-propulsion-generating vehicle may be a railway vehicle carrying passengers and / or cargo. Alternatively, the propulsion-generating vehicle may be another type of railway vehicle other than a locomotive. According to another embodiment, the vehicle system may be one or more automobiles, ships, aircraft, mining vehicles, agricultural vehicles, or other off-highway vehicle (OHV) systems (e.g., vehicle systems that are not legally permitted and / or not designed for operation on public roads). Although some examples provided herein describe the route as a track, not all embodiments are limited to railway vehicles running on railway tracks. One or more embodiments may be used in connection with non-railway vehicles and non-railway routes such as roads, paths, waterways, etc.

[0019] The image data collection and communication system can include one or more visual sensors 112 that can capture or collect data when the vehicle system travels along a route. According to one embodiment, the visual sensor can be an imaging device. For example, the visual sensor can be a camera that can capture or collect still images, a video camera that can capture or collect video images, an infrared camera, a high-resolution camera, a radar, a sonar, or a lidar. The visual sensor can be arranged to obtain image data associated with the route. The image data can include an image of the route. The image data can include an image of the area surrounding the route. According to one embodiment, the vehicle system can be a railway vehicle, and the image data can include an image of the track on which the railway vehicle travels. According to one embodiment, the vehicle system can be a vehicle system traveling on a road, and the image data can include an image of the road. According to one embodiment, the vehicle system is an off-road vehicle system, and the image data can include an image of the off-road vehicle system route.

[0020] According to one embodiment, the image data collection and communication system can be fully arranged on one vehicle of the vehicle system, for example, on one propulsion vehicle. According to one embodiment, one or more components of the image data collection and communication system can be distributed among the vehicles of the vehicle system. For example, some components can be distributed among two or more propulsion vehicles that are connected to or form a group with each other.

[0021] According to one embodiment, at least some of the components of the map data collection and communication system can be arranged remotely from the vehicle system, such as at a dispatch location or a back office location. The remote components of the image data collection and communication system can communicate with the vehicle system and the components of the map data collection and communication system arranged on the vehicle system.

[0022] The image data can include an image of an area surrounding the travel route. For example, the image data can include a panoramic view (e.g., a 360° field of view) of the area surrounding the travel route. The image data can include an image of an area within a specific field of view angle of one or more visual sensors. The image data can include one or more images of terrain (e.g., hills, water areas, etc.), vegetation, buildings, traffic signals, and / or other vehicles on the travel route. The image data collection and communication system can include a communication system 126 that includes a vehicle communication assembly 128 and a remote communication assembly 130. The vehicle communication assembly can be mounted on a leading vehicle, e.g., a leading propulsion generating vehicle. The remote communication assembly can be located at a location far from the vehicle system, such as a dispatch location or a back office location. The vehicle communication assembly can communicate wirelessly with the remote communication assembly.

[0023] The vehicle system can have a controller 136 or a control unit that can be a hardware and / or software system that operates to perform one or more functions for the vehicle system. The controller receives information from components of the image data collection and communication system, such as one or more visual sensors, analyzes the received information, and generates a communication signal. The positioning system 106 can determine the position of the vehicle system along the route. According to one embodiment, the positioning system can be a global positioning system (GPS). The vehicle communication assembly can transmit the position of the vehicle system to the remote communication assembly.

[0024] Referring to Figure 2, the map generation system 150 may include a communication system, a vehicle communication assembly, a vehicle controller, one or more vision sensors, and a telecommunication assembly. The map generation system may include a memory 114, an input 116, and a display 124 mounted on the vehicle system. The map generation system may include one or more processors 132, a memory 134, an input 138, and a display 140 in a remote location 142, which includes a telecommunication assembly. The remote location may be, for example, a dispatch location or a back office location. The remote location may include or communicate with a cloud computing service. The memory mounted on the vehicle system may implement one or more processors in the remote location to carry out the methods disclosed herein, or include instructions that can be executed by a controller mounted on the vehicle system operating on this processor. The memory in the remote location may implement a controller mounted on the vehicle system to carry out the methods disclosed herein, or include instructions that can be executed by one or more processors operating on this controller.

[0025] The vehicle system's built-in memory can store data collected by one or more vision sensors. The vehicle's built-in memory can also store a route map generated before the start of the trip. The map may be stored in remote memory. The map may be made up of multiple image tiles. The image tiles may include image data collected by one or more vision sensors of the vehicle system and of other vehicle systems that have traveled along the route the vehicle system is currently traveling and have taken image data while traveling along the route. The image tiles may include image data collected by the vehicle system or other vehicle systems that have traveled along other routes that include points or areas shared by the route.

[0026] As a vehicle system operates within a transport network, the visual sensors mounted on the vehicle system capture or collect image data of the transport network as it moves along its route within the network. A set of image tiles is formed from the collected image data. Each image tile may be created using a combination or variety of image data from multiple visual sensors. For example, each image tile may be created from image data from cameras, video cameras, radar, LiDAR, sonar, infrared, and / or other visual sensors.

[0027] Multiple image tiles can be grouped together to form a set of image tiles. Multiple image tiles may be joined together to form a tile map. According to one embodiment, the tile map may be a 3D tile map. The region represented by the 3D tile map is a spatial volume within a transport network. A set includes all the images that are combined to form the tile map. A larger set view includes the 3D tile map. A larger set view may also include a 2D view formed from two or more images. For example, a larger set view of image tiles may include two or more images, video frames, or data output from one or more visual sensors that are joined together to form a larger set view of a spatial volume.

[0028] Image tiles may include metadata that includes information such as the date and time the image data for the image tile was captured or collected. The metadata may include location data indicating the location where the image data was captured or collected. Location data may be obtained, for example, from a location determination system. The metadata may include a consecutive count of image data collections for image tiles that have been confirmed to have unchanged image data. The metadata may include a time period count of collections that have been confirmed to have unchanged image data. For example, the metadata may include data indicating that the image data for an image tile has not changed for one year. Each tile may include metadata about the last time the image data collection deviated from the previous image data collection for the tile, as well as the number of collections and the time elapsed between the deviation and the most recent image data collection. The metadata can then be used to calculate the frequency of changes in a particular image tile, thereby determining when a particular image tile is scheduled for its next image data collection.

[0029] If no image tile is scheduled to be collected, a vehicle system passing through the area represented by the image tile may be assigned a list of tiles that may be the oldest tile since the last collection, or which may be known to change frequently based on previous collections and comparisons. When the vehicle system's vision sensor is capturing or collecting image data of the list of tiles, the vision sensor does not need to collect image data of the entire image tile. The vision sensor may sample a smaller portion of the image tile to confirm or deny that the image tile is the same as the one in a previous image data collection. If the smaller sampled portion is the same, the timestamp of the image file may be updated. If the smaller sampled portion is different, the vision sensor may immediately collect image data of the entire image tile for comparison.

[0030] If the image tiles formed from the collected image data differ significantly from expectations, the map can be revised, and the image tiles can be flagged for further collection until they are again considered stable and unchanged. Image acquisition or collection may be performed by other transit vehicle systems within the transport network. By collecting further image data to confirm that the image tiles are stable and unchanged, it is possible to prevent the image tiles from being permanently revised due to temporary conditions, such as the accumulation of leaves or snow along the route.

[0031] Metadata can include an optimal minimum sample size. Machine learning (ML) models can be used to optimize the minimum number of image tile variations required for a useful comparison of single tiles. For example, in an environment with minimal lighting and weather variations, a single tile may be used for a year for comparison purposes. In another example, in an environment with significant lighting and weather differences, two or three image tiles may be required for comparison purposes. The machine learning model can detect which of the image tile variations match before reporting the differences between the image tiles. The best tile matches may be logged or recorded with contextual metadata, such as time, season, and lighting measures. The machine learning model can be used to compare image tiles with priority based on the contextual metadata. The comparison may be stopped after the first match. The machine learning model can then determine the optimal minimum sample size again. The machine learning model can create and store as little variation as possible to enable a single match comparison throughout the year, while monitoring seasonal and time drift patterns.

[0032] Each vehicle system may have a priority level for collecting or acquiring image data. A vehicle system may have a low priority, which may include having no assigned image data collection or acquisition. The vehicle system's visual sensors may be used to perform rolling passive sampling inspection. The acquired or collected image data may not be a complete collection of image tiles, but may be a smaller sample of scattered image tiles.

[0033] A vehicle system may have a medium priority for image data acquisition or collection. The vehicle system may be assigned specific image tiles in a list for scheduled collection and / or verification as unchanged from the most recent image data. A visual sensor may perform partial or complete collection of image data for each image tile in the list. Specific tiles may be designated to the vehicle system as a work list. The list may differ from just-in-time collection but may depend on the current priority level. Priority levels may include checking or updating the oldest tiles, along with business and / or context rules that direct changes to the priority level in areas where rare but high-risk changes exist. For example, if the vehicle system is operating in an area known to experience washouts or mudflows, the vehicle system priority may be changed or updated. Similar to speed limits for the vehicle system, business and / or context rules may be permanent or temporary. For example, business and / or context rules may be permanent or temporary, such as seasonal rules to change priority levels due to snow, or temporary rules for one-off events such as during or after a hurricane.

[0034] Vehicle systems may have a high priority for data ingestion or collection. Vehicle systems may be required to confirm or refute previous observations, such as inconsistencies. Visual sensors can perform partial or complete collections on one or more image tiles and report what the changes are, or submit complete collections for remote or cloud processing, or for human review. Vehicle systems can perform onboard analysis of changes, for example, using edge devices.

[0035] Vehicle systems can operate according to different priority levels while within a transport network. A vehicle system can operate according to a high priority level, and simultaneously operate according to medium and low priority levels. The priority level of a vehicle system can change while operating within the transport network. For example, a vehicle system may be operating according to a low priority level, but can operate at a medium and / or high priority level by receiving communication or transmission from a remote location. The vehicle system controller can determine the capabilities of the vision sensors to operate according to each priority level. The vehicle system controller can determine, for example, the resource capacity of the vehicle system to operate according to each priority level, the requirements for vision sensors for other vehicle system operations, the availability or capability of vision sensors to perform image data acquisition or collection, the data storage capacity installed in the vehicle system, and / or the communication availability with remote locations and / or the cloud, in addition to the vehicle system's resource capacity for operation according to each priority level, the requirements for vision sensors for other vehicle system operations, and, for example, with respect to the vehicle system's uptime.

[0036] Referring to Figure 3, Method 300 according to one embodiment includes step 310, in which the vehicle system downloads matching samples for use in environmental comparison before initiating a trip. The vehicle system may download matching samples of image tiles to an edge device mounted on the vehicle system. The edge device may include hardware that connects the vehicle system to a remote telecommunications assembly or cloud storage system. The method may also include step 320, in which matching samples of image tiles are transmitted or transmitted to the vehicle system for comparison while the vehicle system is tripping within the transport network.

[0037] The method may include a step 330 in which matching samples of image tiles are performed in a rolling comparison with image data captured or collected during the trip. While the vehicle system is traveling along a route within the transport network, the captured or collected image data is formed into image tiles. The image tiles may be compared with matching samples of image tiles downloaded before the trip and / or with matching samples of image tiles transmitted or transmitted to the vehicle system during the trip. The method may include a step 340 in which mismatches are observed between one or more matching samples of image tiles and one or more image tiles formed from image data captured or collected during the trip. The method may include a step 350 in which one or more observed mismatches are logged or recorded or stored. One or more observed mismatches may be stored in memory on the vehicle system, remote memory, and / or in the cloud.

[0038] The method may include step 360 of adding data to a map collection simultaneously with recording or logging observed discrepancies, if it is possible to add data simultaneously. The method may include step 370 of reporting observed discrepancies to the cloud simultaneously with recording or logging observed discrepancies, if it is possible to report them. If it is not possible to report observed discrepancies, the method may include storing the image data and the observed discrepancies and reports until the system can connect to a remote location, for example, via a network.

[0039] Referring to Figure 4, Method 400 may include a step 410 in which a first vehicle (vehicle 1) creates or updates an image tile. The Method may also include a step 420 in which a second vehicle (vehicle 2) samples an image tile and compares it to a record of the image tile in the map. The Method may also include a step 430 in which it observes any discrepancies between the sampled image tile and the record of the image tile in the map. The Method may also include a step 435 in which it observes any matches between the sampled image tile and the record of the image tile in the map. The Method may also include a step 440 in which it resets the collection expiration date to reflect the observed matches, i.e., confirms that the most recent sample of the image tile matches the record of the image tile in the map collection.

[0040] The method may include step 445 of requiring a third vehicle (vehicle 3) to collect a wider sample of the image tile or a complete sample of the image tile to confirm or negate any observed discrepancies. The method may also include step 450 of confirming any observed discrepancies from the wider or complete sample by the third vehicle, and step 460 of updating the image tile with new image data and a new expiration date.

[0041] The method may include step 455 to negate the observed mismatch in the case of a wider or complete sample of the image tile by a third vehicle. The observed mismatch may be negated if the wider or complete sample by the third vehicle is the same mismatch observed by the second vehicle. The method may include step 465 to log or report the false positive (mismatch observed by the second vehicle). The method may include step 470 to review the image data acquisition for false positive diagnosis if the visual sensor or location repeatedly shows a false positive observed mismatch.

[0042] Referring to Figure 5, Method 500 may include step 505 of observing unexpected image data of one or more image tiles using a visual sensor of a passing vehicle system. The visual sensor may not be able to determine what the unexpected image data is. The Method includes step 510 of the vehicle system reporting or uploading the observed unexpected image data to a remote location or cloud. The Method includes step 515 of the remote location or cloud flagging the image tiles for confirmation by collecting additional image data. According to one embodiment, the collection of additional image data may not be a complete collection of tile image data.

[0043] The method may include step 520, in which a remote location or cloud assigns the task to the next available vehicle system passing over the area where image data for image tiles should be sampled. The method may include step 525, in which the vehicle system takes a scattered sample of the image tile and denies a mismatch (i.e., determines that the image tile matches an image tile in the map collection). This step may include step 530, in which the vehicle system takes a scattered sample of the image tile and confirms a mismatch (i.e., determines that the image tile does not match an image tile in the map collection).

[0044] The method may include step 535 of sending scattered samples to a remote location or cloud for human review, and step 540 of queuing scattered sample images for human review and intervention, depending on the circumstances. The method may also include step 545 of the remote location or cloud automatically prioritizing image tiles for complete new image data collection, and step 550 of the next available vehicle system performing new image data collection for the image tiles. The method may also include step 55 of updating the metadata of the image tiles. The updated metadata may influence the priority of future image data collection.

[0045] Referring to Figure 6, method 600 may include a step 610 to update the metadata of the image tile. The method may also include a step 620 to determine whether the time between a confirmed change to the image tile and a previous confirmed change to the image tile is shorter than the time between two previous confirmed changes. If the time between the confirmed change is shorter than the time between two previous confirmed changes (S620: yes), the method may include a step 630 to collect the image data of the image tile at an earlier time than the current priority level. If the time between the confirmed change is longer than the time between two previously confirmed changes (S620: no), the method may include a step 640 to collect the image data of the image tile at a later time than the current priority level.

[0046] Referring to Figure 7, Method 700 includes step 710 of examining image deviation data associated with one or more image tiles of a set of image tiles used to form a larger set view of the spatial volume. The Method may include step 720 of determining, based on the image deviation data being examined, whether it is time for revision of one or more image tiles of the set in order to update the larger set view of the spatial volume. The Method may include step 730 of either or both of the following: (a) a first vehicle, accompanied by one or more sensors, moving through or beside the spatial volume to capture one or more updated image tiles; or (b) a second vehicle, accompanied by one or more sensors, moving through, beside, or toward the spatial volume in order to capture one or more updated image tiles in response to the determination that it is time for revision of one or more image tiles of the set.

[0047] The method may include examining image deviation data associated with one or more image tiles in a set of image tiles used to form a larger set view of the spatial volume, and determining, based on the examined image deviation data, whether it is time for revision of one or more image tiles in the set in order to update the larger set view of the spatial volume. The method may include either or both of the following: (a) a first vehicle, accompanied by one or more sensors, moves in or near the spatial volume to capture one or more updated image tiles; or (b) in response to the determination that it is time for revision of one or more image tiles in the set, a second vehicle, accompanied by one or more sensors, is moving in, near, or toward the spatial volume to capture one or more updated image tiles.

[0048] Image deviation data may include the difference between the image data of a first image tile out of one or more image tiles and the image data of a second image tile out of one or more image tiles.

[0049] Image deviation data may include the time difference between the first time a first image tile of one or more image tiles was acquired and the image data in which a second image tile of one or more image tiles was acquired.

[0050] At least one of the one or more image tiles in the set may be a combination of different sensor outputs.

[0051] A larger set view of image tiles may include two or more images, video frames, or data output from an optical sensor that are stitched together to form a larger set view of spatial volume.

[0052] A larger set view of image tiles can include a three-dimensional image of the spatial volume.

[0053] Determining whether one or more image tiles in a set are due for revision may include determining whether the data content of one or more previously acquired image tiles differs from the data content of more recently acquired image tiles by more than a threshold amount.

[0054] Determining whether one or more image tiles in a set are due for revision may include determining whether the period between (c) an earlier time when the data content of one or more previously acquired image tiles was obtained and (d) a later time when the data content of a more recently acquired image tile was obtained is longer than a threshold period.

[0055] (a) scheduling the first vehicle, or (b) identifying the second vehicle, or both, may include instructing the first or second vehicle to use its onboard sensors to detect data to update at least one of the image tiles that is older than one or more of the other image tiles, or that is associated with an increased frequency of previous changes.

[0056] (a) scheduling a first vehicle, or (b) identifying a second vehicle, or both, may include instructing the first or second vehicle to use one or more onboard sensors to detect partial data of a sampled portion of at least one of the image tiles, rather than the whole. The method may include examining the partial data of at least one of the image tiles detected by the onboard sensor to determine whether the partial data of at least one of the image tiles indicates that at least one of the image tiles has changed.

[0057] This method may include updating the timestamp of at least one image tile associated with partial data in response to determining that at least one of the image tiles has not changed, or instructing the onboard sensor to acquire additional full data for at least one of the image tiles in response to determining that at least one of the image tiles has changed.

[0058] The method may include either or both of the following: (c) a third vehicle, accompanied by one or more sensors, plans to move through or beside the spatial volume to capture one or more updated image tiles; or (d) in response to a determination that at least one of the image tiles has changed by a threshold amount, a fourth vehicle, accompanied by one or more sensors, is moving through, beside, or toward the spatial volume to capture one or more updated image tiles.

[0059] The method may include transmitting a set of image tiles to at least a third vehicle so that the at least third vehicle can control or modify its movement while it is moving through a spatial volume.

[0060] The system may include one or more processors. One or more processors may examine image deviation data associated with one or more image tiles of a set of image tiles used to form a larger set view of the spatial volume. Based on the image deviation data being examined, one or more processors may determine whether it is time for revision of one or more image tiles in the set to update the larger set view of the spatial volume. One or more processors may either (a) schedule a first vehicle moving through or beside the spatial volume with one or more sensors to capture one or more updated image tiles, or (b) in response to determining that it is time for revision of one or more image tiles in the set, identify a second vehicle moving through, beside, or toward the spatial volume with one or more sensors to capture one or more updated image tiles.

[0061] Image deviation data may include either or both of the following: the difference between the image data of a first image tile from a set of one or more image tiles and the image data of a second image tile from a set of one or more image tiles; or the time difference between a first time when the first image tile from a set of one or more image tiles was acquired and the image data when the second image tile from a set of one or more image tiles was acquired.

[0062] A larger set view of image tiles can include a three-dimensional image of the spatial volume.

[0063] Determining whether one or more image tiles in a set are due for revision may include determining whether the data content of one or more previously acquired image tiles differs from the data content of more recently acquired image tiles by more than a threshold amount.

[0064] The system may include a controller. The controller can examine image deviation data associated with one or more image tiles in a set of image tiles used to form a larger set view of the spatial volume. Based on the image deviation data being examined, the controller can determine whether it is time for revision of one or more image tiles in the set to update the larger set view of the spatial volume. The controller can use one or more onboard sensors to instruct a vehicle moving within, near, or toward the spatial volume to capture partial data of one or more updated image tiles in response to determining that it is time for revision of one or more image tiles in the set. The controller can examine the partial data to determine whether it indicates that one or more updated image tiles have changed. The controller can update the timestamp of one or more updated image tiles in response to determining that one or more updated image tiles have not changed, or instruct one or more sensors to capture additional full data of one or more updated image tiles in response to determining that one or more updated image tiles have changed.

[0065] The vehicle may be a first vehicle, and the controller may either or both schedule a second vehicle moving in or near the spatial volume with one or more sensors to capture one or more updated image tiles, or, in response to determining that one or more image tiles have changed by a threshold amount, identify a third vehicle moving in, near, or toward the spatial volume with one or more sensors to capture one or more updated image tiles.

[0066] The vehicle may be a first vehicle, and the controller transmits a set of image tiles to at least a second vehicle so that the at least second vehicle can control or modify the movement of the at least second vehicle while the at least second vehicle is moving through the spatial volume.

[0067] In one embodiment, the control system may have a deployed local data collection system that can use machine learning to enable derivation-based learning results. The controller can learn from and determine a dataset (including data provided by various sensors) by performing data-driven predictions and fitting them according to the dataset. In various embodiments, machine learning may include performing multiple machine learning tasks by a machine learning system such as supervised learning, unsupervised learning, and reinforcement learning. Supervised learning may include presenting a set of input examples and a desired output to the machine learning system. Unsupervised learning may include a learning algorithm that structures its input by methods such as pattern detection and / or feature learning. Reinforcement learning may include the machine learning system running in a dynamic environment and then providing feedback on correct and incorrect decisions. In various examples, machine learning may include multiple other tasks based on the output of the machine learning system. In various examples, the tasks may be machine learning problems such as classification, regression, clustering, density estimation, dimensionality reduction, and anomaly detection. In various examples, machine learning may include multiple mathematical and statistical techniques. In various examples, many types of machine learning algorithms can include decision tree-based learning, correlation rule learning, deep learning, artificial neural networks, genetic learning algorithms, guided logic programming, support vector machines (SVMs), Bayesian networks, reinforcement learning, representation learning, rule-based machine learning, sparse dictionary learning, similarity and metric learning, learning classifier systems (LCS), logistic regression, random forests, K-means, gradient boosting, K-nearest neighbors (KNNs), and a priori algorithms. In various embodiments, a particular machine learning algorithm may be used (for example, to solve both constrained and unconstrained optimization problems that can be based on natural selection). In one example, an algorithm may be used to address a mixed-integer programming problem where some components are restricted to integer values.Algorithms, machine learning techniques, and machine learning systems can be used in computational intelligence systems, computer vision, natural language processing (NLP), recommendation systems, reinforcement learning, graphical model building, and more. For example, machine learning can be used for vehicle performance and behavioral analysis.

[0068] In one embodiment, the control system may include a policy engine to which one or more policies can be applied. These policies may be based at least in part on the characteristics of a given item of equipment or environment. With respect to the control policy, the neural network may receive inputs of several environmental and task-related parameters. These parameters may include identification of a determined trip plan for a group of vehicles, data from various sensors, and location and / or positional data. The neural network may be trained to produce an output based on these inputs, the output representing an action or set of actions that the group of vehicles should take to achieve the trip plan. During operation in one embodiment, the decision can be made by processing the input through the parameters of the neural network to generate a value at the output node that designates that action as the desired action. This action can be converted into a signal that operates the vehicle. This can be achieved by backpropagation, a feedforward process, closed-loop feedback, or open-loop feedback. Alternatively, the controller's machine learning system may use evolutionary strategy techniques to tune various parameters of the artificial neural network instead of using backpropagation. The controller can use a neural network architecture with functions that may not always be solvable using backpropagation, such as non-convex functions. In one embodiment, the neural network has a set of parameters representing the weights of its node connections. Several copies of this network are generated, then different tunings are made to the parameters, and simulations are performed. Once outputs are obtained from the various models, these models can be evaluated for their performance using a determined success metric. The best model is selected, and the vehicle controller executes its plan to realize the desired input data and reflect the best-case outcome scenario predicted thereby. Furthermore, the success metric may be a combination of optimized results that can be weighted against each other.

[0069] As used herein, the terms “processor” and “computer,” as well as related terms such as “processing device,” “computing device,” and “controller,” are not limited to integrated circuits referred to as computers in the art, but may also refer to microcontrollers, microcomputers, programmable logic controllers (PLCs), field-programmable gate arrays, and application-specific integrated circuits, as well as other programmable circuits. Suitable memory may include, for example, computer-readable media. Computer-readable media may be, for example, computer-readable non-volatile media such as random-access memory (RAM) and flash memory. The term “non-transitory computer-readable media” refers to tangible computer-based devices implemented for short-term and long-term storage of information such as computer-readable instructions, data structures, program modules and submodules, or other data within any device. Thus, the methods described herein may be encoded as executable instructions embodied in tangible non-transitory computer-readable media, including, without limitation, storage devices and / or memory devices. When such instructions are executed by a processor, they cause the processor to perform at least a portion of the methods described herein. Thus, the term includes tangible computer-readable media, including non-temporary computer storage devices, including volatile and non-volatile media, without limitation; firmware, physical and virtual storage devices, removable and non-removable media such as CD-ROMs and DVDs; and other digital sources such as networks and the internet.

[0070] Where used herein, elements or steps listed in the singular and followed by the word “a” or “an” do not exclude the plural form of such element or action unless such exclusion is expressly stated. Furthermore, references to “one embodiment” of the present invention do not preclude the existence of additional embodiments incorporating the listed features. Furthermore, unless expressly stated to the contrary, embodiments “comprising,” “comprises,” “including,” “includes,” “having,” or “has” one or more elements having a particular characteristic may include additional such elements that do not possess that characteristic. In the appended claims, the terms “including” and “in which” are used as plain English synonyms for the terms “comprising” and “wherein,” respectively. Furthermore, in the following claims, terms such as “first,” “second,” and “third” are used merely as labels and do not impose numerical requirements on their subjects. Furthermore, the following limitation of claims is not intended to be construed under 112(f) of the U.S. Patent Act unless such limitation of claims explicitly uses the phrase “means for” followed by a description of a function lacking further structure.

[0071] The above description is illustrative and not limiting. For example, the embodiments (and / or aspects thereof) described above can be used in combination with one another. Furthermore, many modifications can be made to adapt specific situations or materials to the teachings of the subject without departing from the scope of the subject. The dimensions and types of materials described herein define the parameters of the subject, but they are illustrative embodiments. Other embodiments will be apparent to those skilled in the art upon closer examination of the above description. Accordingly, the scope of this subject should be determined by referring to the appended claims, together with the entire scope of equivalents to which such claims are granted.

[0072] This specification uses examples to disclose several embodiments of the subject matter, including the best mode, and to enable a person skilled in the art to carry out embodiments of the subject matter, including constructing and using any device or system, and performing any incorporated method. The patentable scope of the subject matter is defined by the claims and may include other examples that a person skilled in the art can conceive. Such other examples are intended to be within the claims if they have structural elements that are not different from the language of the claims, or if they include equivalent structural elements that are not substantially different from the language of the claims.

[0073] References herein to other matters identified in patent documents or prior art should not be construed as an acknowledgment that the documents or other matters were known or that the information contained herein was part of the common general knowledge as of the priority date of any of the claims.

Claims

1. Examining image deviation data associated with one or more image tiles in a set of image tiles used to form a larger set view of spatial volume, Based on the image deviation data being examined, determine whether it is time for revision of one or more image tiles in the set in order to update the larger set view of the spatial volume. (a) scheduling a first vehicle (102) to move in or near the spatial volume with one or more sensors (112) to capture one or more updated image tiles, and (b) identifying a second vehicle (102) moving in, near, or toward the spatial volume with one or more sensors (112) to capture the one or more updated image tiles in the set, in response to the determination that it is time for revision of one or more image tiles of the set, Methods that include...

2. The method according to claim 1, wherein the image deviation data includes the difference between the image data of a first image tile among the one or more image tiles and the image data of a second image tile among the one or more image tiles.

3. The method according to claim 1, wherein the image deviation data includes a time difference between a first time when the first image tile among the one or more image tiles was acquired and image data when the second image tile among the one or more image tiles was acquired.

4. The method according to claim 1, wherein at least one of the one or more image tiles in the set is a combination of different sensor outputs.

5. The method according to claim 1, comprising two or more images, video frames, or data output from an optical sensor (112), wherein a larger set view of the image tiles is joined together to form a larger set view of the spatial volume.

6. The method according to claim 1, wherein a larger set view of the image tiles includes a three-dimensional image of the spatial volume.

7. The method according to claim 1, wherein determining whether one or more image tiles in the set are due for revision includes determining whether the data content of a previously acquired image tile among the one or more image tiles differs from the data content of a more recently acquired image tile by more than a threshold amount.

8. The method according to claim 1, wherein determining whether one or more image tiles in the set are due for revision includes determining whether the period between an earlier time when the data content of a previously acquired image tile among the one or more image tiles was acquired and (d) a later time when the data content of a more recently acquired image tile was acquired is longer than a threshold period.

9. (a) scheduling the first vehicle (102), and (b) identifying the second vehicle (102), either or both of these, The method according to claim 1, comprising instructing the first vehicle (102) or the second vehicle (102) to use onboard sensors to detect data for updating at least one of the image tiles that is older than one or more of the other image tiles or is associated with an increasing frequency of preceding changes.

10. (a) scheduling the first vehicle (102), and (b) identifying the second vehicle (102), either or both of these, This includes instructing the first vehicle (102) or the second vehicle (102) to use one or more of the mounted sensors (112) to detect partial data of a sampled portion, rather than the entirety, of at least one of the image tiles, This method further, To examine the partial data of at least one of the image tiles detected by the mounted sensor and determine whether the partial data of at least one of the image tiles indicates that at least one of the image tiles has changed. The method according to claim 1, including the method described in claim 1.

11. In response to determining that at least one of the image tiles has not changed, update the timestamp of the at least one of the image tiles associated with the partial data, or In response to determining that at least one of the image tiles has changed, instruct the mounted sensor to acquire additional data for the entirety of at least one of the image tiles. The method according to claim 10, further comprising:

12. (c) scheduling a third vehicle (102) to move in or near the spatial volume with the one or more sensors (112) to capture the one or more updated image tiles, and (d) identifying a fourth vehicle (102) moving in, near, or toward the spatial volume with the one or more sensors (112) to capture the one or more updated image tiles in response to a determination that at least one of the image tiles has changed by more than a threshold amount. The method according to claim 11, further comprising:

13. The set of image tiles is transmitted to at least a third vehicle (102) so that the movement of at least the third vehicle (102) can be controlled or modified while the third vehicle (102) is moving through the spatial volume. The method according to claim 1, further comprising:

14. One or more processors (132) configured to examine image deviation data associated with one or more image tiles of a set of image tiles used to form a larger set view of spatial volume, A system (150) comprising, The one or more processors (132) are configured to determine, based on the image deviation data being examined, whether it is time for revision of the one or more image tiles in the set to update the larger set view of the spatial volume. The one or more processors (132) also A system (150) configured to (a) schedule a first vehicle (102) to move in or near the spatial volume with one or more sensors (112) to capture one or more updated image tiles, and (b) in response to it being determined that one or more of the image tiles in the set are due for revision, identify a second vehicle (102) moving in, near, or toward the spatial volume with one or more sensors (112) to capture the one or more updated image tiles.

15. The aforementioned image deviation data, The difference between the image data of the first image tile among the one or more image tiles and the image data of the second image tile among the one or more image tiles, and The time difference between the first time when the first image tile among the one or more image tiles was acquired and the image data when the second image tile among the one or more image tiles was acquired, The system (150) according to claim 14, comprising one or both of the above.

16. The system (150) according to claim 14, wherein a larger set view of the image tiles includes a three-dimensional image of the spatial volume.

17. The system (150) according to claim 14, wherein determining whether one or more image tiles in the set are due for revision includes determining whether the data content of a previously acquired image tile among the one or more image tiles differs from the data content of a more recently acquired image tile by more than a threshold amount.

18. A controller (136) configured to examine image deviation data associated with one or more image tiles of a set of image tiles used to form a larger set view of spatial volume, A system (150) comprising, The controller (136) is also configured to determine, based on the image deviation data being inspected, whether it is time for revision of one or more image tiles in the set in order to update a larger set view of the spatial volume. The controller (136) is configured to, in response to determining that one or more image tiles in the set are due for revision, instruct a vehicle (102) moving within, near, or toward the spatial volume, accompanied by one or more onboard sensors (112), to capture partial data of the updated image tiles. The controller (136) is configured to examine the partial data and determine whether the partial data indicates that one or more updated image tiles have changed. The controller (136) In response to determining that one or more of the updated image tiles have not changed, update the timestamp of the one or more updated image tiles, In response to the determination that one or more of the updated image tiles have changed, the one or more sensors (112) are instructed to acquire additional data for the entirety of the one or more updated image tiles. A system (150) configured as follows.

19. The aforementioned vehicle (102) is the first vehicle (102), The controller (136) is Schedule a second vehicle (102) to move in or near the spatial volume with the one or more sensors (112) to capture the one or more updated image tiles; and identify a third vehicle (102) moving in, near, or toward the spatial volume with the one or more sensors (112) to capture the one or more updated image tiles in response to a determination that the one or more image tiles have changed by more than a threshold amount. The system (150) according to claim 18, configured as follows.

20. The aforementioned vehicle (102) is the first vehicle (102), The controller (136) The set of image tiles is transmitted to at least a second vehicle (102) so that the at least second vehicle (102) controls or modifies the movement of the at least second vehicle (102) while it is moving through the spatial volume. The system (150) according to claim 18, configured as follows.

Images