Mapping system, method of using the mapping system, and program

Abstract

To provide a mapping system that is hard to generate inaccuracy and generates a high quality map.SOLUTION: A mapping system is configured to: receive an input signal including image data of a scene; determine a position of a sensor used to capture the image data relative to a reference map of the scene; determine a change point and a change score in the scene based on the determined position of the sensor and the reference map; generate a change map based on the change point and the change score; generate an update map based on a comparison between the change map and the reference map; and maintain a content of the reference map unchanged.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a mapping system, a method of using the mapping system, and a program.

Background Art

[0002] A mapping system can be used to capture scans or images of scenes such as rooms, factories, etc. and create a three-dimensional (3D) map of the scene. The mapping system performs subsequent scans or imaging of the scene and determines changes in the scene, such as the movement of objects in the scene, new objects in the scene, or the removal of objects in the scene. The mapping system then updates the 3D map based on the determined changes in the scene. The maps generated and updated by the mapping system can be used in a wide range of technologies, including augmented reality (AR) games, virtual reality (VR) games, autonomous vehicle control, and other suitable activities.

Summary of the Invention

Problems to be Solved by the Invention

[0003] There is a need for a mapping system that generates high-quality maps with low susceptibility to inaccuracies.

Means for Solving the Problems

[0004] Aspects of the present specification relate to a mapping system including a non-transitory computer-readable medium configured to store instructions and a processor connected to the non-transitory computer-readable medium. The processor is configured to receive an input signal including image data of a scene, determine a position of a sensor used to capture the image data with respect to a reference map of the scene, determine a change point and a change score of the scene based on the determined position of the sensor and the reference map, generate a change map based on the change point and the change score, generate an update map based on a comparison between the change map and the reference map, and execute the instructions to maintain the content of the reference map without change.

[0005] Aspects of the present specification relate to a method of using a mapping system, including receiving an input signal including image data of a scene, determining a position of a sensor used to capture the image data with respect to a reference map of the scene, determining a change point and a change score of the scene based on the determined position of the sensor and the reference map, generating a change map based on the change point and the change score, generating an update map based on a comparison between the change map and the reference map, and maintaining the content of the reference map without change.

[0006] Aspects of the present specification relate to a program for causing a processor to receive an input signal including image data of a scene, determine a position of a sensor used to capture the image data with respect to a reference map of the scene, determine a change point and a change score of the scene based on the determined position of the sensor and the reference map, generate a change map based on the change point and the change score, generate an update map based on a comparison between the change map and the reference map, and maintain the content of the reference map without change.

Brief Description of the Drawings

[0007] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. Note that various features are not drawn to scale in accordance with standard industry practice. In fact, the dimensions of the various features may be arbitrarily increased or decreased for clarity of explanation.

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

DETAILED DESCRIPTION OF THE INVENTION

[0009] The following disclosure provides many different embodiments or examples for implementing different features of the provided subject matter. To simplify the present disclosure, specific examples of components, values, operations, materials, arrangements, etc. are set forth below. Of course, these are merely examples and are not intended to be limiting. Other components, values, operations, materials, arrangements, etc. are contemplated. For example, in the following description, forming a first feature over or on a second feature can include embodiments in which the first and second features are formed in direct contact, and can also include embodiments in which additional features can be formed between the first and second features so that the first and second features cannot be in direct contact. Further, the present disclosure may repeat reference numerals and / or letters in various examples. This repetition is for the purpose of simplicity and clarity and does not in itself define a relationship between the various embodiments and / or configurations described.

[0010] A mapping system that directly updates a map to determine changes from a previous image or scan of a scene has a high risk of inaccurately determining changes in the updated map. The increased risk of inaccuracy is due to several factors including the quality of the imaging or scanning device, threshold processing during change registration, failure to verify objects in the image or scan, or other such drawbacks. To reduce costs, low-resolution imaging or scanning devices are often used in mapping systems. With these low-resolution devices, it becomes more difficult to identify objects, and the risk increases that an object will be detected under certain lighting conditions and then not detected under different lighting conditions. In such situations, the mapping system may consider an object not detected in a later image or scan as a change to the scene when in fact the object still exists but was simply not detected. Threshold processing is a technique used to reduce the computational load of the mapping system. Threshold processing is an attempt to account for minor differences in the scene due to lighting conditions, transient moving objects (e.g., people moving), or other such situations. Setting the threshold value too severely increases the risk of not being able to identify changes in the scene. In contrast, setting the threshold too loosely increases the risk of false positives for changes in the scene. If the geometry of an object in the scene cannot be verified, there is a higher risk that a different object in a similar position to the previous object will be treated as the same object and differences in the scene will not be identified.

[0011] In addition to items that increase the risk of inaccuracy, directly updating the map of a scene propagates errors through successive iterations of the scene's image or scan. For example, in simultaneous localization and mapping (SLAM) techniques, thresholding and assumptions are utilized while analyzing an image or scan. The SLAM technique measures the map's data points one at a time to determine the position of objects in a scene. If there are any errors in a previous analysis, inaccuracies will occur in subsequent analyses. Error propagation degrades the overall reliability of maps generated by such mapping systems.

[0012] Mapping systems that generate high-quality maps with low likelihood of inaccuracy are useful for vehicle movement automation, improvement of realistic game environments, and advancement of other applications. To encourage improving the quality of map generation while avoiding the cost of continuously using a high-resolution imager or scanner, the mapping systems according to some embodiments herein utilize a reference map that remains unchanged during the operation of the mapping system. Utilization of the reference map provides high-quality fixed reference points to reduce the risk of errors that propagate through successive iterations of imaging or scanning a scene. Further, the mapping systems according to some embodiments herein also utilize object geometry verification to improve the accuracy of the mapping system compared to other approaches. As a result, the mapping systems of some embodiments herein can utilize low-resolution imaging or scanning devices during implementation of the mapping system while generating accurate maps for use in various applications.

[0013] For the sake of brevity, the following description focuses on images of scenes. Those skilled in the art will recognize that the images are merely illustrative and that other types of scene detection, such as point clouds, are within the scope of this description. The following description also refers to sensors for capturing data regarding scenes. In some embodiments, the sensors include one or more cameras, one or more thermal cameras, one or more video cameras, one or more light and range detectors (LiDAR), combinations of these elements, or other suitable sensors.

[0014] Figure 1 is a schematic diagram of a mapping system 100 according to some embodiments. The mapping system 100 is configured to receive an input signal. The input signal includes both an image and depth data. In some embodiments, the image data includes red, green, blue (RGB) image data. In some embodiments, the image data includes another type of image data such as grayscale, thermal, or another suitable type of image data. The mapping system 100 includes a registration module 105 configured to receive image data and a reference three-dimensional (3D) map 125 from the input signal. The registration module 105 is configured to generate a global pose indicating the position of the sensor used to collect the input signal with respect to the reference 3D map 125. The mapping system 100 further includes a change detection module 110 configured to receive depth data from the input signal, the global pose from the registration module 105, and the reference 3D map 125. The change detection module 110 is configured to generate a change score and change points for each object in the scene. The mapping system 100 further includes a change improvement module 115 configured to receive the change score and change points from the change detection module 110. The change improvement module 115 is configured to perform post-processing to determine whether an object in the scene has changed with respect to a previous iteration of the mapping of the scene. The mapping system 100 further includes a change 3D map 120 configured to receive the final change points from the change improvement module 115. The change 3D map 120 stores information regarding changes in the scene based on the input signal with respect to the reference 3D map 125. The mapping system 100 avoids updating the reference 3D map in order to facilitate reducing the propagation of errors due to successive iterations of mapping the scene. The mapping system 100 further includes a map update module 130 configured to combine the reference 3D map 125 with the change map 120 to determine an updated map of the scene.The mapping system 100 further includes a downstream task module 135 configured to generate a determination or instruction based on the updated map received from the map update module 130.

[0015] The input signal includes both image data for capturing the texture / color characteristics in the scene and depth data for facilitating the detection of the relative positions of points in the scene. In some embodiments, the image data includes color image data. In some embodiments, the image data includes grayscale data. This specification focuses on color image data. However, those skilled in the art will recognize that the present application is not limited to color image data. In some embodiments, the input signal is received from a single sensor that includes both an image and a depth detection function. In some embodiments, the input signal is received from two or more sensors. In some embodiments, the plurality of sensors includes sensors of the same type, such as image detection sensors. In some embodiments, the plurality of sensors includes sensors of different types, such as LiDAR sensors and image detection sensors. In some embodiments, the depth data is generated based on stereo image detection using triangulation. In some embodiments, the depth data is generated using a structured light sensor. In some embodiments, the depth data is generated using a time of flight (ToF) sensor.

[0016] In some embodiments, the input signal is decomposed into frames to assist with the processing load on the mapping system 100. A frame is a smaller portion of the scene. All frames of the input signal are captured simultaneously, and the capture is after the creation of the reference 3D map 125. The mapping system 100 analyzes each frame separately to determine change points and change scores, as described below. The identification of geometrically meaningful objects within the input signal is performed using multiple frames to help improve the accuracy of the determination of geometrically meaningful objects. In some embodiments, all of the frames are analyzed during the determination of geometrically meaningful objects. In some embodiments, when enough frames have been analyzed to identify a geometrically meaningful object, the frames are analyzed for the identified geometrically meaningful object.

[0017] The registration module 105 is configured to receive the reference 3D map 125 and the image data from the input signal. The registration module 105 is configured to determine the position of the sensor used to capture the image data. The registration module 105 is configured to output a global pose based on the determined position of the sensor relative to the reference 3D map 125. The global pose includes the image data and the position of the sensor. By using the image data without considering the depth data, the processing load on the registration module 105 is reduced compared to a system that generates a global pose using the entire input signal. In some embodiments that include multiple sensors, the registration module 105 is configured to determine the position of each of the sensors used to capture the input signal. The registration module 105 utilizes permanent objects or points within the reference 3D map 125 to determine the position of the sensors. The registration module 105 is implemented using one or more processors, such as the processor described with respect to the following mapping system 1000 (FIG. 10).

[0018] The change detection module 110 is configured to receive a reference 3D map 125, a global pose from the registration module 105, and depth data from the input signal. The change detection module 110 is configured to compare the information of the reference 3D map 125 with the data from the global pose and the depth data to determine whether a change has occurred in the scene with respect to the reference 3D map 125. The determination of change by the change detection module 110 is with respect to the reference 3D map 125, not with respect to the previous iteration before the mapping of the scene by the mapping system 100. The change detection module 110 is configured to identify the region of the 3D reference map 125 corresponding to the currently analyzed frame based on the global pose. The change detection module 110 is further configured to identify the positions of points in the scene and the detected points in the scene using depth or point-to-point distance data. In some embodiments, the change detection module 110 is configured to identify the objects in the scene and the positions of the detected objects based on depth or point-to-point distance data. The change detection module 110 is configured to compare the identified points and the positions of the identified points with the reference 3D map 125 to determine whether there are any identified points that have been added to the scene or moved within the scene. The change detection module 110 is also configured to determine whether any points from the reference 3D map 125 have been removed from the scene since the reference 3D map 125 was generated based on the global pose and the depth data. The change detection module 110 is configured to generate a change point indicating the position of the identified change and a change score indicating the likelihood of the identified change. In some embodiments, the change detection module 110 is further configured to generate a change point indicating color in addition to the position of the detected change if color image data is available. Those skilled in the art will recognize that a change point indicating grayscale is also possible if grayscale image data is available.

[0019] The change points indicate positions in scenes that are different from the same positions in the reference 3D map 125. The change score indicates the likelihood that a change point is an actual change. That is, the change points indicate that some kind of change has occurred at the positions in the scene, and the change score indicates how likely it is that the identified change is an actual change in the scene rather than an artifact generated by different lighting conditions or other factors that affect the accuracy of object detection. The following description presents examples for clarifying the change points and change scores. Those skilled in the art will understand that the mapping system 100 is not limited to the examples described below.

[0020] In at least one example, the reference 3D map 125 includes a table with no objects placed on it. The input signal includes an image of a scene that includes a box placed on the table. Using the depth data, the change detection module 110 can determine that a position in the scene has changed. For example, the distance between the closest object in the reference 3D map 125, such as the wall behind the scene, and the input signal, i.e., the box, is different. The change detection module 110 indicates change points based on the difference in depth data between the reference 3D map 125 and the input signal. Next, the change detection module 110 generates a change score for each point where a change is found. Generating the change scores helps reduce false positives for the identified change points. Since the distance between the back wall of the scene and the top of the box is large, the change score for the top of the box is large. In contrast, since the distance between the top surface of the table and the bottom of the box is small, the change score for the bottom of the box is small. As described below, the combination of the change points and change scores helps the change improvement module 115 determine the boundaries of the detected points of potential change in the input signal.

[0021] The change detection module 110 is implemented using one or more processors, such as a processor described with respect to the following mapping system 1000 (FIG. 10). In some embodiments, the change detection module 110 is implemented using the same processor as the registration module 105. In some embodiments, the change detection module 110 is implemented using a processor different from the registration module 105.

[0022] The change improvement module 115 is configured to receive change point and change score data from the change detection module 110. The change improvement module 115 is further configured to receive image data. The change improvement module 115 is configured to perform a geometric analysis of the change point and change score data to improve the determination of the boundaries of the objects associated with the change point and change score data. In some embodiments, the change improvement module 115 is further configured to receive a reference 3D map 125 or to assist in improving the boundaries of the objects.

[0023] Returning to the non - limiting example of a box on a table from above, the change improvement module 115 is configured to assist in determining the boundaries of the box. As described above, the change score at the bottom of the box is low. In a system that only performs threshold processing to determine that a change score below a certain value is not a change, the risk of "floating objects" increases. That is, the upper part of the box is displayed as a change, but the lower part of the box is not displayed as a change. As a result, in the scene map update, the upper part of the box appears to float on the table. However, the change improvement module 115 is configured to determine whether the geometry of the change points and change score data generates a meaningful shape for addition to the scene map. To make such a determination, the change improvement module 115 is configured to utilize the change scores of adjacent change points to assist in determining whether a geometrically meaningful object is represented by the change points and change score data. Geometrically meaningful objects include objects that have a clear boundary and a spatial relationship with other objects that have physical meaning. For example, a "floating object" has no physical meaning, while an object resting on a table has physical meaning. Geometrically meaningful objects are constructed from one or more geometrically meaningful shapes. In some embodiments, a single object includes multiple geometrically meaningful shapes. For example, a bicycle may, in some cases, include multiple circles as well as at least one rectangle.

[0024] Continuing with the non-limiting example of the box on the table, the change improvement module 115 is configured to determine that the change score at the top of the box indicates that there is a high likelihood that an object exists. The change improvement module 115 is configured to adjust the threshold of nearby change points, such as the bottom of the box, in an attempt to identify a geometrically meaningful object. The change improvement module 115 is configured to reject the overall likelihood of a potential object in response to indicating a change related to the entire geometrically meaningful object or being unable to determine a geometrically meaningful object. Those skilled in the art will understand that indicating a change can include adding an object, moving an object within the scene, or removing an object from the scene. By implementing changes to the entire object, the risk of error when updating the 3D map is reduced, and the updated 3D map is more likely to resemble a realistic scene, such as having no "floating objects". The change improvement module 115 is configured to analyze the change points, for example using clustering or segmentation, to estimate the boundaries of the object. The change improvement module 115 can then utilize the change points potentially within the estimated boundaries of the object to determine whether a geometrically meaningful object has been detected. In some embodiments, the change improvement module 115 is configured to utilize the average of all change scores within the estimated boundaries of the potential object to determine whether to determine that the object is a change. In some embodiments, the change improvement module 115 is configured to determine whether the ratio of the threshold of the change points within the estimated boundaries of the potential object to the change score having a change score above the threshold score to determine whether the object is a change. In some embodiments, the user can select the algorithm used by the change improvement module 115 based on the sensor used to capture the input signal.For example, in some embodiments, using the percentage of change points that exceed the change score threshold is more likely to produce accurate results when the sensor generates a noisy input signal compared to the average change score algorithm.

[0025] The change improvement module 115 is implemented using one or more processors, such as a processor described with respect to the following mapping system 1000 (FIG. 10). In some embodiments, the change improvement module 115 is implemented using the same processor as the registration module 105 and the change detection module 110. In some embodiments, the change improvement module 115 is implemented using a processor different from the registration module 105 or the change detection module 110.

[0026] The change 3D map 120 is generated based on the final change points received from the change improvement module 115. The final change points indicate the location of the changes in the objects throughout the scene. In some embodiments, the changes indicate the addition of an object, the removal of an object, or the movement of an object within the scene. In some embodiments, the change 3D map 120 includes image data along with the final change points to determine the changes to the 3D map relative to the reference 3D map 125. The change 3D map 120 is stored in a non-transitory computer-readable medium (computer-readable medium), such as the memory of the mapping system 1000 (FIG. 10) described below.

[0027] The reference 3D map 125 includes a map generated using a high-resolution sensor to capture a scene. In some embodiments, the reference 3D map 125 includes the dimensions of objects in the scene. In some embodiments, the dimensions of the reference 3D map 125 are scaled dimensions of the true dimensions of the objects in the scene. In some embodiments, the reference 3D map 125 is generated using the system 200 (FIG. 2) described below. The reference 3D map 125 remains constant during use of the mapping system 100. The reference 3D map 125 can be used as a stable basis for comparison during use of the mapping system 100 to assist in preventing or reducing error propagation during subsequent iterations of mapping a scene using the mapping system 100. The reference 3D map 125 is stored in a non-transitory computer-readable medium such as the memory of the mapping system 1000 (FIG. 10) described below. In some embodiments, the reference 3D map 125 is stored in the same non-transitory computer-readable medium as the change 3D map 120. In some embodiments, the reference 3D map 125 is stored in a non-transitory computer-readable medium different from the change 3D map 120.

[0028] The map update module 130 is configured to combine changes from the change 3D map 120 with the reference 3D map 125 to generate an updated map. The updated map includes changes to the entire object identified by the change improvement module 115. In some embodiments, the updated map is stored in a non-transitory computer-readable medium such as the memory of the mapping system 1000 (FIG. 10) described below. In some embodiments, the updated map is stored in the same non-transitory computer-readable medium as the change 3D map 120 and the reference 3D map 125. In some embodiments, the change map is stored in a non-transitory computer-readable medium different from the change 3D map 120 or the reference 3D map 125.

[0029] The downstream task module 135 is configured to execute instructions based on the received change map. The following description utilizes a non-limiting example of autonomous vehicle control. Those skilled in the art will understand that this specification is not limited to this example. For example, in some embodiments, the downstream task module 135 utilizes the updated map to instruct a vehicle, such as a factory vehicle, to navigate around newly added objects in a scene, such as a pallet of materials. In some embodiments, the downstream task module 135 is configured to directly transmit the instructions to the vehicle. In some embodiments, the downstream task module 135 is configured to provide the instructions to an external device that can be used to control the vehicle. In some embodiments, the instructions are transmitted wirelessly. In some embodiments, the instructions are transmitted via a wired connection.

[0030] The downstream task module 135 is implemented using one or more processors, such as a processor described with respect to the following mapping system 1000 (FIG. 10). In some embodiments, the downstream task module 135 is implemented using the same processor as the registration module 105, the change detection module 110, and the change improvement module 115. In some embodiments, the downstream task module 135 is implemented using a processor different from the registration module 105, the change detection module 110, or the change improvement module 115.

[0031] Using the mapping system 100, since the determination of changes is performed for the entire object, an updated map with higher accuracy can be generated compared to other methods. Furthermore, maintaining the reference 3D map 125 in its original state during the mapping of the scene helps reduce or prevent the propagation of errors due to multiple iterations of scene mapping. As a result, the instructions given to the external device based on the updated map are more accurate than the instructions using maps generated using other systems.

[0032] FIG. 2 is a schematic diagram of a system 200 for generating a reference map according to some embodiments. The system 200 can be used to create a reference 3D map 125. In some embodiments, the system 200 can be used to create a reference 3D map 125 that can be used by a mapping system 100 (FIG. 1). In some embodiments, the system 200 can be used to create a reference 3D map 125 that can be used by a mapping system different from the mapping system 100 (FIG. 1). The system 200 includes a map creation module 205 configured to receive a high-resolution scan of a scene. The map creation module 205 is configured to generate a map that is stored as the reference 3D map 125.

[0033] High-resolution scanning is performed using at least one high-resolution sensor configured to capture both image data and depth data regarding a scene. In some embodiments, high-resolution scanning is performed using a sensor having a higher resolution than that used to capture the input signal of the mapping system 100 (FIG. 1). In some embodiments, high-resolution scanning is performed at multiple positions with respect to the scene to assist in ensuring the accuracy and precision of the reference 3D map 125. In some embodiments, high-resolution scanning is performed using a single sensor that includes both image and depth detection capabilities. In some embodiments, high-resolution scanning is performed using two or more sensors. In some embodiments, the multiple sensors include sensors of the same type, such as image detection sensors. In some embodiments, the multiple sensors include sensors of different types, such as LiDAR sensors and image detection sensors. In some embodiments, depth data is generated based on stereo image detection using triangulation. In some embodiments, depth data is generated using a structured light sensor. In some embodiments, depth data is generated using a ToF sensor. In some embodiments, high-resolution scanning is performed using multiple sensors configured to receive the same type of information, such as image data or depth data, to assist in ensuring the accuracy and precision of the reference 3D map 125.

[0034] The map creation module 205 is configured to receive a high-resolution scan and generate a map. The map creation module 205 is configured to perform object identification to identify objects in the scene. The map creation module 205 is configured to utilize depth data to determine the placement of the identified objects in the scene relative to each other. In some embodiments, the map creation module 205 is configured to perform object recognition using, for example, a trained neural network, to identify permanent and movable objects in the scene. In some embodiments, the trained neural network includes a database of object types likely to occur in the scene. In some embodiments, the map creation module 205 is configured to generate a map that includes metadata indicating whether the identified objects are permanent objects or movable objects. The map creation module 205 is configured to instruct that the map be stored in a non-transitory computer-readable medium as a reference 3D map 125.

[0035] The map creation module 205 is implemented using one or more processors, such as a processor described with respect to the following mapping system 1000 (FIG. 10). In some embodiments, the map creation module 205 is implemented using the same processor as each module of the mapping system 100 (FIG. 1). In some embodiments, the map creation module 205 is implemented using a processor different from at least one of the modules of the mapping system 100 (FIG. 1).

[0036] Figure 3 is a flowchart of a method 300 of using a mapping system according to some embodiments. Method 300 is implemented by a mapping system to generate a change map and generate instructions for downstream implementation. In some embodiments, method 300 is implemented using mapping system 100 (FIG. 1). In some embodiments, method 300 is implemented using a mapping system other than mapping system 100 (FIG. 1).

[0037] In operation 305, input data is received by the mapping system. The input data includes both image data that enables object detection in a scene and depth data that facilitates detection of the relative positions of points in the scene. In some embodiments, the image data includes color image data. In some embodiments, the image data includes grayscale data. This specification focuses on color image data. However, those skilled in the art will recognize that the present application is not limited to color image data. In some embodiments, the input data is received from a single sensor that includes both an image and a depth detection function. In some embodiments, the input data is received from more than one sensor. In some embodiments, the plurality of sensors includes sensors of the same type, such as image detection sensors. In some embodiments, the plurality of sensors includes sensors of different types, such as LiDAR sensors and image detection sensors. In some embodiments, the depth data is generated based on stereo image detection using triangulation. In some embodiments, the depth data is generated using a structured light sensor. In some embodiments, the depth data is generated using a ToF sensor.

[0038] In operation 310, registration of image data from input data is performed. The registration is performed based on the image data and the reference 3D map 125. In some embodiments, the registration is performed using the registration module 105 (FIG. 1). In some embodiments, the registration is performed using a device different from the registration module 105 (FIG. 1). The registration determines the position of the sensor used to capture the image data. The registration outputs a global pose based on the determined position of the sensor relative to the 3D reference map 125. The global pose includes the image data and the position of the sensor. By using the image data without considering the depth data, the processing load during registration is reduced compared to a method of generating a global pose using the entire input signal. In some embodiments including multiple sensors, the registration determines the position of each of the sensors used to capture the input signal. The registration utilizes permanent objects in the reference 3D map 125 to determine the position of the sensors.

[0039] In operation 315, change detection is performed using the global pose and depth data from the input data. The change detection is performed by comparing the reference 3D map 125 with the depth data. In some embodiments, the change detection is performed using the change detection module 110 (FIG. 1). In some embodiments, the change detection is performed using a device different from the change detection module 110 (FIG. 1). The change detection is configured to compare the information of the reference 3D map 125 with the data from the global pose and depth data to determine whether a change has occurred in the scene with respect to the reference 3D map 125. The determination of change is made with respect to the reference 3D map 125, not with respect to the iterations before mapping the scene. The change detection compares the identified regions of the 3D reference map 125 corresponding to the currently analyzed frame based on the global pose. The change detection further identifies the positions of points in the scene and the detected points in the scene using depth data or point-to-point distance data. In some embodiments, the change detection identifies the positions of points in the scene and the detected points based on the depth data. The change detection also determines whether any point from the reference 3D map 125 has been removed from the scene since the reference 3D map 125 was generated based on the global pose and depth data. The change detection generates a change point indicating the position of the identified change and a change score indicating the likelihood of the identified change. In some embodiments, the change detection includes generating a change point indicating color in addition to the position of the detected change when color image data is available. Those skilled in the art will recognize that a change point indicating grayscale is also possible when grayscale image data is available.

[0040] Following operation 315, operations 305 through 315 are repeated until all frames of the input data have been analyzed to determine whether any change has occurred within the scene. In some embodiments, to reduce the processing load for performing registration and change detection, the input data is decomposed into frames or sections of the entire scene. Each frame is captured simultaneously. In some embodiments where multiple sensors are used to capture the input data, the analysis of all frames includes the analysis of the input data from all sensors.

[0041] In operation 320, change improvement is performed using the change points and change scores from the detection of changes in operation 315. The change improvement helps ensure that the entire object is considered when evaluating potential changes in a scene. In some embodiments, the change improvement is performed using change improvement module 115 (FIG. 1). In some embodiments, the change improvement is performed using a device other than change improvement module 115 (FIG. 1). The change improvement helps determine the boundaries of the identified objects. The change improvement determines whether the geometry of the change point and change score data generates a meaningful shape for addition to the map of the scene. To make such a determination, the change improvement utilizes the change scores of adjacent change points to assist in determining whether a geometrically meaningful object is represented by the change point and change score data. The change improvement analyzes the change points, for example using clustering or segmentation, to estimate the boundaries of the objects. In some embodiments, the change improvement determines whether an object has changed by utilizing the average of all potentially existing change scores within the estimated object boundaries of the potential object. In some embodiments, the change improvement determines whether an object has changed by determining whether the percentage of change points within the estimated object boundaries of the potential object exceeds a threshold score having a change score above the threshold. In some embodiments, the user can select an algorithm for performing change improvement based on the sensors used to capture the input signal. For example, in some embodiments, using the percentage of change points above the change score threshold is more likely to produce accurate results when the sensor generates a noisy input signal compared to the average change score algorithm.

[0042] The improvement of operation 320 outputs the final change point stored in the change 3D map 120. The change 3D map 120 and the reference 3D map 125 are the same as the change 3D map 120 and the reference 3D map 125 described above, and will not be described in detail here for the sake of brevity.

[0043] In operation 325, an instruction is output based on the final change point determined by the change improvement 320. The instruction is generated based on an updated map formed based on the comparison between the change 3D map 120 and the reference 3D map 125. In some embodiments, the updated map is stored in a non-transitory computer-readable medium such as the memory of the mapping system 1000 (FIG. 10) described below. In some embodiments, the instruction is output using the downstream task module 135 (FIG. 1). In some embodiments, the instruction is output using a device other than the downstream task module 135 (FIG. 1). In some embodiments, the instruction is directly transmitted to an external device, such as a vehicle, to control the external device based on the updated map. In some embodiments, the instruction is provided to an external controller that can be used to control the external device. In some embodiments, the instruction is transmitted wirelessly. In some embodiments, the instruction is transmitted via a wired connection.

[0044] Those skilled in the art will recognize that the method 300 is adjustable. In some embodiments, at least one operation is added to the method 300. For example, in some embodiments, the method 300 further includes an operation of generating an updated map. In some embodiments, at least one operation is omitted from the method 300. For example, in some embodiments, operation 325 is omitted, and the change 3D map 120 is stored in a non-transitory computer-readable medium for use by a separate method. In some embodiments, the order of operations of the method 300 is adjusted. For example, in some embodiments, operation 320 is included as part of the iterative operation for each frame.

[0045] Using method 300, the change 3D map 120 has higher accuracy compared to other methods because the determination of changes is performed for the entire object. Further, maintaining the reference 3D map 125 in its original state during the mapping of the scene helps reduce or prevent the propagation of errors due to multiple iterations of scene mapping. As a result, the instructions given to the external device based on the change 3D map 120 are more accurate than the instructions using maps generated using other systems.

[0046] FIG. 4 is a schematic diagram of a mapping system 400 according to some embodiments. The mapping system 400 includes elements similar to the mapping system 100 (FIG. 1). Elements having the same reference numerals in the mapping system 400 are similar to the corresponding elements having the same reference numerals in the mapping system 100 (FIG. 1). The description of elements having the same reference numerals is abbreviated for brevity.

[0047] Compared to the mapping system 100 (FIG. 1), the mapping system 400 includes a segmentation module 405 configured to receive both image data and depth data from an input signal. The segmentation module 405 is configured to output two-dimensional (2D) segments to a segment-based improvement module 410. Further, the segmentation module 405 is configured to output depth data to a change detection module 110. In some embodiments, the change detection module 110 is configured to directly receive depth data from the input signal without the input signal passing through the segmentation module 405. The segment-based improvement module 410 is configured to receive the 2D segments from the segmentation module 405, as well as change points and change scores from the change detection module 110. The segment-based improvement module 410 is configured to output final change points to the change 3D map 120.

[0048] The segmentation module 405 is configured to analyze an input signal that includes only depth data or both image data and depth data in order to identify objects in a scene. The segmentation module 405 utilizes an algorithm to classify the pixels of the input signal and helps to identify the boundaries of the objects in the scene. In some embodiments, the segmentation module 405 utilizes a k-means clustering algorithm, a fuzzy c-means clustering (FCM) algorithm, a neural network, or another suitable algorithm. The segmentation module 405 identifies the boundaries of the objects and outputs 2D segments that can be used for segment-based improvement in order to improve the accuracy of the determination of changes in the scene. The 2D segments include the boundaries of the objects identified by the segmentation module 405. In some embodiments, the segmentation module 405 is configured to generate 3D segments, but generating 3D segments utilizes more processing load than generating 2D segments.

[0049] The segmentation module 405 helps to improve the identification of objects in the mapping system 400 as compared to other techniques that do not include image segmentation. However, the segmentation module 405 increases the processing load of the mapping system 400 as compared to other techniques that do not include image segmentation. The segmentation module 405 is implemented using one or more processors, such as the processor described with respect to the following mapping system 1000 (FIG. 10). In some embodiments, the segmentation module 405 is implemented using the same processor as the registration module 105, the change detection module 110, the segment-based improvement module 410, and the downstream task module 135. In some embodiments, the segmentation module 405 is implemented using a processor different from the registration module 105, the change detection module 110, the segment-based improvement module 410, or the downstream task module 135.

[0050] The segment-based improvement module 410 is configured to receive 2D segments from the segmentation module 405, change points from the change detection module 110, and change scores. The segment-based improvement module 410 functions in the same manner as the above-described change improvement module 115 (FIG. 1). Similar to the change improvement module 115 (FIG. 1), the segment-based improvement module 410 analyzes the received data to identify geometrically meaningful objects in the scene to determine changes in the scene. Compared with the change improvement module 115 (FIG. 1), the segment-based improvement module 410 can compare the change points and change scores with the 2D segments to improve the accuracy of change determination. Returning to the non-limiting example of the box in the above-described table, the 2D segment is useful for more accurately identifying the bottom of the box than depending only on the change score as described above for the change improvement module 115 (FIG. 1). The 2D segment provides data related to the determined position of the bottom of the box to assist in determining whether the box is a geometrically meaningful object that can be used to indicate a change in the scene. After determining whether there is one or more changes in the scene based on the input signal, the segment-based improvement module 410 outputs the final change points to the change 3D map 120.

[0051] The segment-based improvement module 410 is implemented using one or more processors, such as a processor described with respect to the following mapping system 1000 (FIG. 10). In some embodiments, the segment-based improvement module 410 is implemented using the same processor as the registration module 105, the change detection module 110, the segmentation module 405, and the downstream task module 135. In some embodiments, the segment-based improvement module 410 is implemented using a processor different from the registration module 105, the change detection module 110, the segmentation module 405, or the downstream task module 135.

[0052] Using the mapping system 400, the change 3D map 120 has higher accuracy compared to other methods because the determination of change is performed for the entire object. Including segmentation analysis in the mapping system 400 helps to further improve the accuracy of object identification. Additionally, maintaining the reference 3D map 125 in its original state during the mapping of the scene helps to reduce or prevent the propagation of errors due to multiple iterations of scene mapping. As a result, the instructions given to the external device based on the change 3D map 120 are more accurate than the instructions using maps generated using other systems.

[0053] Figure 5 is a flowchart of a method 500 of using a mapping system according to some embodiments. The method 500 is performed by a mapping system to generate a change map and generate instructions for downstream implementation. In some embodiments, the method 500 is performed using the mapping system 400 (FIG. 4). In some embodiments, the method 500 is performed using a mapping system other than the mapping system 400 (FIG. 4). The method 500 includes elements similar to the method 300 (FIG. 3). Elements having the same reference numerals in the method 500 are similar to the corresponding elements having the same reference numerals in the method 300 (FIG. 3). The description of elements having the same reference numerals is abbreviated for brevity.

[0054] Compared to the method 300 (FIG. 3), the method 500 includes an operation 505 of performing segmentation of the input data and generating 2D segments. The method 500 also includes an operation 510 of performing an improvement of the change based on the change points and change scores and the 2D segments to assist in improving the accuracy of object identification.

[0055] In operation 505, segmentation is performed on the input signal. The segmentation is performed using only depth data, or both image data and depth data. The segmentation contributes to identifying the boundaries of potential objects in the scene to assist in the determination of changes. The segmentation uses an algorithm to classify the pixels of the input signal and helps to identify the boundaries of the objects in the scene. In some embodiments, the segmentation uses a k-means clustering algorithm, an FCM algorithm, or another suitable algorithm. The segmentation outputs 2D segments that can be used by segment-based improvements to identify the boundaries of the objects and improve the accuracy of the determination of changes in the scene. The 2D segments include the boundaries of the objects identified during segmentation.

[0056] In some embodiments, operation 505 is performed using the segmentation module 405 (FIG. 4) of the mapping system 400 (FIG. 4). In some embodiments, operation 505 is performed using a device other than the segmentation module 405 (FIG. 4). The segmentation helps to improve the identification of objects in method 500 compared to other techniques that do not include segmentation. However, the segmentation increases the processing load for implementing method 500 compared to other techniques that do not include segmentation.

[0057] In operation 510, a segment-based refinement is performed to assist in identifying geometrically meaningful objects in the scene and determining changes in the scene. The segment-based refinement is performed using the 2D segments from operation 505 in addition to the change points and change scores from operation 315. The segment-based refinement functions in a similar manner to the change refinement of operation 320 (FIG. 3). Similar to the change refinement of operation 320 (FIG. 3), the segment-based refinement of operation 510 analyzes the received data to identify geometrically meaningful objects in the scene to determine changes in the scene. Compared to the change refinement of operation 320 (FIG. 3), the segment-based refinement of operation 510 can compare the change points and change scores with the 2D segments to improve the accuracy of change determination. The 2D segments provide data related to the determined positions of the boundaries of potential objects to assist in determining whether the objects are geometrically meaningful objects that can be used to indicate changes in the scene. The segment-based refinement outputs the final change points to the change 3D map 120 following a determination of whether one or more changes exist in the scene based on the input signal. In some embodiments, operation 510 is performed using the segment-based refinement module 410 (FIG. 4) of the mapping system 400 (FIG. 4). In some embodiments, operation 510 is performed using a device other than the segment-based refinement module 410 (FIG. 4).

[0058] One of ordinary skill in the art will recognize that method 500 is adjustable. In some embodiments, at least one operation is added to method 500. For example, in some embodiments, method 500 further includes an operation of generating an updated map. In some embodiments, at least one operation is omitted from method 500. For example, in some embodiments, operation 325 is omitted, and the changed 3D map 120 is stored in a non-transitory computer-readable medium for use by a separate method. In some embodiments, the order of operations of method 500 is adjusted. For example, in some embodiments, operation 505 is executed before operation 315, and change detection is performed based on the 2D segments output from operation 505.

[0059] Using method 500, the changed 3D map 120 has higher accuracy compared to other methods because the determination of change is performed for the entire object. Including segmentation analysis in the mapping system 400 helps to further improve the accuracy of object identification. Further, maintaining the reference 3D map 125 in its original state during scene mapping helps to reduce or prevent the propagation of errors due to multiple iterations of scene mapping. As a result, the instructions given to the external device based on the changed 3D map 120 are more accurate than the instructions using maps generated using other systems.

[0060] FIG. 6 is a schematic diagram of a mapping system 600 according to some embodiments. The mapping system 600 includes elements similar to those of the mapping system 400 (FIG. 4). Elements having the same reference numerals in the mapping system 600 are similar to the corresponding elements having the same reference numerals in the mapping system 400 (FIG. 4). The description of elements having the same reference numerals is abbreviated for brevity.

[0061] Compared with the mapping system 400 (FIG. 4), the mapping system 600 is configured to receive an input signal lacking depth data. As a result, the mapping system 600 includes a 3D reconstruction module 605 configured to receive a global pose from the registration module 105 and generate 3D map data usable by the change detection module 110 to generate change points and change scores.

[0062] The 3D reconstruction module 605 is configured to receive image data and a global pose to generate 3D map data. In some embodiments, when generating an input signal using multiple sensors, the accuracy of the 3D map data is improved. Based on the known positions of the sensors by the global pose, the 3D reconstruction module 605 can determine the relative distances between the objects of the image data. Based on these relative distances, the 3D reconstruction module 605 can generate 3D map data usable by the change detection module 110.

[0063] The 3D reconstruction module 605 assists in performing scene mapping using low-cost sensors lacking depth data collection. This enables the mapping system 600 to be utilized in a wider variety of situations. The 3D reconstruction module 605 is implemented using one or more processors, such as the processor described with respect to the following mapping system 1000 (FIG. 10). In some embodiments, the 3D reconstruction module 605 is implemented using the same processor as the registration module 105, the change detection module 110, the segmentation module 405, the segment-based improvement module 410, and the downstream task module 135. In some embodiments, the 3D reconstruction module 605 is implemented using a processor different from the registration module 105, the change detection module 110, the segmentation module 405, the segment-based improvement module 410, or the downstream task module 135.

[0064] In some embodiments, the change detection module 110 that utilizes 3D map data uses a point-to-point distance thresholding technique to compensate for inaccuracies in the reconstruction of the 3D map data. The point-to-point thresholding technique helps reduce the risk of false positives when determining change points and change scores because the input signal does not contain depth data.

[0065] In some embodiments, the segmentation module 405 is omitted from the mapping system 600. In some embodiments where the segmentation module 405 is omitted, the mapping system utilizes a change improvement module 115 (FIG. 1) instead of the segment-based improvement module 410.

[0066] Using the mapping system 600, the change 3D map 120 has higher accuracy compared to other techniques because the determination of changes is performed for the entire object. Including 3D reconstruction helps in using the mapping system 600 in situations where sensors capable of capturing depth data are not available. Further, maintaining the reference 3D map 125 in its original state during the mapping of the scene helps reduce or prevent the propagation of errors due to multiple iterations of scene mapping. As a result, the commands given to the external device based on the change 3D map 120 are more accurate than the commands using maps generated using other systems.

[0067] Figure 7 is a flowchart of a method 700 of using a mapping system according to some embodiments. Method 700 is implemented by a mapping system to generate a change map and generate instructions for downstream implementation. In some embodiments, method 700 is implemented using mapping system 600 (FIG. 6). In some embodiments, method 700 is implemented using a mapping system other than mapping system 600 (FIG. 6). Method 700 includes elements similar to method 500 (FIG. 5). Elements having the same reference numerals in method 700 are similar to the corresponding elements having the same reference numerals in method 500 (FIG. 5). The description of elements having the same reference numerals is abbreviated for brevity.

[0068] Compared with method 500 (FIG. 5), method 700 includes an operation 705 for performing 3D reconstruction of image data. In operation 705, 3D map data is reconstructed based on the image data and the global pose determined in operation 310. In some embodiments, when using multiple sensors to generate input signals, the accuracy of the 3D map data is improved. Based on the known positions of the sensors, the global pose from operation 310 is used to determine the relative distances between the objects in the image data. Based on these relative distances, a 3D map for determining scene changes is generated.

[0069] In some embodiments, the operation 505 for performing segmentation is omitted from method 700. In some embodiments where operation 505 is omitted, method 700 utilizes operation 320 (FIG. 3) instead of operation 510.

[0070] One skilled in the art will recognize that method 700 is adjustable. In some embodiments, at least one operation is added to method 700. For example, in some embodiments, method 700 further includes an operation of generating an updated map. In some embodiments, at least one operation is omitted from method 700. For example, in some embodiments, operation 325 is omitted, and the changed 3D map 120 is stored in a non-transitory computer-readable medium for use by a separate method. In some embodiments, the order of operations of method 700 is adjusted. For example, in some embodiments, operation 505 is executed before operation 315, and the detection of changes is executed based on the 2D segments output from operation 505.

[0071] Using method 700, the changed 3D map 120 has higher accuracy compared to other techniques because the determination of changes is executed for the entire object. Including 3D reconstruction enables the use of method 700 in situations where sensors capable of capturing depth data are not available. Further, maintaining the reference 3D map 125 in its original state during the mapping of the scene helps reduce or prevent the propagation of errors due to multiple iterations of scene mapping. As a result, the commands given to the external device based on the changed 3D map 120 are more accurate than the commands using maps generated using other systems.

[0072] FIG. 8 is a schematic diagram of a mapping system 800 according to some embodiments. The mapping system 800 includes elements similar to those of the mapping system 100 (FIG. 1). Elements having the same reference numerals in the mapping system 800 are similar to the corresponding elements having the same reference numerals in the mapping system 100 (FIG. 1). The description of elements having the same reference numerals is abbreviated for brevity.

[0073] Compared with the mapping system 100 (FIG. 1), the mapping system 800 is configured to provide the updated map generated by the map update module 130 to the change detection module 110. By supplying the updated map to the change detection module 110, the mapping system 800 can take into account temporary objects in the scene. For example, if an object does not exist during the generation of the reference 3D map 125, the object is added later, detected by the change detection module 110, and the object is identified as a change. Following the detection of an object, if the object is removed during subsequent mapping of the scene, the comparison between the reference 3D map and the input signal (after removing the object) indicates no change. As a result, no change points occur, and the risk of failing to remove the object when generating the updated map increases. By feeding back the updated map to the change detection module 110, the change detection module 110 can generate change points and change scores based on the newer map.

[0074] In some embodiments, the feedback of the updated map is determined based on a query to the change 3D map 120 from a previous mapping of the scene. In response to a determination that the change 3D map 120 is empty, i.e., there is no change from the reference 3D map, the updated map is not fed back to the change detection module 110. The change detection module 110 generates change points and change scores based on a comparison with the reference 3D map 125. In response to a determination that the change 3D map 120 contains at least one change, the updated map is fed back to the change detection module 110.

[0075] Those skilled in the art will recognize that the feedback of the updated map from the map update module 130 to the change detection module 110 can also be used in the mapping system 400 (FIG. 4) and the mapping system 600 (FIG. 6).

[0076] Using the mapping system 800, the changed 3D map 120 has higher accuracy compared to other methods because the determination of changes is performed for the entire object. Including the feedback of the updated map in the change detection module 110 helps to account for temporary objects in the scene. Further, maintaining the reference 3D map 125 in its original state during the mapping of the scene helps to reduce or prevent the propagation of errors due to multiple iterations of scene mapping. As a result, the commands given to the external device based on the changed 3D map 120 are more accurate than the commands using maps generated using other systems.

[0077] Figure 9 is a flowchart of a method 900 of using a mapping system according to some embodiments. The method 900 is performed by a mapping system to generate a change map and generate commands for downstream implementation. In some embodiments, the method 900 is performed using the mapping system 800 (FIG. 8). In some embodiments, the method 900 is performed using a mapping system other than the mapping system 800 (FIG. 8). The method 900 includes elements similar to the method 300 (FIG. 3). Elements having the same reference numerals in the method 900 are similar to the corresponding elements having the same reference numerals in the method 300 (FIG. 3). The description of the elements having the same reference numerals is abbreviated for brevity.

[0078] Compared to the method 300 (FIG. 3), the method 900 includes an updated map 130 generated based on the comparison between the reference 3D map 125 and the changed 3D map 120. The updated map 130 is used in operation 315 to assist in accounting for temporary objects in the scene.

[0079] In some embodiments, the use of the updated map 130 in operation 315 is determined based on a query to the change 3D map 120 from a previous mapping of the scene. In response to a determination that the change 3D map 120 is empty, i.e., there is no change from the reference 3D map, the updated map is not used in operation 315. Operation 315 instead relies on the reference 3D map 125. In response to a determination that the change 3D map 120 includes at least one change, the updated map is used in operation 315.

[0080] One of ordinary skill in the art will recognize that the feedback of the updated map 130 to operation 315 can also be used in method 500 (FIG. 5) and method 700 (FIG. 7).

[0081] One of ordinary skill in the art will recognize that method 900 is adjustable. In some embodiments, at least one operation is added to method 900. For example, in some embodiments, method 900 further includes an operation of generating an updated map. In some embodiments, at least one operation is omitted from method 900. For example, in some embodiments, the use of the updated map 130 in operation 315 is omitted if the change 3D map 120 from the previous scene mapping is empty. In some embodiments, the order of operations of method 900 is adjusted. For example, in some embodiments, operation 320 is included as part of the iterative operation for each frame.

[0082] Using method 900, the change 3D map 120 has higher accuracy compared to other techniques because the determination of change is performed for the entire object. Using the updated map 130 helps method 900 account for temporary objects in the scene. Further, maintaining the reference 3D map 125 in its original state during the mapping of the scene helps reduce or prevent the propagation of errors due to multiple iterations of scene mapping. As a result, the commands given to the external device based on the change 3D map 120 are more accurate than commands using maps generated using other systems.

[0083] Figure 10 is a block diagram of a mapping system 1000 according to some embodiments. The mapping system 1000 includes a hardware processor 1002 and a non-transitory computer-readable storage medium 1004 encoded with, i.e., storing, computer program code 1006, i.e., a set of executable instructions. The computer-readable storage medium 1004 is also encoded with instructions 1007 for interfacing with external devices. The processor 1002 is electrically coupled to the computer-readable storage medium 1004 via a bus 1008. The processor 1002 is also electrically coupled by the bus 1008 to an input / output (I / O) interface 1010. A network interface 1012 is also electrically connected to the processor 1002 via the bus 1008. A network interface 1021 is connected to a network 1014 such that the processor 1002 and the computer-readable storage medium 1004 can be connected to external elements via the network 1014. The processor 1002 is configured to execute computer program code 1006 encoded in the computer-readable storage medium 1004 to enable the mapping system 1000 to perform some or all of the operations described in mapping system 100 (FIG. 1), mapping system 200 (FIG. 2), method 300 (FIG. 3), mapping system 400 (FIG. 4), method 500 (FIG. 5), mapping system 600 (FIG. 6), method 700 (FIG. 7), mapping system 800 (FIG. 8), or method 900 (FIG. 9).

[0084] In some embodiments, the processor 1002 is a central processing unit (CPU), a multiprocessor, a distributed processing system, an application specific integrated circuit (ASIC), and / or a suitable processing device.

[0085] In some embodiments, the computer-readable storage medium 1004 is an electronic, magnetic, optical, electromagnetic, infrared, and / or semiconductor system (or apparatus or device). For example, the computer-readable storage medium 1004 includes semiconductor or solid state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and / or optical disks. In some embodiments that use optical disks, the computer-readable storage medium 1004 includes compact disk read-only memory (CD-ROM), compact disk read / write (CD-R / W), and / or digital video disk (DVD).

[0086] In some embodiments, the memory medium 1004 stores computer program code 1006 configured to cause the mapping system 1000 to execute some or all of the operations described in mapping system 100 (FIG. 1), mapping system 200 (FIG. 2), method 300 (FIG. 3), mapping system 400 (FIG. 4), method 500 (FIG. 5), mapping system 600 (FIG. 6), method 700 (FIG. 7), mapping system 800 (FIG. 8), or method 900 (FIG. 9). In some embodiments, the memory medium 1004 also stores information used for the execution of some or all of the operations described in mapping system 100 (FIG. 1), mapping system 200 (FIG. 2), method 300 (FIG. 3), mapping system 400 (FIG. 4), method 500 (FIG. 5), mapping system 600 (FIG. 6), method 700 (FIG. 7), mapping system 800 (FIG. 8), or method 900 (FIG. 9), as well as information generated during the execution of some or all of the operations as described in mapping system 100 (FIG. 1), mapping system 200 (FIG. 2), method 300 (FIG. 3), mapping system 400 (FIG. 4), method 500 (FIG. 5), mapping system 600 (FIG. 6), method 700 (FIG. 7), mapping system 800 (FIG. 8), or method 900 (FIG. 9), such as depth parameters 1016, image parameters 1018, reference map parameters 1020, change map parameters 1022, update map parameters 1024, and / or a set of executable instructions to execute some or all of the operations as described in mapping system 100 (FIG. 1), mapping system 200 (FIG. 2), method 300 (FIG. 3), mapping system 400 (FIG. 4), method 500 (FIG. 5), mapping system 600 (FIG. 6), method 700 (FIG. 7), mapping system 800 (FIG. 8), or method 900 (FIG. 9).

[0087] In some embodiments, the storage medium 1004 stores instructions 1007 for interfacing with an external device. The instructions 1007 enable the processor 1002 to generate instructions readable by the external device to effectively perform some or all of the operations described in mapping system 100 (FIG. 1), mapping system 200 (FIG. 2), method 300 (FIG. 3), mapping system 400 (FIG. 4), method 500 (FIG. 5), mapping system 600 (FIG. 6), method 700 (FIG. 7), mapping system 800 (FIG. 8), or method 900 (FIG. 9).

[0088] The mapping system 1000 includes an I / O interface 1010. The I / O interface 1010 is coupled to an external circuit. In some embodiments, the I / O interface 1010 includes a keyboard, keypad, mouse, trackball, trackpad, and / or cursor direction keys for communicating information and commands to the processor 1002.

[0089] The mapping system 1000 also includes a network interface 1012 coupled to a processor 1002. The network interface 1012 enables the mapping system 1000 to communicate with a network 1014 to which one or more other computer systems are connected. The network interface 1012 includes a wireless network interface such as BLUETOOTH®, WIFI, WIMAX, GPRS, or WCDMA®, or a wired network interface such as ETHERNET, USB, IEEE-1394. In some embodiments, some or all of the operations described in mapping system 100 (FIG. 1), mapping system 200 (FIG. 2), method 300 (FIG. 3), mapping system 400 (FIG. 4), method 500 (FIG. 5), mapping system 600 (FIG. 6), method 700 (FIG. 7), mapping system 800 (FIG. 8), or method 900 (FIG. 9) are implemented in two or more mapping systems 1000, and information such as depth parameter 1016, image parameter 1018, reference map parameter 1020, change map parameter 1022, or update map parameter 1024 is exchanged between different mapping systems 1000 via the network 1014.

[0090] Some or all of the above embodiments may also be described as follows, but are not limited thereto.

[0091] (Appendix 1) A non-transitory computer-readable medium configured to store instructions, and a processor coupled to the non-transitory computer-readable medium, the processor receives an input signal including image data of a scene, determines a position of a sensor used to capture the image data with respect to a reference map of the scene, determines a change point and a change score of the scene based on the determined position of the sensor and the reference map, generates a change map based on the change point and the change score, Generate an update map based on a comparison between the change map and the reference map, and maintain the content of the reference map without change A mapping system configured to execute the instructions for this purpose.

[0092] (Appendix 2) The processor is further configured to execute the instructions for generating the change map by determining whether the change points and the change scores indicate a geometrically meaningful shape in the image data using the change points and the change scores. The mapping system according to Appendix 1, further configured to execute the instructions for this purpose.

[0093] (Appendix 3) The processor is further configured to execute the instructions for generating a change map that does not indicate a change related to the point with respect to the reference map in response to the change points and the change scores not indicating the geometrically meaningful shape. The mapping system according to Appendix 1 or Appendix 2, further configured to execute the instructions for this purpose.

[0094] (Appendix 4) The processor is further configured to execute the instructions for generating a change map that indicates a change related to the point with respect to the reference map in response to the change points and the change scores indicating the geometrically meaningful shape. The mapping system according to any one of Appendices 1 to 3, further configured to execute the instructions for this purpose.

[0095] (Appendix 5) The processor is further configured to receive the input signal including depth data The mapping system according to any one of Appendices 1 to 4, further configured to execute the instructions for this purpose.

[0096] (Appendix 6) The processor is further configured to execute the instructions for: segmenting the input signal to generate a two-dimensional (2D) segment, and generating the change map using the 2D segment, wherein the mapping system is as described in any one of Appendices 1 to 5.

[0097] (Appendix 7) The processor is further configured to execute the instructions for: reconstructing three-dimensional (3D) data based on the input signal, and determining the change points and the change scores based on the reconstructed 3D data, wherein the mapping system is as described in any one of Appendices 1 to 6.

[0098] (Appendix 8) receiving an input signal including image data of a scene, determining the position of a sensor used to capture the image data with respect to a reference map of the scene, determining change points and change scores of the scene based on the determined position of the sensor and the reference map, generating a change map based on the change points and the change scores, generating an update map based on a comparison between the change map and the reference map, and maintaining the content of the reference map without changing it, wherein the method includes using a mapping system.

[0099] (Appendix 9) Generating the change map includes using the change points and the change scores to determine whether the change points and the change scores indicate a geometrically meaningful shape in the image data, as described in Appendix 8.

[0100] (Appendix 10) Generating the change map includes, in response to the change points and the change scores not indicating the geometrically meaningful shape, not indicating the change related to the points with respect to the reference map, a method of using the mapping system described in Appendix 8 or Appendix 9.

[0101] (Appendix 11) Generating the change map includes, in response to the change points and the change scores indicating the geometrically meaningful shape, indicating the change related to the points with respect to the reference map, a method of using the mapping system described in any one of Appendix 8 to Appendix 10.

[0102] (Appendix 12) Receiving the input signal includes receiving depth data, a method of using the mapping system described in any one of Appendix 8 to Appendix 11.

[0103] (Appendix 13) Segmenting the input signal to generate a two-dimensional (2D) segment, Generating the change map using the 2D segment and further includes a method of using the mapping system described in any one of Appendix 8 to Appendix 12.

[0104] (Appendix 14) Reconstructing three-dimensional (3D) data based on the input signal, Determining the change points and the change scores based on the reconstructed 3D data and further includes a method of using the mapping system described in any one of Appendix 8 to Appendix 13.

[0105] (Appendix 15) Receiving an input signal including image data of a scene, Determining the position of the sensor used to capture the image data with respect to the reference map of the scene, Determine a change point and a change score of the scene based on the determined position of the sensor and the reference map, generate a change map based on the change point and the change score, generate an update map based on a comparison between the change map and the reference map, maintain the content of the reference map without changing it A program for causing a processor to execute the above.

[0106] (Appendix 16) The program according to Appendix 15, wherein the processor is caused to generate the change map by using the change point and the change score to determine whether the change point and the change score indicate a geometrically meaningful shape in the image data.

[0107] (Appendix 17) The program according to Appendix 15 or 16, wherein the processor is caused to generate the change map that does not indicate a change related to the point with respect to the reference map in response to the change point and the change score not indicating the geometrically meaningful shape.

[0108] (Appendix 18) The program according to any one of Appendices 15 to 17, wherein the processor is caused to generate the change map that indicates a change related to the point with respect to the reference map in response to the change point and the change score indicating the geometrically meaningful shape.

[0109] (Appendix 19) segment the input signal to generate a two-dimensional (2D) segment, generate the change map by using the 2D segment The program according to any one of Appendices 15 to 18, for causing the processor to execute the above.

[0110] (Appendix 20) Reconstruct 3D data based on the input signal, Determine the change points and the change scores based on the reconstructed 3D data The program according to any one of Appendices 15 to 19, which causes the processor to perform the above.

[0111] (Appendix 21) The mapping system includes a non-transitory computer-readable medium configured to store instructions. The mapping system includes a processor connected to the non-transitory computer-readable medium. The processor is configured to execute instructions for receiving an input signal including image data of a scene. The processor is configured to execute instructions for determining the position of a sensor used to capture image data relative to a reference map of the scene. The processor is configured to execute instructions for determining change points and change scores of the scene based on the determined position of the sensor, the reference map, and a first change map from a previous mapping of the scene. The processor is configured to execute instructions for generating a second change map based on the change points and the change scores. The processor is configured to execute instructions for generating an updated map based on a comparison between the second change map and the reference map. The processor is configured to execute instructions for maintaining the content of the reference map without change.

[0112] (Appendix 22) The mapping system includes a non-transitory computer-readable medium configured to store instructions. The mapping system includes a processor connected to the non-transitory computer-readable medium. The processor is configured to execute instructions for receiving an input signal of a scene, where the input signal includes image data and depth data. The processor is configured to execute instructions for determining a position of a sensor used to capture the image data relative to a reference map of the scene based on the image data. The processor is configured to execute instructions for determining change points and change scores of the scene based on the depth data, the determined position of the sensor, and the reference map. The processor is configured to execute instructions for determining whether an object is a geometrically meaningful object based on the change points and the change scores. In response to determining that the object is a geometrically meaningful object, the processor is configured to execute instructions for generating an updated map. In response to determining that the object is not a geometrically meaningful object, the processor is configured to execute instructions for not indicating a change to the reference map.

[0113] (Appendix 23) The mapping system includes a non-transitory computer-readable medium configured to store instructions. The mapping system further includes a processor connected to the non-transitory computer-readable medium. The processor is configured to execute instructions for receiving an input signal including image data of a scene. The processor is configured to execute instructions for determining the position of a sensor used to capture the image data relative to a reference map of the scene. The processor is configured to execute instructions for segmenting the image data to generate two-dimensional (2D) segments. The processor is configured to execute instructions for determining change points and change scores of the scene based on the determined position of the sensor and the reference map. The processor is configured to execute instructions for generating a change map based on the change points and change scores and the 2D segments. The processor is configured to execute instructions for generating an updated map based on a comparison of the change map and the reference map. The processor is configured to execute instructions for maintaining the content of the reference map without changing it.

[0114] (Appendix 24) The mapping system includes a non-transitory computer-readable medium configured to store instructions. The mapping system includes a processor connected to the non-transitory computer-readable medium. The processor is configured to execute instructions for generating a reference map based on data from a first sensor, the first sensor having a first resolution. The processor is configured to execute instructions for receiving an input signal including image data of a scene. The processor is configured to execute instructions for determining a position of a second sensor used to capture the image data relative to the reference map of the scene, the second sensor having a second resolution smaller than the first resolution. The processor is configured to execute instructions for determining a change point and a change score of the scene based on the determined position of the sensor and the reference map. The processor is configured to execute instructions for generating a change map based on the change point and the change score. The processor is configured to execute instructions for generating an updated map based on a comparison between the change map and the reference map. The processor is configured to execute instructions for maintaining the content of the reference map without change.

[0115] The foregoing outlines features of several embodiments so that those skilled in the art may better understand the aspects of the present disclosure. Those skilled in the art should appreciate that they may readily use the present disclosure as a basis for designing or modifying other processes and structures for carrying out the same purposes and / or achieving the same advantages as the embodiments introduced herein. Those skilled in the art should also understand that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that various changes, substitutions, and alterations may be made therein without departing from the spirit and scope of the present disclosure.

[0116] This application claims priority to U.S. Patent Application No. 18 / 180,861, filed Mar. 9, 2023, the entire disclosure of which is hereby incorporated herein by reference.

Description of Reference Numerals

[0117] 100 Mapping System 105 Registration Module 110 Change Detection Module 115 Change Improvement Module 120 Change 3D Map 125 Reference 3D Map 130 Map Update Module 135 Downstream Task Module 200 Mapping System 205 Map Creation Module 300 Method 400 Mapping System 405 Segmentation Module 410 Segment-Based Improvement Module 500 Method 600 Mapping System 605 3D Reconstruction Module 700 Method 800 Mapping System 900 Method 1000 Mapping System 1002 Processor 1004 Computer-Readable Storage Medium 1006 Computer Program Code 1007 Instruction 1008 Bus 1010 I / O Interface 1012 Network Interface 1014 Network 1016 Depth Parameter 1018 Image Parameter 1020 Reference Map Parameter 1021 Network Interface 1022 Change Map Parameter 1024 Update Map Parameter

Claims

1. a non-transitory computer-readable medium configured to store instructions; a processor coupled to the non-transitory computer readable medium, the processor comprising: receiving an input signal containing image data of a scene; Utilizing permanent objects or points in a reference map of the scene to determine the location of a sensor used to capture the image data; determining change points of the scene based on the determined positions of the sensors, the reference map and the image data, and generating a change score indicative of the likelihood that the change points are actual changes based on a depth difference between the reference map and the image data; Segmenting the input signal to generate two-dimensional (2D) segments; generating a change map by comparing the change points and the change scores to the 2D segment that includes a boundary of the object; The contents of said reference map are maintained unchanged. A mapping system configured to execute the instructions for:

2. The processor, Utilizing the change points and the change scores to determine whether the change points indicate geometrically significant shapes in the image data, thereby generating the change map. The mapping system of claim 1 , further configured to execute the instructions for:

3. The processor, generating the change map in response to the change points not exhibiting the geometrically meaningful shape, the change map not exhibiting the changes associated with the points relative to the reference map; The mapping system of claim 2 , further configured to execute the instructions for:

4. The processor, generating a change map indicative of changes for the points relative to the reference map in response to the change points indicating the geometrically significant shape; The mapping system of claim 2 , further configured to execute the instructions for:

5. The processor, receiving the input signal including depth data; The mapping system of claim 1 or claim 2, further configured to execute the instructions for:

6. The processor, reconstructing three-dimensional (3D) data based on the input signals; Determining the change points based on the reconstructed 3D data and generating the change scores. The mapping system of claim 1 or claim 2, further configured to execute the instructions for:

7. Receiving an input signal including image data of a scene; Utilizing permanent objects or points in a reference map of the scene to determine a position of a sensor used to capture the image data; determining change points of the scene based on the determined positions of the sensors, the reference map and the image data, and generating a change score indicative of the likelihood that the change points are real changes based on a depth difference between the reference map and the image data; segmenting the input signal to generate two-dimensional (2D) segments; generating a change map by comparing the change points and the change scores to the 2D segment that includes a boundary of the object; maintaining the content of said reference map unchanged; A method of using a mapping system, comprising:

8. 8. The method of claim 7, wherein generating the change map comprises utilizing the change points and the change scores to determine whether the change points indicate geometrically meaningful shapes in the image data.

9. receiving an input signal containing image data of a scene; Utilizing permanent objects or points in a reference map of the scene to determine the location of a sensor used to capture the image data; determining change points of the scene based on the determined positions of the sensors, the reference map and the image data, and generating a change score indicative of the likelihood that the change points are actual changes based on a depth difference between the reference map and the image data; Segmenting the input signal to generate two-dimensional (2D) segments; generating a change map by comparing the change points and the change scores to the 2D segment that includes a boundary of the object; The contents of said reference map are maintained unchanged. A program that causes a processor to do something.

Images