Method and system for camera-assisted mapping and navigation - Patents.com

Abstract

To provide a method and system for a camera assisted map and navigation.SOLUTION: A computer-implemented method and computer system for use in navigating in a facility are provided. For example, the computer-implemented method includes the steps of: receiving at least one image from a camera; estimating the current location of the camera in the facility on the basis of at least one image and model data of the facility by a processor; generating a virtual path from the current location of the camera to a destination location in the facility using the model data of the facility by the processor; and generating and outputting navigation information to the destination location according to the virtual path.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The subject disclosure relates generally to camera-assisted maps and navigation, and more particularly to methods and systems for use in navigating within facilities. [Background technology]

[0002] Typically, people find it difficult to navigate within a facility and therefore require special assistance. For example, travelers require help from maps, sign boards, customer care, other fellow travelers, indoor maps, etc. Current infrastructure and known systems do not yet address these issues.

[0003] Known systems for indoor guidance use a combination of the Global Positioning System (GPS) with other technologies such as Bluetooth, Infrared, Wi-Fi, RFID, etc. to provide detailed and accurate location information to users. For example, a representative system in this category is disclosed in US Pat. No. 9,539,164. However, such systems are not feasible due to the additional hardware required to provide the location information. Furthermore, GPS services are satellite-based and are not available within facilities because GPS services require line-of-sight to the satellites.

[0004] The subject disclosure addresses such shortcomings and provides methods and systems for use in navigating within a facility that do not require the use of additional hardware and do not rely on the availability of GPS services. Summary of the Invention [Means for solving the problem]

[0005] A first aspect of the subject disclosure provides a computer-implemented method for use in navigating a facility, the method including receiving at least one image from a camera; estimating, by a processor, a current location of the camera within the facility based on the at least one image and facility model data; generating, by the processor, a virtual route from the current location of the camera within the facility to a destination location using the facility model data; and generating and outputting navigation information to the destination location according to the virtual route.

[0006] In some examples, the facility model data includes image data for a plurality of images, with the image data for each image including location information corresponding to a location within the facility where the image was captured, object information corresponding to an object of the facility in the image, distance information corresponding to a distance between the object and the location, first relationship information defining a distance and relative direction for navigating from one object in the image to another object in the image as a first relationship, and second relationship information defining a distance and relative direction for navigating from the location where the image was captured to the location where another image was captured as a second relationship.

[0007] In some examples, estimating the current location includes dividing at least one image into one or more image blocks, by the processor; detecting, by the processor, object candidates within the one or more image blocks corresponding to facility objects based on object information from the facility model data; determining, by the processor, distance values ​​to the detected object candidates based on object information and distance information of the corresponding objects from the facility model data; determining, by the processor, a distance between the object candidates based on the distance values; and estimating, by the processor, a current location of the camera based on location information from the facility model data and the distance values ​​to the detected object candidates.

[0008] In some examples, estimating the current location further includes performing, by the processor, object classification on candidate objects in the image based on the distance value and the distance to detect objects in the facility.

[0009] In some examples, the computer-implemented method further includes receiving, via the input device, information regarding a destination object within a facility to which the camera is to navigate; searching, by the processor, for at least one image block of the image, object information in the model data of the facility that corresponds to the information regarding the destination object; and estimating, by the processor, a location of the destination object based on image data of the image including the destination object as a destination location.

[0010] In some examples, generating the virtual route includes determining, by the processor, a relationship between candidate objects in an image block of the image and the destination object based on first and second relationship information in the facility model data, and deriving, by the processor, the virtual route based on the determined relationship.

[0011] In some examples, outputting the navigation information includes displaying at least one of the image and the navigation information on a display.

[0012] In some examples, the computer-implemented method further includes generating, by the processor, model data of the facility. The generating step includes acquiring, by the camera, one or more images from a plurality of locations within the facility. The generating step includes, for each of the plurality of images, determining, by the processor, depth information based on the image and image information provided by the camera, generating, by the processor, location information based on the location at which the image was acquired, dividing, by the processor, the image into one or more image blocks, detecting, by the processor, objects of the facility in the one or more image blocks and generating object information defining characteristics of the detected objects, the object information including information indicative of the image blocks of the image, determining, by the processor, distances between the detected objects in the one or more image blocks and locations using the depth information and generating distance information corresponding to the detected objects in the image blocks, and determining, by the processor, distances between the detected objects in the one or more image blocks and how the distances between the detected objects in the one or more image blocks are determined. The method further includes the steps of: calculating a distance between a location where the image was captured and another location where the other image was captured based on location information of the image and the other image and the relative direction describing how to navigate from one location to another location, and generating first relationship information based on the distance and the relative direction, the first relationship information including information indicative of the first and second image blocks of the image; determining, by the processor, a distance between a location where the image was captured and another location where the other image was captured based on location information of the image and the other image and the relative direction describing how to navigate from one location to another location, and generating second relationship information based on the distance and the relative direction, the second relationship information including information indicative of the image and the other image; and generating, by the processor, image data for the image including the location information, the object information, the first relationship information, and the second relationship information.

[0013] In some examples, the computer-implemented method further includes storing, by the processor, the at least one image, and performing, by the processor, a machine learning operation using the at least one image and the model data of the facility to generate updated model data of the facility.

[0014] A second aspect of the subject disclosure provides a computing system for use in navigating a facility including a processor, a camera device, and at least one memory device accessible by the processor. The memory device includes a number of program instructions that, when executed by the processor, cause the computing system to perform a method including receiving at least one image from the camera, estimating a current location of the camera within the facility based on the at least one image and facility model data, generating a virtual route from the current location of the camera within the facility to a destination location using the facility model data, and generating and outputting navigation information to the destination location according to the virtual route.

[0015] In some examples, the system is further configured to perform a method as described in the examples of the first aspect of the subject disclosure.

[0016] According to a third aspect there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to perform a method according to the first aspect and examples thereof.

[0017] The above aspects and examples present a simplified summary to provide a basic understanding of some aspects of the methods and computing systems disclosed herein. This summary is not an extensive overview of the methods and computing systems disclosed herein. It is not intended to identify key / critical elements or to delineate the scope of such methods and computing systems. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

[0018] The accompanying drawings illustrate various examples of the subject disclosure, with the general description provided above and the detailed description of the examples provided below serving to explain the examples of the subject disclosure, in which like reference numerals are used to designate like parts in the various figures. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is an overview illustrating the high-level components of a computing system for use in navigating a facility. [Diagram 2] FIG. 2 illustrates an exemplary system that encompasses the computing system of FIG. 1. [Diagram 3] FIG. 1 illustrates a method for use in navigating a facility, according to an example. [Figure 4A] FIG. 4 illustrates an example for estimating the current location of the camera according to block 330 of the method of FIG. [Figure 4B] 4 illustrates an example for generating a virtual path according to block 340 of the method of FIG. 3. [Figure 4C] FIG. 4 illustrates an example for generating model data according to block 310 of the method of FIG. 3. [Diagram 5] FIG. 2 illustrates an example for dividing an image into one or more image blocks. [Figure 6A]4 illustrates an exemplary image received from a camera at block 320 of the method of FIG. 3. [Figure 6B] FIG. 6B illustrates the example image of FIG. 6A divided into image blocks. [Figure 6C] FIG. 6B illustrates an image block of the example image of FIG. 6A with detected object candidates. [Figure 6D] FIG. 6B illustrates detection of distance values ​​to object candidates within an image block of the example image of FIG. 6A. [Figure 6E] FIG. 6B illustrates classification of object candidates within an image block of the example image of FIG. 6A. [Figure 6F] FIG. 6B illustrates the derivation of relationships between objects within an image block of the example image of FIG. 6A. [Figure 6G] FIG. 6B shows the example image of FIG. 6A with image blocks marked as suitable for a path. [Figure 6H] FIG. 6B illustrates the example image of FIG. 6A with image blocks marked as virtual paths. [Figure 7A] FIG. 2 illustrates a first example of determining a relationship between a detected object and a destination object. [Figure 7B] FIG. 11 illustrates a second example of determining a relationship between a detected object and a destination object. [Figure 8] 3 is a diagrammatic representation of a computer system that provides the functionality of a computing system for use in navigating a facility such as that shown in FIG. 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Before proceeding with a detailed description of the examples, some broader aspects of the related techniques will first be described.

[0021] The subject disclosure generally relates to navigating within a facility. The term "navigate" has its general meaning and is specifically understood as determining a location and a direction to a destination. The term "facility" includes all types of buildings and structures. Examples of facilities include airport buildings having one or more floors (e.g., check-in areas and terminal buildings), hospitals, shopping malls, etc. The subject disclosure more specifically relates to navigating within a facility (i.e., indoors) from one location within the facility to another location within the facility. As will be appreciated, the subject disclosure is not limited to indoor navigation, but may also be used for outdoor navigation, i.e., navigating outside a facility, e.g., within a city. In the subject disclosure, elements, components, and objects within a facility are generally referred to as objects of the facility. Examples of objects include walls, pillars, doors, seats, sign boards, desks, kiosks, etc.

[0022] The subject disclosure employs machine learning (ML) techniques and applies to algorithms and statistical models used by computer systems to perform specific tasks without explicit instructions, instead relying primarily on patterns and inference. Machine learning algorithms build mathematical models based on sample data, known as "training data," to make predictions or decisions without being explicitly programmed to perform the task. The subject disclosure uses a mathematical model, referred to herein as model data of the facility, as its data foundation. The model data is based on multiple images from the facility.

[0023] As described, the subject disclosure uses object detection and object classification techniques to detect instances of a class of semantic objects, such as people, buildings, cars, etc., in digital images and videos. Each object class has its own special features that help classify the class. Techniques for object detection and object classification can be, for example, ML-based or deep learning-based. Known ML-based approaches include Histogram of Oriented Gradients (HOG) features. Object detection also includes feature extraction, which extracts features from a digital image, and feature recognition, which recognizes the extracted features that are useful features in classification.

[0024] FIG. 1 illustrates an overview illustrating the broad components of a computing system 100 for use in navigating a facility. The system 100 includes a component 110 that performs the function of receiving at least one image from a camera. The component 110 may also receive multiple images in sequence or multiple frames of a video. The received image / frame is pre-processed by a component 120 that performs one or more pre-processing functions such as creating a grid and dividing the image into image blocks, as well as augmentation including filters and perspectives. The image is then further processed using ML techniques by a collection of several components that interact with each other. The collection includes a component 130 for performing the functions of an object detection model, a component 140 for performing the functions of an object classification model, and a component 150 for performing the functions of a path classification model. The component 130 detects candidates for objects in the image (i.e., the component detects regions of interest in the image). The component 140 classifies the candidates for the object, thereby recognizing the object in the image. Component 150 identifies areas in the image that are suitable for a path (e.g., the floor of the facility is suitable, whereas the ceiling, walls, or obstacles are not suitable). Based on the detected candidates or objects, component 160 for performing a function of building relationships derives relationships between them. The relationships define a path for navigating from the current location to a destination location in the facility. Component 160 takes into account the output of component 150 and therefore also areas that are not suitable for a path. Component 170 for performing a function of building a map uses the relationships built by component 160 to, for example, generate a virtual path along which a user can travel from the current location to the destination location. Component 170 also generates navigation information according to the virtual path.Navigation information is provided and displayed to the user along with the image. Thus, the subject disclosure provides a camera-assisted map and navigation system that does not require additional hardware such as that found in U.S. Pat. No. 9,539,164.

[0025] FIG. 2 illustrates an exemplary system that encompasses the computing system of FIG. 1. The computing system 100 corresponds to the system shown in FIG. 1. That is, the computing system 100 may include a processor and at least one memory device accessible by the processor. The memory device includes a number of program instructions that, when executed by the processor, cause the computing system to perform a method including receiving at least one image, estimating a current location of a camera in a facility based on the at least one image and the facility model data, generating a virtual route from the current location of the camera in the facility to a destination location using the facility model data, and generating and outputting navigation information to the destination location according to the virtual route. The computing system 100 may communicate with a computing device 110, such as a mobile device, via any suitable communication connection. The computing device 110 may include a camera device 112, a display device 114, and an input device 116. The camera device 112 captures at least one image, and the computing device 110 transmits the at least one image to the computing system 110. The computing device 110 receives navigation information from the computing system 110 and displays the same on the display device 114, e.g., along with at least one captured image. Information regarding a destination location is input via the input device 116, and the computing device 110 transmits information regarding the destination location to the computing system 100. The computing system 100 and the computing device 110 may communicate, e.g., API request / response, with the backend system 120. The backend system 120 may be a continuous learning system and / or system for data analysis and prediction. For example, the backend system 120 generates and updates model data.

[0026] An exemplary method for use in navigating a facility will now be described in light of Figure 3. Note that not all blocks illustrated in Figure 3 are necessarily required to perform the exemplary method. At least the dashed blocks are optional and / or may be performed only once.

[0027] The method illustrated in Figure 3 may be performed by one or more computing devices, such as a personal computer, a server, a PDA, a smartphone, a mobile phone, a tablet, or the like. In one example, a computing system as described herein may be used to perform the method of Figure 3, the computing system including a processor and at least one memory device accessible by the processor. The memory device includes a number of program instructions that, when executed by the processor, cause the computing system to perform the method of Figure 3. The computing system may also include a camera, or the camera may be connected to the computing system. The computing system may also be connected to the computing device, or the camera may be connected to or incorporated in the computing device.

[0028] The method 300 begins at block 310 with generating model data for the facility. The process of generating the model data will be described in more detail below with reference to Figure 4C. As explained above, the model data forms the data basis for the method of Figure 3. The model data may be stored in a memory device of the computing system and thereby accessible by the processor.

[0029] In one example, the resulting facility model data in block 310 includes image data of a plurality of images. As described with reference to FIG. 4C, the plurality of images are captured by a camera from within the facility and used as sample or training data. The plurality of images are processed to generate a model of the facility that makes predictions or decisions using ML techniques. In particular, the model includes data regarding objects of the facility that enable the ML techniques to predict and determine that structures in the images correspond to particular objects of the facility. In the example, the image data of each image used as sample or training data includes location information corresponding to a location within the facility where the image was captured, object information corresponding to an object of the facility in the image, distance information corresponding to a distance between the object and the location, first relationship information that specifies a distance and a relative direction for navigating from one object in the image to the other object, and second relationship information that specifies a distance and a relative direction for navigating from a location where the image was captured to a location where another image was captured. Here, the first relationship information may also be referred to as a first relationship, and the second relationship information may also be referred to as a second relationship.

[0030] At block 320, at least one image (also referred to as a tile) is received from the camera. In one particular example, the at least one image may be represented by a frame of a video stream captured by a video camera. At a location within the facility, a camera is used to capture at least one image from the periphery of the location (i.e., a physical area in a direction from the location). The image may be transmitted by the camera in any suitable format, for example, over a connection, to a computing system that receives the image. Alternatively, the image may be stored in any suitable format in a memory connected to the camera and read by the computing system.

[0031] An example of an image 600 received from a camera is shown in Fig. 6A. In this example (also referred to as an airport example below), the facility is an airport, specifically an arrivals lobby. As shown, the arrivals lobby includes several objects such as columns, baggage conveyors, windows, a floor, a lighted ceiling, a door, and a sign board with the number "1" hanging from the ceiling. These objects are also depicted (included) in the image.

[0032] Optionally, information regarding a destination object within the facility to which the camera should navigate is received in block 325. In block 325, the destination location is estimated.

[0033] In one example, the acts of block 325 include receiving information about the destination object (e.g., a name of the destination object) via an input device. The information may be entered in any suitable format, including text input, voice input, etc. The information about the destination object is used to search for an image or image blocks of an image that contain object information corresponding to the destination object. For example, if the object information includes a name of the object, the model data may be searched using the name of the destination object as a search key. If found, the location from which the image containing the destination object was obtained is estimated based on the image data of the image.

[0034] In block 330, the current location of the camera within the facility is estimated. The estimation process is based on the image and the model data of the facility. To navigate within the facility, the current location of the camera represents the starting point of the path along which to navigate to the destination point. In block 330, the location where the image was requested, i.e., the location where the camera was located when the image was acquired, is estimated and used as the current location. Also, the camera's direction (i.e., the direction the camera was pointing in when the image was acquired, such as north, east, south, west, etc.) may be estimated along with the estimated location as a base or reference point.

[0035] According to an example, the estimation process in block 330 of the method of FIG. 3 is performed as shown in FIG. 4A.

[0036] First, in block 420, the image is divided into one or more image blocks. The image blocks are non-overlapping and contiguous (directly adjacent to each other). An example of dividing an image is illustrated in FIG. 5. As illustrated, the image may be divided into n×n (n is an integer) image blocks, all having the same size. Alternatively, the image may be divided into n×m (n, m are integers) image blocks, which may also have different sizes. The division process may also be performed in an adaptive and / or dynamic manner depending on the image characteristics. For example, if some areas of the image are blurred and others are not, the size of the image blocks of the blurred areas may be larger than the size of the image blocks of the other non-blurred areas. Also, in some examples, the image may be left as it is (i.e., "divided" into only one image block) in the first step. Based on the results of further processing of the image blocks, it may be decided to further divide the image into, for example, 2×2 image blocks in a next step, as described in detail below. Also, if necessary, adjacent image blocks may be merged to form a single image block that includes the facility object but not only a portion of it. In a similar manner, an image block may also be divided into multiple image sub-blocks.

[0037] 6B illustrates the process of dividing the airport example image 600 into multiple image blocks. A grid 610 is applied to image 600, dividing image 600 into image blocks, e.g., image block 620-1 and image block 620-2.

[0038] Then, in block 422, object candidates corresponding to the facility objects are detected in one or more image blocks. In one example, the object candidate detection process is performed separately for each image block, and therefore can be performed in parallel. This process is also referred to as object detection, which uses an object detection model as described above to detect regions of interest in the image blocks. For example, features may be extracted from the image blocks and compared with feature information in the object information, i.e., features extracted from the facility objects.

[0039] In Fig. 6C, two image blocks 630 and 640 of an example airport image 600 are shown. It is assumed that object detection as described has been performed on each of the two image blocks 630 and 640. Thereby, a first region 632 in the image block 630 has been identified as having features (e.g., histogram values) corresponding to features of a facility object (e.g., a baggage conveyor). The first region 632 represents an object candidate in the image block 630. Similarly, a first region 642 and a second region 644 are identified as object candidates in the image block 640. The first region 642 has features corresponding to features of a facility object (e.g., a baggage conveyor or an escalator or a handrail), while the second region 644 has features corresponding to features of a facility object (e.g., a sign board suspended from the ceiling). In particular, second region 644 may be detected as containing features corresponding to the number "1," thereby narrowing the object candidates to a sign board hanging from the ceiling having the number "1" written on it.

[0040] At block 424, a distance value to the object candidates detected in block 422 is determined. As described above, the model data includes object information corresponding to objects of the facility and distance information defining a distance between the object and a location of a camera that captured an image having the object. From the model data, object information for each object that is a detected object candidate and its corresponding distance information are obtained from the model data. Based on the characteristics of the object candidates (e.g., width in number of pixels) and the objects in the model data, a distance value can be determined based on the distance information for the object. This determination process can include triangulation or ML techniques, as will be understood by those skilled in the art.

[0041] In Fig. 6D, an example is illustrated for determining distance values ​​to object candidates 632 in image block 630 (denoted as O1) and object candidates 642, 644 in image block 640 (denoted as O2) shown in Fig. 6C. From the model data, distance information related to object O1 is retrieved from the model data. As explained, the distance information defines the distance from the object to the location where the sample or training image with the object was taken. This distance is used to estimate, and thereby determine, distance values ​​d11 and d12 from an estimated camera location (denoted as Camera Location in Fig. 6D) to object O1. Distance values ​​d21 and d22 are similarly determined for object O2.

[0042] Further, in block 426, the distance between the object candidates detected in block 422 is determined based on the distance values ​​determined in block 424. Again, triangulation or ML techniques may be applied, as will be appreciated by those skilled in the art.

[0043] In the example of FIG. 6D, the distance D1 between object candidates O1 and O2 (ie, between image blocks 630 and 640) is determined based on one or more of the distance values ​​d11, d12, d21, and d22.

[0044] Finally, in block 430, the current location of the camera is estimated based on the location information from the model data and the distance value determined in block 426. As described, the location information defines a location in the facility where the camera was located when acquiring the sample or training image from which the image data and the object information of the object in the sample or training image are generated. For example, the location can be assumed as a reference location to which the distance information to the object corresponds. Based thereon, the relative location of the object can be determined. From the relative location of the object, the location of the camera is derived using the determined distance value (e.g., at the intersection of the distance values ​​d11 and d12 from the relative location of O1 and the distance values ​​d21 and d22 from the relative location of O2). As understood by those skilled in the art, other techniques including triangulation or ML techniques can also be used.

[0045] Optionally, in one example, the estimation process in block 330 of the method of FIG. 3 may further include performing object classification on the object candidate in block 428 (e.g., using an object classification model as described herein). Object classification refers to classifying and recognizing the object candidate as an object of the facility. In this example, the object classification may be based on the distance value determined in block 424 and the distance determined in block 426. As described above, the model data includes distance information defining a distance between the object and a location, and first relationship information defining a distance between two objects of the facility. Based on the distance between them and the distance to the location, the score indicates the likelihood that the object candidate corresponds to a particular object in the facility. Also, the likelihood that the object candidate corresponds to a particular object in the facility may be derived from the object information (e.g., using the object features).

[0046] FIG. 6E illustrates the results of performing object classification of block 428 on object candidates 632, 642, and 644 in image blocks 630 and 640 as illustrated in FIG. 6C. As illustrated, object candidate 632 in image block 630 is classified as a luggage conveyor based on features such as histogram, color, location, texture, etc. Similarly, object candidate 642 in image block 640 is classified as a sign board suspended from the ceiling based on features such as the number "1". Object candidate 644 in image block 640 is not classified (i.e., unclassified) if it may correspond to the same likelihood as a different object such as a luggage conveyor or an escalator. Thus, object candidate 644 may be ignored in further processing of the image blocks.

[0047] The classified objects may be used to derive distance values ​​and relationships between the classified objects. Figure 6F illustrates, for example, a relationship R1 between classified objects O1 and O2 derived using first relationship information from model data. In addition, the distance values ​​determined in block 422 for the candidate objects may be adapted or modified for the classified objects. For example, the distance values ​​d11, d12, d21, and d22 in Figure 6D may be replaced with distance values ​​c11, c12, c21, and c22, respectively. The determination of the distance values ​​c11, c12, c21, and c22 is performed in a manner similar to block 422 as described above.

[0048] Returning to FIG. 3, a virtual path from the current location of the camera estimated in block 330 to a destination location in the facility is generated in block 340. The virtual path along which the current location will navigate to the destination location can be generated using model data of the facility. For example, as described above, the objects of the facility are determined in the images taken from the current location. From the model data, relationships that define the distance and direction between the objects are derived. Thus, the virtual path corresponds to a set of relationships from a first object in the current location to an object in the vicinity of the destination location. In other words, the virtual path may be understood as a set of footprints that move the camera a given distance in a given direction (e.g., move the camera 50 meters north and then 30 meters east to reach the destination location).

[0049] According to an example, the generating process at block 340 of the method of FIG. 3 is performed as shown in FIG. 4B.

[0050] First, in block 440, a relationship between the object candidate and a destination object is determined. The destination object may be estimated according to instructions or information in block 325, as described in more detail below. The relationship is determined based on first and / or second relationship information from the model data. Based on the relationship, the distance and relative direction from one image block to the other image block and from one image to the other image are determined. If multiple relationships exist, a score is determined and only the relationship having the highest score is used.

[0051] An example for determining relationships according to block 442 is shown in Figures 7A and 7B.

[0052] The example of FIG. 7A illustrates a relationship from an object in image block T42 (denoted as start point S and corresponding to the current location) in tile 1 (image 1) to an object in image block T14 (denoted as destination point D and corresponding to the destination location) in tile 2 (image 2). Based on the second relationship information, the image block T42 of tile 1 is navigated according to relationship R12 (the relative direction that can be normalized is east) to reach the image block T42 of tile 2. The distance from tile 1 to tile 2 is derived from the second relationship information. Then, based on the first relationship information, for example, the image block T42 of tile 2 is navigated north to the image block T12 of tile 2 and from the image block T12 of tile 2 to the image block T14 of tile 2 (i.e., according to relationships R2 and R3). As will be understood by those skilled in the art, the diagram of FIG. 7A only represents an example. In any case, the direction may be North (ie, toward the top of the image), East (ie, to the right of the image), South (ie, toward the bottom of the image), or West (ie, to the left of the image).

[0053] The example of FIG. 7B also illustrates a relationship from a starting point S in an image block T42 in tile 1 to a destination point D in an image block T14 in tile 4 (image 2). In the example of FIG. 7B, it is determined based on the second relationship information of tile 2, which may be navigated to tile 3 (i.e., relationship R23) or to tile n (i.e., relationship R2n). To determine which relationship should be followed, it is determined which relationship is stronger. For example, a score based on distance is calculated according to the second relationship information (e.g., the shorter the distance, the higher the score), and a higher score is determined as stronger. In the example of FIG. 7B, the score of relationship R23 is higher than the score of relationship R2n (e.g., the distance in relationship R23 is shorter), indicating that the relationship in the direction to tile 3 is stronger. As will be understood by those skilled in the art, the diagram of FIG. 7B is merely an example.

[0054] 4B, a virtual path is derived in block 442. The virtual path is derived based on the relationship derived in block 440.

[0055] In one example of generating a virtual path in block 340 of the method of FIG. 3, a path classification is performed (e.g., using a path classification model). The operation of the path classification means determining which image blocks of an image represent a suitable path, i.e., can be used to navigate to a destination location / destination object. As will be understood by those skilled in the art, the path classification can be formed using ML techniques based on corresponding features (i.e., floor tiles, floor colors, guiding lines, or signs on the floor, etc.). For example, determining image blocks that correspond to floor objects of a facility.

[0056] An example of path classification for image 600 of FIG. 6A is shown in FIG. 6G. In this non-limiting example, image blocks 650-1 and 650-2 are detected and classified as floor objects and therefore suitable image blocks for deriving a path. Other image blocks, such as the image block to the left of image block 650-2, are not suitable because they contain objects other than the floor object that may be determined to be obstacles.

[0057] In FIG. 6H, the example image 600 of FIG. 6A is depicted with image blocks marked as virtual paths derived in block 442. For example, image blocks 660-1 through 660-5 are image blocks of the virtual path. As described above with respect to relationships, scores (not shown) may be associated with the image blocks. For example, the scores associated with image blocks 660-3 through 660-5 may have a value of "99," while the score of image block 660-2 has a value of "50" and the score of image block 660-1 has a value of "0." A threshold of "50" may be used such that only image blocks 660-3 through 660-5 are used in determining which image blocks to follow, i.e., which relationships. As will be appreciated by those skilled in the art, the diagram of FIG. 6H represents an example only.

[0058] 3, navigation information to the destination location according to the virtual route generated in block 340 is generated and output. In one example, the navigation information may be output by displaying at least one image and the navigation information on a display device.

[0059] In the method of Figure 3, at least one image received from the camera may be stored in a memory device. The image may be used as sample or training data in a method for updating model data, which may be similar to generating model data according to block 310, as described below. That is, the at least one image is used in performing machine learning operations to generate updated model data for the facility.

[0060] FIG. 4C illustrates an example for generating model data according to block 310 of the method of FIG.

[0061] As discussed above, the method and subject disclosure of Figure 3 is based on model data and uses the model data in navigating the facility, which thus forms the data foundation for the method of Figure 3. As is widely known in the field of ML algorithms, generating model data for a facility involves building a mathematical model based on sample or training data.

[0062] The generation process in block 310 of FIG. 3 begins with capturing a number of images from within a facility in block 480. The images that form the sample or training data are captured using a camera. For example, the camera may be moved to a number of locations within the facility. At each location, one or more images may be taken by the camera from the perimeter of the location (i.e., the physical area around the location). Because a typical facility such as an airport terminal includes a number of different objects such as pillars, doors, seats, desks, etc., objects around the location are also captured and therefore included in the image. For example, at each location, four images may be captured by the camera in the north, east, south, and west directions. The number of images at a location is not limited to four, and one image having a panoramic or wide-angle characteristic (i.e., 360 degrees) may also be captured. Additional information may be associated with the image at a location, such as the location (i.e., coordinates in a coordinate system applicable to the facility), lighting conditions, orientation, camera settings such as zoom ratio and aperture, etc. Such additional information may be referred to as image metadata and may be stored in Exchangeable Image File Format (Exif) format. The format of the images may be JPEG format or any other suitable format, and the images may be stored on a storage device connected to the camera. In one particular example, the images may also be represented by frames of a video stream captured by a video camera.

[0063] The following processing is performed on each of the multiple images, either sequentially, in parallel, in batches, positionally, or in any other suitable manner, by a computer system that receives the multiple images from a camera or reads the multiple images from a storage device.

[0064] At block 482, depth information is determined. The depth information defines the distance from the location to the object. In reality, the image is a two-dimensional representation of the physical area around the location, so the depth information represents a third dimension of information. The depth information may be determined by a sensor associated with the camera or by applying techniques such as stereo triangulation or time of flight. The depth information may also be determined using ML techniques based on the image (i.e., image data) and image information provided by the camera, such as the metadata mentioned above. The depth information may be determined for each individual pixel or group of pixels of the image.

[0065] At block 484, location information is generated based on the location where the image was captured. As described above, additional information, such as metadata, associated with the image includes information about the location such that the location information may correspond to or be derived from information about the location in the additional information. The location information may be represented by coordinates in a coordinate system that applies to the facility, or may be relative to an adjacent or reference location. For example, the location information includes information that the location is 5 meters north and 2 meters east of the reference location. Those skilled in the art will appreciate that any suitable representation of the location information may be used, provided that the location is uniquely identified within the facility.

[0066] In block 486, the image is divided into one or more image blocks. The image blocks are non-overlapping and contiguous (directly adjacent to each other). An example of dividing an image is shown in FIG. 5. An image may be divided into only one image block. That is, the entire image is taken as an image block. An image may also be divided into 2×2, 4×4, and generally n×n (n is an integer) image blocks that all have the same size. However, the image division process is not limited to the example of FIG. 5, and an image may be divided into m×n (m, n are integers) image blocks that do not have the same size. For example, an image may be divided such that the image blocks at the edge of the image are larger than the image blocks closer to the center of the image. In block 486, an image may be divided several times with different numbers or sizes of image blocks (e.g., 2×2 and 4×4).

[0067] In block 488, objects of the facility are detected in one or more image blocks. More specifically, the detection process in block 488 is performed in each image block. For example, as will be appreciated by those skilled in the art, ML techniques may be used to detect objects in the image blocks. Other techniques for detecting objects will also be apparent to those skilled in the art. The detection process results in object information describing features and characteristics of the detected objects. For example, the object information may include information such as the object's histogram, color, size, texture, etc. The object information also includes information indicative of the image block of the image (e.g., an identifier for the image block).

[0068] At block 490, a distance between the detected objects and the location is determined. In determining, the depth information determined at block 482 is used. For example, the distance between the detected objects may be derived based on the distance of the detected objects from the location, for example, by using triangulation or ML techniques. In an example, the distance between the object and the location may also be measured. For each of the detected objects or each of the image blocks, distance information is generated based on the determined distance.

[0069] A distance between the detected object in the image block and a relative direction is calculated in block 492. For example, the distance between the objects in the image block may be calculated based on depth information and / or distance information. The relative direction describes how to navigate from one detected object in a first image block to another detected object in a second image block. For example, the distance may be 5 meters and the relative direction may be northeast to describe that one will navigate and move 5 meters from the first object to the northeast to reach the second object. In an example, the distance and the relative direction form a first relationship based on which the first relationship information is generated. In addition, the first relationship information may indicate an image block that includes the detected object (e.g., using an identifier of the image block). The first relationship between the image blocks is illustrated in FIG. 7A. For example, the first relationship between image block T11 and image block T12 of tile 1 is R1. Similarly, the first relationship between image block T13 and image block T14 of tile 1 is R3.

[0070] Further, in block 494, the distance between the locations is determined. The distance between the location where the first image was captured and another location where the second image was captured is determined together with the relative direction to describe a relationship between them (referred to as a second relationship). The determination process is based on the location information of each of the images (e.g., the first and second images). Similar to the relative direction described above, the relative direction with respect to the location where the image was captured also describes how to navigate from the location where the first image was captured to the location where the second image was captured. For example, the distance can be 50 meters and the relative direction can be north to describe that one will navigate 50 meters north from the location where the first image was captured to reach the location where the second image was captured. In one example, the distance and the relative direction are used to generate the second relationship information. In addition, the second relationship information can indicate the image and / or the location (e.g., using an image identifier or location coordinates). The second relationship between the images is illustrated in FIG. 7B. For example, the second relationship between tile 1 (first image) and tile 2 (second image) is R12. Similarly, the second relationship between tile 3 (first image) and tile 4 (second image) is R34.

[0071] Image data for the image is generated at block 496. The image data includes at least the location information, the object information, the first relationship information, and the second relationship information.

[0072] The steps of blocks 482-496 are performed on a plurality of sample or training images acquired in block 480 to generate model data by generating image data for each image. The model data forms a model of the facility.

[0073] Finally, FIG. 8 is a diagrammatic representation of a computer system providing the functionality of a computing system for use in navigating a facility such as that shown in FIG. 2. Within the computer system 800, a set of instructions may be executed that causes the computer system to perform any of the methods disclosed herein. The computer system 800 includes at least one processor 820, a main memory 840, and a network interface device 880, which communicate with each other via a bus 810. Optionally, a static memory 860 and a disk drive unit 870 may also be included. A video display, an alphanumeric input device, and a cursor control device may be provided as examples of a user interface 830. The network interface device 880 connects the computer system 800 to a client or device equipped with a camera, a display, and input means via the Internet and / or any other network. The client or device is used, for example, by a user, in navigating a facility. Also, model data 842 and images 844 (e.g., sample or training images, images received from a client or device) may be stored within the main memory 840. A set of computer executable instructions (i.e., computer program code) 846 embodying any one or all of the methods described above resides, in whole or at least in part, in or on a machine-readable medium, e.g., main memory 840 and / or at least one processor 820. The machine-readable medium on which the code 844 resides may also be a non-volatile data carrier (e.g., a non-removable magnetic hard disk or an optical or magnetic removable disk), which is part of the disk drive unit 870. The code 846 may further be transmitted or received as a propagated signal over the Internet via a network interface device 880. The basic operation of the computer system 800, including the user interface and network communications, is controlled by an operating system (not shown).

[0074] In general, the routines executed to implement the examples of the subject disclosure may be referred to herein as "computer program code" or simply "program code", whether implemented as part of an operating system or as a specific application, component, program, object, module, or sequence of instructions, or a subset thereof. Program code typically resides at various times in various memory and storage devices within a computer and contains computer readable instructions that, when read and executed by one or more processors within a computer, cause the computer to perform the operations necessary to execute the operations and / or elements embodying various aspects of the examples of the subject disclosure. The computer readable program instructions for carrying out the operations of the examples of the subject disclosure may be, for example, source code or object code written in assembly language or any combination of one or more programming languages.

[0075] Various program code described herein may be identified based on the application in which it is implemented in a particular example of the subject disclosure. However, it should be appreciated that any specific program names below are used merely for convenience and thus the subject disclosure should not be limited to use only in any particular application identified and / or implied by such names. Moreover, given the generally limitless ways in which computer programs can be organized into routines, procedures, methods, modules, objects, etc., and the various ways in which program functionality can be allocated among the various software layers (e.g., operating systems, libraries, APIs, applications, applets, etc.) present in a typical computer, it should be appreciated that examples of the subject disclosure are not limited to the specific configurations and allocations of program functionality described herein.

[0076] The program code embodied in any of the applications / modules described herein may be distributed alone or together as a program product in a variety of different forms. In particular, the program code may be distributed using a computer-readable storage medium having computer-readable program instructions that cause a processor to perform example aspects of the subject disclosure.

[0077] Computer-readable storage media that are non-transitory in nature may include volatile and non-volatile, as well as removable and non-removable tangible media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media may further include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other solid-state memory technology, portable compact disk read-only memory (CD-ROM) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired information and can be read by a computer. Computer-readable storage media should not be interpreted as transitory signals in the strict sense (e.g., electric waves or other propagating electromagnetic waves, electromagnetic waves propagating through a transmission medium such as a wave guide, or electric signals transmitted through wires). The computer readable program instructions may be downloaded from the computer readable storage medium into a computer, another type of programmable data processing apparatus, or another device, or via a network to an external computer or external storage device.

[0078] Computer-readable program instructions stored on a computer-readable medium may be used to instruct a computer, other type of programmable data processing apparatus, or other device to function in a particular manner, such that the instructions stored on the computer-readable medium produce an article of manufacture including instructions that implement the functions, behaviors, and / or operations specified in the flowcharts, sequence diagrams, and / or block diagrams. The computer program instructions may be provided to one or more processors of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a mechanism, such that the instructions, executed via the one or more processors, cause a series of calculations to implement the functions, behaviors, and / or operations specified in the flowcharts, sequence diagrams, and / or block diagrams.

[0079] In certain other examples, the functions, behaviors, and / or operations specified in the flowcharts, sequence diagrams, and / or block diagrams may be reordered, processed sequentially, and / or processed in parallel, consistent with examples of the subject disclosure. Further, any of the flowcharts, sequence diagrams, and / or block diagrams may include more or fewer blocks than those illustrated, consistent with examples of the subject disclosure.

[0080] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit the examples of the subject disclosure. It is further understood that the terms "comprise" and / or "comprising" as used in the subject disclosure specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, to the extent the terms "include," "have," "have," "comprise," "consisting of," or variations thereof are used, such terms are intended to be inclusive in the same manner as the term "comprising."

[0081] Although all of the examples have been illustrated by way of illustration of various examples, and although these examples have been described in considerable detail, it is not intended to restrict or in any way limit the scope to such details. Additional advantages or modifications will be readily apparent to those skilled in the art. Thus, the disclosure of the subject matter in its broader aspects is not limited to the specific details, representative apparatus and methods, and illustrative examples shown and described. Thus, departures may be made from such details without departing from the scope of the general concept. [Explanation of symbols]

[0082] 110 Image / Video Frame Receiving Component 120 Pre-processing Components 130 Object Detection Model Components 140 Object Classification Model Components 150 Route Classification Model Components 160 Relationship Building Components 170 Map Building Components

Claims

1. 1. A computer-implemented method for use in navigating a facility, comprising: receiving at least one image from a camera; estimating, by a processor, a current location of the camera within the facility based on the at least one image and model data of the facility; generating, by the processor, a virtual path from the current location of the camera within the facility to a destination location using the model data of the facility; generating and outputting navigation information to the destination location according to the virtual route; Including, the model data of the facility includes image data of a plurality of images, the image data of each image including location information corresponding to a location within the facility where the image was acquired, object information corresponding to an object of the facility in the image, distance information corresponding to a distance between the object and the location, first relationship information defining a distance and relative direction for navigating from one object in the image to another object in the image as a first relationship, and second relationship information defining a distance and relative direction for navigating from the location where the image was acquired to a location where another image was acquired as a second relationship; The step of estimating a current location includes: dividing, by the processor, the at least one image into one or more image blocks; detecting, by the processor, object candidates within the one or more image blocks corresponding to objects of the facility based on the object information from the model data of the facility; obtaining, by the processor, the object information and the distance information of the object candidate from the model data of the facility; determining, by the processor, a distance value from the current location of the camera to the object candidate based on the object information, the distance information, and a pixel width in the at least one image of the object candidate; determining, by the processor, a distance between the object candidates based on the distance values; estimating, by the processor, the current location of the camera based on the location information from the model data of the facility and the distance values ​​to the detected object candidates; the virtual path corresponds to a relationship from the object candidate at the current location to an object in the vicinity of the destination location based on the first and second relationship information. Computer-implemented method.

2. The step of estimating a current location includes: The computer-implemented method of claim 1 , further comprising performing, by the processor, object classification on the candidate objects in the images based on the distance values ​​and the distances to detect the objects in the facility.

3. receiving, via an input device, information regarding a destination object within the facility to which the camera should navigate; searching, by the processor, for at least one image block of an image in the model data for the object information in the model data of the facility that corresponds to the information regarding the destination object; estimating, by the processor, a location of the destination object based on image data of an image including the destination object as the destination location; 3. The computer-implemented method of claim 1, further comprising:

4. The step of generating a virtual path includes: determining, by the processor, a relationship between the candidate objects and the destination object in the image block of the image based on the first and second relationship information in the model data of the facility; deriving, by the processor, the virtual path based on the determined relationship; 4. The computer-implemented method of claim 3, comprising:

5. 5. The computer-implemented method of claim 1, wherein outputting the navigation information comprises displaying the at least one image and the navigation information on a display.

6. The method further includes generating, by the processor, the model data of the facility, the generating the model data of the facility comprising: acquiring, with the camera, one or more images from multiple locations within the facility; For each of the plurality of images, the processor determines depth information based on the image and image information provided by the camera; the processor generates location information based on the location at which the image was captured; the processor divides the image into one or more image blocks; the processor detects objects of the facility in the one or more image blocks and generates object information defining characteristics of the detected objects, the object information including information indicative of the image block of the image; the processor uses the depth information to determine a distance between the detected objects in the one or more image blocks and the location and generates distance information corresponding to the detected objects in the image block; and the processor determines how the distance between the detected objects in the one or more image blocks is determined. determining, by the processor, a distance between the location where the image was acquired and the other location where the other image was acquired based on the location information of the image and the other image and a relative direction describing how to navigate from the location to the other location, and generating, by the processor, second relationship information based on the distance and the relative direction, the second relationship information including information indicative of the image and the other image; and generating, by the processor, image data for the image including the location information, the object information, the first relationship information, and the second relationship information.

6. The computer-implemented method of claim 1 , comprising:

7. storing, by the processor, the at least one image; 7. The computer-implemented method of claim 1, further comprising: performing, by the processor, a machine learning operation using the at least one image and the model data of the facility to generate updated model data of the facility.

8. 1. A computing system for use in navigating a facility, comprising: A processor; A camera device; at least one memory device accessible by said processor; the memory device includes a number of program instructions which, when executed by the processor, cause the computing system to perform a method including the steps of receiving at least one image from the camera device; estimating a current location of the camera device within the facility based on the at least one image and model data of the facility; generating a virtual route from the current location of the camera device within the facility to a destination location using the model data of the facility; and generating and outputting navigation information to the destination location according to the virtual route; the model data of the facility includes image data of a plurality of images, the image data of each image including location information corresponding to a location within the facility where the image was acquired, object information corresponding to an object of the facility in the image, distance information corresponding to a distance between the object and the location, first relationship information defining a distance and relative direction for navigating from one object in the image to another object in the image as a first relationship, and second relationship information defining a distance and relative direction for navigating from the location where the image was acquired to a location where another image was acquired as a second relationship; The step of estimating a current location includes: Dividing the at least one image into one or more image blocks; detecting object candidates corresponding to objects of the facility based on the object information from the model data of the facility within the one or more image blocks; acquiring the object information and the distance information of the object candidate from the model data of the facility; determining a distance value from the current location of the camera device to the object candidate based on the object information, the distance information, and a pixel width in the at least one image of the object candidate; determining a distance between the object candidates based on the distance values; estimating the current location of the camera device based on the location information from the model data of the facility and the distance values ​​to the detected object candidates; the virtual path corresponds to a relationship from the object candidate at the current location to an object in the vicinity of the destination location based on the first and second relationship information. system.

9. 9. The system of claim 8, further configured to perform the method of any one of claims 2 to 7.

10. 8. A computer program product comprising program code instructions stored on a computer readable medium, the program code instructions being adapted to perform the steps of the method according to any one of claims 1 to 7 when executed on a computer.

Images