Terminal device, service server and method for object recognition-based indoor location

Abstract

A terminal device, a service server, and a method thereof for object recognition-based indoor location are disclosed. The terminal device according to one aspect of the present invention includes: a storage unit storing an indoor map to which preset coordinate values ​​for each object are matched; an image capture unit; and a control unit that recognizes first and second objects in an image acquired through the image capture unit, estimates virtual space positions of the first and second objects, respectively, estimates virtual space distances between the first and second objects and the user, respectively, using the virtual space positions of the first and second objects and the virtual space position of a user, and estimates the indoor location of the user using the virtual space distances.

Description

[Technical field]

[0001] The present invention relates to a terminal device, a service server, and a method for an object recognition-based indoor site, and more particularly, to a terminal device, a service server, and a method for an object recognition-based indoor site, which can recognize an object in an image captured by a camera of a terminal device, and match the recognized object on an indoor map having vectorized coordinates to estimate a user's position. [Background technology]

[0002] Many services are provided that provide selective data to a large number of users at their desired locations, i.e., various Location-Based Services (LBS) based on the user's current location, such as real-time data pop-up services, selective data transmission services based on user location, and indoor navigation services.

[0003] Such services are based on technology for measuring the user's location, and location-based services can provide services such as indoor navigation by measuring the user's location using WiFi, beacons, etc. In order to provide such location-based services properly, it is important to accurately grasp the user's location. However, when WiFi is used to determine the user's location, it is difficult to provide a proper location-based service because the fluctuation in the magnitude of the WiFi reception signal used for location calculation is very large indoors, and when a beacon transmitter is used, it may be difficult to measure the user's location due to the placement interval of the beacon transmitter. For example, in order to measure the user's location using a beacon, the distance between the beacon and the user must be accurately measured, but the greater the distance between the beacon and the user, the greater the error. In addition, conventional indoor location estimation technologies have the problem of requiring expensive equipment and infrastructure to be constructed.

[0004] Therefore, there is a demand for the development of technology that can estimate indoor positions without constructing expensive equipment or infrastructure.

[0005] Prior art related to the present invention includes Korean Patent Publication No. 2011-0025025. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been devised to solve the above problems, and an object of the present invention is to provide a terminal device, a service server, and a method thereof for an object recognition-based indoor location that can recognize an object in an image captured by a camera of the terminal device and estimate a user's position using the object.

[0007] The problems to be solved by the present invention are not limited to the problem(s) mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0008] According to one aspect of the present invention, a terminal device includes a storage unit in which an indoor map to which preset coordinate values ​​for each object are matched is stored, a photographing unit, and a control unit that recognizes first and second objects in an image acquired through the photographing unit, estimates virtual space positions of the first and second objects, respectively, estimates virtual space distances between the first and second objects and the user, respectively, using the virtual space positions of the first and second objects and a virtual space position of a user, and estimates an indoor position of the user using the virtual space distances.

[0009] In the present invention, a display unit may further be included, and the control unit may display the user's location on the indoor map together with an image acquired through the photographing unit through the display unit.

[0010] In the present invention, the control unit can recognize the first and second objects using deep learning, and estimate the virtual space positions of the first and second objects, respectively, using a point cloud generated in a virtual space.

[0011] In the present invention, the control unit can estimate the virtual space position of the user by using a Dead Reckoning algorithm.

[0012] In the present invention, the control unit can calculate a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user, and calculate a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user.

[0013] In the present invention, the control unit acquires indoor coordinates of the first and second objects from the indoor map, respectively, estimates a first indoor predicted position and a second indoor predicted position of the user using the first and second Euclidean distances and the indoor coordinates of the first and second objects, calculates a virtual space cross product which is a cross product of a first virtual vector and a second virtual vector which are vectors to the virtual space positions of the first and second objects based on the virtual space position of the user, calculates a first indoor cross product which is a cross product of a first real vector and a second real vector which are vectors to the indoor coordinates of the first and second objects based on the first indoor predicted position and a second indoor cross product which is a cross product of a first real vector and a second real vector which are vectors to the indoor coordinates of the first and second objects based on the second indoor predicted position, and estimates an indoor predicted position of an indoor cross product having a sign that matches that of the virtual space cross product among the first and second indoor cross products as the indoor position of the user.

[0014] In the present invention, the system further includes a communication unit which communicates with a service server through a communication network, and when an image collection application stored in the storage unit is executed, the control unit photographs pre-selected objects through the photographing unit, stores images of each of the photographed objects, and transmits the photographed images of each of the stored objects to the service server through the communication unit.

[0015] According to another aspect of the present invention, a service server includes a communication unit that receives photographed images of each object set on an indoor map, and an object recognition model generation unit that learns the photographed images of each object received through the communication unit and matches objects and their coordinates through the learned photographed images of each object to generate an object recognition model for object recognition.

[0016] In the present invention, the service server may further include a position estimation unit that, when a position estimation request signal including a photographed image is received from the terminal device via the communication unit, inputs the photographed image into the object recognition model to recognize first and second objects, estimates virtual space positions of the first and second objects, respectively, estimates virtual space distances between the first and second objects and the user, respectively, using the virtual space positions of the first and second objects and the virtual space position of the user, estimates an indoor position of the user using the virtual space distances, and transmits the estimated indoor position to the terminal device.

[0017] According to yet another aspect of the present invention, an object recognition based indoor location service method includes a step of a terminal device recognizing first and second objects in an image acquired through a photographing unit and estimating virtual space positions of the first and second objects, respectively, a step of the terminal device estimating virtual space distances between the first and second objects and the user, respectively, using the virtual space positions of the first and second objects and a virtual space position of a user, and a step of the terminal device estimating an indoor location of the user using the virtual space distances.

[0018] In the present invention, in the step of estimating the virtual space positions of the first and second objects, respectively, the terminal device can recognize the first and second objects using deep learning, and estimate the virtual space positions of the first and second objects, respectively, using a point cloud generated in the virtual space.

[0019] In the present invention, the step of estimating the virtual space distances may include a step of the terminal device estimating the virtual space position of the user using a dead reckoning algorithm, and a step of the terminal device calculating a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user, and calculating a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user.

[0020] In the present invention, the step of estimating the indoor position of the user may include a step of the terminal device acquiring indoor coordinates of the first and second objects from the indoor map, and estimating a first indoor predicted position and a second indoor predicted position of the user using the first and second Euclidean distances and the indoor coordinates of the first and second objects; a step of the terminal device calculating a virtual space cross product that is a cross product of a first virtual vector and a second virtual vector that are vectors to the virtual space positions of the first and second objects based on the virtual space position of the user; a step of the terminal device calculating a first indoor cross product that is a cross product of a first real vector and a second real vector that are vectors to the indoor coordinates of the first and second objects based on the first indoor predicted position and a second indoor cross product that is a cross product of a first real vector and a second real vector that are vectors to the indoor coordinates of the first and second objects based on the second indoor predicted position; and a step of the terminal device estimating an indoor predicted position of an indoor cross product having a sign matching with that of the virtual space cross product among the first and second indoor cross products as the indoor position of the user. Effect of the Invention

[0021] According to the present invention, the user's location is estimated using an image captured by a camera of a user terminal device in an indoor environment, thereby making it possible to accurately estimate the indoor location without constructing expensive equipment or infrastructure.

[0022] Meanwhile, the effects of the present invention are not limited to the effects mentioned above, and may include various effects within a range that is obvious to a person skilled in the art from the contents described below. [Brief description of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram illustrating an object recognition-based indoor site according to an embodiment of the present invention; [Diagram 2] 1 is a diagram illustrating an object recognition based indoor site system according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a block diagram showing a configuration of a terminal device according to an embodiment of the present invention. [Figure 4] 11 is an exemplary diagram illustrating a location estimation screen according to an embodiment of the present invention; FIG. [Diagram 5] 1 is an exemplary diagram illustrating a method for estimating a user's indoor position using a cross product according to an embodiment of the present invention; FIG. [Figure 6] FIG. 2 is a block diagram illustrating a configuration of a service server according to an embodiment of the present invention. [Figure 7] 4 is a flowchart illustrating an object recognition-based indoor site method according to an embodiment of the present invention. [Figure 8] 4 is a flowchart illustrating a method for estimating a user's position using a cross product according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, a terminal device, a service server and a method for an object recognition-based indoor site according to an embodiment of the present invention will be described with reference to the accompanying drawings. In this process, the thickness of lines and the size of components illustrated in the drawings may be exaggerated for clarity and convenience of explanation.

[0025] In addition, the terms described below are defined in consideration of the functions of the present invention, and may vary depending on the intention or practice of a user or operator. Therefore, the definitions of such terms should be based on the overall contents of this specification.

[0026] Also, the implementations described herein may be embodied in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only one form of implementation (e.g., discussed only as a method), the implementation of the discussed features may be embodied in other forms (e.g., an apparatus or a program). An apparatus may be embodied in appropriate hardware, software, firmware, and the like. A method may be embodied in an apparatus, such as, for example, a processor, which generally refers to a processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0027] FIG. 1 is a conceptual diagram illustrating an object recognition-based indoor site according to an embodiment of the present invention.

[0028] 1, a user acquires an image through a photographing unit of a terminal device 100 and recognizes preselected first and second objects in the acquired image. Then, the terminal device 100 matches the recognized first and second objects on a vectorized indoor map to estimate the user's indoor position.

[0029] A system for estimating a user's indoor position through object recognition will be described with reference to FIG.

[0030] FIG. 2 is a diagram illustrating an object recognition based indoor site system according to an embodiment of the present invention.

[0031] 2, the object recognition based indoor site system according to an embodiment of the present invention includes an administrator terminal 100a, a service server 200, and a user terminal 100b. At this time, the administrator terminal 100a, the service server 200, and the user terminal 100b may be connected to various types of wireless communication networks such as Wifi, 3G, and LTE.

[0032] The administrator terminal 100a maps the objects selected for use in estimating the user's position on an indoor map. That is, the administrator can select the objects to be used in estimating the user's position. At this time, the objects can be selected from static objects (e.g., store signs, display boards, fire hydrants, etc.) and can be selected based on objects having unique characteristics in the room. Then, the administrator maps the selected objects on a pre-created indoor map. That is, the administrator stores the coordinate values ​​on the indoor map for each selected object. Here, the indoor map is a digitized (vectorized) map created using CAD drawings, point cloud maps, Lidar maps, image maps, etc., and the digitized map can be a map that can be used by the administrator terminal 100a and the user terminal 100b. In addition, the indoor map can include main information about the corresponding room. For example, in the case of a shopping mall, the indoor map can include boundaries that divide the stores, the names of the stores, etc.

[0033] In addition, the administrator terminal 100a stores an image collection application, stores images of objects captured through the image collection application for deep learning network learning for object recognition, and transmits the stored captured images of each object to the service server 200. At this time, the captured images may be images captured from various directions including the object.

[0034] As described above, the administrator terminal 100a photographs images including preselected objects from various angles and provides the photographed images of the objects to the service server 200 so that the photographed images can be used as learning data for object recognition.

[0035] The service server 200 collects photographed images of each object set on an indoor map from the administrator terminal 100a, and generates an object recognition model by learning the collected photographed images of each object. At this time, the service server 200 can generate the object recognition model using deep learning. Specifically, when the service server 200 receives a photographed image of each object, it learns a deep learning network that stores the image pixel coordinate values ​​of the four vertices of the smallest rectangle (hereinafter referred to as a "bounding box") that includes the corresponding object together with the object name. At this time, the deep learning network can be designed using various models related to the object recognition issue, and for example, the YOLO network can be used.

[0036] Furthermore, when the service server 200 receives a location estimation request signal including a photographed image from the user terminal 100b, the service server 200 inputs the photographed image into a location estimation model to estimate the location of the user terminal 100b.

[0037] For a detailed description of the service server 200, please refer to FIG.

[0038] The user terminal 100b stores a location estimation application, and when the user photographs the surrounding environment through the location estimation application, the user terminal 100b recognizes objects (e.g., signs, fire hydrants, picture frames, doors, etc.) in the photographed image and estimates the user's location using the location coordinates and distance estimates of the recognized objects.

[0039] On the other hand, in this embodiment, the administrator terminal 100a and the user terminal 100b are described separately, but the administrator terminal 100a and the user terminal 100b may be the same terminal. Therefore, for the sake of convenience, the administrator terminal 100a and the user terminal 100b will be referred to as the terminal device 100 in the following description.

[0040] FIG. 3 is a block diagram showing the configuration of a terminal device according to one embodiment of the present invention, FIG. 4 is an example diagram for explaining a location estimation screen according to one embodiment of the present invention, and FIG. 5 is an example diagram for explaining a method for estimating a user's indoor location using a cross product according to one embodiment of the present invention.

[0041] Referring to FIG. 3, a terminal device 100 according to an embodiment of the present invention includes a communication unit 110, a storage unit 120, a photographing unit 130, a display unit 140, and a control unit 150.

[0042] The communication unit 110 is configured to communicate with the service server 200 through a communication network, and can transmit and receive various information such as an image captured through the photographing unit 130. At this time, the communication unit 110 can be realized in various forms such as a short-range communication module, a wireless communication module, a mobile communication module, a wired communication module, etc.

[0043] The storage unit 120 is configured to store data related to the operation of the terminal device 100. Here, the storage unit 120 may use a known storage medium, for example, any one or more of known storage media such as ROM, PROM, EPROM, EEPROM, RAM, etc. In particular, the storage unit 120 may store an indoor map to which coordinate values ​​of preselected objects are matched. In addition, the storage unit 120 may store an image collection application that can drive the photographing unit 130 to obtain photographed images of preselected objects in order to collect learning data for generating an object recognition model. In addition, the storage unit 120 may store a location estimation application that can estimate a current location using an image.

[0044] When an image collection application or an image estimation application is executed, the photographing unit 130 captures an image and transmits the captured image to the control unit 150. The photographing unit 130 may be, for example, a camera.

[0045] The display unit 140 is configured to display various information related to the operation of the terminal device 100. In particular, the display unit 140 can display an image collection screen when an image collection application is executed, and can display a location estimation screen when a location estimation application is executed. The display unit 140 can also operate as an input unit for receiving information input from a user.

[0046] When the image collection application stored in the storage unit 120 is executed, the control unit 150 drives the photographing unit 130, stores the images of each object photographed by the photographing unit 130, and transmits the photographed images of each stored object to the service server 200. That is, when the control unit 150 completes image collection from various directions for a preselected object, it transmits the collected objects and the coordinates of the corresponding objects on the indoor map to the service server 200. At this time, the number of objects on the indoor map and the photographing directions can be preset by an administrator.

[0047] In addition, when the location estimation application stored in the storage unit 120 is executed, the control unit 150 drives the photographing unit 130, recognizes objects (e.g., signs, fire hydrants, picture frames, doors, etc.) in the image photographed by the photographing unit 130, and estimates the user's location using the position coordinates and distance estimates of the recognized objects.

[0048] That is, the control unit 150 recognizes the first and second objects in the image acquired through the photographing unit 130, estimates the virtual space positions of the first and second objects, respectively, estimates the virtual space distances between the first and second objects and the user using the virtual space positions of the first and second objects and the virtual space position of the user, respectively, and estimates the indoor position of the user using the virtual space distances. Here, the virtual space may refer to a space seen on a screen.

[0049] Hereinafter, a method in which the control unit 150 estimates the indoor position of the user will be described in detail.

[0050] When the control unit 150 acquires an image captured by the photographing unit 130 by executing a position estimation application, the control unit 150 generates a point cloud in the space of the image (a virtual space visible on the screen, hereinafter referred to as the "virtual space"). Each point cloud has coordinates in the virtual space. Therefore, the coordinates at the time when the point cloud map begins to be generated (i.e., when the user begins to move) may be set to [0, 0]. If the user walks around the entire area of ​​the indoor space, a point cloud map for the indoor environment will be generated and each point will have coordinates in the virtual space.

[0051] The control unit 150 recognizes pre-specified first and second objects in an image acquired through the photographing unit 130, and estimates the virtual space positions of the first and second objects, respectively, using a point cloud generated in the virtual space.

[0052] That is, the control unit 150 recognizes the first object and the second object using a deep learning technique, and outputs a first bounding box including the first object and a second bounding box including the second object. At this time, the control unit 150 may recognize the first object and the second object using the Yolo Network, and at least one of the object name, coordinate values, and the length of the bounding box may be displayed in the first and second bounding boxes. When the control unit 150 recognizes the first and second objects, it may output a recognition result such as a photographed image 310 of the position estimation screen 300 illustrated in FIG. 4. Referring to FIG. 4, a bounding box (A) including an object called "sun visor" is displayed, and the bounding box may display coordinates [x, y] and [x1, y1], and lengths d and h of both sides.

[0053] As described above, when an image including a learned object is input to the deep learning network, the control unit 150 outputs the coordinates of the four vertices of the image pixels of a bounding box including the object and the name of the object.

[0054] Thereafter, the control unit 150 selects the most central points in the first and second bounding boxes and estimates them as the virtual space positions of the first and second objects, respectively. That is, the control unit 150 may select the most central points in the bounding boxes of the first and second objects from the point clouds generated on the image and estimate the selected point coordinates as the virtual space coordinates.

[0055] Once the virtual space positions of the first and second objects are estimated as described above, the control unit 150 estimates the virtual space distances between the first and second objects and the user, respectively, using the virtual space positions of the first and second objects and the virtual space position of the user.

[0056] That is, the control unit 150 continuously tracks the user's position in the virtual space starting from [0,0] through sensors (e.g., a gyroscope, an accelerometer, a geomagnetic sensor, etc.) mounted in the terminal device 100. At this time, the control unit 150 may track the user's position in the virtual space using a dead reckoning algorithm. The dead reckoning algorithm may be an algorithm that estimates the user's movement distance and direction based on a sensor (not shown) mounted in the terminal device 100 to track the user's movement.

[0057] When the virtual space position of the user is estimated, the control unit 150 calculates a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user, and calculates a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user. At this time, since the first Euclidean distance and the second Euclidean distance are scalar values, the distance value in the virtual space and the distance value on the indoor map may be the same.

[0058] The control unit 150 may estimate the actual position of the user on the indoor map by using the virtual space positions of the first and second objects, the virtual space position of the user, and the Euclidean distance between the virtual space positions of the first and second objects and the virtual space position of the user. In this case, the actual positions of the first and second objects on the indoor map may be preset coordinate values.

[0059] Therefore, the control unit 150 can obtain the indoor coordinates (actual positions) of the first and second objects from the indoor map, and estimate the first and second predicted indoor positions of the user using the first and second Euclidean distances and the indoor coordinates of the first and second objects. That is, since the position of the user and the positions of the first and second objects are known in the virtual space, the control unit 150 can draw a triangular figure. On the indoor map, the positions of the first and second objects and two distance values ​​from the first and second objects to the user are known. That is, if the figure of the virtual space is matched on the indoor space, the user can be located at only two points on the indoor map. In order to select the user's position from the two points, it is necessary to know in which direction the first and second objects are located with respect to the user.

[0060] Therefore, the control unit 150 can use the concept of outer product (or vector product). The outer product of two vectors from the user to the first and second objects in the virtual space has a direction. This direction must be the same on the indoor map. In other words, the sign of the outer product of two vectors from the user in the virtual space to the first and second objects must be the same as the sign of the outer product of two vectors from the user in the indoor map to the first and second objects in the virtual space. As a result, among two points where the user can be located on the indoor map, the point where the sign of the outer product is the same as the sign of the outer product of the virtual space can be estimated as the user's position.

[0061] Specifically, the control unit 150 calculates the virtual space cross product of a first virtual vector, which is a vector to the virtual space positions of the first and second objects, and a second virtual vector, based on the virtual space position of the user. That is, the control unit 150 may calculate the virtual space cross product, which is the cross product of the first virtual vector from the virtual space position of the user to the virtual space position of the first object and the second virtual vector from the virtual space position of the user to the virtual space position of the second object.

[0062] Thereafter, the control unit 150 obtains a first indoor outer product, which is a cross product of a first real vector, which is a vector to the indoor coordinates of the first and second objects, with respect to the first indoor predicted position, and a second indoor outer product, which is a cross product of a first real vector, which is a vector to the indoor coordinates of the first and second objects, with respect to the second indoor predicted position. That is, the control unit 150 may obtain a first indoor outer product, which is a cross product of a first real vector from the first indoor predicted position to the indoor coordinates of the first object, and a second real vector from the first indoor predicted position to the indoor coordinates of the second object. Also, the control unit 150 may obtain a second indoor outer product, which is a cross product of a first real vector from the second indoor predicted position to the indoor coordinates of the first object, and a second real vector from the second indoor predicted position to the indoor coordinates of the second object.

[0063] Thereafter, the control unit 150 may estimate the indoor predicted position of the indoor cross product, of the first and second indoor cross products, whose sign is the same as that of the cross product of the virtual space, as the indoor position of the user.

[0064] 5, it is assumed that the actual coordinates of the first and second objects are p1 and p2, and the distances from the actual coordinates of the first and second objects are d1 and d2. In this case, the possible user positions on the indoor map can be two points, a and b, and the user position must be determined from among the two points. In order to determine the user position from among the two possible points, the directions of the first and second objects as seen in the virtual space must be considered.

[0065] If p2 is to the right of p1 in the virtual space, the cross product d2*d1 of the two vectors may have a + sign, and if p2 is to the left of p1 in the virtual space, the cross product d2*d1 of the two vectors may have a - sign. The cross product (d2*d1) of the two vectors is calculated for possible user positions a and b on the indoor map, and the user position on the indoor map where the sign of the cross product calculated in the virtual space matches can be determined as the user's final position.

[0066] As described above, the control unit 150 can recognize two objects through the photographing unit 130 and estimate the user's position using the position coordinates and distance estimates of the recognized two objects. In this way, the present invention utilizes a vectorized indoor map so that the user can accurately estimate his / her position even if the user only recognizes two objects through the photographing unit 130 of the terminal device 100. Specifically, the object to be used for estimating the user's position is mapped on the indoor map in advance. That is, the coordinates of the object on the indoor map are stored. After that, when the two objects are recognized in the virtual space visible on the screen of the terminal device 100, the user's position can be estimated by matching the geometric patterns drawn by the user and the two objects on the indoor map with the geometric patterns drawn by the user and the two objects in the virtual space.

[0067] When the user's indoor position is estimated, the control unit 150 displays the user's position on an indoor map and displays it together with the photographed image through the display unit 140. That is, when the user's position is estimated through a position estimation application, the control unit 150 can display a photographed image 310 and a position estimation screen 300 in which the user's position (B) is displayed on an indoor map 320, as shown in FIG.

[0068] Such a control unit 150 may include at least one computing device, which may be a general-purpose central processing unit (CPU), a programmable logic device implemented for a specific purpose (CPLD, FPGA), an application specific semiconductor computing device (ASIC), or a microcontroller chip.

[0069] Meanwhile, the terminal device 100 configured as described above can capture the surrounding environment through the photographing unit 130 and can be an electronic device that can be applied to various wired and wireless environments. For example, the terminal device 100 includes a PDA (Personal Digital Assistant), a smartphone, a cellular phone, a PCS (Personal Communication Service) phone, a GSM (Global System for Mobile) phone, a W-CDMA (Wideband CDMA) phone, a CDMA-2000 phone, an MBS (Mobile Broadband System) phone, etc. Here, the terminal device 100 can refer to a small portable device, but if it includes a video camera or a laptop computer, it can be referred to as a mobile communication terminal, so the embodiment of the present invention is not particularly limited thereto.

[0070] FIG. 6 is a block diagram showing the configuration of a service server according to an embodiment of the present invention.

[0071] Referring to FIG. 6, a service server 200 according to an embodiment of the present invention includes a communication unit 210, a storage unit 220, an object recognition model generating unit 230, and a control unit 250.

[0072] The communication unit 210 receives a photographed image of each object from the terminal device 100 .

[0073] The storage unit 220 is configured to store data related to the operation of the service server 200. In particular, the storage unit 220 may store an indoor map in which preselected coordinate values ​​of each object are stored.

[0074] The object recognition model generating unit 230 receives photographed images of objects from the terminal device 100 through the communication unit 210, learns the photographed images of each object received, and generates an object recognition model for object recognition. At this time, the object recognition model generating unit 230 may generate the object recognition model using deep learning. The object recognition model may be in a form in which coordinate values ​​for each object are mapped. Therefore, when an image with an unknown position is input, the object recognition model may calculate an object in the image and the coordinates of the object as output values.

[0075] Meanwhile, the service server 200 according to the present invention may further include a position estimation unit 240 that, when a position estimation request signal including a photographed image is received from the terminal device 100 via the communication unit 210, inputs the photographed image into an object recognition model to recognize the first and second objects, estimates virtual space positions of the first and second objects, respectively, estimates virtual space distances between the first and second objects and the user, respectively, using the virtual space positions of the first and second objects and the virtual space position of the user, estimates an indoor position of the user using the virtual space distances, and transmits the estimated indoor position to the terminal device. When the first and second objects are recognized, the position estimation unit 240 may display the recognized first and second objects on the screen of the terminal device 100.

[0076] Meanwhile, the location estimation model generating unit 230 and the location estimating unit 240 may each be implemented by a processor required to execute a program on a computing device, etc. As described above, the location estimation model generating unit 230 and the location estimating unit 240 may be implemented as physically independent components, or may be implemented in a form in which the components are functionally separated within a single processor.

[0077] The control unit 250 is configured to control the operation of various components of the service server 200, including the communication unit 210, the storage unit 220, the location estimation model generation unit 230, and the location estimation unit 240, and may include at least one computing device, where the computing device may be a general-purpose central processing unit (CPU), a programmable logic device implemented for a specific purpose (CPLD, FPGA), an application specific semiconductor computing device (ASIC), or a microcontroller chip.

[0078] FIG. 7 is a flowchart illustrating an object recognition-based indoor site detection method according to an embodiment of the present invention.

[0079] 7, when the location estimation application is executed, the terminal device 100 drives the photographing unit 130 to photograph an image and recognizes the first and second objects in the photographed image (S510). At this time, the terminal device 100 may recognize the first and second objects using a deep learning technique.

[0080] When step S510 is performed, the terminal device 100 estimates the virtual space positions of the first and second objects, respectively (S520). At this time, the terminal device 100 can estimate the virtual space positions of the first and second objects, respectively, using a point cloud generated in the virtual space.

[0081] When step S520 is performed, the terminal device 100 estimates the virtual space distance between each of the first and second objects and the user using the virtual space positions of the first and second objects and the virtual space position of the user (S530). At this time, the terminal device 100 can track the virtual space position of the user using a dead reckoning algorithm. When the virtual space position of the user is estimated, the terminal device 100 can calculate a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user, and can calculate a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user.

[0082] When step S530 is performed, the terminal device 100 estimates the user's position on the indoor map by using the virtual space positions of the first and second objects, the virtual space position of the user, and the Euclidean distance between the virtual space positions of the first and second objects and the virtual space position of the user (S540). For a detailed description of how the terminal device 100 estimates the user's position on the indoor map, please refer to FIG. 8.

[0083] FIG. 8 is a flowchart illustrating a method for estimating a user's position using a cross product according to an embodiment of the present invention.

[0084] 8, the terminal device 100 calculates the cross product of two vectors from the user to the first and second objects in the virtual space (S610). That is, the terminal device 100 may calculate the cross product of the virtual space, which is the cross product of a first virtual vector from the virtual space position of the user to the virtual space position of the first object and a second virtual vector from the virtual space position of the user to the virtual space position of the second object.

[0085] When step S610 is performed, the terminal device 100 obtains the cross product of two vectors from the user of the indoor map to the first and second objects (S620). That is, the terminal device 100 may obtain the first indoor cross product of the first and second real vectors, which are vectors from the first indoor predicted position to the indoor coordinates of the first and second objects, and the second indoor cross product of the first and second real vectors, which are vectors from the second indoor predicted position to the indoor coordinates of the first and second objects.

[0086] When step S620 is performed, the terminal device 100 estimates a point on the indoor map where the user can be located, which has the same sign as the cross product of the virtual space, as the user's location (S630). That is, the terminal device 100 can estimate an indoor predicted location of the indoor cross product of the first and second indoor cross products, which has the same sign as the cross product of the virtual space, as the user's indoor location.

[0087] As described above, the terminal device, service server, and method for an object recognition-based indoor location according to one embodiment of the present invention estimates a user's location using an image captured by a camera of the user terminal device in an indoor environment, thereby enabling accurate estimation of an indoor location without constructing expensive equipment or infrastructure.

[0088] Although the present invention has been described with reference to the embodiments illustrated in the drawings, this is by way of example only, and those skilled in the art will recognize that various modifications and equivalent alternative embodiments are possible.

[0089] Therefore, the true technical scope of the present invention should be defined by the following claims.

Claims

1. A storage unit in which an indoor map is stored with matching preset object-specific coordinate values; Photography department; and a control unit that recognizes first and second objects in an image acquired by the photographing unit, estimates virtual space positions of the first and second objects, estimates virtual space distances between the first and second objects and the user using the virtual space positions of the first and second objects and a virtual space position of a user, and estimates an indoor position of the user using the virtual space distances; The control unit is a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user; a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user; indoor coordinates of the first and second objects are acquired from the indoor map; a first indoor predicted position and a second indoor predicted position of the user are estimated using the first and second Euclidean distances and the indoor coordinates of the first and second objects; and a first virtual vector and a second virtual vector which are vectors to the virtual space positions of the first and second objects based on the virtual space position of the user. a first indoor outer product which is the outer product of a first real vector and a second real vector which are vectors to the indoor coordinates of the first and second objects based on the first indoor predicted position and a second indoor outer product which is the outer product of a first real vector and a second real vector which are vectors to the indoor coordinates of the first and second objects based on the second indoor predicted position, and an indoor predicted position of an indoor outer product of the first and second indoor outer products which has a sign matching that of the virtual space outer product among the first and second indoor outer products is estimated as the indoor position of the user.

2. Further comprising a display unit, The terminal device of claim 1, wherein the control unit displays the user's location on the indoor map together with the image captured by the photographing unit on the display unit.

3. The control unit is The terminal device according to claim 1 , further comprising: a first object detecting unit that detects the first and second objects based on the first and second objects; and a point cloud generated in a virtual space that detects the first and second objects based on the first and second objects.

4. The control unit is The terminal device according to claim 1 , wherein the virtual space position of the user is estimated using a Dead Reckoning algorithm.

5. The communication unit further includes a communication unit for communicating with a service server through a communication network, The control unit is 2. The terminal device of claim 1, wherein, when an image collection application stored in the storage unit is executed, the terminal device photographs preselected objects through the photographing unit, stores images of the photographed objects, and transmits the photographed images of the stored objects to the service server through the communication unit.

6. A communication unit for receiving a photographed image of each object set on the indoor map; and 6. A service server comprising: an object recognition model generation unit that learns the photographed image of each object received from the terminal device of claim 5 through the communication unit, and matches the object and its coordinates through the photographed image of each learned object to generate an object recognition model for object recognition.

7. 7. The service server of claim 6, further comprising a position estimation unit that, when a position estimation request signal including a photographed image is received from a terminal device through the communication unit, inputs the photographed image into the object recognition model to recognize first and second objects, respectively, estimates virtual space positions of the first and second objects, respectively, estimates virtual space distances between the first and second objects and the user using the virtual space positions of the first and second objects and the virtual space position of a user, respectively, estimates an indoor position of the user using the virtual space distances, and transmits the estimated indoor position to the terminal device.

8. recognizing first and second objects in an image acquired by the terminal device through a photographing unit, and estimating virtual space positions of the first and second objects, respectively; The terminal device estimates virtual space distances between the first and second objects and the user, respectively, using the virtual space positions of the first and second objects and the virtual space position of the user; and The terminal device estimates an indoor position of the user by using the virtual space distance; The step of estimating an indoor position of the user includes: the terminal device calculating a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user, and calculating a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user; acquiring indoor coordinates of the first and second objects from an indoor map to which preset coordinate values ​​of each object are matched, and estimating a first predicted indoor position and a second predicted indoor position of the user by using the first and second Euclidean distances and the indoor coordinates of the first and second objects; The terminal device calculates a cross product of a virtual space, which is a cross product of a first virtual vector and a second virtual vector, which are vectors from the virtual space position of the user to the virtual space positions of the first and second objects; calculating a first indoor outer product, which is a cross product of a first real vector and a second real vector, which are vectors from the first indoor predicted position to the indoor coordinates of the first and second objects, and a second indoor outer product, which is a cross product of a first real vector and a second real vector, which are vectors from the second indoor predicted position to the indoor coordinates of the first and second objects; and The method for providing an object recognition-based indoor location service includes estimating an indoor predicted location of an indoor / outer product of the first and second indoor / outer products, the indoor / outer product having a sign that is the same as the outer product of the virtual space, as the indoor location of the user.

9. In the step of estimating virtual space positions of the first and second objects, 9. The object recognition based indoor position service method of claim 8, wherein the terminal device recognizes the first and second objects using deep learning, and estimates virtual space positions of the first and second objects, respectively, using a point cloud generated in a virtual space.

10. The step of estimating each of the virtual space distances includes: The method of claim 8, further comprising: estimating the virtual space position of the user by the terminal device using a dead reckoning algorithm.

Images