Terminal device, service server, and method for indoor positioning based on object recognition

Abstract

A terminal device, a service server, and a method thereof for indoor positioning based on object recognition are disclosed. The terminal device according to an aspect of the present invention includes: a storage unit for storing an indoor map in which preset coordinate values are matched with each object; a photographing unit; and a control unit for recognizing a first object and a second object from an image obtained through the photographing unit, estimating corresponding virtual space positions of the first object and the second object, estimating virtual space distances between the user and each of the first object and the second object by using the virtual space position of the user and the virtual space positions of the first object and the second object, and estimating the indoor position of the user by using the virtual space distances.

Description

Technical Field

[0001] The present invention relates to a terminal device, a service server, and a method for indoor positioning based on object recognition, and more particularly, to a terminal device, a service server, and a method for indoor positioning based on object recognition that can recognize an object from an image captured by a capturing unit of a terminal device and can match the recognized object to an indoor map having vectorized coordinates to estimate the position of a user. Background Art

[0002] Various services for providing selective data to multiple users at a desired location, that is, various location-based services (LBSs) based on the current location of a user, such as a real-time data pop-up service, a selective data transmission service according to the user's location, and an indoor navigation service, are being provided.

[0003] These services are based on technologies for measuring the position of a user. According to the LBS, the position of a user can be measured using WiFi, beacons, etc., and services such as an indoor navigation service can be provided. To appropriately provide such an LBS, it is important to accurately find the position of the user. Here, in the case of using WiFi to find the position of a user, the magnitude of the WiFi received signal for position calculation fluctuates very greatly, making it difficult to provide an appropriate LBS. In the case of using a beacon transmitter, it may be difficult to measure the position of a user according to the arrangement interval of the beacon transmitters. For example, in order to measure the position of a user using a beacon, the distance between the beacon and the user should be accurately measured, but the error increases as the distance between the beacon and the user increases. In addition, a problem with conventional indoor positioning technologies is that expensive equipment or infrastructure is required.

[0004] Therefore, it is necessary to develop a technology for estimating an indoor position without expensive equipment or infrastructure.

[0005] As a related art of the present invention, there is Korean Patent Publication No. 2011-00250250. Summary of the Invention

[0006] Technical Problem

[0007] The present invention aims to solve the above problems and aims to provide a terminal device, a service server, and a method for object recognition for indoor positioning that can recognize an object from an image captured by a capturing unit of a terminal device and can use the object to estimate the position of a user.

[0008] The object of the present invention is not limited to the above-described object, and other objects not described yet can be clearly understood by those of ordinary skill in the art from the following description.

[0009] Technical Solution

[0010] One aspect of the present invention provides a terminal device, which includes: a storage unit configured to store an indoor map, on which a coordinate value is preset to match each object; a photographing unit; and a control unit configured to identify a first object and a second object from an image obtained through the photographing unit, estimate a virtual space position of the first object and a virtual space position of the second object, estimate a virtual space distance between the user and each of the first object and the second object by using the virtual space position of the first object, the virtual space position of the second object, and the virtual space position of the user, and estimate the indoor position of the user by using the virtual space distance.

[0011] In the present invention, the terminal device may further include a display unit, and the control unit may display the position of the user on the indoor map through the display unit while displaying the image obtained through the photographing unit.

[0012] In the present invention, the control unit may use deep learning to identify the first object and the second object, and use a point cloud generated in the virtual space to estimate each of the virtual space position of the first object and the virtual space position of the second object.

[0013] In the present invention, the control unit may use a dead reckoning algorithm to estimate the virtual space position of the user.

[0014] In the present invention, the control unit may calculate a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user, and calculate a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user.

[0015] In the present invention, the control unit may obtain the indoor coordinates of each of the first object and the second object from the indoor map, estimate the first indoor predicted position and the second indoor predicted position of the user by using the first Euclidean distance and the second Euclidean distance and the indoor coordinates of the first object and the second object, calculate a virtual space outer product corresponding to the outer product of a first virtual vector and a second virtual vector that are corresponding vectors from the virtual space position of the user to the virtual space positions of the first object and the second object, calculate a first indoor outer product corresponding to the outer product of a first real vector and a second real vector that are vectors from the first indoor predicted position to the indoor coordinates of the first object and the second object and a second indoor outer product corresponding to the outer product of the first real vector and the second real vector that are vectors from the second indoor predicted position to the indoor coordinates of the first object and the second object, and estimate the indoor predicted position where the first indoor outer product or the second indoor outer product has the same sign as the virtual space outer product as the indoor position of the user.

[0016] In the present invention, the terminal device may further include a communication unit configured to communicate with a service server through a communication network. Wherein, when an image collection application stored in the storage unit is executed, the control unit captures a predetermined object through the photographing unit, stores the captured image of each of the objects, and sends the captured image of each of the stored objects to the service server through the communication unit.

[0017] Another aspect of the present invention provides a service server, which includes: a communication unit configured to receive the captured images of each object set on an indoor map; and an object recognition model generation unit configured to learn the captured images of each object received through the communication unit and generate an object recognition model for object recognition by matching each object with the coordinates of the object through the captured images of the learned object.

[0018] In the present invention, the service server may further include a position estimation unit configured to: when receiving a position estimation request signal including a captured image from a terminal device through the communication unit, identify a first object and a second object by inputting the captured image into the object recognition model, estimate the virtual space position of the first object and the virtual space position of the second object, estimate the virtual space distance between the user and each of the first object and the second object by using the virtual space position of the first object, the virtual space position of the second object, and the virtual space position of the user, estimate the indoor position of the user by using the virtual space distance, and send the estimated indoor position to the terminal device.

[0019] Another aspect of the present invention provides an indoor positioning service method based on object recognition, the method including: identifying, by a terminal device, a first object and a second object from an image acquired by a capturing unit and estimating the virtual space position of the first object and the virtual space position of the second object; estimating, by the terminal device, the virtual space distance between the user and each of the first object and the second object by using the virtual space position of the first object, the virtual space position of the second object, and the virtual space position of the user; and estimating, by the terminal device, the indoor position of the user by using the virtual space distance.

[0020] In the present invention, estimating the virtual space position of the first object and the virtual space position of the second object may include: identifying, by the terminal device, the first object and the second object by using deep learning and estimating each of the virtual space position of the first object and the virtual space position of the second object by using a point cloud generated in the virtual space.

[0021] In the present invention, estimating the virtual space distance may include: estimating, by the terminal device, the virtual space position of the user by using a dead reckoning algorithm; and calculating, by the terminal device, a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user and a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user.

[0022] In the present invention, estimating the indoor position of the user may include: obtaining, by the terminal device, each of the indoor coordinates of the first object and the indoor coordinates of the second object from an indoor map, and estimating a first indoor predicted position and a second indoor predicted position of the user by using the first Euclidean distance, the second Euclidean distance, the indoor coordinates of the first object, and the indoor coordinates of the second object; calculating, by the terminal device, a virtual space outer product corresponding to the outer product of a first virtual vector and a second virtual vector that are corresponding vectors from the virtual space position of the user to the virtual space positions of the first object and the second object; calculating, by the terminal device, a first indoor outer product corresponding to the outer product of a first real vector and a second real vector that are vectors from the first indoor predicted position to the indoor coordinates of the first object and the indoor coordinates of the second object, and a second indoor outer product corresponding to the outer product of the first real vector and the second real vector that are vectors from the second indoor predicted position to the indoor coordinates of the first object and the indoor coordinates of the second object; and estimating, by the terminal device, an indoor predicted position in which the first indoor outer product or the second indoor outer product has the same sign as the virtual space outer product as the indoor position of the user.

[0023] Advantageous Effects

[0024] According to the present invention, the position of the user is estimated by using an image captured in an indoor environment by a camera of a user terminal device, so that the indoor position can be accurately estimated without expensive equipment or infrastructure.

[0025] Meanwhile, the effects of the present invention are not limited to the above-described effects, and may include various effects within the scope that are obvious to those of ordinary skill in the art from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a conceptual diagram illustrating indoor positioning based on object recognition according to an embodiment of the present invention.

[0027] Figure 2 is a diagram illustrating an indoor positioning system based on object recognition according to an embodiment of the present invention.

[0028] Figure 3 is a block diagram illustrating a configuration of a terminal device according to an embodiment of the present invention.

[0029] Figure 4 is an example diagram illustrating a position estimation screen according to an embodiment of the present invention.

[0030] Figure 5 is an example diagram illustrating a method of estimating the indoor position of a user by using an outer product according to an embodiment of the present invention.

[0031] Figure 6 is a block diagram showing the configuration of a service server according to an embodiment of the present invention.

[0032] Figure 7 is a flowchart showing a method for indoor positioning based on object recognition according to an embodiment of the present invention.

[0033] Figure 8 is a flowchart showing a method for estimating a user's position using an outer product according to an embodiment of the present invention. Detailed implementation manners

[0034] Hereinafter, a terminal device, a service server, and a method thereof for indoor positioning based on object recognition according to an embodiment of the present invention will be described with reference to the accompanying drawings. In this process, the thickness of the lines or the size of the components shown in the drawings may be exaggerated for clarity and convenience of description.

[0035] The terms described below are terms defined in consideration of their functions in the present invention and may vary according to the intention or habit of the user or operator. Therefore, these terms should be defined based on the entire content of this specification.

[0036] In addition, the embodiments described herein can be implemented as, for example, a method or process, a device, a software program, or a data stream or signal. Even when an embodiment is discussed only in one implementation form (e.g., only as a method), the features discussed can be implemented in another form (e.g., a device or a program). The device can be implemented as appropriate hardware, software, firmware, etc. The method can be implemented by a device such as a processor, which generally includes a processing device, and the processing device includes a computer, a microprocessor, an integrated circuit, a programmable logic device, etc. The processor also includes a communication device, such as a computer, a cellular phone, a portable / personal information terminal (personal digital assistant (PDA)), or other devices that facilitate information communication between end users.

[0037] Figure 1 is a conceptual diagram showing indoor positioning based on object recognition according to an embodiment of the present invention.

[0038] Referring to Figure 1 , the user acquires an image through the photographing unit of the terminal device 100, and the terminal device 100 identifies a preset first object and a second object from the acquired image. Thereafter, the terminal device 100 matches the identified first object and second object to a vectorized indoor map, thereby estimating the user's indoor position.

[0039] The following will refer to Figure 2 to describe such a system for estimating a user's indoor position through object recognition.

[0040] Figure 2 FIG. is a diagram showing an indoor positioning system based on object recognition according to an embodiment of the present invention.

[0041] Referring to Figure 2 , an indoor positioning system based on object recognition according to an embodiment of the present invention includes a manager terminal 100a, a service server 200, and a user terminal 100b. Here, the manager terminal 100a, the service server 200, and the user terminal 100b can be connected through various forms of wireless communication networks such as WiFi, 3G, Long Term Evolution (LTE), etc.

[0042] The manager terminal 100a maps the objects selected for user position estimation to an indoor map. In other words, the manager can select the objects to be used for user position estimation. Here, the objects can be selected from stationary objects (e.g., billboards, signs, fire hydrants, etc.), and objects with unique features indoors can be mainly selected. Thereafter, the manager maps the selected objects to a pre-made indoor map. In other words, the manager stores the coordinate values of each of the selected objects on the indoor map. Here, the indoor map can be a digital (vectorized) map made using a computer-aided design (CAD) drawing, a point cloud map, a lidar map, an image map, etc., and the digital map can be a map that can be used by the manager terminal 100a and the user terminal 100b. In addition, the indoor map can include key information of the corresponding indoor area. For example, in the case of a shopping mall, the indoor map can include boundaries for differentiating between stores, store names, etc.

[0043] In addition, the manager terminal 100a stores an image collection application, stores images of the objects taken through the image collection application to train a deep learning network for object recognition, and sends the stored object-specific captured images to the service server 200. Here, the captured images can be images of the objects taken in various directions.

[0044] As described above, the manager terminal 100a captures images including the preselected objects in various directions and provides the object-specific captured images to the service server 200 so that the object-specific captured images can be used as training data for object recognition.

[0045] The service server 200 collects the captured images of each object set on the indoor map from the manager terminal 100a, and generates an object recognition model by learning the collected object-specific captured images. Here, the service server 200 can use deep learning to generate the object recognition model. Specifically, when receiving the object-specific captured image, the service server 200 trains a deep learning network that stores the object name and the image pixel coordinate values of the four vertices of the minimum quadrilateral (hereinafter referred to as "bounding box") including the corresponding object. The deep learning network can be designed with various models related to object recognition. For example, the YOLO (you only look once) network can be used.

[0046] In addition, when receiving a position estimation request signal including a captured image from the user terminal 100b, the service server 200 inputs the captured image into the position estimation model to estimate the position of the user terminal 100b.

[0047] The service server 200 will be described in detail below with reference to Figure 6 Detailed description.

[0048] A position estimation application is stored in the user terminal 100b. When capturing the surrounding environment through the position estimation application, the user terminal 100b recognizes objects (such as billboards, fire hydrants, frames, doors, etc.) from the captured image, and estimates the user's position using the position coordinates and distance estimation values of the recognized objects.

[0049] On the other hand, although the manager terminal 100a and the user terminal 100b are described separately in this embodiment, the manager terminal 100a and the user terminal 100b can be the same terminal. Therefore, for ease of description, the manager terminal 100a and the user terminal 100b are hereinafter referred to as the terminal device 100.

[0050] Figure 3 is a block diagram showing the configuration of a terminal device according to an embodiment of the present invention, Figure 4 is an example diagram showing a position estimation screen according to an embodiment of the present invention, and Figure 5 is an example diagram showing a method for estimating a user's indoor position using the outer product according to an embodiment of the present invention.

[0051] Referring to Figure 3 According to an embodiment of the present invention, the terminal device 100 includes a communication unit 110, a storage unit 120, a photographing unit 130, a display unit 140, and a control unit 150.

[0052] The communication unit 110 is a component for communicating with the service server 200 via a communication network and can send and receive various types of information, such as images obtained by the photographing unit 130. The communication unit 110 can be implemented in one of various forms such as a short-range communication module, a wireless communication module, a mobile communication module, a wired communication module, etc.

[0053] The storage unit 120 is a component for storing data related to the operation of the terminal device 100. As the storage unit 120, a known storage medium can be used. For example, one or more of known storage media such as read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), random access memory (RAM), etc. can be used. Specifically, an indoor map in which coordinate values match each preselected object can be stored in the storage unit 120. In addition, an image collection application for operating the photographing unit 130 to obtain a photographed image of each preselected object to collect training data for generating an object recognition model can be stored in the storage unit 120. In addition, a position estimation application for estimating the current position using an image can be stored in the storage unit 120.

[0054] When the image collection application or the image estimation application is executed, the photographing unit 130 obtains an image and sends the obtained image to the control unit 150. The photographing unit 130 can be, for example, a camera.

[0055] The display unit 140 is a component for displaying various types of information related to the operation of the terminal device 100. In particular, the display unit 140 can display an image collection screen when the image collection application is executed, and can display a position estimation screen when the position estimation application is executed. The display unit 140 can also operate as an input unit for receiving information from the user.

[0056] When the image collection application stored in the storage unit 120 is executed, the control unit 150 operates the photographing unit 130, stores the object-specific image photographed by the photographing unit 130, and sends the photographed image of each object stored to the service server 200. In other words, when the image collection for the preselected object is completed in each direction, the control unit 150 sends the collected object and the coordinates of the object on the indoor map to the service server 200. Here, the number of objects on the indoor map, the photographing direction, etc. can be preset by the administrator.

[0057] In addition, when the position estimation application stored in the storage unit 120 is executed, the control unit 150 operates the imaging unit 130 to identify an object (e.g., a billboard, a fire hydrant, a frame, a door, etc.) from the image captured by the imaging unit 130, and estimates the user's position by using the position coordinates and distance estimation values of the identified object.

[0058] In other words, the control unit 150 identifies a first object and a second object from the image obtained by the imaging unit 130, estimates the virtual space position of each of the first object and the second object, estimates the virtual space distance between the user and each of the first object and the second object by using the virtual space positions of the first object and the second object and the user's virtual space position, and estimates the user's indoor position by using the virtual space distance. Here, the virtual space may be the space shown on the screen.

[0059] A method for the control unit 150 to estimate the user's indoor position will be described in detail below.

[0060] When an image captured by the imaging unit 130 is obtained by executing the position estimation application, the control unit 150 generates a point cloud in the space of the image (the virtual space shown on the screen, hereinafter referred to as "virtual space"). Each point cloud has coordinates in the virtual space. Therefore, at the time point when the generation of the point cloud map starts (i.e., the time point when the user starts moving), the coordinates can be set to [0,0]. When the user moves around in the indoor space, a point cloud map for the indoor environment is generated, and each point has coordinates in the virtual space.

[0061] The control unit 150 identifies a pre-specified first object and a second object from the image obtained by the imaging unit 130, and estimates each of the virtual space positions of the first object and the second object by using the point cloud generated in the virtual space.

[0062] In other words, the control unit 150 uses deep learning technology to identify the first object and the second object, and outputs a first bounding box including the first object and a second bounding box including the second object. Here, the control unit 150 can use the YOLO network to identify the first object and the second object, and at least one of the object name, coordinate value, and bounding box length can be displayed together with the first bounding box and the second bounding box. When the control unit 150 identifies the first object and the second object, the control unit 150 can output recognition results such as Figure 4 a captured image 310 of the position estimation screen 300 shown in. Refer to Figure 4 ., a bounding box A including an object called "sunshade" can be displayed, and the coordinates [x,y] and [x1,y1] and two side lengths d and h can be displayed together with the bounding box.

[0063] As described above, when an image including a learned object is input to the deep learning network, the control unit 150 outputs an object name and four vertex coordinates based on the image pixels including the bounding box of the object.

[0064] Subsequently, the control unit 150 selects the most central point within each of the first bounding box and the second bounding box, and estimates the selected point as the virtual space positions of the first object and the second object. In other words, the control unit 150 may select the most central point within each of the bounding box of the first object and the bounding box of the second object, and estimate the coordinates of the selected point as virtual space coordinates.

[0065] When the virtual space positions of the first object and the second object are estimated as described above, the control unit 150 estimates the virtual space distances between the user and each of the first object and the second object by using the virtual space positions of the first object and the second object and the virtual space position of the user.

[0066] In other words, the control unit 150 continuously tracks the position of the user starting from [0,0] in the virtual space through sensors (e.g., gyroscope, accelerometer, geomagnetic sensor, etc.) installed in the terminal device 100. The control unit 150 may use a dead reckoning algorithm to track the virtual space position of the user. The dead reckoning algorithm may be an algorithm for estimating the travel distance and direction of the user based on sensors (not shown) installed in the terminal device 100 and tracking the movement of the user.

[0067] When the virtual position of the user is estimated, the control unit 150 calculates a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user and a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user. Since the first Euclidean distance and the second Euclidean distance are scalar values, the distance values in the virtual space may be equal to the distance values on the indoor map.

[0068] The control unit 150 may estimate the actual position of the user on the indoor map by using the virtual space positions of the first object and the second object, the virtual space position of the user, and the Euclidean distances between the virtual space position of the user and the virtual space positions of the first object and the second object. Here, the actual positions of the first object and the second object on the indoor map may have preset coordinate values.

[0069] Therefore, the control unit 150 can obtain the indoor coordinates (actual positions) of the first object and the second object from the indoor map, and estimate the first indoor predicted position and the second indoor predicted position of the user by using the first Euclidean distance, the second Euclidean distance, and the indoor coordinates of the first object and the second object. In other words, since the positions of the user, the first object, and the second object in the virtual space are known, the control unit 150 can draw a figure in a triangular shape. The positions of the first object and the second object and the two distances between the user and the first object and the second object can be obtained from the indoor map. As a result, when the figure in the virtual space is matched to the indoor space, the user can appear at only two points on the indoor map. In order to select the position of the user from the two points, it is necessary to know in what directions the first object and the second object are located with respect to the user.

[0070] Therefore, the control unit 150 can utilize the concept of the cross product or vector product. In the virtual space, the cross product of the two vectors from the user to the first object and the second object has a direction. This direction should be the same as the direction in the indoor map. In other words, the cross product of the two vectors from the user to the first object and the second object in the virtual space should have the same sign as the cross product of the two vectors from the user to the first object and the second object on the indoor map. As a result, among the two points where the user may appear on the indoor map, the point on the indoor map where the cross product has the same sign as the cross product in the virtual space can finally be estimated as the position of the user.

[0071] Specifically, the control unit 150 calculates the virtual space cross product of the first virtual vector and the second virtual vector, which are vectors from the virtual space position of the user to the virtual space positions of the first object and the second object. In other words, the control unit 150 can calculate the virtual space cross product, which is the cross product of the first virtual vector from the virtual space position of the user to the virtual space position of the first object and the second virtual vector from the virtual space position of the user to the virtual space position of the second object.

[0072] Thereafter, the control unit 150 may calculate a first indoor-outdoor product and a second indoor-outdoor product. The first indoor-outdoor product is the cross product of a first real vector and a second real vector, which are vectors from the first indoor predicted position to the indoor coordinates of the first object and the second object. The second indoor-outdoor product corresponds to the cross product of the first real vector and the second real vector, which are vectors from the second indoor predicted position to the indoor coordinates of the first object and the second object. In other words, the control unit 150 may calculate the first indoor-outdoor product, which is the cross product of the first real vector from the first indoor predicted position to the indoor coordinates of the first object and the second real vector from the first indoor predicted position to the indoor coordinates of the second object. In addition, the control unit 150 may calculate the second indoor-outdoor product, which is the cross product of the first real vector from the second indoor predicted position to the indoor coordinates of the first object and the second real vector from the second indoor predicted position to the indoor coordinates of the second object.

[0073] Subsequently, the control unit 150 may estimate the indoor predicted position where the first indoor-outdoor product or the second indoor-outdoor product has the same sign as the virtual space cross product as the user's indoor position.

[0074] Referring to Figure 5 , assume that the actual coordinates of the first object and the second object are p1 and p2, and the distances from the actual coordinates of the first object and the second object are d1 and d2. In this case, the possible positions of the user on the indoor map may be two points a and b, and it is necessary to determine one of the two points as the user's position. To determine the user's position from the two possible points, the directions of the first object and the second object shown in the virtual space should be considered.

[0075] When p2 is on the right side of p1 in the virtual space, the cross product d2×d1 of the two vectors may have a positive (+) sign, and when p2 is on the left side of p1 in the virtual space, the cross product d2×d1 of the two vectors may have a negative (-) sign. For the possible positions a and b of the user on the indoor map, the cross product d2×d1 may be calculated, and the user position where the cross product on the indoor map has the same sign as the cross product calculated in the virtual space may be determined as the user's final position.

[0076] As described above, the control unit 150 can identify two objects through the photographing unit 130, and estimate the user's position by using the position coordinates and distance estimation values of the two identified objects. In this way, according to the present invention, a vectorized indoor map is used so that the user can estimate his or her position by only identifying two objects via the photographing unit 130 of the terminal device 100. Specifically, objects for estimating the user's position are pre-mapped onto the indoor map. In other words, the coordinates of the objects on the indoor map are stored. After that, when two objects are identified from the virtual space shown on the screen of the terminal device 100, the user's position can be estimated by matching the geometric shape formed by the user and the two objects on the indoor map with the geometric shape formed by the user and the two objects in the virtual space.

[0077] When the indoor position of the user is estimated, the control unit 150 displays the user's position on the indoor map through the display unit 140 while displaying the captured image. In other words, when the user's position is estimated through the position estimation application, the control unit 150 can display a position estimation screen 300, and the position estimation screen 300 includes a captured image 310 and an indoor map 320 on which the position B of the user is shown.

[0078] The control unit 150 may include at least one computing device. The computing device may be a general-purpose central processing unit (CPU), a programmable device element (complex programmable logic device (CPLD) or field programmable array (FPGA)) appropriately implemented for a specific purpose, an on-demand semiconductor processing unit (application specific integrated circuit (ASIC)), or a microcontroller chip.

[0079] On the other hand, the terminal device 100 configured as described above may be an electronic device that can photograph the environment through the photographing unit 130 and can be applied to various wired and wireless environments. For example, the terminal device 100 may be a PDA, a smart phone, a cellular phone, a personal communication service (PCS) phone, a global system for mobile communications (GSM) phone, a wideband code division multiple access (CDMA) phone, a CDMA-2000 phone, a mobile broadband system (MBS) phone, etc. The terminal device 100 may be a small portable device. However, when the terminal device 100 is a portable camera or a laptop computer, it may be referred to as a mobile communication terminal. Therefore, the terminal device 100 is not limited to a small portable device.

[0080] Figure 6 is a block diagram showing the configuration of a service server according to an embodiment of the present invention.

[0081] Refer to Figure 6, a service server 200 according to an embodiment of the present invention includes a communication unit 210, a storage unit 220, an object recognition model generation unit 230, and a control unit 250.

[0082] The communication unit 210 receives an object-specific captured image from the terminal device 100.

[0083] The storage unit 220 is a component for storing data related to the operation of the service server 200. In particular, an indoor map storing coordinate values of each preselected object can be stored in the storage unit 220.

[0084] The object recognition model generation unit 230 receives an object-specific captured image from the terminal device 100 through the communication unit 210, learns the received captured image of each object, and generates an object recognition model for object recognition. Here, the object recognition model generation unit 230 can use deep learning to generate the object recognition model. The object recognition model can have coordinate values of each object. Therefore, when an image with an unknown position is input, the object recognition model can calculate the object in the image and the coordinates of the object as output values.

[0085] On the other hand, the service server 200 according to the present invention may further include a position estimation unit 240. When the position estimation unit 240 receives a position estimation request signal including a captured image from the terminal device 100 through the communication unit 210, it identifies a first object and a second object by inputting the captured image into the object recognition model, estimates the virtual space positions of the first object and the second object, estimates the virtual space distances between the user and each of the first object and the second object using the virtual space positions of the first object and the second object and the virtual space position of the user, estimates the indoor position of the user using the virtual space distances, and sends the estimated indoor position to the terminal device. When the first object and the second object are identified, the position estimation unit 240 can display the first object and the second object on the screen of the terminal device 100.

[0086] On the other hand, the object recognition model generation unit 230 and the position estimation unit 240 can be respectively implemented by a processor or the like required to execute a program on a computing device. As described above, the object recognition model generation unit 230 and the position estimation unit 240 can be implemented by physically independent components, or in a form functionally divided in one processor.

[0087] The control unit 250 is a component for controlling the operations of various constituent units of the service server 200, including the communication unit 210, the storage unit 220, the object recognition model generation unit 230, and the position estimation unit 240, and may include at least one computing device. The computing device may be a general-purpose CPU, a programmable device element (CPLD or FPGA) appropriately implemented for a specific purpose, an on-demand semiconductor processing unit (ASIC), or a microcontroller chip.

[0088] Figure 7 is a flowchart showing a method for indoor positioning based on object recognition according to an embodiment of the present invention.

[0089] Referring to Figure 7 , when the position estimation application is executed, the terminal device 100 captures an image by operating the capture unit 130 and recognizes a first object and a second object from the captured image (S510). Here, the terminal device 100 may use deep learning technology to recognize the first object and the second object.

[0090] After performing operation S510, the terminal device 100 estimates the virtual space position of each of the first object and the second object (S520). Here, the terminal device 100 may use the point cloud generated in the virtual space to estimate the virtual space position of each of the first object and the second object.

[0091] After performing operation S520, the terminal device 100 estimates the virtual space distance between the user and each of the first object and the second object using the virtual space positions of the first object and the second object and the virtual space position of the user (S530). Here, the terminal device 100 may use a dead reckoning algorithm to track the virtual space position of the user. When the virtual space position of the user is estimated, the terminal device 100 may calculate the first Euclidean distance between the virtual space position of the first object and the virtual space position of the user and the second Euclidean distance between the virtual space position of the second object and the virtual space position of the user.

[0092] After performing operation S530, the terminal device 100 estimates the position of the user on the indoor map using the virtual space positions of the first object and the second object, the virtual space position of the user, and the Euclidean distances between the virtual space position of the user and the virtual space positions of the first object and the second object (S540). The method by which the terminal device 100 estimates the position of the user on the indoor map will be described in detail with reference to Figure 8 The method for the terminal device 100 to estimate the position of the user on the indoor map will be described in detail.

[0093] Figure 8 is a flowchart showing a method for estimating the position of a user using an outer product according to an embodiment of the present invention.

[0094] Reference Figure 8 As shown in Figure 8 , the terminal device 100 calculates the cross product of two vectors from the user to the first object and the second object in the virtual space (S610). In other words, the terminal device 100 can calculate the virtual space cross product corresponding to the cross product of the first virtual vector from the virtual space position of the user to the virtual space position of the first object and the second virtual vector from the virtual space position of the user to the virtual space position of the second object.

[0095] After performing operation S610, the terminal device 100 calculates the cross product of two vectors from the user to the first object and the second object on the indoor map (S620). In other words, the terminal device 100 can calculate the first indoor cross product of the first real vector and the second real vector from the first indoor predicted position to the indoor coordinates of the first object and the second object, and the second indoor cross product of the first real vector and the second real vector from the second indoor predicted position to the indoor coordinates of the first object and the second object.

[0096] After performing operation S620, the terminal device 100 estimates the position of the user as the point among the two points where the user may be present on the indoor map and where the cross product has the same sign as the virtual space cross product (S630). In other words, the terminal device 100 can estimate the indoor predicted position where the first indoor cross product or the second indoor cross product has the same sign as the virtual space cross product as the indoor position of the user.

[0097] As described above, the terminal device, service server, and method for indoor positioning based on object recognition according to an embodiment of the present invention can accurately estimate the position of the user by using an image captured in an indoor environment by the camera of the user terminal device, so that the indoor position can be accurately estimated without expensive equipment or infrastructure.

[0098] The present invention has been described above with reference to the embodiments shown in the drawings. However, the embodiments are merely examples, and those of ordinary skill in the art should understand that various modifications and equivalents can be derived from the embodiments.

[0099] Therefore, the technical scope of the present invention should be determined according to the appended claims.

Claims

1. A terminal device, comprising: a storage unit configured to store an indoor map, on which a preset coordinate value is matched with each object; a photographing unit; and a control unit configured to identify a first object and a second object from an image obtained through the photographing unit, estimate a virtual space position of the first object and a virtual space position of the second object, estimate a virtual space distance between the user and each of the first object and the second object by using the virtual space positions of the first object and the second object and the virtual space position of the user, and estimate the indoor position of the user by using the virtual space distance; wherein the control unit calculates a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user, and calculates a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user, obtains indoor coordinates of each of the first object and the second object from the indoor map, estimates a first indoor predicted position and a second indoor predicted position of the user by using the first Euclidean distance, the second Euclidean distance, the indoor coordinates of the first object, and the indoor coordinates of the second object, calculates a virtual space outer product corresponding to an outer product of a first virtual vector and a second virtual vector, which are corresponding vectors from the virtual space position of the user to the virtual space positions of the first object and the second object, calculates a first indoor outer product corresponding to an outer product of a first real vector and a second real vector, which are vectors from the first indoor predicted position to the indoor coordinates of the first object and the indoor coordinates of the second object, and a second indoor outer product corresponding to an outer product of a first real vector and a second real vector, which are vectors from the second indoor predicted position to the indoor coordinates of the first object and the indoor coordinates of the second object, and estimates the indoor predicted position where the first indoor outer product or the second indoor outer product has the same sign as the virtual space outer product as the indoor position of the user.

2. The terminal device according to claim 1, further comprising a display unit, Among them, wherein the control unit displays the position of the user on the indoor map through the display unit and simultaneously displays the image obtained through the photographing unit.

3. The terminal device according to claim 1, wherein, The control unit uses deep learning to identify the first object and the second object, and uses a point cloud generated in the virtual space to estimate each of the virtual space positions of the first object and the second object.

4. The terminal device according to claim 1, wherein, The control unit uses a dead reckoning algorithm to estimate the virtual space position of the user.

5. The terminal device according to claim 1, further comprising a communication unit configured to communicate with a service server through a communication network, Among them, When the image collection application stored in the storage unit is executed, the control unit captures a predetermined object through the capturing unit, stores the captured images of each of the objects, and transmits the captured images of each of the stored objects to the service server through the communication unit.

6. A server, comprising: A communication unit configured to receive captured images of each object set on an indoor map; An object recognition model generation unit configured to learn the captured images of each object received through the communication unit and generate an object recognition model for object recognition by matching each object with the coordinates of the object via the captured images of the learned objects; And A position estimation unit configured to: when a position estimation request signal including a captured image is received from a terminal device through the communication unit, identify a first object and a second object by inputting the captured image into the object recognition model, estimate the virtual space positions of the first object and the second object, estimate the virtual space distances between the user and each of the first object and the second object by using the virtual space positions of the first object and the second object and the virtual space position of the user, estimate the indoor position of the user by using the virtual space distances, and transmit the estimated indoor position to the terminal device, wherein the position estimation unit calculates a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user, and calculates a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user, obtains the indoor coordinates of each of the first object and the second object, estimates a first indoor predicted position and a second indoor predicted position of the user by using the first Euclidean distance, the second Euclidean distance, the indoor coordinates of the first object, and the indoor coordinates of the second object, calculates a virtual space outer product corresponding to the outer product of a first virtual vector and a second virtual vector that are corresponding vectors from the virtual space position of the user to the virtual space positions of the first object and the second object, calculates a first indoor outer product corresponding to the outer product of a first real vector and a second real vector that are vectors from the first indoor predicted position to the indoor coordinates of the first object and the second object, and a second indoor outer product corresponding to the outer product of the first real vector and the second real vector that are vectors from the second indoor predicted position to the indoor coordinates of the first object and the second object, and estimates the indoor predicted position in which the first indoor outer product or the second indoor outer product has the same sign as the virtual space outer product as the indoor position of the user.

7. An indoor positioning service method based on object recognition, the method comprising: The terminal device identifies a first object and a second object from the images acquired by the imaging unit and estimates the virtual space positions of the first object and the virtual space position of the second object; The terminal device estimates the virtual space distances between the user and each of the first object and the second object by using the virtual space positions of the first object and the second object and the virtual space position of the user; And The terminal device estimates the indoor position of the user by using the virtual space distances, wherein coordinate values are preset and matched with each object on the indoor map, wherein estimating the virtual space distances includes: the terminal device calculates a first Euclidean distance between the virtual space position of the first object and the virtual space position of the user and a second Euclidean distance between the virtual space position of the second object and the virtual space position of the user, wherein estimating the indoor position of the user includes: the terminal device obtains each of the indoor coordinates of the first object and the indoor coordinates of the second object from the indoor map, estimates a first indoor predicted position and a second indoor predicted position of the user by using the first Euclidean distance and the second Euclidean distance and the indoor coordinates of the first object and the indoor coordinates of the second object, the terminal device calculates a virtual space outer product corresponding to the outer product of a first virtual vector and a second virtual vector that are corresponding vectors from the virtual space position of the user to the virtual space positions of the first object and the second object, the terminal device calculates a first indoor outer product corresponding to the outer product of a first real vector and a second real vector that are vectors from the first indoor predicted position to the indoor coordinates of the first object and the indoor coordinates of the second object and a second indoor outer product corresponding to the outer product of a first real vector and a second real vector that are vectors from the second indoor predicted position to the indoor coordinates of the first object and the indoor coordinates of the second object, and the terminal device estimates the indoor predicted position in which the first indoor outer product or the second indoor outer product has the same sign as the virtual space outer product as the indoor position of the user.

8. The indoor positioning service method according to claim 7, wherein, Estimating the virtual space positions of the first object and the second object includes: the terminal device uses deep learning to identify the first object and the second object and uses the point cloud generated in the virtual space to estimate each of the virtual space positions of the first object and the virtual space position of the second object.

9. The indoor positioning service method according to claim 7, wherein, Estimating the virtual space distances includes: The terminal device estimates the virtual space position of the user by using a dead reckoning algorithm.

Images