User equipment and control method for the same

Abstract

A user terminal according to one embodiment may include a camera that captures surrounding images, a display, a communication unit that communicates with a server, a database that stores augmented reality-based VPS (Visual Positioning System) map information and 2D (2 Dimension) map information, and a control unit that generates 2D route information based on delivery information received from the server and the 2D map information, generates augmented reality-based 3D route information based on the delivery information and the VPS map information, and controls the display to display the 2D route information or 3D route information based on the distance between the destination and the user included in the delivery information.

Description

Technology Field

[0001] The present invention relates to a user device (UE) that provides augmented reality (AR) and a method for controlling the same. Background Technology

[0002] Recently, services utilizing Augmented Reality (AR) have been increasing. Users can receive augmented reality services using User Equipment (UE), and can experience more vivid services compared to existing ones.

[0003] For example, map and navigation services utilizing augmented reality can display augmented reality images of Points of Interest (POIs) for various business sectors, such as restaurants, cafes, and grocery stores, at the location of the corresponding POIs. That is, when a user captures a point of interest using a user terminal, the augmented reality image of the point of interest can be superimposed on the captured image and displayed in the area where the point of interest is located.

[0004] However, while conventional augmented reality-based map and navigation services can be effectively applied to finding destinations in densely populated shopping or building areas, continuous updates to the Visual Positioning System (VPS) map are essential due to changes in terrain features and the environment; this imposes a Total Cost of Ownership (TCO) burden on the business operator, which limits commercialization. The problem to be solved

[0005] A user terminal and a control method thereof are provided that can enhance the usability of destination search regardless of indoor / outdoor (environmental attributes) or the presence or absence of GPS (Global Positioning System), and reduce Total Cost of Ownership (TCO) through user participation. means of solving the problem

[0006] A user terminal (User Equipment; UE) according to one embodiment may include a camera that captures surrounding images, a display, a communication unit that communicates with a server, a database that stores augmented reality-based VPS (Visual Positioning System) map information and 2D (2 Dimension) map information, and a control unit that generates 2D route information based on delivery information received from the server and the 2D map information, generates augmented reality-based 3D route information based on the delivery information and the VPS map information, and controls the display to display the 2D route information or 3D route information based on the distance between the destination and the user included in the delivery information.

[0007] The control unit may control the display to display 3D path information when the distance between the destination and the user is less than a predetermined distance.

[0008] The above control unit may generate path information such that at least one of time, travel distance, and congestion is minimized.

[0009] The control unit may identify the user's location information based on the VPS map information and the surrounding image.

[0010] The above control unit may generate real-time route information based on the user's location information and the delivery information.

[0011] The control unit may control the display to composite a destination represented as an augmented reality 3D object onto the surrounding image and display the composited image.

[0012] The above database may update the VPS map information based on the surrounding images.

[0013] The control unit may assign a weight to the surrounding image based on the characteristics of the surrounding image, and provide a reward score when the weight is higher than a predetermined standard.

[0014] The above compensation score may be weighted based on at least one of the resolution, capacity, and length of the surrounding image, and may vary according to the size of the sum of each weight.

[0015] A user terminal control method according to one aspect of the disclosed invention may include storing augmented reality-based VPS (Visual Positioning System) map information and 2D (2 Dimension) map information, receiving delivery information from a server, generating 2D route information based on the delivery information and the 2D map information, generating 3D route information based on the delivery information and the VPS map information, and controlling the display to display 2D route information or 3D route information based on the distance between the destination and the user included in the delivery information. Effects of the invention

[0016] According to a user terminal and a control method thereof in one aspect, the usability of destination search can be enhanced regardless of whether it is indoors or outdoors (environmental attributes) and whether GPS is present, and total cost of ownership (TCO) can be reduced through user participation. Brief explanation of the drawing

[0017] FIG. 1 illustrates an augmented reality (AR) mobility system according to one embodiment. FIG. 2 is a block diagram showing the configuration of a user terminal according to one embodiment. FIG. 3 is a diagram illustrating a case where a user terminal according to one embodiment photographs a destination. FIG. 4 is a diagram illustrating a case where a user terminal according to one embodiment displays path information to a destination. FIG. 5 is a diagram illustrating the case where a user terminal according to one embodiment is close to a destination. FIG. 6 is a diagram illustrating how a user terminal according to one embodiment captures an image and updates a VPS map. FIG. 7 is a flowchart illustrating an embodiment of a user terminal control method according to one embodiment that provides path information to a destination. FIG. 8 is a flowchart illustrating an embodiment of updating a user-participated VPS map based on a captured surrounding image among a user terminal control method according to one embodiment. Specific details for implementing the invention

[0018] Throughout the specification, the same reference numerals refer to the same components. This specification does not describe all elements of the embodiments, and general content in the art to which the invention pertains or content that overlaps between embodiments is omitted.

[0019] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0020] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0021] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0022] In addition, terms such as "~part," "~unit," "~block," "~part," and "~module" may refer to a unit that processes at least one function or operation. For example, the above terms may refer to at least one piece of hardware such as an FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), at least one piece of software stored in memory, or at least one process processed by a processor.

[0023] The symbols attached to each step are used to identify each step and do not indicate the order of the steps relative to one another; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0024] Hereinafter, an embodiment relating to a user terminal and a control method thereof according to one aspect will be described in detail with reference to the attached drawings.

[0025] FIG. 1 illustrates an augmented reality (AR) mobility system (1000) according to one embodiment.

[0026] An augmented reality (AR) mobility system (1000) according to one embodiment may be intended to allow personnel delivering goods, etc., to easily search for information regarding the location of a destination (Point Of Interest), etc., using indoor and outdoor AR navigation based on a map based on a Visual Positioning System (VPS), which is a core technology of augmented reality (AR), and for personnel to directly participate in continuously updating the VPS map.

[0027] In other words, the augmented reality (AR) mobility system (1000) can be applied to services or companies that require the transport or delivery of specific goods, services, etc., such as order brokers, E-commercial companies, department store food halls, convenience stores, courier companies, cosmetic distributors, event operators including conventions, last-mile mobility service providers, and airport corporations.

[0028] Referring to FIG. 1, an augmented reality (AR) mobility system (1000) includes a user terminal (100) that displays a route to a destination or a place adjacent to a destination to a user, or displays an augmented reality image of a destination according to the user's selection, a server (200) that manages VPS map information, 2D map information, and delivery information, and a network (1) that provides communication between the user terminal (100) and the server (200).

[0029] Delivery information may include, for example, location information of the destination. More specifically, delivery information may include, for example, information regarding the goods to be delivered, information regarding the delivery origin, information regarding the delivery destination, information regarding the delivery person's current location, and telephone number information including the telephone numbers of the delivery person and / or the person requesting delivery, but is not limited thereto. Furthermore, for example, information regarding the delivery origin and delivery destination may include not only location information but also all location-related information, such as surrounding video footage and photos taken at the said location.

[0030] Examples of networks (1) include, but are not limited to, 3GPP (3rd Generation Partnership Project) networks, LTE (Long Term Evolution) networks, 5G networks, WIMAX (World Interoperability for Microwave Access) networks, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth networks, satellite broadcasting networks, analog broadcasting networks, DMB (Digital Multimedia Broadcasting) networks.

[0031] Examples of user terminals (100) may include all kinds of wired and wireless communication devices capable of input and output, such as PCS (Personal Communication System), GSM (Global System for Mobile communication), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (WCode Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, smartpad, tablet PC, laptop, wearable device, digital signage, etc., but are not limited thereto.

[0032] Additionally, the user terminal (100) may include a terminal of a delivery person acting on behalf of the delivery, a terminal of a delivery requester requesting a delivery, a terminal of an administrator, etc., but for convenience of explanation, the user terminal (100) will be described below as a terminal of a delivery person acting on behalf of the delivery.

[0033] For example, the user terminal (100) can calculate location information of the user terminal (100) based on surrounding images captured through the user terminal (100), and can generate a path to a destination based on said location information. In addition, the user terminal (100) can synthesize a path expressed as an augmented reality-based 3D object based on the captured surrounding images and provide it to the user. In addition, the user terminal (100) can update a map based on the captured surrounding images to ensure the reliability of the map.

[0034] A user terminal (100) according to one embodiment may assign a weight to a surrounding image captured by a user through the user terminal (100) when the user transmits the image to a server (200), and may provide a reward score to the user based on the weight. For example, the reward score may be in the form of points or mileage for purchasing other products, or it may be a form of accumulated funds that can be used like cash. However, it is not limited thereto.

[0035] Additionally, the user terminal (100) can control the displayed image based on the user's location and the location relative to the destination. That is, the user terminal (100) may output 2D route information based on 2D map information or output 3D route information based on augmented reality-based VPS map information based on criteria such as the distance between the user's location and the destination, and whether it is indoors or outdoors.

[0036] The server (200) can receive requests for information regarding augmented reality (AR) based VPS map information and 2D map information from the user terminal (100). For example, the server (200) may be a server or device that collects information including surrounding images, routes, and the current location of a delivery person captured by the user terminal (100), and may be the same server or device as the user terminal (100), but is not limited thereto.

[0037] FIG. 2 is a block diagram showing the configuration of a user terminal (100) according to one embodiment.

[0038] Referring to FIG. 2, a user terminal (100) according to one embodiment may include a camera (110), a communication unit (120), a display (130), a database (140), a control unit (150), and a sensor unit (160).

[0039] The camera (110) may be provided, for example, on the front and / or rear of the user terminal (100) to acquire images of the front and / or rear of the user terminal (100). That is, the camera (110) may capture and acquire images of the surroundings of the user terminal (100). The camera (110) may be a camera of a type known in the prior art, and is not limited to any specific form or type, such as by using various types of image sensors.

[0040] Meanwhile, the surrounding image captured by the camera (110) may include, for example, image information as well as image information regarding the front and / or rear of the user terminal (100).

[0041] The communication unit (120) can communicate with the server (200) through the network (1) and can receive VPS map information, 2D map information, delivery information, etc. from the server (200) or transmit surrounding images captured by the camera (110) to the server (200). The communication unit (120) may be composed of a wireless communication module of a known type.

[0042] A display (130) is provided on the front and / or rear of a user terminal (100) and can display surrounding images captured by a camera (110), and can superimpose augmented reality 3D objects or images onto surrounding images captured by the camera (110). To this end, the display (130) may be provided as a display module of a known type. Additionally, the display (130) may be provided integrally with an input device capable of receiving user input, for example, and may also be used as an input device for inputting user operations.

[0043] That is, the display (130) may be a device that transmits user input to the control unit (150) based on user input (e.g., operation), but is not limited thereto.

[0044] Meanwhile, in one embodiment, another display (130) may include a CRT (Cathode Ray Tube), FPD (Flat Panel Display), LCD (Liquid Crystal Display), PDP (Plasma Display Panel), LED (Light Emitting Diode), and OLED (Organic Light Emitting Diode) method, but is not limited thereto. Accordingly, conventionally known or future display forms or types may be applied.

[0045] The database (140) can store surrounding image information captured by the camera (110), delivery information, 2D map information, and augmented reality-based VPS map information. For example, the database (140) may store information received by the communication unit (120) from the server (200), including delivery information, 2D map information, and augmented reality-based VPS map information. More specifically, the database (140) may update or update information already stored in the database (140) based on information received from the server (200).

[0046] In another embodiment, the database (140) may be the same device or server as the server (200), but is not limited thereto.

[0047] Augmented reality-based VPS map information may include, for example, information regarding a 3D spatial map of an extensive indoor or outdoor space. More specifically, the VPS map information may be a 3D spatial map generated based on panoramic geometry, or 3D map information previously generated using known technologies, such as generating a 3D spatial map using surrounding images captured after training a spatial map generation model with image data as input through an artificial intelligence learning model, but is not limited thereto. The VPS map information may be information generated through a spatial map generation method to be developed in the future.

[0048] The control unit (150) can generate 2D route information based on delivery information received by the communication unit (120) from the server (200) and 2D map information stored in the database (140). The 2D route information may include, for example, information regarding a 2D route from a user's location to information regarding a destination included in the delivery information, and map information around the route. However, it is not limited thereto.

[0049] The destination may include, for example, the place where the delivery person must pick up the delivered item, as well as the final delivery location, but is not limited thereto.

[0050] Additionally, the control unit (150) can generate 3D route information based on delivery information received from the server (200) by the communication unit (120) and VPS map information stored in the database (140). The 3D route information may include, for example, information regarding a 3D route expressed as an augmented reality 3D object from the user's location to information regarding the destination included in the delivery information, and VPS map information around the route.

[0051] More specifically, the delivery origin point for the user to pick up the delivery product and the delivery destination for completing the delivery are often indoors rather than outdoors. In indoor spaces, such as the interior of a building, for example, in areas with a high concentration of buildings or shops, the destination display on a 2D map may cause difficulties for the user in finding the destination due to errors. The control unit (150) is configured to solve these problems by generating 3D path information expressed as an augmented reality-based 3D object based on VPS map information and delivery information, and can control the display (130) to display the 3D path information.

[0052] That is, the control unit (150) can help the user find a more accurate destination by controlling the display (130) to display 3D path information on the VPS map information.

[0053] Meanwhile, a control unit (150) according to one embodiment of the present invention can identify the user's location information based on VPS map information and surrounding images captured by a camera (110). That is, by comparing the VPS map information stored in the database (140) with images or image information similar to the surrounding images captured by the camera (110), the location information of the user terminal (100) may be identified by calculating location information associated with the images stored in the VPS map information. However, it is not limited thereto.

[0054] In another embodiment, the database (140) may store a VPS location identification model learned from a dataset generated based on, for example, a video or image and metadata including location information associated with the video or image. Accordingly, the control unit (150) may input a surrounding video captured by the camera (110) to the VPS location identification model and output a location associated with the surrounding video captured by the camera (110).

[0055] Meanwhile, in another embodiment, the user terminal (100) may further include a learning processing unit. The learning processing unit may store a deep learning algorithm for training a VPS location identification model. However, it is not limited thereto, and machine learning algorithms known in the past or machine learning algorithms to be developed in the future, including unsupervised learning, reinforcement learning, and supervised learning, may be applied.

[0056] The learning processing unit may, for example, generate metadata that associates surrounding images captured by the camera (110) with location information of the user terminal (100), and may train a VPS location identification model based on said metadata. Here, the location information of the user terminal (100) may, for example, be location information of the user terminal (100) generated based on a GPS sensor included in the sensor unit (160), or location information of the user terminal (100) generated by the control unit (150) based on VPS map information and surrounding images captured by the camera (110), but is not limited thereto.

[0057] Accordingly, the user terminal (100) may, for example, generate a large amount of metadata based on surrounding images captured from a camera (110) and location information of the user terminal (100), and input the metadata into an artificial intelligence algorithm to train a VPS location identification model, thereby improving the reliability and stability of the VPS location identification model.

[0058] Meanwhile, the learning processing unit may input video or image data regarding a location into the learned VPS location identification model to verify the VPS location identification model learned in the manner described above, and output location information corresponding to the video or image data to display on the display (130). In this case, by the user determining whether the video or image data output on the display (130) and the associated location information are correct or incorrect, the learning processing unit may generate metadata with improved reliability and stability. Additionally, the reliability and stability of the VPS location identification model may be improved by inputting enhanced metadata with high reliability and stability. However, it is not limited thereto.

[0059] Meanwhile, a control unit (150) according to one embodiment may generate route information based on delivery information and 2D map information such that at least one of the time taken by the user to reach the destination, travel distance, and congestion is minimized. More specifically, the control unit (150) may generate route information based on the weight of each item when the user sets a weight for each item (time, travel distance, congestion, etc.) through the display (130) of the user terminal (100).

[0060] For example, the control unit (150) can generate route information such that the estimated arrival time is minimized even if the travel distance increases, when the user maximizes the weight for time through the display (130). However, it is not limited to this.

[0061] Meanwhile, a control unit (150) according to one embodiment can generate route information in real time based on VPS map information and surrounding images captured by a camera, and delivery information, and the location information of a user identified based on the surrounding images captured by the camera (e.g., location information generated based on a VPS location identification model). This may be intended to update route information to a destination regardless of whether the user is indoors or outdoors by identifying the location of the moving user terminal (100) based on surrounding images captured by the camera (110) and VPS map information when the user is moving. However, it is not limited thereto.

[0062] Meanwhile, the control unit (150) can control the display (130) to display route information differently depending on the distance between the user's location and the destination (e.g., delivery origin or destination, etc.) as determined by the distance.

[0063] More specifically, the control unit (150) can control the display (130) to display augmented reality-based 3D path information on the surrounding image captured by the camera (110) when the distance between the user's location and the destination is less than a predetermined distance.

[0064] Additionally, the control unit (150) can control the display (130) to display 2D path information on the 2D map information when the distance between the user's location and the destination is greater than a predetermined distance.

[0065] The control unit (150) controls the display (130) to display 2D path information or 3D path information depending on the distance between the user's location and the destination. For example, since the destination (e.g., delivery origin or destination) is often indoors rather than outdoors, there is difficulty in finding the destination due to the user's location error and the properties of the 2D map in the case of indoors, so this may be done to solve the problem.

[0066] In other words, the control unit (150) can control the display (130) to display 3D path information and the destination, which are represented as augmented reality-based 3D objects, on the surrounding image captured by the camera (110) so that it is easier to search for the destination when the distance between the user's location and the destination is smaller than a predetermined distance, as the user is imminent to find the destination.

[0067] Conversely, the control unit (150) can control the display (130) to display 2D path information that is composited with 2D map information to make it easier to search for a path when the distance between the user's location and the destination is greater than a set distance, as finding the correct path is imminent.

[0068] Meanwhile, in this case, the set distance may be, for example, an actual distance entered by the user through the display (130) (e.g., 10m, 20m, etc.), but is not limited thereto.

[0069] In another embodiment, the set distance may be changed based on other criteria, such as the boundary of a building, the width of the road, congestion, or the speed of the user. In other words, when the user's location enters the boundary of a building or passes through a narrow commercial street, the control unit (150) may control the display (130) to display the 3D path by synthesizing it with the surrounding image captured by the camera (110). However, it is not limited thereto. In yet another embodiment, depending on the user's settings, the control unit (150) may control the display (130) to display 2D path information in parallel according to the set distance or other criteria described above.

[0070] In another embodiment, the control unit (150) can control the display (130) to display a 2D path on a 2D map information when there is no VPS map information corresponding to the user's location and thus 3D path information cannot be supported based on the user's location. However, it is not limited thereto, and the control unit (150) can control the display (130) to display a 2D path information depending on the situation in which 3D path information cannot be provided, including network abnormalities.

[0071] Meanwhile, the control unit (150) according to one embodiment can correct the user's location information more quickly and accurately based on the user location information generated from the sensor unit (160).

[0072] More specifically, the control unit (150) generates user location information based on surrounding images captured by the camera (110) and VPS map information. In this case, when generating user location information based on surrounding images captured by the camera (110) and VPS map information, the amount of data processing may be greater. Accordingly, the control unit (150) can extract a group of candidate VPS map information to be compared with surrounding images captured by the camera (110) based on user location information measured by the sensor unit (160). In this case, the control unit (150) may generate user location information more quickly and accurately by comparing the surrounding images captured by the camera (110) with the group of candidate VPS map information. However, it is not limited to this.

[0073] The database (140) can update VPS map information based on surrounding images captured by the camera (110). That is, the database (140) can update the previously stored VPS map information based on surrounding images captured by the camera (110), thereby updating information about alleys, indoors, etc., where terrain changes frequently, through updates to the changed 3D spatial map. However, it is not limited to this.

[0074] In another embodiment, the user terminal (100) may transmit surrounding images captured by the camera (110) to the server (200) via the communication unit (120), and the server (200) may update the VPS map based on the received surrounding images captured by the camera (110). In another embodiment, the administrator of the server (200) may update the VPS map information after conducting an inspection regarding the update of the VPS map based on the received images.

[0075] Meanwhile, the control unit (150) may, for example, assign weights to the image based on the surrounding image captured by the camera (110) in order to provide a reward score. The weights may, for example, be a standard value for evaluating the quality of the image provided by the user, but are not limited thereto.

[0076] More specifically, the control unit (150) may assign weights based on at least one of the resolution, capacity, and length of the image based on the surrounding image captured by the camera (110) in order to prevent updating the VPS map based on image or photo data of poor image quality. For example, the control unit (150) may assign higher weights as the resolution, capacity, and length increase or decrease.

[0077] In this case, the control unit (150) can provide a reward score when the sum of the weights for each item (e.g., resolution, capacity, length, etc.) is higher than the set standard.

[0078] In other words, the control unit (150) can assign weights to each item of the surrounding image captured by the camera (110) based on image evaluation items (e.g., resolution, capacity, length, etc.), and can provide a reward score when the sum of the assigned weights is higher than a set standard.

[0079] Additionally, the control unit (150) may provide a higher reward score as the sum of the assigned weights increases above a predetermined standard. However, it is not limited to this.

[0080] Meanwhile, the established criteria may be, for example, weights set on video or image data of sufficient quality to update the VPS map.

[0081] The sensor unit (160) may include a location sensing sensor, such as a GPS, for example. That is, the sensor unit (160) can generate location information of the user terminal (100) and may further include a tilt sensor, an inertial sensor, etc., capable of detecting the direction and tilt of the user terminal (100). Accordingly, the sensor unit (160) can detect the direction and tilt of the camera (110) of the user terminal (100) and generate shooting direction and tilt information.

[0082] Meanwhile, a learning processing unit according to one embodiment may improve reliability and stability by generating metadata including location information related to video or image data based on the shooting direction of the camera (110) of the user terminal (100) based on the sensors included in the sensor unit (160), and by inputting the metadata into an artificial intelligence algorithm to train a VPS location identification model.

[0083] More specifically, video or image data captured facing the floor may have limitations in determining the location based on VPS map information using only the video or image. Accordingly, the learning processing unit may train a VPS location determination model by excluding video or image data that includes the floor, which makes it difficult to determine the location of the user terminal (100), based on the shooting direction information of the camera (110) and video or image data generated from the sensor unit (160), for example. However, it is not limited thereto.

[0084] FIG. 3 is a diagram illustrating the case where a user terminal (100) according to one embodiment photographs a destination.

[0085] As shown in FIG. 3, the display (130) of a user terminal (100) according to one embodiment may include a status display area (31), a delivery information area (32), a first route information output area (33), and a second route information output area (35).

[0086] The status display area (31) may include, for example, time, wireless signal strength (intensity), and battery level information, but is not limited thereto.

[0087] Additionally, the delivery information area (32) may provide, for example, the type of delivery (pickup, delivery, etc.), whether the delivery item is ready when the type of delivery is pickup, location information, etc., and may provide an interface for connecting to a phone at the delivery origin and connecting to a VPS map update.

[0088] The first path information output area (33) can control the display (130) to output 2D path information or 3D path information output by the control unit (150), for example, and may include the remaining distance to the destination.

[0089] Additionally, the first path information output area (33) may display 3D path information (36) as a 3D virtual object implemented in augmented reality, for example, when the control unit (150) controls the display (130) to output 3D path information. In this case, the second path information output area (35) may control the display (130) to output 2D path information.

[0090] Here, the 3D path information (36) may be generated based on, for example, surrounding images captured by the camera (110) generated by the control unit (150) and VPS map information, but is not limited thereto.

[0091] Meanwhile, the first path information output area (33) can control the display (130) so that the control unit (150) displays a 3D virtual object (34) representing the location of the destination in the surrounding image captured by the camera (110), based on, for example, the surrounding image captured by the camera (110), VPS map information, and delivery information.

[0092] The 3D virtual object (34) may be, for example, information included in the delivery information. In other words, the manager (or user) of the delivery origin can display the pickup location based on video or image information about the place of the delivery origin and transmit it to the server (200), and the control unit (150) can control the display to display the 3D virtual object (34) in the first route information output area (33) based on the delivery information received from the server (200).

[0093] FIG. 4 is a diagram illustrating a case in which a user terminal (100) according to one embodiment displays path information to a destination.

[0094] As shown in FIG. 4, the display (130) of a user terminal (100) according to one embodiment may include a status display area (41), a destination guide area (42), and a route information output area (43).

[0095] The destination guide area (42) may, for example, be a control unit (150) that controls the display (130) to display the attributes of the path (TBT mode), remaining distance, estimated time to the destination, etc. of the user terminal (100).

[0096] Here, the 2D path information (44) may be, for example, generated by the control unit (150) based on 2D map information and delivery information stored in the database (140), but is not limited thereto.

[0097] FIG. 5 is a diagram illustrating the case where a user terminal (100) according to one embodiment photographs a destination.

[0098] As shown in FIG. 5, the display (130) of a user terminal (100) according to one embodiment may include a status display area (51), a delivery information area (52), a first route information output area (53), and a second route information output area (55).

[0099] The status display area (51) may include, for example, time, wireless signal strength (intensity), and battery level information, but is not limited thereto.

[0100] Additionally, the delivery information area (52) may provide, for example, the type of delivery (pickup, delivery, etc.), whether the delivery item is ready if the type of delivery is pickup, location information, a phone connection to the delivery destination, and an interface for connecting to a VPS map update.

[0101] The first path information output area (53) can control the display (130) to output 2D path information or 3D path information output by the control unit (150), for example, and may include the remaining distance to the destination.

[0102] Additionally, the first path information output area (53) can display 3D path information as a 3D virtual object implemented in augmented reality, for example, when the control unit (150) controls the display (130) to output 3D path information. In this case, the second path information output area (55) can control the display (130) to output 2D path information.

[0103] Here, the 3D path information may be generated based on, for example, surrounding images captured by the camera (110) generated by the control unit (150) and VPS map information, but is not limited thereto.

[0104] Meanwhile, the first path information output area (53) can control the display (130) so that the control unit (150) displays a 3D virtual object (54) representing the location of the destination in the surrounding image captured by the camera (110), based on, for example, the surrounding image captured by the camera (110), VPS map information, and delivery information.

[0105] The 3D virtual object (54) may be, for example, information included in the delivery information. In other words, the manager of the delivery destination (or the delivery requester) may display the delivery location based on video or image information about the place of the delivery destination and transmit it to the server (200), thereby allowing the control unit (150) to control the display to display the 3D virtual object (34) in the first path information output area (33) based on the delivery information received from the server (200).

[0106] FIG. 6 is a diagram illustrating how a user terminal (100) according to one embodiment captures an image and updates a VPS map.

[0107] As illustrated in FIG. 6, the display (130) of a user terminal (100) according to one embodiment may include a location display area (61) and an image display area (62).

[0108] The location display area (61) may, for example, be user location information sensed by the sensor unit (160), and in another embodiment, may be user location information generated by the control unit (150) based on VPS map information and surrounding images captured by the camera (110). However, it is not limited thereto.

[0109] That is, the control unit (150) can control the display (130) so that video or image information associated with the location displayed in the location display area (61) is displayed in the video display area (62). In this case, the video or image displayed in the video display area (62) may be, for example, a surrounding video captured by the camera (110).

[0110] As illustrated in FIG. 6, the user may update the VPS map through the user's location and video or image data associated with the user's location, and accordingly, the user may generate metadata for training the VPS location determination model or receive a reward score as described above. However, it is not limited thereto.

[0111] Below, based on the details described above, we will briefly examine the operation flow of the present invention.

[0112] FIG. 7 is a flowchart illustrating an embodiment of a user terminal control method according to one embodiment that guides a path to a destination, and FIG. 8 is a flowchart illustrating an embodiment of a user terminal control method according to one embodiment that updates a user-participated VPS map based on captured surrounding images.

[0113] The user terminal control method illustrated in FIGS. 7 and FIGS. 8 can be performed by the user terminal (100) described above. Therefore, even if the content described below is omitted, the content described for the user terminal (100) can be applied equally to the description of the user terminal control method.

[0114] Referring to FIG. 7, the user terminal (100) can store VPS map information and 2D map information (310).

[0115] The user terminal (100) can receive delivery information from the server (200) (320).

[0116] The user terminal (100) can identify the user's location based on GPS information from the sensor unit (160), VPS map information, and images captured from the camera (110) (330).

[0117] The user terminal (100) can determine the distance between the user location and the destination (340).

[0118] When the distance between the user location and the destination is lower than a predetermined distance, the user terminal (100) can display 3D path information represented as an augmented reality-based 3D object on a surrounding image captured by the camera (110) (350).

[0119] If the distance between the user location and the destination is greater than the set distance, the user terminal (100) can display the route in the 2D map information (360).

[0120] Referring to FIG. 8, the user terminal (100) can capture an image with a camera (110) (410).

[0121] The user terminal (100) can assign weights to the image (420).

[0122] The user terminal (100) can provide a reward score based on weights (430).

[0123] The user terminal (100) can update the VPS map based on the above image (440).

[0124] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. The instructions may be stored in the form of program code and, when executed by a processor, may generate a program module to perform the operation of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0125] Computer-readable recording media include all types of recording media that store instructions that can be decoded by a computer. Examples include ROM (read-only memory), RAM (random access memory), magnetic tape, magnetic disk, flash memory, optical data storage devices, etc.

[0126] As described above, the disclosed embodiments have been explained with reference to the attached drawings. Those skilled in the art will understand that the present invention may be practiced in forms different from the disclosed embodiments without changing the technical spirit or essential features of the invention. The disclosed embodiments are illustrative and should not be interpreted restrictively. Explanation of the symbols

[0127] 100: User terminal 110: Camera 120: Communications unit 130: Display 140: Database 150: Control unit 160: Sensor unit 200: Server 1000: Augmented Reality (AR) Mobility System

Claims

Claim 1 A camera for capturing surrounding images; a display; a communication unit for communicating with a server; a database for storing augmented reality-based VPS (Visual Positioning System) map information and 2D (2 Dimension) map information; and a control unit for generating 2D route information based on delivery information received from the server and the 2D map information, generating augmented reality-based 3D route information based on the delivery information and the VPS map information, and controlling the display to display the 2D route information or the 3D route information based on the distance between the destination included in the delivery information and the location of the user terminal; wherein the control unit controls the display to display the 2D route information when the distance between the destination and the location of the user terminal is greater than a predetermined distance, and controls the display to display the 3D route information when the distance between the destination and the location of the user terminal is less than a predetermined distance, and the predetermined distance is adaptively determined based on at least one of the boundary of the building entered by the user terminal, the width of the road at the location of the user terminal, the congestion at the location of the user terminal, or the speed of the user. Claim 2 delete Claim 3 In claim 1, the control unit is a user terminal that identifies the user's location information based on the VPS map information and the surrounding image. Claim 4 In paragraph 3, the control unit is a user terminal that controls the display to display the synthesized image, and synthesizes a destination represented as an augmented reality 3D object onto the surrounding image. Claim 5 In paragraph 3, the database is a user terminal that updates the VPS map information based on the surrounding image. Claim 6 In claim 5, the control unit assigns a weight to the surrounding image based on the characteristics of the surrounding image and provides a reward score when the weight is higher than a predetermined standard. Claim 7 In claim 6, the above compensation score is weighted based on at least one of the resolution, capacity, and length of the surrounding image, and varies according to the size of the sum of each weight. Claim 8 In claim 1, the control unit is a user terminal that generates path information such that at least one of time, travel distance, and congestion is minimized. Claim 9 In claim 8, the control unit is a user terminal that generates real-time route information based on the user's location information and the delivery information. Claim 10 A user terminal control method comprising: storing augmented reality-based VPS (Visual Positioning System) map information and 2D (2-Dimension) map information; receiving delivery information from a server; generating 2D route information based on the delivery information and the 2D map information; generating 3D route information based on the delivery information and the VPS map information; and controlling a display to display 2D route information or 3D route information based on the distance between a destination included in the delivery information and the location of a user terminal, wherein controlling the display comprises controlling the display to display the 2D route information when the distance between the destination and the location of the user terminal is greater than a predetermined distance, and controlling the display to display the 3D route information when the distance between the destination and the location of the user terminal is less than a predetermined distance, wherein the predetermined distance is adaptively determined based on at least one of the boundary of a building entered by the user terminal, the width of a road at the location of the user terminal, the congestion level at the location of the user terminal, or the speed of the user. Claim 11 delete Claim 12 In claim 10, the user terminal control method further comprises: capturing surrounding images by a camera; and identifying the user's location information based on the VPS map information and the surrounding images. Claim 13 In claim 12, the method of controlling the display comprises synthesizing a destination represented as an augmented reality 3D object onto the surrounding image and controlling the display to display the synthesized image. Claim 14 In claim 12, storing the map information is a user terminal control method that updates the VPS map information based on the surrounding image. Claim 15 In claim 14, the user terminal control method further comprises: assigning a weight to the surrounding image based on the characteristics of the surrounding image, and providing a reward score when the weight is higher than a predetermined standard. Claim 16 A user terminal control method according to claim 15, wherein the above-mentioned compensation score is weighted based on at least one of the resolution, capacity, and length of the surrounding image, and varies according to the size of the sum of each weight. Claim 17 A user terminal control method according to claim 10, wherein generating the above-mentioned path information is such that at least one of time, travel distance, and congestion is minimized. Claim 18 In claim 17, the method for controlling a user terminal, wherein generating the above-mentioned route information is to generate real-time route information based on the user's location information and the above-mentioned delivery information. Claim 19 A computer-readable recording medium having a program capable of executing the user terminal control method of paragraph 10.

Images