Ride-hailing service provision method, ride-hailing service provision system, and ar service provision device interworking therewith

Abstract

A ride-hailing service provision system, according to an embodiment of the present invention, may be configured to include an AR service provision device and a service platform interworking with the AR service provision device. Herein, the AR service provision device renders an AR image, based on the location information and vehicle information of a hailed vehicle, onto the hailed vehicle and displays on a three-dimensional map. In addition, the service platform may request, in response to the hailed vehicle entering a pickup zone on the basis of the location information of the passenger, that a camera of the hailed vehicle take an image in such a way that a plurality of prospective passengers are included therein, so as to generate a plurality of thumbnails therefrom, and may provide the AR service provision device with a user interface for selecting one of the plurality of generated thumbnails. The service platform may also transmit information on the thumbnail selected by the AR service provision device to the hailed vehicle so as to track the location of the prospective passenger.

Description

Method for providing a vehicle call service, system for providing a vehicle call service, and AR service providing device linked thereto

[0001] The present invention relates to a vehicle call service providing method, system, and linked AR service providing device, and more specifically, to a vehicle call service providing method, system, and linked AR service providing device for providing vehicle call, authentication, and location designation services through mutual interaction between an unmanned vehicle and a passenger terminal.

[0002] A vehicle is a device that allows the user to move in the desired direction. A representative example is an automobile.

[0003] Meanwhile, there's a growing trend toward incorporating various sensors and electronic devices into vehicles to enhance user convenience. In particular, research is actively underway on Advanced Driver Assistance Systems (ADAS) to enhance user convenience. Furthermore, development of autonomous vehicles (AVs) is also actively underway.

[0004] Recently, development of UI / UX and services that assist in driving vehicles is actively underway using augmented reality (AR), virtual reality (VR), and mixed reality (MR) technologies that combine AR and VR.

[0005] When utilizing mixed reality technology, it is possible to provide various information necessary for driving a vehicle based on the actual real world, or to provide various interfaces related to driving a vehicle in a virtual world (or digital twin) that is almost identical to the real world, and additionally, it has the advantage of being able to provide information and content from various fields to vehicle passengers in addition to driving.

[0006] Meanwhile, the market for providing passenger transportation services through autonomous vehicles, known as autonomous taxis or robotaxis, is growing. Since these transportation services utilize autonomous vehicles without human drivers, extensive research is needed to improve passenger interaction.

[0007] Specifically, from the passenger's perspective, many changes are needed to take into account the characteristics of autonomous vehicles in relation to the various interactions and communications that occur with conventional human drivers before, during, and after boarding the vehicle.

[0008] For example, in the past, after a passenger designated a boarding location, the called vehicle approached the agreed-upon boarding location and stopped. The passenger visually inspected the vehicle's license plate number and brand before deciding to board. Furthermore, after boarding, passengers could easily confirm the vehicle they had requested through a simple conversation.

[0009] However, in autonomous vehicles, even if passengers check the license plate number and brand to confirm the vehicle they requested and board, an additional authentication process is required to confirm that the vehicle they requested is the correct one. Furthermore, passengers are increasingly anxious about whether the autonomous vehicle will safely reach their designated destination.

[0010] Additionally, in cases where the area around the boarding location, such as an airport, is crowded and it is difficult for the called vehicle to approach the promised boarding location, there is a problem in that it is difficult to easily change the boarding location through visual confirmation and possible hand gestures.

[0011] The present invention aims to solve the above-mentioned problems and other problems.

[0012] According to some embodiments of the present invention, the purpose is to provide a vehicle call service providing method, system, and linked AR service providing device that enable coordination through mutual recognition and interaction between a passenger and an unmanned vehicle using AR and MR technologies when an unmanned vehicle called by a passenger approaches a calling location within a predetermined distance.

[0013] In addition, according to some embodiments of the present invention, the purpose is to provide a vehicle calling service providing method, system, and linked AR service providing device, which enable not only a passenger who has called a vehicle to accurately recognize the called vehicle, but also the called vehicle to accurately recognize the called passenger to provide automatic authentication and accurate boarding service.

[0014] In addition, according to some embodiments of the present invention, when the boarding or disembarking location promised to a passenger is not suitable as a boarding or disembarking location, the purpose is to provide a vehicle call service providing method, system, and linked AR service providing device that can inform of a suitable location for boarding or disembarking using AR and MR technology and even specify a detailed location.

[0015] In addition, according to some embodiments of the present invention, when a passenger designates a stopover, the purpose is to provide a vehicle call service providing method, system, and linked AR service providing device that can interact with a passenger at a stopover using AR and MR technology so that the unmanned vehicle can recognize the exact location of the stopover designated by the passenger.

[0016] To this end, a vehicle call service providing system according to an embodiment of the present invention includes an AR service providing device and a service platform linked with the AR service providing device.

[0017] At this time, the AR service providing device can render an AR image based on the location information and vehicle information of the called vehicle to the called vehicle and display it on a 3D map. In addition, the service platform linked with the AR service providing device can, in response to the called vehicle entering a boarding area based on the passenger's location information, request the called vehicle to generate a plurality of thumbnails by taking pictures of a plurality of candidate passengers with the camera of the called vehicle, and provide the AR service providing device with a user interface for selecting one of the plurality of generated thumbnails. In addition, the service platform can transmit information about the thumbnail selected by the AR service providing device to the called vehicle to track the location of the candidate passenger.

[0018] According to an embodiment, the rideable section satisfies the range of the camera equipped in the calling vehicle and the range of the camera equipped in the AR service providing device within the distance range where the calling vehicle and the AR service providing device can each photograph each other.

[0019] According to an embodiment, the service platform may transmit additional information input from the AR service providing device when calling a vehicle to the calling vehicle, extract an area related to the additional information from an image captured by a camera of the calling vehicle, and generate the plurality of thumbnails including the extracted area.

[0020] According to an embodiment, the AR service providing device may receive a selection input for one of a plurality of thumbnails using the user interface, and the service platform may transmit an ID and a feature point of a thumbnail corresponding to the selection input to the calling vehicle, so that the calling vehicle may recognize a candidate passenger corresponding to the transmitted thumbnail ID, and track the location of the recognized candidate passenger based on the feature point.

[0021] According to an embodiment, the AR service providing device may, in response to the selection input, change a first AR image based on the location information and vehicle information of the called vehicle into a second AR image based on passenger customization information and render the image on the called vehicle.

[0022] According to an embodiment, the service platform may transmit a re-shooting command to the calling vehicle in response to a non-selection input for a plurality of thumbnails, and provide a thumbnail corresponding to the re-shooting command to the AR service providing device.

[0023] According to an embodiment, the service platform may, in response to the stopping of the called vehicle, cause the called vehicle to perform passenger authentication by comparing a first feature corresponding to a thumbnail corresponding to the selection input with a second feature obtained from a photographed image of a candidate passenger close to the door of the called vehicle.

[0024] According to an embodiment, the service platform may cause the calling vehicle to perform an action related to guiding a passenger to board in response to determining that the similarity between the first feature point and the second feature point satisfies a threshold value or higher based on the comparison.

[0025] According to an embodiment, the service platform may, in response to the calling vehicle determining that the similarity between the first feature point and the second feature point is less than a threshold value based on the comparison, cause the service platform to perform an action related to passenger mismatch and transmit vehicle arrival guidance information to the AR service providing device.

[0026] In addition, a method for providing a vehicle call service according to an embodiment of the present invention may include: obtaining location information of a call vehicle; rendering an AR image based on the location information and vehicle information of the call vehicle to the call vehicle and displaying the AR image on a three-dimensional map; determining that the call vehicle on which the AR image is rendered has entered a boarding area; requesting the generation of a plurality of thumbnails by taking pictures including a plurality of candidate passengers through a camera of the call vehicle; providing a user interface for selecting one of the plurality of generated thumbnails to an AR service providing device; and transmitting information on a thumbnail selected through the user interface to the call vehicle to track the location of the candidate passenger.

[0027] The effects of the vehicle call service providing method and system according to the present invention, and the AR service providing device therefor are described as follows.

[0028] According to a vehicle call service providing method, system, and AR service providing device therefor according to an embodiment of the present invention, remote interaction can be naturally achieved between an unmanned vehicle without a human driver and a passenger before boarding, and the convenience of authentication for boarding and changing of boarding and alighting locations is further enhanced by mutual recognition.

[0029] In addition, according to an embodiment of the present invention, not only can the vehicle called by the passenger be accurately identified, but the vehicle can also identify and authenticate the passenger in advance through a thumbnail selected by the passenger before boarding, thereby facilitating smooth interaction and authentication, and providing high-quality services such as accurately guiding passenger boarding in an unmanned vehicle.

[0030] In addition, according to an embodiment of the present invention, if the boarding or disembarking location promised to the passenger is not suitable as a boarding or disembarking location, AR and MR technology can be used to accurately inform the passenger of a suitable boarding or disembarking location, and allow the passenger to intuitively recognize the changing travel route.

[0031] In addition, according to some embodiments of the present invention, even if the first passenger designates a stopover, the second passenger can provide the stopover location, so that carpooling can be done naturally using AR and MR technology even in a situation where the first passenger does not know the second passenger.

[0032] FIG. 1 is a drawing illustrating an example of a vehicle related to an embodiment of the present invention.

[0033] FIG. 2 is a drawing of a vehicle related to an embodiment of the present invention viewed from various angles.

[0034] FIGS. 3 and 4 are drawings showing the interior of a vehicle related to an embodiment of the present invention.

[0035] FIG. 5 and FIG. 6 are drawings for reference in explaining various objects related to driving of a vehicle related to an embodiment of the present invention.

[0036] FIG. 7 is a block diagram for reference in explaining a vehicle and an AR service providing device related to an embodiment of the present invention.

[0037] FIG. 8 is a drawing showing an example of a detailed configuration of an entire system for providing a vehicle call service and an AR service providing device related to an embodiment of the present invention.

[0038] FIGS. 9A and 9B are drawings for explaining distance conditions related to vehicle calling according to an embodiment of the present invention.

[0039] FIGS. 10a, 10b, 10c, 10d, and 10e are drawings showing various examples of displaying the location and driving of a called vehicle in an MR map view and guiding a passenger to board, according to an embodiment of the present invention.

[0040] FIG. 11 is an exemplary flowchart illustrating a method for determining and authenticating a passenger who has called a vehicle using a thumbnail related to an embodiment of the present invention.

[0041] FIGS. 12 and 13 are drawings for explaining how a vehicle related to an embodiment of the present invention generates and provides thumbnails to candidate passengers.

[0042] FIG. 14 is a drawing for explaining selecting a passenger who has called a vehicle using a thumbnail in an AR service providing device related to an embodiment of the present invention.

[0043] FIG. 15 is an exemplary flowchart illustrating a method for generating a thumbnail based on prior information of a passenger who has called a vehicle related to an embodiment of the present invention.

[0044] FIG. 16 is an exemplary flowchart illustrating a method for authenticating a passenger who has called a vehicle based on feature points of a selected thumbnail related to an embodiment of the present invention.

[0045] FIG. 17 is an exemplary flowchart for explaining a method for deriving and selecting a boarding location using three-dimensional map information related to an embodiment of the present invention.

[0046] FIG. 18 and FIG. 19 are drawings for explaining recommending boarding and disembarking locations based on difficulty information of a boarding and disembarking area of ​​a vehicle related to an embodiment of the present invention.

[0047] FIG. 20 is a drawing for explaining selecting a vehicle disembarkation direction using a highlight structure on a three-dimensional map related to an embodiment of the present invention.

[0048] FIG. 21 is a drawing for explaining changing the path of a vehicle using three-dimensional map information related to an embodiment of the present invention.

[0049] FIG. 22 and FIG. 23 are exemplary diagrams and flowcharts for explaining providing location information of a waypoint using three-dimensional map information related to an embodiment of the present invention.

[0050] FIG. 24 is a drawing for explaining displaying a ride-sharing applicant using three-dimensional map information related to an embodiment of the present invention.

[0051] FIGS. 1 and 2 are views showing the exterior of a vehicle related to an embodiment of the present invention, and FIGS. 3 and 4 are views showing the interior of a vehicle related to an embodiment of the present invention.

[0052] FIGS. 5 and 6 are drawings illustrating various objects related to driving of a vehicle according to an embodiment of the present invention.

[0053] Fig. 7 is a block diagram used for reference in explaining a vehicle related to an embodiment of the present invention. Fig. 7 is a block diagram used for reference in explaining a vehicle according to an embodiment of the present invention.

[0054] Referring to FIGS. 1 to 7, the vehicle (100) may include wheels that rotate by a power source and a steering input device (510) for controlling the direction of travel of the vehicle (100).

[0055] The vehicle (100) may be an autonomous vehicle. The vehicle (100) may be switched between an autonomous driving mode and a manual driving mode based on user input. For example, the vehicle (100) may be switched from a manual mode to an autonomous driving mode, or from an autonomous driving mode to a manual mode, based on user input received through a user interface device (hereinafter, referred to as a "user terminal") (200).

[0056] The vehicle (100) can be switched to autonomous driving mode or manual driving mode based on driving situation information. The driving situation information can be generated based on object information provided by the object detection device (300). For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on the driving situation information generated by the object detection device (300). For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information received through the communication device (400).

[0057] The vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on information, data, and signals provided from an external device.

[0058] When the vehicle (100) is operated in autonomous driving mode, the autonomous vehicle (100) may be operated based on the driving system (700). For example, the autonomous vehicle (100) may be operated based on information, data, or signals generated from the driving system (710), the exit system (740), and the parking system (750).

[0059] When the vehicle (100) is driven in manual mode, the autonomous vehicle (100) can receive user input for driving through the driving control device (500). Based on the user input received through the driving control device (500), the vehicle (100) can be driven.

[0060] The overall length refers to the length from the front to the rear of the vehicle (100), the overall width refers to the width of the vehicle (100), and the overall height refers to the length from the bottom of the wheel to the roof. In the following description, the overall length direction (L) may refer to the direction that serves as a reference for measuring the overall length of the vehicle (100), the overall width direction (W) may refer to the direction that serves as a reference for measuring the overall width of the vehicle (100), and the overall height direction (H) may refer to the direction that serves as a reference for measuring the overall height of the vehicle (100).

[0061] As illustrated in FIG. 7, the vehicle (100) may include a user interface device (hereinafter, referred to as a 'user terminal') (200), an object detection device (300), a communication device (400), a driving operation device (500), a vehicle driving device (600), a driving system (700), a navigation system (770), a sensing unit (120), a vehicle interface unit (130), a memory (140), a control unit (170), and a power supply unit (190).

[0062] Depending on the embodiment, the vehicle (100) may include other components in addition to the components described herein, or may not include some of the components described herein.

[0063] The user interface device (200) is a device for communication between a vehicle (100) and a user. The user interface device (200) can receive user input and provide information generated in the vehicle (100) to the user. The vehicle (100) can implement a UI (User Interfaces) or UX (User Experience) through the user interface device (hereinafter, referred to as a 'user terminal') (200).

[0064] The user interface device (200) may include an input unit (210), an internal camera (220), a biometric detection unit (230), an output unit (250), and a processor (270). Depending on the embodiment, the user interface device (200) may include other components in addition to the described components, or may not include some of the described components.

[0065] The input unit (210) is for receiving information from a user, and data collected from the input unit (210) can be analyzed by a processor (270) and processed into a user's control command.

[0066] The input unit (210) may be placed inside the vehicle. For example, the input unit (210) may be placed in an area of ​​a steering wheel, an area of ​​an instrument panel, an area of ​​a seat, an area of ​​each pillar, an area of ​​a door, an area of ​​a center console, an area of ​​a head lining, an area of ​​a sun visor, an area of ​​a windshield, or an area of ​​a window.

[0067] The input unit (210) may include a voice input unit (211), a gesture input unit (212), a touch input unit (213), and a mechanical input unit (214).

[0068] The voice input unit (211) can convert a user's voice input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170). The voice input unit (211) can include one or more microphones.

[0069] The gesture input unit (212) can convert a user's gesture input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0070] The gesture input unit (212) may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input. According to an embodiment, the gesture input unit (212) may detect a user's three-dimensional gesture input. To this end, the gesture input unit (212) may include a light output unit that outputs a plurality of infrared lights or a plurality of image sensors.

[0071] The gesture input unit (212) can detect a user's 3D gesture input through a TOF (Time of Flight) method, a structured light method, or a disparity method.

[0072] The touch input unit (213) can convert a user's touch input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0073] The touch input unit (213) may include a touch sensor for detecting a user's touch input. In some embodiments, the touch input unit (213) may be formed integrally with the display unit (251), thereby implementing a touch screen. Such a touch screen may provide both an input interface and an output interface between the vehicle (100) and the user.

[0074] The mechanical input unit (214) may include at least one of a button, a dome switch, a jog wheel, and a jog switch. An electrical signal generated by the mechanical input unit (214) may be provided to a processor (270) or a control unit (170). The mechanical input unit (214) may be placed on a steering wheel, a center fascia, a center console, a cockpit module, a door, etc.

[0075] The internal camera (220) can capture images of the vehicle interior. The processor (270) can detect the user's status based on the images of the vehicle interior. The processor (270) can obtain information about the user's gaze from the images of the vehicle interior. The processor (270) can detect the user's gestures from the images of the vehicle interior.

[0076] The biometric detection unit (230) can obtain the user's biometric information. The biometric detection unit (230) includes a sensor capable of obtaining the user's biometric information, and can use the sensor to obtain the user's fingerprint information, heartbeat information, etc. The biometric information can be used for user authentication.

[0077] The output unit (250) is for generating output related to visual, auditory, or tactile sensations. The output unit (250) may include at least one of a display unit (251), an audio output unit (252), and a haptic output unit (253).

[0078] The display unit (251) can display graphic objects corresponding to various pieces of information. The display unit (251) can include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0079] The display unit (251) can be formed as a layer structure with the touch input unit (213) or as an integral part, thereby implementing a touch screen.

[0080] The display unit (251) may be implemented as a HUD (Head Up Display). When the display unit (251) is implemented as a HUD, the display unit (251) may be equipped with a projection module to output information through an image projected onto a windshield or window.

[0081] The display unit (251) may include a transparent display. The transparent display may be attached to a windshield or a window. The transparent display may have a predetermined transparency and display a predetermined screen. In order to have transparency, the transparent display may include at least one of a transparent TFEL (Thin Film Electroluminescent), a transparent OLED (Organic Light-Emitting Diode), a transparent LCD (Liquid Crystal Display), a transparent display, and a transparent LED (Light Emitting Diode) display. The transparency of the transparent display may be adjusted.

[0082] Meanwhile, the user interface device (200) may include a plurality of display units (251a to 251g).

[0083] The display unit (251) may be arranged in one area of ​​the steering wheel, one area of ​​the instrument panel (521a, 251b, 251e), one area of ​​the seat (251d), one area of ​​each pillar (251f), one area of ​​the door (251g), one area of ​​the center console, one area of ​​the head lining, one area of ​​the sun visor, or may be implemented in one area of ​​the windshield (251c), one area of ​​the window (251h).

[0084] The audio output unit (252) converts an electric signal provided from the processor (270) or the control unit (170) into an audio signal and outputs the converted signal. To this end, the audio output unit (252) may include one or more speakers.

[0085] The haptic output unit (253) generates a tactile output. For example, the haptic output unit (253) can operate by vibrating a steering wheel, a seat belt, or a seat (110FL, 110FR, 110RL, 110RR) so that the user can perceive the output.

[0086] The processor (hereinafter, referred to as a “control unit”) (270) can control the overall operation of each unit of the user interface device (200). Depending on the embodiment, the user interface device (200) may include a plurality of processors (270) or may not include a processor (270).

[0087] If the user interface device (200) does not include a processor (270), the user interface device (200) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0088] Meanwhile, the user interface device (200) may be referred to as a vehicle display device. The user interface device (200) may be operated under the control of the control unit (170).

[0089] The object detection device (300) is a device for detecting an object located outside a vehicle (100). The object may be various objects related to the operation of the vehicle (100). Referring to FIGS. 5 and 6, the object (O) may include a lane (OB10), another vehicle (OB11), a pedestrian (OB12), a two-wheeled vehicle (OB13), a traffic signal (OB14, OB15), a light, a road, a structure, a speed bump, a terrain, an animal, etc.

[0090] A lane (OB10) may be a driving lane, a lane adjacent to a driving lane, or a lane in which opposing vehicles drive. A lane (OB10) may be a concept that includes lines on the left and right sides that form a lane.

[0091] Another vehicle (OB11) may be a vehicle driving around the vehicle (100). The other vehicle may be a vehicle located within a predetermined distance from the vehicle (100). For example, the other vehicle (OB11) may be a vehicle preceding or following the vehicle (100).

[0092] A pedestrian (OB12) may be a person located around a vehicle (100). A pedestrian (OB12) may be a person located within a predetermined distance from a vehicle (100). For example, a pedestrian (OB12) may be a person located on a sidewalk or roadway.

[0093] A two-wheeled vehicle (OB12) may refer to a vehicle that is positioned around a vehicle (100) and moves using two wheels. The two-wheeled vehicle (OB12) may be a vehicle with two wheels that is positioned within a predetermined distance from the vehicle (100). For example, the two-wheeled vehicle (OB13) may be a motorcycle or bicycle positioned on a sidewalk or roadway.

[0094] Traffic signals may include traffic lights (OB15), traffic signs (OB14), and patterns or text painted on the road surface.

[0095] The light may be generated from a lamp installed in another vehicle. The light may be generated from a streetlight. The light may be sunlight.

[0096] A road may include slopes such as road surfaces, curves, uphill and downhill slopes, etc.

[0097] Structures may be objects located along roads and fixed to the ground. For example, structures may include streetlights, street trees, buildings, utility poles, traffic lights, and bridges.

[0098] Landforms may include mountains, hills, etc.

[0099] Meanwhile, objects can be classified into moving objects and fixed objects. For example, moving objects may include concepts such as other vehicles and pedestrians. For example, fixed objects may include concepts such as traffic signals, roads, and structures.

[0100] The object detection device (300) may include a camera (310), a radar (320), a lidar (330), an ultrasonic sensor (340), an infrared sensor (350), and a processor (370).

[0101] Depending on the embodiment, the object detection device (300) may include other components in addition to the described components, or may not include some of the described components.

[0102] The camera (310) may be positioned at an appropriate location outside the vehicle to capture images of the vehicle's exterior. The camera (310) may be a mono camera, a stereo camera (310a), an AVM (Around View Monitoring) camera (310b), or a 360-degree camera.

[0103] For example, the camera (310) may be positioned inside the vehicle, close to the front windshield, to capture an image of the front of the vehicle. Alternatively, the camera (310) may be positioned around the front bumper or radiator grill.

[0104] For example, the camera (310) may be positioned inside the vehicle, close to the rear glass, to capture images of the rear of the vehicle. Alternatively, the camera (310) may be positioned around the rear bumper, trunk, or tailgate.

[0105] For example, the camera (310) may be positioned close to at least one of the side windows inside the vehicle to obtain an image of the side of the vehicle. Alternatively, the camera (310) may be positioned around a side mirror, fender, or door.

[0106] The camera (310) can provide the acquired image to the processor (370).

[0107] The radar (320) may include an electromagnetic wave transmitter and receiver. The radar (320) may be implemented in a pulse radar or continuous wave radar manner based on the principle of radio wave emission. Among continuous wave radar methods, the radar (320) may be implemented in a frequency modulated continuous wave (FMCW) manner or a frequency shift keying (FSK) manner depending on the signal waveform.

[0108] The radar (320) can detect an object using electromagnetic waves, based on a TOF (Time of Flight) method or a phase-shift method, and can detect the location of the detected object, the distance to the detected object, and the relative speed.

[0109] The radar (320) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0110] The lidar (330) may include a laser transmitter and receiver. The lidar (330) may be implemented using a TOF (Time of Flight) method or a phase-shift method.

[0111] The lidar (330) can be implemented as a driven or non-driven type.

[0112] When implemented as a drive type, the lidar (330) is rotated by a motor and can detect objects around the vehicle (100).

[0113] When implemented in a non-driven manner, the lidar (330) can detect an object located within a predetermined range relative to the vehicle (100) through optical steering. The vehicle (100) can include a plurality of non-driven lidars (330).

[0114] Lidar (330) can detect an object based on a time-of-flight (TOF) method or a phase-shift method using laser light as a parameter, and can detect the position of the detected object, the distance to the detected object, and the relative speed.

[0115] The lidar (330) can be placed at an appropriate location outside the vehicle to detect objects located in front, behind, or to the side of the vehicle.

[0116] The ultrasonic sensor (340) may include an ultrasonic transmitter and a receiver. The ultrasonic sensor (340) may detect an object based on ultrasonic waves, and may detect the location of the detected object, the distance from the detected object, and the relative speed.

[0117] The ultrasonic sensor (340) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0118] The infrared sensor (350) may include an infrared transmitter and a receiver. The infrared sensor (340) may detect an object based on infrared light, and may detect the location of the detected object, the distance to the detected object, and the relative speed.

[0119] The infrared sensor (350) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0120] The processor (370) can control the overall operation of each unit of the object detection device (300).

[0121] The processor (370) can detect and track an object based on the acquired image. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object through an image processing algorithm.

[0122] The processor (370) can detect and track an object based on the reflected electromagnetic waves that are returned when the transmitted electromagnetic waves are reflected by the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the electromagnetic waves.

[0123] The processor (370) can detect and track an object based on the reflected laser light that is reflected back by the transmitted laser beam from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the laser light.

[0124] The processor (370) can detect and track an object based on the reflected ultrasonic waves that are returned when the transmitted ultrasonic waves are reflected off the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the ultrasonic waves.

[0125] The processor (370) can detect and track an object based on the reflected infrared light that is reflected back by the transmitted infrared light from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the infrared light.

[0126] Depending on the embodiment, the object detection device (300) may include multiple processors (370) or may not include a processor (370). For example, each of the camera (310), radar (320), lidar (330), ultrasonic sensor (340), and infrared sensor (350) may individually include a processor.

[0127] If the object detection device (300) does not include a processor (370), the object detection device (300) can be operated under the control of the processor or control unit (170) of the device in the vehicle (100).

[0128] The object detection device (400) can be operated under the control of the control unit (170).

[0129] The communication device (400) is a device for communicating with an external device. Here, the external device may be another vehicle, a mobile terminal, or a server.

[0130] The communication device (400) may include at least one of a transmitting antenna, a receiving antenna, an RF (Radio Frequency) circuit capable of implementing various communication protocols, and an RF element to perform communication.

[0131] The communication device (400) may include a short-range communication unit (410), a location information unit (420), a V2X communication unit (430), an optical communication unit (440), a broadcast transmission / reception unit (450), and a processor (470).

[0132] Depending on the embodiment, the communication device (400) may include other components in addition to the described components, or may not include some of the described components.

[0133] The short-range communication unit (410) is a unit for short-range communication. The short-range communication unit (410) can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0134] The short-range communication unit (410) can form a short-range wireless communication network (Wireless Area Networks) to perform short-range communication between the vehicle (100) and at least one external device.

[0135] The location information unit (420) is a unit for obtaining location information of a vehicle (100). For example, the location information unit (420) may include a GPS (Global Positioning System) module or a DGPS (Differential Global Positioning System) module.

[0136] The V2X communication unit (430) is a unit for performing wireless communication with a server (V2I: Vehicle to Infrastructure), another vehicle (V2V: Vehicle to Vehicle), or a pedestrian (V2P: Vehicle to Pedestrian). The V2X communication unit (430) may include an RF circuit capable of implementing protocols for communication with infrastructure (V2I), communication between vehicles (V2V), and communication with pedestrians (V2P).

[0137] The optical communication unit (440) is a unit for communicating with an external device via light. The optical communication unit (440) may include an optical transmission unit that converts an electrical signal into an optical signal and transmits it to the outside, and an optical reception unit that converts a received optical signal into an electrical signal.

[0138] According to an embodiment, the light transmitting unit may be formed to be integrated with a lamp included in the vehicle (100).

[0139] The broadcast transmitter / receiver (450) is a unit for receiving broadcast signals from an external broadcast management server via a broadcast channel, or transmitting broadcast signals to the broadcast management server. The broadcast channels may include satellite channels and terrestrial channels. The broadcast signals may include TV broadcast signals, radio broadcast signals, and data broadcast signals.

[0140] The processor (470) can control the overall operation of each unit of the communication device (400).

[0141] Depending on the embodiment, the communication device (400) may include a plurality of processors (470) or may not include a processor (470).

[0142] If the communication device (400) does not include a processor (470), the communication device (400) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0143] Meanwhile, the communication device (400) may implement a vehicle display device together with the user interface device (200). In this case, the vehicle display device may be referred to as a telematics device or an AVN (Audio Video Navigation) device.

[0144] The communication device (400) can be operated under the control of the control unit (170).

[0145] The driving control device (500) is a device that receives user input for driving.

[0146] When in manual mode, the vehicle (100) can be driven based on signals provided by the driving control device (500).

[0147] The driving control device (500) may include a steering input device (510), an acceleration input device (530), and a brake input device (570).

[0148] The steering input device (510) can receive input for the direction of travel of the vehicle (100) from the user. The steering input device (510) is preferably formed in the form of a wheel so that steering input can be provided by rotation. Depending on the embodiment, the steering input device may be formed in the form of a touch screen, a touch pad, or a button.

[0149] The acceleration input device (530) can receive an input from a user for accelerating the vehicle (100). The brake input device (570) can receive an input from a user for decelerating the vehicle (100). The acceleration input device (530) and the brake input device (570) are preferably formed in the form of a pedal. Depending on the embodiment, the acceleration input device or the brake input device may also be formed in the form of a touch screen, a touch pad, or a button.

[0150] The driving operation device (500) can be operated under the control of the control unit (170).

[0151] The vehicle driving device (600) is a device that electrically controls the driving of various devices in the vehicle (100).

[0152] The vehicle driving device (600) may include a power train driving unit (610), a chassis driving unit (620), a door / window driving unit (630), a safety device driving unit (640), a lamp driving unit (650), and an air conditioning driving unit (660).

[0153] Depending on the embodiment, the vehicle drive device (600) may include additional components other than the described components, or may not include some of the described components.

[0154] Meanwhile, the vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0155] The power train drive unit (610) can control the operation of the power train device.

[0156] The power train drive unit (610) may include a power source drive unit (611) and a transmission drive unit (612).

[0157] The power source driving unit (611) can perform control over the power source of the vehicle (100).

[0158] For example, if a fossil fuel-based engine is the power source, the power source drive unit (610) can perform electronic control of the engine. This can control the engine output torque, etc. The power source drive unit (611) can adjust the engine output torque according to the control of the control unit (170).

[0159] For example, if an electric energy-based motor is the power source, the power source driving unit (610) can perform control over the motor. The power source driving unit (610) can adjust the rotation speed, torque, etc. of the motor according to the control of the control unit (170).

[0160] The transmission drive unit (612) can perform control over the transmission. The transmission drive unit (612) can adjust the state of the transmission. The transmission drive unit (612) can adjust the state of the transmission to forward (D), reverse (R), neutral (N), or parking (P).

[0161] Meanwhile, when the engine is the power source, the transmission drive unit (612) can adjust the gear engagement state in the forward (D) state.

[0162] The chassis drive unit (620) can control the operation of the chassis device. The chassis drive unit (620) can include a steering drive unit (621), a brake drive unit (622), and a suspension drive unit (623).

[0163] The steering drive unit (621) can perform electronic control of the steering apparatus within the vehicle (100). The steering drive unit (621) can change the direction of travel of the vehicle.

[0164] The brake drive unit (622) can perform electronic control of the brake apparatus within the vehicle (100). For example, the speed of the vehicle (100) can be reduced by controlling the operation of the brakes placed on the wheels.

[0165] Meanwhile, the brake driving unit (622) can individually control each of the plurality of brakes. The brake driving unit (622) can control the braking force applied to the plurality of wheels differently.

[0166] The suspension drive unit (623) can perform electronic control of the suspension apparatus within the vehicle (100). For example, when there is a curve in the road surface, the suspension drive unit (623) can control the suspension apparatus to reduce vibration of the vehicle (100). Meanwhile, the suspension drive unit (623) can individually control each of the plurality of suspensions.

[0167] The door / window actuator (630) can perform electronic control of a door apparatus or window apparatus in a vehicle (100).

[0168] The door / window driving unit (630) may include a door driving unit (631) and a window driving unit (632).

[0169] The door driving unit (631) can control the door device. The door driving unit (631) can control the opening and closing of a plurality of doors included in the vehicle (100). The door driving unit (631) can control the opening or closing of a trunk or tail gate. The door driving unit (631) can control the opening or closing of a sunroof.

[0170] The window driving unit (632) can perform electronic control of a window apparatus. It can control the opening or closing of a plurality of windows included in a vehicle (100).

[0171] The safety device driving unit (640) can perform electronic control of various safety devices in the vehicle (100).

[0172] The safety device drive unit (640) may include an airbag drive unit (641), a seat belt drive unit (642), and a pedestrian protection device drive unit (643).

[0173] The airbag driving unit (641) can perform electronic control of the airbag apparatus within the vehicle (100). For example, the airbag driving unit (641) can control the airbag to deploy when a danger is detected.

[0174] The seat belt drive unit (642) can perform electronic control of the seat belt apparatus within the vehicle (100). For example, the seat belt drive unit (642) can control the passenger to be secured to the seat (110FL, 110FR, 110RL, 110RR) using the seat belt when a danger is detected.

[0175] The pedestrian protection device drive unit (643) can perform electronic control of the hood lift and pedestrian airbag. For example, the pedestrian protection device drive unit (643) can control the hood lift up and the pedestrian airbag to deploy when a collision with a pedestrian is detected.

[0176] The lamp driving unit (650) can perform electronic control of various lamp apparatuses within the vehicle (100).

[0177] The air conditioning drive unit (660) can perform electronic control of the air conditioning device (air cinditioner) within the vehicle (100). For example, the air conditioning drive unit (660) can control the air conditioning device to operate and supply cool air to the vehicle when the temperature inside the vehicle is high.

[0178] The vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0179] The vehicle driving device (600) can be operated under the control of the control unit (170).

[0180] The driving system (700) is a system that controls various operations of the vehicle (100). The driving system (700) can be operated in autonomous driving mode.

[0181] The driving system (700) may include a driving system (710), an exiting system (740), and a parking system (750).

[0182] Depending on the embodiment, the driving system (700) may include other components in addition to the described components, or may not include some of the described components.

[0183] Meanwhile, the driving system (700) may include a processor. Each unit of the driving system (700) may individually include a processor.

[0184] Meanwhile, depending on the embodiment, if the driving system (700) is implemented in software, it may be a sub-concept of the control unit (170).

[0185] Meanwhile, according to an embodiment, the driving system (700) may be a concept including at least one of a user interface device (200), an object detection device (300), a communication device (400), a vehicle driving device (600), and a control unit (170).

[0186] The driving system (710) can drive the vehicle (100).

[0187] The driving system (710) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to drive the vehicle (100). The driving system (710) can receive object information from the object detection device (300) and provide a control signal to the vehicle driving device (600) to drive the vehicle (100). The driving system (710) can receive a signal from an external device through the communication device (400) and provide a control signal to the vehicle driving device (600) to drive the vehicle (100).

[0188] The exit system (740) can perform exit of a vehicle (100).

[0189] The exit system (740) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to perform exit of the vehicle (100). The exit system (740) can receive object information from the object detection device (300) and provide a control signal to the vehicle driving device (600) to perform exit of the vehicle (100). The exit system (740) can receive a signal from an external device through the communication device (400) and provide a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0190] The parking system (750) can perform parking of a vehicle (100).

[0191] The parking system (750) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100). The parking system (750) can receive object information from the object detection device (300) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100). The parking system (750) can receive a signal from an external device through the communication device (400) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0192] A navigation system (770) can provide navigation information. The navigation information can include at least one of map information, set destination information, route information based on the set destination, information on various objects along the route, lane information, and current vehicle location information.

[0193] The navigation system (770) may include memory and a processor. The memory may store navigation information. The processor may control the operation of the navigation system (770).

[0194] According to an embodiment, the navigation system (770) may receive information from an external device via the communication device (400) and update previously stored information.

[0195] Depending on the embodiment, the navigation system (770) may be classified as a subcomponent of the user interface device (200).

[0196] The sensing unit (120) can sense the status of the vehicle. The sensing unit (120) can include a posture sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight detection sensor, a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0197] The sensing unit (120) can obtain sensing signals for vehicle attitude information, vehicle collision information, vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle inclination information, vehicle forward / backward information, battery information, fuel information, tire information, vehicle lamp information, vehicle internal temperature information, vehicle internal humidity information, steering wheel rotation angle, vehicle external illumination, pressure applied to an accelerator pedal, pressure applied to a brake pedal, etc.

[0198] The sensing unit (120) may further include, in addition, an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0199] The vehicle interface unit (130) can serve as a conduit for various types of external devices connected to the vehicle (100). For example, the vehicle interface unit (130) may be equipped with a port capable of connecting to a mobile terminal, and may be connected to the mobile terminal through the port. In this case, the vehicle interface unit (130) can exchange data with the mobile terminal.

[0200] Meanwhile, the vehicle interface unit (130) may serve as a conduit for supplying electrical energy to a connected mobile terminal. When the mobile terminal is electrically connected to the vehicle interface unit (130), the vehicle interface unit (130) may provide the mobile terminal with electrical energy supplied from the power supply unit (190) under the control of the control unit (170).

[0201] The memory (140) is electrically connected to the control unit (170). The memory (140) can store basic data for the unit, control data for controlling the operation of the unit, and input / output data. The memory (140) can be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware. The memory (140) can store various data for the overall operation of the vehicle (100), such as programs for processing or controlling the control unit (170).

[0202] Depending on the embodiment, the memory (140) may be formed integrally with the control unit (170) or implemented as a sub-component of the control unit (170).

[0203] The control unit (170) can control the overall operation of each unit within the vehicle (100). The control unit (170) can be referred to as an ECU (Electronic Control Unit).

[0204] The power supply unit (190) can supply power required for the operation of each component under the control of the control unit (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0205] One or more processors and control units (170) included in the vehicle (100) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0206] In the embodiments of the present invention, the term "vehicle" or "call vehicle" assumes an unmanned vehicle. Therefore, the configurations and examples applicable only to non-unmanned vehicles in the descriptions provided with reference to FIGS. 1 through 7 above may be excluded. Furthermore, the terms "call vehicle" and "vehicle" may be used interchangeably in the present invention.

[0207] In embodiments of the present invention, the terms "passenger" or "client" are used to refer to the person or device (e.g., an AR service providing device) that summoned the vehicle. Accordingly, the location of the "passenger" or "client" is used to refer to the current location of the person or device that summoned the vehicle. Furthermore, in the present invention, there may be one or more "passengers" or "clients."

[0208] In addition, the AR service providing device (800) according to the present invention may be installed in a vehicle as part of the vehicle as shown in FIG. 7, or may be implemented as a terminal device separate from the vehicle.

[0209] In the former case, the AR service providing device (800) may be implemented in the form of a display placed in a vehicle, etc. In this case, the AR service providing device (800) is described as operating in communication and linkage with a terminal carried by a passenger.

[0210] In the latter case, the AR service providing device (800) may be implemented in the form of a terminal carried by a passenger. In this case, the AR service providing device (800) is described as operating in communication and interoperability with the vehicle (100).

[0211] FIG. 8 is a drawing showing an example of a detailed configuration of a system (1000) for providing a vehicle call service and an AR service providing device (800) related to an embodiment of the present invention.

[0212] In the embodiments of the present invention, the AR service providing device (800) is described as meaning a terminal device carried by a passenger (or a second passenger) unless otherwise stated.

[0213] Additionally, in embodiments of the present invention, the AR service providing device (800) may be used to mean a terminal device carried by a passenger (or a second passenger) before the passenger boards the unmanned vehicle, and may be used to mean either a terminal device carried by the passenger or a display placed in the vehicle after the passenger boards the unmanned vehicle.

[0214] There may be one or more AR service providing devices (800). For example, the first passenger and the second passenger may each have their own AR service providing devices (800). In this case, each AR service providing device (800) may operate in a manner of directly communicating and linking with the vehicle (100), or indirectly communicating and linking with the vehicle (100) through one of the other devices.

[0215] The AR service providing device (800) can be connected to a vehicle to enable communication through a communication module.

[0216] In addition, the AR service providing device (800) has an application installed for providing a vehicle call service according to an embodiment of the present invention, and can operate in conjunction with a service platform through execution of the application.

[0217] That is, the AR service providing device (800) according to an embodiment of the present invention can operate in conjunction with a service platform for providing vehicle calling, authentication, and location-designating services. In this case, the AR service providing device (800) and the service platform can be said to constitute a vehicle calling service providing system.

[0218] The service platform can provide both AR and MR services.

[0219] Specifically, the AR service provided by the service platform may provide an augmented reality experience related to displaying information related to the car-hailing service, such as recognizing a driving car. Furthermore, the MR service provided by the service platform may provide a digital world experience based on mixed reality related to the car-hailing service. In some embodiments, the MR service provided by the service platform may be performed only based on requests (e.g., passenger input / requests).

[0220] The AR service providing device (800) may be configured to include a communication unit (810), a display (820), and a processor (830).

[0221] The communication unit (810) may be configured to communicate with a vehicle, a service platform, and / or other devices / servers / clouds. In addition, the communication unit (810) may transmit a request to enable the service platform to communicate with the vehicle and / or other devices / servers / clouds, and may receive information about the results of a task / request from the vehicle and / or other devices / servers / clouds via the service platform.

[0222] The display (820) can display inputs and outputs related to the vehicle call service. In addition, the display (820) can display requests and / or execution results of user interfaces, AR services, and MR services provided through the service platform.

[0223] The processor (830) may include or be linked to an AR engine for providing AR services and / or MR services. That is, the processor (830) may be formed integrally with the AR engine or may operate to be linked to a device / server / cloud that includes the AR engine.

[0224] The processor (830) may include or operate in conjunction with an AR renderer and / or an MR renderer. For example, the processor (830) may operate in conjunction with an MR renderer that generates a digital twin map based on a three-dimensional polygon map received from a DTaaS server and an image collected through a camera of the vehicle (100), or may include an MR renderer that performs such operations.

[0225] The processor (830) may be linked to a service platform for providing a vehicle call service. For example, the processor (830) may be linked to the service platform to receive requests (e.g., dispatch requests, vehicle customization information, etc.) input by the AR service providing device (800) and transmit them to a related node (e.g., a vehicle dispatch server). In addition, for example, the processor (830) may be linked to the service platform to transmit information (e.g., dispatch result information, vehicle information, driving information, etc.) transmitted from a vehicle and / or a related server to the AR service providing device (800), and may cause a user interface and results related thereto to be output to the AR service providing device (800).

[0226] Referring to FIG. 8, a vehicle call service providing system (1000) according to an embodiment of the present invention may be implemented by including a service platform (900) that communicates with at least one node and an AR service providing device (800). This means that in embodiments of the present invention, the operation of the vehicle call service providing system (1000) is performed through the linkage between the service platform (900) and the AR service providing device (800).

[0227] The service platform (900) can communicate with one or more nodes to provide a vehicle call service, and can operate to provide a user interface related to the communication result to the AR service providing device (800).

[0228] Here, at least one node may include a vehicle (100), a cloud server (50), a vehicle customizing server (901), a vehicle dispatch server (902), and an authentication server (903).

[0229] The cloud server (50) can receive vehicle information and driving information from the vehicle (100) and provide them to the service platform (900). The cloud server (50) can transmit vehicle reservation information transmitted through the service platform (900) to the vehicle (100), i.e., the vehicle (100) that received the dispatch request.

[0230] The vehicle customization server (901) receives prior information (e.g., vehicle exterior color, passenger designation code, etc.) input by the AR service providing device (800) through the service platform (900) and transmits it to the vehicle (100) that has received the dispatch request.

[0231] The vehicle dispatch server (902) transmits a dispatch request to a candidate vehicle based on the location information of the AR service providing device (800) and one or more requests, and transmits the dispatch request result to the AR service providing device (800) through the service platform (900).

[0232] The authentication server (903) authenticates whether the passenger who called the vehicle is the passenger based on the passenger information transmitted through the service platform (900) and the image acquired through the camera (310) of the vehicle (100), and transmits the authentication result to the AR service provision device (800) of the vehicle (100) and the passenger through the service platform (900).

[0233] The service platform (900) can transmit vehicle designation and customization information to the vehicle dispatch server (902) when a vehicle is called. The service platform (900) can request the vehicle customization server (901) to store and / or retrieve the customization information received from the AR service providing device (800). The service platform (900) can request the authentication server (903) to generate a passenger authentication key, perform authentication based on feature points of an image, and receive the authentication result. In addition, the service platform (900) can additionally communicate with another node (e.g., FMS server, driving route server, data relay server, etc.) for updating and managing the vehicle's destination, calculating a driving route, and relaying real-time data between the server and the client.

[0234] The service platform (900) can receive information about the response result from one or more nodes and transmit it to the AR service providing device (800), so that the AR service providing device (800) can directly output the result of rendering an AR image corresponding to the vehicle customization information on a (real or virtual) vehicle or output a user interface that can execute it.

[0235] Although not shown, the service platform (900) may further include a Machine Learning as a Service (MLaaS) platform that transmits data necessary for providing car-hailing services to a hybrid computing system. In this case, the service platform (900) can provide more customized car-hailing services to regular passengers based on learning from the MLaaS platform.

[0236] Meanwhile, the AR service providing device (800) according to an embodiment of the present invention can display a 3D map in which an AR image generated based on vehicle information and location information of the calling vehicle is rendered on the calling vehicle.

[0237] In addition, the AR service providing device (800) according to an embodiment of the present invention can, in response to a request received while a 3D map in which an AR image is rendered on a called vehicle is displayed, convert the 3D map into a 3D surrounding area map based on the passenger's location information, and additionally display an AR graphic object at a location related to possible boarding and disembarking based on the boarding and disembarking difficulty attribute.

[0238] According to an embodiment, the AR graphic object may include a first AR graphic object rendered in a location where boarding and disembarking is possible and a second AR graphic object rendered in a location where boarding and disembarking is not possible.

[0239] A vehicle call service providing system (1000) according to an embodiment of the present invention includes a service platform (900) that is linked to an AR service providing device (800) and communicates with one or more nodes. Here, the one or more nodes may include a calling vehicle (100), a cloud server related to the vehicle call service, etc.

[0240] The service platform (900) may recognize that the called vehicle has entered the boarding area based on the passenger's location information and request that the camera of the called vehicle take pictures including multiple candidate passengers to generate multiple thumbnails.

[0241] The service platform (900) can receive information about multiple thumbnails from the calling vehicle and provide a user interface for selecting one of the multiple generated thumbnails to the AR service providing device (800).

[0242] The service platform (900) can track the location of a candidate passenger by transmitting information about a thumbnail selected by the AR service providing device (800) to the calling vehicle (100).

[0243] In this way, the AR service providing device (800) can check, select, and provide detailed boarding and disembarking locations based on boarding and disembarking difficulty attributes through AR and / or MR services. Furthermore, the AR service providing device (800) works in conjunction with the service platform (900) to enable the calling vehicle and passengers to mutually verify each other through AR and / or MR services, thereby ensuring safe and accurate boarding.

[0244] Below, FIGS. 9a and 9b are drawings for explaining distance conditions related to vehicle calling according to an embodiment of the present invention.

[0245] In an embodiment of the present invention, a case where the distance between the calling vehicle and the passenger is not included in the field of view of each other's cameras is defined as a "first distance."

[0246] In an embodiment of the present invention, the distance between the calling vehicle and the passenger within the field of view of each other's cameras is defined as "second distance." Furthermore, "second distance" includes the absence of static obstructions (e.g., structures such as buildings, street trees, and bridge piers) along the straight-line distance between the calling vehicle and the passenger, allowing for mutual filming.

[0247] In embodiments of the present invention, under conditions that satisfy the first distance requirement, the passenger determines the current location of the summoned vehicle through a 3D map rendered with AR images based on the vehicle's location information and vehicle information. Furthermore, under conditions that satisfy the first distance requirement, the summoned vehicle recognizes the passenger's location based on a location specified by the passenger, such as through GPS.

[0248] Referring to FIG. 9a, the visual field (920) of the passenger (910) (or, the shooting range (920) of the AR service providing device of the passenger (910)) and the shooting range (950) of the driving call vehicle (100) do not overlap at all, thus satisfying the first distance (D1). In this case, the passenger and the call vehicle cannot yet actually recognize each other.

[0249] In embodiments of the present invention, under conditions that satisfy the second distance, the calling vehicle can recognize that the passenger has entered the boarding area based on the passenger's location information.

[0250] Referring to FIG. 9b, the viewing range (920') of the passenger (910) (or, the shooting range (920') of the AR service providing device of the passenger (910)) and the shooting range (950') of the calling vehicle (100) close to the passenger overlap each other to satisfy the second distance (D2).

[0251] In this case, the passenger can actually recognize the vehicle he or she called through information such as the vehicle number of the called vehicle.

[0252] Additionally, in this embodiment, when a summoned vehicle enters the field of vision or shooting range, the passenger can more accurately recognize that the vehicle he or she summoned is the one he or she has summoned by seeing an AR object rendered based on vehicle information and location information displayed on the actual summoned vehicle. This will be described in more detail with reference to FIGS. 10a to 10e below.

[0253] However, even when the second distance (D2) requirement is met, the called vehicle cannot accurately identify the passenger until the passenger approaches and attempts to board. Consequently, it fails to provide quality service to passengers, and the anxiety of passengers with luggage increases in crowded airports and other places where passengers are waiting to board.

[0254] Accordingly, in the present invention, under conditions that satisfy the second distance, the vehicle provides an image of a candidate passenger and tracks the passenger based on the image selected by the passenger, thereby accurately recognizing the passenger even before boarding, thereby providing high-quality service.

[0255] Additionally, in the present invention, the vehicle can recommend to the passenger to change the boarding / disembarking position (to a position that is easy to board / disembark), and the passenger can also change the boarding / disembarking position in advance and then provide the vehicle with the exact position.

[0256] In this way, the present invention provides various embodiments in which a calling vehicle and a passenger can recognize each other and interact with each other even before the passenger boards.

[0257] FIGS. 10a, 10b, 10c, 10d, and 10e are drawings showing various examples of displaying the location and driving of a called vehicle in an MR map view and guiding a passenger to board, according to an embodiment of the present invention.

[0258] Figures 10a to 10c are examples of providing a mixed reality (MR)-based 3D map and a call vehicle image on the display of an AR service providing device (800) under conditions that satisfy the aforementioned 'first distance'.

[0259] Specifically, as illustrated in FIG. 10a, the AR service providing device provides a driving calling vehicle (100M) in the form of a bird's-eye view on an MR map view generated based on the location information and driving information of the calling vehicle.

[0260] To this end, the AR service providing device may be linked to a service platform that provides metadata for providing MR services (e.g., service metadata, 3D assets), 3D polygon maps, 3D maps such as digital twin maps (e.g., digital twin maps (DT)), or may be linked to one or more nodes (e.g., clouds or servers, etc.) through the service platform.

[0261] In the bird's eye view of Fig. 10a, traffic light information (1004) based on C-ITS or camera recognition, a driving route (1001) to a boarding location designated by a passenger, and objects (e.g., other vehicles) (1002) recognized by sensors of the calling vehicle and related caution information (1003) may be provided together on a 3D map.

[0262] In Fig. 10b, when a bird's-eye view of a summoned vehicle driving along a driving route (1001) on a 3D map is provided, various AR objects based on customized information specified by the passenger who summoned the vehicle may be rendered and displayed. For example, a passenger-set color may be rendered on a lampshade (1005) in the summoned vehicle, and a passenger-set unique code (1006) and vehicle number (1007) may be rendered on the external display of the summoned vehicle. In addition, an area (e.g., a right side area) of the display illustrated in Fig. 10b may additionally display identification code information (1008) and QR information (1009) for passenger authentication.

[0263] Continuing, as illustrated in FIG. 10c, while the bird view call vehicle (100M) is driving along the driving path (1001), driving objects recognized through various sensors equipped in the actual vehicle, such as pedestrians (1010), signal information (1011), and driving-related information, such as a speech bubble object (1012) containing the text 'stopped at traffic light', can be rendered and displayed in real time.

[0264] Meanwhile, the MR map view illustrated in FIGS. 10a to 10c can perform 3D screen rotation, 3D object rotation, view zoom in / out, etc. based on various types of input.

[0265] Continuing, FIG. 10d is an example in which a real view is displayed on the display of an AR service providing device (800) under conditions that satisfy the second distance described above, and an AR object based on customizing information is rendered on the real call vehicle of the Real View.

[0266] FIG. 10d illustrates a situation where an actual called vehicle (100) has entered the shooting range of a passenger's AR service providing device (800). At this time, the exact location is tracked based on the unique information of the called vehicle (100), and an AR graphic object (1013) generated using customized information provided by the passenger as prior information is rendered in real time on the actual called vehicle (100), for example, on an external display, thereby implementing the system. Identification information and authentication information of the called vehicle may be provided in one area (e.g., the right side area) of the AR service providing device (800).

[0267] Specifically, the AR service providing device (800) recognizes multiple surrounding vehicles within the camera's preview image, matches the feature points of the recognized surrounding vehicles with the feature points based on the customized information set by the passenger, determines the actual vehicle called based on the similarity, and tracks its location. Here, the feature points of the vehicle may include one or more of the vehicle color, the vehicle type / model, the shape of the hood, whether there is a shade light, the shape of the wheels, the shape of the headlights, the vehicle number, and the external identification display display characters / identification code (QR).

[0268] The AR service providing device (800) can render and track an image box around the determined call vehicle in real time so that the passenger can easily track the determined call vehicle. That is, an AR object highlighting the final determined call vehicle can be rendered, or the background of the real view excluding the final determined call vehicle can be processed to black and white. In addition, although not shown, an AR speech bubble image indicating that the final determined call vehicle is the vehicle I will ride can be displayed, or a highlight can be displayed on a unique feature part of the vehicle's exterior customized by the passenger (e.g., a wheel). In addition, although not shown, for users who are sensitive to personal information, the E-Ink and / or transparent display on the exterior of the actual vehicle may not be changed, and the AR graphic object may be applied only to the vehicle displayed on the AR service providing device (800). Also, in some embodiments, if the final determined call vehicle is obscured by surrounding obstacles (e.g., surrounding vehicles), the expected location of the vehicle at the current time point ('second time point') determined by considering the vehicle location immediately before ('first time point') and the direction / speed of movement of the vehicle can be predicted, and tracking can be performed by outputting a partial highlight at the predicted location.

[0269] Continuing, FIG. 10e shows various display examples according to the location of the summoned vehicle on the 3D map in the AR service providing device (800). If the summoned vehicle does not appear on the displayed 3D map, the AR service providing device (800) provides guidance on the approximate location of the summoned vehicle by displaying direction information (1015). In addition, if the summoned vehicle exists on the displayed 3D map but is far away and difficult to recognize, only the location information (1016) can be rendered to provide guidance. In addition, when the summoned vehicle appears head-on on the 3D map of the AR service providing device (800), an AR object (1017) based on the passenger's customized information is rendered on the external display or the entire vehicle. Meanwhile, although not shown, the AR service providing device (800) can display POI information surrounding the driving route of the summoned vehicle upon request, and the passenger can change the initially set boarding location based on the displayed POI information.

[0270] In this way, by rendering AR objects of various shapes according to the location of the called vehicle on a 3D map, the vehicle called by the passenger can be identified more accurately.

[0271] Meanwhile, in some embodiments, the summoned vehicle may change the E-Ink and / or transparent display on the exterior of the vehicle to apply text, images, colors, etc. based on the passenger's customized information. Here, the passenger's customized information may include the vehicle's color, texture, and preset unique ID information customized by the user.

[0272] In this regard, the passenger's customized information set through the AR service providing device (800) can be uploaded to the vehicle customization server (901) through the service platform. In addition, the customized information stored in the database within the vehicle customization server (901) can be transmitted to the vehicle dispatch server (902) together with a unique ID, current GPS location, destination, and other information required when calling a vehicle. Accordingly, the AR service providing device (800) reflects the external features customized in advance by the user on the actual dispatched vehicle and renders them on a 3D map. In other words, features of the dispatched vehicle's external features that can be identified with the naked eye are directly reflected in the 3D vehicle model and displayed on the 3D map.

[0273] The above has examined a method for recognizing the unmanned vehicle I will board among multiple vehicles using a passenger AR service providing device (800), for example, a camera on a passenger terminal. Below, embodiments related to a method for accurately recognizing passengers even before boarding a vehicle by providing multiple passenger images acquired through the vehicle's camera to the AR service providing device (800) are described.

[0274] FIG. 11 is an exemplary flowchart illustrating a method for determining and authenticating a passenger who has called a vehicle using a thumbnail related to an embodiment of the present invention.

[0275] Each process of FIG. 11 is performed by one of the service platforms and linked AR service providing devices that constitute the vehicle call service providing system. Furthermore, some of the processes of FIG. 11 may be performed by the AR service providing device, while others may be performed via the service platform communicating with one or more nodes. Here, one or more nodes may include a calling vehicle and a cloud server that communicate with the service platform.

[0276] Specifically, some of the processes of FIG. 11 may be performed by the service platform receiving data / commands / signals from the calling vehicle, one or more servers, and / or the AR service providing device, or transmitting data / commands / signals to them. In addition, some of the processes of FIG. 11 may be performed by the linked AR service providing device transmitting data / commands / signals to the calling vehicle, and / or one or more servers via the service platform, or outputting or changing information to the AR service providing device based on data / commands / signals received from one of them.

[0277] Referring to FIG. 11, the AR service providing device (800) can obtain real-time location information of a called vehicle through the service platform (900) or directly (S1110). To this end, the AR service providing device (800) or the service platform (900) can receive location data directly from the called vehicle, or the service platform (900) can receive GPS-based location data through one or more nodes.

[0278] Accordingly, the AR service providing device (800) can render an AR image based on the location information and vehicle information of the calling vehicle to the calling vehicle and display it on a 3D map (S1120).

[0279] Here, the summoned vehicle displayed on the 3D map is presented in a bird's-eye view, taking into account driving information, such as the vehicle's direction and speed. Furthermore, the summoned vehicle displayed on the 3D map displays external characteristics based on customization information pre-configured by the passenger.

[0280] In this way, while the calling vehicle is displayed in a bird's-eye view on the AR service providing device (800), additionally displayed signal information, POI information, etc. have been described in detail with reference to FIGS. 10a to 10c above, so a description thereof will be omitted here.

[0281] When the distance between the called vehicle and the passenger (or the passenger's AR service providing device (800)) changes from the aforementioned 'first distance' to the 'second distance', the AR service providing device (800) can determine that the called vehicle with the AR image rendered has entered the boarding area (S1130).

[0282] Here, the boarding possible section may be a case where the distance between the vehicle and the passenger satisfies the second distance (i.e., a case where the shooting range of the camera equipped on the calling vehicle and the shooting range of the camera equipped on the AR service providing device are within the distance range where the calling vehicle and the AR service providing device can each shoot each other), or a case where the location of the vehicle satisfies a case where the location of the vehicle is within a predetermined radius (e.g., 10 to 15 m) from the boarding location input / set by the passenger. In addition, the boarding possible section may mean a case where the second distance is satisfied and the calling vehicle is not in a no-stop zone.

[0283] In this way, when the called vehicle enters the boarding area, the AR service providing device (800) can recognize the called vehicle through changes in the external characteristics of the called vehicle.

[0284] While vehicles can estimate passengers to some extent using location and passenger information entered and configured in advance, it can be difficult to accurately identify passengers in crowded places (e.g., bus stops). Therefore, the following steps are being implemented to clearly identify passengers in vehicles, alleviate their anxiety, and provide improved service.

[0285] Specifically, in response to the call vehicle entering the boarding area, the service platform may request the camera of the call vehicle to capture multiple candidate passengers and generate multiple thumbnails (S1140).

[0286] Specifically, the service platform (900) captures images of all surrounding passengers within the boarding area via multiple external cameras of the calling vehicle, analyzes the captured images, and generates thumbnails for each candidate passenger. Each generated thumbnail is matched with a corresponding unique ID. The service platform (900) transmits information about the thumbnail and ID generated by the calling vehicle to the AR service providing device (800). Furthermore, the service platform (900) can upload information about the thumbnail and ID generated by the calling vehicle to at least one node.

[0287] Next, the service platform (900) may provide the AR service providing device (800) with a user interface for selecting one of the generated multiple thumbnails (S1150). Here, the user interface may include an API related to displaying multiple thumbnails, selecting one, and transmitting the selected information.

[0288] For example, the service platform (900) may display multiple thumbnails in order of similarity to passenger information entered / set by the passenger, and provide a user interface for selecting a thumbnail that matches the passenger's input. Furthermore, for example, the service platform (900) may display multiple thumbnails in the form of cards that can be flicked horizontally / vertically, and provide a user interface for selecting one of the thumbnails. Furthermore, for example, the service platform (900) may provide a user interface for selecting one of the thumbnails while overlapping each thumbnail at a corresponding position in the overall image.

[0289] Next, the service platform (900) can receive information about the selected thumbnail from the AR service providing device (800) through the user interface and transmit the information about the selected thumbnail to the calling vehicle to track the location of the candidate passenger (S1160).

[0290] Specifically, the service platform (900) transmits and shares the ID and feature points for the received, selected thumbnails with the calling vehicle and one or more nodes, and allows the calling vehicle to identify a passenger based on the feature points of the thumbnail matching the ID (hereinafter, “first feature points”) and to track the identified passenger through a camera.

[0291] Meanwhile, the AR service providing device (800) may display a consensus message according to the selection of the thumbnail.

[0292] If the third distance between the called vehicle and the passenger is satisfied, the vehicle may stop. Here, the third distance may mean that the location information of the called vehicle matches or is nearly identical to the input / set boarding location information, with thumbnail selection by the AR service providing device (800) as an initiating condition.

[0293] In an embodiment, the service platform (900) may compare the feature points (hereinafter, “second feature points”) of the image of a candidate passenger (or pedestrian) recognized through the side camera of the vehicle when the vehicle is stopped with the first feature point to determine the final passenger.

[0294] For example, when the vehicle is stopped, the service platform (900) can compare whether a passenger approaching the vehicle matches the passenger tracked based on the first characteristic point by a standard value or more, and cause the calling vehicle to perform automatic passenger authentication and / or boarding guidance operations.

[0295] Here, automatic passenger authentication means that if the comparison result between the first and second characteristic points satisfies a similarity level higher than the standard, authentication is considered successful, and if the similarity level is lower than the standard, authentication is considered failed.

[0296] Additionally, the boarding guidance operation herein may include displaying information guiding passenger boarding on the external features of the called vehicle (e.g., E-Ink and / or an external transparent display) or outputting similar voice guidance information externally.

[0297] In this way, according to an embodiment of the present invention, not only can the vehicle called by the passenger be accurately identified, but the vehicle can also identify and authenticate the passenger in advance through a thumbnail selected by the passenger before boarding, thereby facilitating smooth mutual interaction and authentication.

[0298] FIGS. 12 and 13 are drawings for explaining how a vehicle generates thumbnails for candidate passengers and provides them to an AR service providing device, in response to a request from a service platform related to an embodiment of the present invention.

[0299] And, FIG. 14 is a drawing for explaining selecting a passenger who called a vehicle using a thumbnail in an AR service providing device related to an embodiment of the present invention.

[0300] Also, FIG. 15 is an exemplary flowchart for explaining a method for generating a thumbnail based on prior information of a passenger who has called a vehicle related to an embodiment of the present invention.

[0301] First, referring to Figure 12, when a summoned vehicle enters a boarding area based on the boarding location entered / set by the passenger, the vehicle's multiple external cameras capture images of potential passengers within the boarding area. The summoned vehicle then generates thumbnails for each potential passenger from the captured images.

[0302] Specifically, the calling vehicle detects boardable objects through multiple external cameras and generates individual thumbnails for the detected objects.

[0303] In an embodiment, when generating multiple thumbnails, it is possible to determine whether passenger information (or, pre-input / set information, additional information) is included in advance, and individual thumbnails can be generated based on the determination.

[0304] In this regard, referring to FIG. 15, the AR service providing device (800) determines, via the service platform (900), that the vehicle has entered a boarding zone based on the passenger's location information (S1510). That is, the service platform (900) can determine that the vehicle has entered a boarding zone based on the passenger's current location information or boarding location information specified by the passenger.

[0305] When a passenger calls a vehicle, the service platform (900) can transmit additional information input from the AR service provision device (800) to the calling vehicle (S1520). Here, the additional information (or passenger information, pre-information) includes the passenger's gender, physical characteristics such as clothing, stored unique information, the number of luggage items, and whether or not passengers are present.

[0306] Next, the service platform (900) can extract an area related to additional information from an image captured through a camera of a calling vehicle and generate a plurality of thumbnails including the extracted area (S1530).

[0307] Now, referring to FIG. 12, a first image object (1201, 1202) and a first recognition box are displayed for each candidate passenger detected in the captured image. Then, a second image object (1203, 1204) is displayed for the recognized object based on passenger information pre-entered / set by the passenger and transmitted to the called vehicle. For example, if the passenger information pre-entered / set by the passenger includes 'bag' and 'passenger present', a second image object (1203, 1204) and a second recognition box are displayed for the object matching each passenger information.

[0308] In some embodiments, when passenger information is included, an edited thumbnail may be generated that includes an object matching the passenger information. To this end, the service platform (900) transmits additional information input from the AR service providing device (800) to the called vehicle when the vehicle is called, extracts an area related to the additional information from an image captured by the camera of the called vehicle, and generates multiple thumbnails including the extracted area.

[0309] Referring to FIG. 13, information regarding a plurality of generated thumbnails is received via the service platform (900) and output to the display of the AR service providing device (800). As illustrated in FIG. 13, a plurality of thumbnails (1301S to 1305S) generated from a single image including all of a plurality of candidate passengers (1301 to 1305) are provided for selection. At this time, selection of one of the generated plurality of thumbnails (1301S to 1305S) is displayed in a manner that matches the location on the image including all of a plurality of candidate passengers (1301 to 1305).

[0310] For example, as illustrated in FIG. 14, if a selection input is made for the first thumbnail (1401S) in the AR service providing device (800), a bounding box is displayed on the candidate passenger (1401) corresponding to the first thumbnail (1401S) in the entire image. If a selection input is made for another thumbnail (1401S) in the AR service providing device (800), a bounding box is displayed on the candidate passenger (1404) corresponding to the other thumbnail (1401S) for which a selection input was received in the entire image. Accordingly, the passenger can accurately and intuitively select his or her own thumbnail by simultaneously checking individual thumbnails and surrounding images.

[0311] Meanwhile, in some embodiments, for privacy reasons, the generated thumbnails and full images may be provided with faces blurred. In such cases, passengers can make selections based on their location, posture, and other external characteristics, excluding their face.

[0312] In Fig. 14, if additional input is received after selecting the first thumbnail (1401S) or another thumbnail (1401S), the selected thumbnail is determined to be selected and a consensus message is output. Thereafter, the AR service providing device (800) transmits information about the ID and feature points of the selected thumbnail to the vehicle (via the service platform). Then, the called vehicle recognizes the candidate passenger corresponding to the transmitted thumbnail ID and tracks the location of the recognized candidate passenger in real time based on the corresponding feature points.

[0313] Meanwhile, since the passenger can now sufficiently identify the summoned vehicle with the naked eye, the exterior of the vehicle or the external appearance of the vehicle, as shown through the AR service providing device (800) as described with reference to FIG. 10d, may be changed based on the passenger's customized information. For example, the hood of the summoned vehicle may be changed to a passenger-preset color, or an identification character and / or a QR code for authentication may be displayed on the exterior or external display of the vehicle.

[0314] According to an embodiment, if a passenger is not included in the plurality of thumbnails provided to the AR service providing device (800), a "non-selectable input" may be executed using the provided user interface. In this way, in response to the non-selectable input for the plurality of thumbnails, the service platform (900) transmits a re-capture command to the calling vehicle. The vehicle re-generates a thumbnail according to the re-capture command and transmits it to the AR service providing device (800) via the service platform (900).

[0315] Additionally, although not shown, in some embodiments, the AR service providing device (800) may, in response to a selection input for a thumbnail, change a first AR image based on the location information and vehicle information of the called vehicle into a second AR image based on passenger customization information and render it on the called vehicle.

[0316] According to this, passenger selection and vehicle matching are performed simultaneously, allowing passengers to more intuitively recognize the called vehicle.

[0317] FIG. 16 is an exemplary flowchart illustrating a method for authenticating a passenger who has called a vehicle based on feature points of a selected thumbnail related to an embodiment of the present invention.

[0318] Referring to FIG. 16, the AR service providing device (800) receives a selection input for one of a plurality of thumbnails using the provided user interface (S1610).

[0319] The ID and characteristic points of the thumbnail corresponding to the selection input are transmitted to the calling vehicle through the service platform (900), and the service platform (900) causes the calling vehicle to recognize a candidate passenger corresponding to the thumbnail ID, and continuously tracks the location of the recognized candidate passenger based on the characteristic points ('first characteristic points') (S1620).

[0320] At this time, the location tracking of the candidate passenger continues until the passenger boards / cancels boarding or passenger authentication is performed.

[0321] When transmitting information about the ID and feature points of a thumbnail corresponding to a selection input, the service platform (900) may transmit a consensus message to the AR service providing device (800). In addition, the service platform (900) may upload information about the ID and feature points of a thumbnail corresponding to a selection input to at least one node in communication with the vehicle.

[0322] Next, the service platform (900) performs passenger authentication by comparing the first feature point of the image searched based on the thumbnail ID corresponding to the selection input of the called vehicle with the second feature point obtained from the captured image of the candidate passenger close to the door of the called vehicle in response to the stop of the called vehicle (S1630).

[0323] Here, the comparison between the first and second feature points can be performed by analyzing whether the number of matches between the feature points is greater than or equal to a predetermined value (e.g., 7 to 8 out of 10). Alternatively, the comparison between the first and second feature points can be performed by analyzing whether the overall degree of similarity between the feature points is greater than or equal to a predetermined ratio (e.g., 80%).

[0324] As a result of determining whether the similarity between feature points is above a standard value (S1610), if it is determined that the similarity is above the standard value, the service platform (900) processes it as “authentication success” (S1650).

[0325] Accordingly, the service platform (900) may cause the vehicle to perform actions related to guiding the passenger to board (S1660). For example, a welcome message such as "Welcome" may be output via E-Ink or an external display on the outside of the summoned vehicle. Alternatively, for example, the vehicle may be guided to unlock the door or scan a QR code. Meanwhile, in some embodiments, QR code authentication or communication between the vehicle and the passenger's AR service providing device, such as UWB / WiFi / BT, may be included as part of the unlocking process.

[0326] Meanwhile, if the judgment result (S1610) determines that the similarity between feature points is below the standard value, the service platform (900) processes it as “authentication failure” (S1670).

[0327] Accordingly, the service platform (900) causes the vehicle to perform actions related to the passenger mismatch and transmits vehicle arrival information to the AR service providing device (800) (S1680). For example, the vehicle may display a mismatch message, such as "Not a passenger," on the vehicle's external display. Additionally, for example, the AR service providing device (800) may display acceptance information for re-performing the previous steps of extracting the second feature point and comparing the feature points.

[0328] Additionally, although not shown, in case of authentication failure through selective input for thumbnails, customized information can be reflected in the vehicle to visually guide passenger boarding, and authentication of passengers can be performed based on whether a UWB signal is captured.

[0329] The embodiments described below may include a vehicle display as an AR service providing device (800). Additionally, some embodiments described below may include multiple AR service providing devices, such as the first and second AR service providing devices.

[0330] In the past, when a passenger in an unmanned vehicle needed to change their itinerary, such as changing their boarding or disembarkation location, changing their destination, requesting a ride-sharing service, or adding a stopover, they had to press the in-vehicle control service center call button to connect to a counselor over the phone and change the necessary itinerary through third-party communication, which was inconvenient.

[0331] Accordingly, the following embodiments describe a method for a passenger to directly interact with a vehicle through an AR service providing device without third-party communication, thereby requesting changes to an itinerary, such as changing a boarding or disembarking location, changing a destination, requesting a ride-sharing service, or adding a stopover.

[0332] Below, FIG. 17 is an exemplary flowchart for explaining a method of deriving and selecting a boarding location using 3D map information related to an embodiment of the present invention.

[0333] Each process of FIG. 17 may be performed by the AR service providing device (800). In some examples, the AR service providing device (800) may display a 3D map and perform the process by communicating or interfacing with one or more nodes for providing AR or MR services. In addition, the AR service providing device (800) may include an AR engine or renderer for rendering AR objects. In addition, unless otherwise stated, each process of FIG. 17 is performed on the premise that it is operated by the processor (830) of the AR service providing device (800).

[0334] Referring to Figure 17, the AR service providing device (800) can first obtain real-time location information of the calling vehicle (S1710). To this end, the AR service providing device (800) can receive location data directly from the calling vehicle, or the service platform (900) can receive GPS-based location data via one or more nodes.

[0335] In addition, the AR service providing device (800) can render an AR image based on the acquired location information and vehicle information (and driving information) of the called vehicle to the called vehicle and display it on a 3D map (S1720).

[0336] Here, the summoned vehicle displayed on the 3D map refers to a vehicle that is displayed on the 3D map in a bird's-eye view, taking into account the vehicle's driving information, such as its direction and speed. Furthermore, the summoned vehicle displayed on the 3D map displays external characteristics based on customization information pre-configured by the passenger.

[0337] Continuing, the AR service providing device (800) can, in response to a request, convert the 3D map into a 3D surrounding area map based on the passenger's location information (S1730).

[0338] Here, the passenger's location information may refer to the current location of the AR service providing device (800) based on GPS. Alternatively, the passenger's location information may be boarding (disembarking) location information previously input / set through the AR service providing device (800).

[0339] Passengers may summon a driverless vehicle in an unfamiliar location or alight from a driverless vehicle in an unfamiliar location. For example, when summoning a driverless vehicle in a new location or at night, passengers may want to alter their itinerary by accurately informing the driverless vehicle of their current location or confirming that their current location is suitable for boarding or alighting.

[0340] Accordingly, the AR service providing device (800) can, in response to a passenger's request, communicate with one or more nodes to generate a 3D surrounding area map based on the passenger's location information, and collect information on polygon and texture data of 3D map buildings, roads, and terrain, and unique IDs for each surrounding building and road area.

[0341] According to an embodiment, the AR service providing device (800) may provide an MR view for a specific location based on an input for the specific location in a 3D surrounding area map.

[0342] To this end, the AR service providing device (800) can communicate with a DTaaS server (not shown) as one or more nodes, and can cause the DTaaS server to provide an MR view around the specific location by linking with a POI database in which POI service data for each building or each area included in the map information is stored.

[0343] Continuing, the AR service providing device (800) can display (render) an AR graphic object at a location related to possible boarding and disembarking on the 3D surrounding area map based on the boarding and disembarking difficulty attribute (S1740).

[0344] Here, the difficulty of boarding and disembarking can be determined based on the availability of the summoned vehicle to stop. Furthermore, the difficulty of boarding and disembarking can be determined based on the passenger's location information, the polygon and texture data of the 3D map buildings, roads, and terrain, and other topographical characteristics, such as unique IDs for each surrounding building and road section. The polygon and texture data of the terrain can include data of a 3D polygon model that provides the volume of the structure and has or does not have texture data formed on its surface.

[0345] In an embodiment, the AR graphic object rendered at the above boarding and disembarking related location may include a first AR graphic object indicating a boarding and disembarking location and a second AR graphic object indicating a non-boarding and disembarking location, based on the boarding and disembarking difficulty attribute.

[0346] In an embodiment, the AR service providing device (800) can change the boarding / disembarking location by selecting an AR graphic object rendered on a 3D surrounding area map. Specifically, the AR service providing device (800) can cancel the existing boarding / disembarking location and designate a new boarding / disembarking location by transmitting the GPS-converted location corresponding to the AR graphic object to the vehicle.

[0347] In relation to Fig. 17, Figs. 18 and 19 are examples showing recommending a boarding location and an alighting location based on the boarding and alighting difficulty attribute of the called vehicle, respectively.

[0348] The AR service providing device (800) can render AR graphic objects at locations related to possible boarding and disembarking to provide boarding / disembarking recommendations / not recommendations based on boarding and disembarking difficulty attributes on a 3D surrounding area map.

[0349] The AR service providing device (800) can render a corresponding real street view, i.e., an MR image, based on a 3D surrounding area map based on a selection input for an AR graphic object indicating a recommendation / disrecommendation for boarding / disembarking.

[0350] Here, the boarding / disembarking difficulty attribute may include one of road information, building information, and terrain information on the 3D surrounding area map. Furthermore, the recommended / non-recommended boarding / disembarking locations based on the boarding / disembarking difficulty attribute may be determined specifically based on polygon and texture data for buildings, roads, and terrain on the 3D map, unique IDs for each building and road section, passenger location information, and vehicle location information.

[0351] Furthermore, the difficulty of boarding and disembarking attribute can be used broadly to encompass all attributes that affect the ability of a driverless vehicle to stop. Examples include public transportation boarding and disembarking areas, shuttle stations, areas where stopping is prohibited under the Road Traffic Act, areas under construction, areas where accidents have occurred, areas with high current traffic congestion, areas with a high number of ride-hailing service requests, and areas with guardrails installed on sidewalks.

[0352] According to an embodiment, the AR service providing device (800) can determine recommended boarding and alighting locations by calculating walking distances for each location assigned a difficulty level. For example, the recommended boarding and alighting locations are sorted based on proximity to the passenger's location information and whether the boarding and alighting location is easy. Then, among the sorted recommended boarding and alighting locations, the top few are finally determined as the recommended boarding and alighting locations.

[0353] According to an embodiment, the AR service providing device (800) may determine non-recommended boarding and alighting locations based on traffic condition information. Here, the traffic condition information is refreshed within a certain time or distance from the vehicle's expected arrival, and the latest collected traffic condition information is used.

[0354] The AR service provider (800) can render distinct color coding for recommended and non-recommended boarding and disembarking locations, allowing passengers to intuitively identify them. By providing a 3D surrounding area map and AR objects, passengers can intuitively understand boarding and disembarking difficulty attributes.

[0355] Referring to FIG. 18, first graphic objects (1811, 1812, 1813, 1830) indicating recommended boarding locations on a 3D surrounding area map are rendered based on the passenger's location information (CL). For example, the first graphic objects (1811, 1812, 1813, 1830) may be displayed at locations corresponding to stopping areas, stopping areas such as bus stops, areas outside of congested traffic areas, and ride-sharing exclusive areas, respectively. In addition, second graphic objects (1821, 1840) indicating non-recommended boarding locations on a 3D surrounding area map may also be rendered based on the passenger's location information (CL). For example, the second graphic objects (1821, 1840) may be displayed at locations corresponding to congested traffic areas and areas with pedestrian guardrails, respectively.

[0356] In Fig. 18, when a selection input is applied to the recommendation icon (1810), only the first graphic object (1811, 1812, 1813, 1830) indicating only the recommended boarding location is highlighted, and the graphic object indicating the non-recommended boarding location information is hidden.

[0357] On the other hand, when a selection input is applied to the impossible icon (1820) in Fig. 18, only the second graphic object (1821, 1840) indicating only the impossible location is highlighted, and the graphic object indicating the recommended location information is hidden.

[0358] Meanwhile, if there is an actual view or MR view for at least some of the recommended ride locations and / or non-recommended ride locations illustrated in FIG. 18, selecting the location will output the actual view or MR view. For example, if there is a selection input at a location where the first graphic object (1830) is displayed, an actual view or MR view (1831) that allows you to check the ride share-only sign will pop up. Also, for example, if there is a selection input at a location where the second graphic object (1840) is displayed, an actual view or MR view (1841) that allows you to check the pedestrian guardrail sign will pop up.

[0359] Figure 19 shows a 3D peripheral area map based on the passenger location information (CL) shown in Figure 18 applied to the destination's disembarkation location. In this case, the passenger location information (CL) of Figure 19 refers to the destination location previously input / set by the passenger.

[0360] In Fig. 19, when a selection input is received for the first graphic object (1830) indicating a recommended get-off location, the AR service providing device (800) renders an AR speech bubble image (1910) including additional information such as the corresponding location information, additional travel distance / travel time information when changing to the corresponding location, and additional fee information. The AR service providing device (800) may transmit GPS-based location information that is converted to match the changed get-off location information to the called vehicle based on the AR speech bubble image (1910) rendered on the 3D surrounding area map, based on the selection of a change in the get-off location.

[0361] In this way, embodiments of the present invention provide a 3D surrounding area map based on passenger location information so that the difficulty of boarding a called vehicle can be easily determined, and can recommend / disrecommend boarding / disembarkation locations to passengers and intuitively provide related additional information through AR images or MR views.

[0362] According to an embodiment, the processor (830) of the AR service providing device (800) may change the boarding and disembarking location based on a selection input for an AR graphic object, and transmit information about the changed boarding and disembarking location to the vehicle through the communication unit (810) so that the called vehicle performs an operation corresponding to the changed boarding and disembarking location.

[0363] At this time, AR route information guiding the changed boarding location is displayed on the 3D map on the display (820) of the AR service providing device (800).

[0364] Specifically, when the walking path changes according to the changed boarding and disembarking locations, the AR service providing device (800) can display the walking path connecting the passenger's current location or the current location of the called vehicle to the changed boarding and disembarking locations as an AR object on the road of a 3D map.

[0365] Additionally, although not shown, if the called vehicle enters a "secondary street" and the pre-entered / set pickup location is determined to be congested, guidance information may be output via the vehicle's external E-Ink or external display to guide the vehicle to move a predetermined distance from the agreed pickup location. For example, information such as "Move forward a little further and stop" may be displayed via the vehicle's hood lights, external E-Ink, or external display.

[0366] According to an embodiment, the processor (830) of the AR service providing device (800) may, in response to the arrival of a called vehicle with an AR image rendered within a boarding area, request the vehicle to check the surroundings based on the passenger's location information through an external camera of the called vehicle. Thereafter, based on the response to the request from the AR service providing device (800), the AR graphic object indicating a recommended boarding and alighting location on a 3D map may be rendered, taking into account the difficulty of boarding and alighting.

[0367] In this way, embodiments of the present invention allow for the identification of the difficulty level of a boarding area, which is difficult to determine using 2D map information. Furthermore, since the actual street view is rendered and displayed based on 3D map information, it is very helpful for passengers to understand the surroundings structurally even when calling a vehicle in an unfamiliar area. Furthermore, if necessary, boarding and alighting locations can be easily changed based on 3D map information and recognized by the vehicle.

[0368] FIG. 20 is a drawing showing a method for selecting a vehicle disembarkation direction using a highlighted structure on a three-dimensional map according to an embodiment of the present invention.

[0369] Additionally, FIG. 21 is a drawing showing a method for changing a vehicle's path using three-dimensional map information related to an embodiment of the present invention.

[0370] First, referring to Figure 20, highlighting is rendered on surrounding structures based on the passenger's location information (CL) displayed on the 3D surrounding area map. Here, the surrounding structures are structures visible to the naked eye when the passenger disembarks from the vehicle, and may include, for example, surrounding buildings, signage, building entrances, bus stops, and sculptures. Furthermore, the ID or name of the surrounding structure may be augmented and displayed in the form of an AR speech bubble.

[0371] At this time, at least some of the front, left and right sides, and / or rear of the structure are displayed with highlight images (2010, 2020) of distinct colors. Passengers can accurately designate the direction in which to disembark from the vehicle by selecting one of the highlight images (2010, 2020).

[0372] Specifically, based on a selection input for one of the highlight images (2010, 2020) of a color that distinguishes the structure, for example, the image (2010) of the first side, the AR service providing device (800) recognizes the direction matching the image (2010) of the first side as the disembarkation location and transmits information about it to the vehicle. In a similar manner, the AR service providing device (800) may also recognize the direction matching the image (2020) of the second side as the disembarkation location and transmit information about it to the vehicle. Accordingly, the passenger can more accurately designate the desired disembarkation direction.

[0373] To this end, the processor (830) of the AR service providing device (800) may render multiple AR images for specifying boarding and disembarking directions for structures included in an actual street view image. Furthermore, the processor (830) may determine a structure direction corresponding to a selected AR image among the multiple AR images as a boarding and disembarking location. Furthermore, the processor (830) may operate to transmit information about the boarding and disembarking location in the structure direction determined through a selection input to the called vehicle via the communication unit (810).

[0374] Meanwhile, even after boarding a called vehicle, the passenger can change the travel route differently from the vehicle settings through the AR service provision device (800) (equipped in the vehicle).

[0375] Referring to FIG. 21, for example, in a state where a first movement path (2110) according to vehicle settings is displayed in an MR map view, a second movement path (2120) is generated based on an input that deviates from the first movement path, and a portion of the second movement path (2120) is rendered in a form that overlaps the first movement path (2110).

[0376] Thereafter, the AR service providing device (800) deletes the display of the first movement path (2110) from the MR map view based on the selection input for the second movement path (2120) and changes the driving path along the second movement path (2120). The AR service providing device (800) can transmit information about the changed second movement path (2120) to the vehicle to change the current driving path.

[0377] Specifically, while the summoned vehicle is driving after the passenger boards the vehicle, the AR service providing device (800) displays a first carpet image indicating the current driving route on the MR view map. Additionally, a second carpet image indicating another possible detour route may be displayed overlapping a portion of the first carpet image. Here, the change to the other possible detour route may be determined by considering both the traffic conditions around the summoned vehicle and the time available for changing the route before the second carpet image reaches a junction with respect to the first carpet image.

[0378] The passenger can change the driving route of the called vehicle by selecting the second carpet image displayed on the MR view map of the AR service providing device (800). In this case, the display (820) of the AR service providing device (800) can provide a preview image of the called vehicle exiting the junction along the selected detour route on the MR view map in the form of a three-dimensional image, and can also output a message or voice such as "Shall we take the detour route?" to guide the passenger's final choice.

[0379] In addition, according to an embodiment, specifically, when a destination change item is entered through an AR service providing device (800), an MR map view centered on a previously input / set first destination is displayed. In this case, the displayed MR map view recommends a plurality of recommended get-off locations within a predetermined radius centered on the first destination, and AR image objects are rendered at the plurality of recommended get-off locations. Next, when one of the plurality of recommended get-off locations is selected, the AR service providing device (800) recognizes the location as a second destination and transmits location information on the second destination to the vehicle so as to change the vehicle's get-off location to the second destination.

[0380] According to the embodiments discussed above, passengers can easily understand surrounding structures through the MR view map and easily change boarding / disembarking locations to provide information to the vehicle. Furthermore, after reviewing multiple travel routes provided through the MR view map, passengers can easily select and change their desired driving route.

[0381] Meanwhile, some embodiments of the present invention describe a method for accurately recognizing the location of a second passenger waiting at a stopover after boarding a vehicle and transmitting the location to the vehicle. The method is similar to the aforementioned method of recognizing and changing the boarding location of the second passenger waiting at a stopover in that the second passenger is "before" boarding the vehicle. However, the method differs in that the first passenger is "already on board" the vehicle. In other words, since the location of the stopover is set by the first passenger, the second passenger has limited ability to change his or her location.

[0382] FIG. 22 and FIG. 23 are exemplary diagrams and flowcharts for explaining a method for providing location information of a transit point to a second passenger using three-dimensional map information related to an embodiment of the present invention.

[0383] First, referring to FIG. 22, the AR service providing device (800) can display a 3D stopover map based on the location information of a second passenger related to a stopover addition request in response to a stopover request by a first passenger (S2210).

[0384] Here, the stopover request is performed through the first passenger's AR service providing device (800). However, it is preferable that the second passenger's location information be selected and provided through the second passenger's AR service providing device (800). Accordingly, in response to proximity to the stopover location input / set through the first passenger's AR service providing device (800), control can be granted to the second passenger's AR service providing device (800) through the first passenger's AR service providing device (800) or the service platform or vehicle. Here, granting control means granting limited control over operations related to the location of the stopover.

[0385] Accordingly, the AR service providing device (800) for the second passenger can render multiple AR images to the MR view map to designate the direction of the stopover location to the structure included in the 3D stopover map (S2220).

[0386] This is to provide accurate location information of the second passenger by using structures on the MR map view, assuming that the location of the second passenger does not change.

[0387] Specifically, referring to FIG. 23, in the MR view map displayed on the AR service providing device (800), highlight images (2302, 2303) may be displayed on surrounding structures based on the location information (CL 2) of the second passenger. The second passenger may select a highlight image corresponding to a structure closer to his / her current location among the highlight images (2302, 2303) to provide his / her current location more accurately.

[0388] For example, if a selection input is received for a highlight image (2302) displayed at the main entrance location of the building in FIG. 23, the AR service providing device (800) transmits information about this to the calling vehicle, and the initial waypoint location (2301) stored in the calling vehicle is changed to a location corresponding to the highlight image (2302).

[0389] Referring again to FIG. 22, the AR service providing device (800) can determine the direction of a structure corresponding to a selected AR image among a plurality of AR images highlighted on the MR view map as a waypoint location (S2230).

[0390] In some embodiments, the AR service providing device (800) may generate a thumbnail image based on the ID of a structure associated with the determined stopover location (S2240). However, this may be performed selectively at the request of the second passenger.

[0391] Next, the AR service provider (800) transmits the GPS coordinates of the determined stopover location to the calling vehicle (S2240). Accordingly, the vehicle can accurately recognize the stopover location where the second passenger is waiting and guide the second passenger to board. The operations related to guiding the second passenger to board can be applied in the same or similar manner as the embodiments described above with reference to FIGS. 11 to 16.

[0392] In some embodiments, the vehicle can recognize a street view of a structure corresponding to a thumbnail image based on GPS coordinate information transmitted from an AR service providing device (800).

[0393] Specifically, when a vehicle is driving toward a stopover location based on the transmitted GPS coordinate information, if a structure corresponding to a thumbnail image is recognized by the first passenger or the vehicle's AR service providing device (800), the vehicle can accurately drive toward the location of the second passenger from that location through a street view around the structure. When the vehicle approaches that location, the vehicle can consider the boarding difficulty attribute to determine the final stopover location and perform an action to guide the second passenger to board.

[0394] Meanwhile, when the addition of a stopover is completed, the stopover location set by the first passenger and the calculated expected arrival time information are transmitted as itinerary information to the AR service providing device (800) of the second passenger through the AR service providing device (800) of the first passenger.

[0395] In some embodiments, considering boarding difficulty attributes such as whether the transit location set by the first passenger is an unboardable area, an AR view or MR view that shows the changed transit boarding location can be provided to the AR service providing device (800) of the second passenger. In other words, if the location of the second passenger can be accurately set through the AR service providing device (800) of the first passenger, there is no need to grant control to the second passenger.

[0396] Conversely, if the second passenger is given control and the stopover location is accurately transmitted through the second passenger's AR service provision device (800), for example, in response to an input for the highlight image (2302) in FIG. 23, feature information of the highlighted structure may be transmitted to the vehicle. This means that, for example, since the first passenger and the vehicle do not know the external characteristics of the second passenger at the stopover location, they instead determine the exact boarding location based on the structures around the waiting street.

[0397] In FIG. 23, when the stopover location is finally determined from the initially input / set stopover location (2301) to the stopover location changed by the second passenger, that is, the structure location matching the highlight image (2302), a movement path (2304) from the first location (2301) to the second location (2302) can be displayed on the MR view map.

[0398] Meanwhile, if there are no structural buildings around the second passenger, the location of the stopover can be changed by transmitting the GPS-based coordinate information or directly input location information of the second passenger's AR service provision device (800) to the vehicle.

[0399] FIG. 24 is a drawing for explaining displaying a ride-sharing applicant using three-dimensional map information related to an embodiment of the present invention.

[0400] For example, in the case of a multi-passenger unmanned vehicle, a first thumbnail (2301) indicating the location of the first passenger who has boarded the vehicle on the MR view map and second and third thumbnails (2302, 2303) indicating the respective locations of the candidate passengers may be rendered so that the first passenger can select whether to ride with multiple candidate passengers who have supported ride-sharing.

[0401] At this time, the second and third thumbnails (2302, 2303) may display preference information (e.g., ratings, etc.) related to selecting one of the candidate passengers. Once a candidate passenger is selected based on the selection input for one of the second and third thumbnails (2302, 2303), the candidate passenger's location may be determined based on the candidate passenger's external characteristics recognized through the vehicle's external camera and the difficulty level of boarding considering the surrounding circumstances.

[0402] In some embodiments, if the first passenger approves the rideshare request, a carpet image (2301) representing the travel path to the location of the determined candidate passenger may be displayed on the MR view map.

[0403] Meanwhile, even if not shown, in the event of an emergency while the unmanned vehicle is in operation, an emergency action protocol that can quickly control the unmanned vehicle from the outside can be guided through augmented reality (AR) through the camera of the rescue team's AR service provision device.

[0404] In another embodiment, in response to a detection of an action such as opening a vehicle door due to an unexpected situation of a passenger while the called vehicle is in motion, a pull-over process may be executed to display the vehicle's stopping location on an MR view map, and then provide relevant information to the passenger by considering the difficulty of getting off.

[0405] Meanwhile, the above-described vehicle call service provision method may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to media directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instructions, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0406] As described above, according to the vehicle call service providing method, system, and AR service providing device therefor according to an embodiment of the present invention, remote interaction can be naturally achieved between an unmanned vehicle without a human driver and a passenger before boarding, and the convenience of authentication for boarding and changing of boarding and alighting locations is further enhanced by recognizing each other.

[0407] In addition, according to an embodiment of the present invention, not only can the vehicle called by the passenger be accurately identified, but the vehicle can also identify and authenticate the passenger in advance through a thumbnail selected by the passenger before boarding, thereby facilitating smooth interaction and authentication, and providing high-quality services such as accurately guiding passenger boarding in an unmanned vehicle.

[0408] In addition, according to an embodiment of the present invention, if the boarding or disembarking location promised to the passenger is not suitable as a boarding or disembarking location, AR and MR technology can be used to accurately inform the passenger of a suitable boarding or disembarking location, and allow the passenger to intuitively recognize the changing travel route.

[0409] In addition, according to some embodiments of the present invention, even if the first passenger designates a stopover, the second passenger can provide the stopover location, so that carpooling can be done naturally using AR and MR technology even in a situation where the first passenger does not know the second passenger.

[0410] Further scope of the applicability of the present invention will become apparent from the detailed description below. However, since various modifications and variations within the spirit and scope of the present invention will become apparent to those skilled in the art, it should be understood that the detailed description and specific examples, such as preferred embodiments of the present invention, are given by way of example only.

[0411] The features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present invention, and are not necessarily limited to just one embodiment. Furthermore, the features, structures, effects, etc. exemplified in each embodiment can be combined or modified in other embodiments by those skilled in the art to which the embodiments pertain. Therefore, the contents related to such combinations and modifications should be construed as falling within the scope of the present invention.

[0412] In addition, although the above description focuses on examples, these are merely examples and do not limit the present invention. Those skilled in the art to which the present invention pertains will appreciate that various modifications and applications not exemplified above are possible without departing from the essential characteristics of the present invention. For example, each component specifically shown in the examples can be modified and implemented. In addition, differences related to such modifications and applications should be interpreted as being included within the scope of the present invention defined in the appended claims.

Claims

1. A vehicle calling system including an AR service providing device and a service platform linked with the AR service providing device, The above AR service providing device, An AR image based on the location information and vehicle information of the called vehicle is rendered to the called vehicle and displayed on a 3D map. The above service platform, In response to the call vehicle entering the boarding area based on the passenger's location information, a request is made to the camera of the call vehicle to capture multiple candidate passengers and generate multiple thumbnails. Providing a user interface for selecting one of the plurality of thumbnails generated above to the AR service providing device, Characterized in that information about the thumbnail selected by the AR service providing device is transmitted to the calling vehicle to track the location of the candidate passenger. Vehicle call service provision system.

2. In paragraph 1, The above boarding zone is, The shooting range of the camera equipped in the above-mentioned calling vehicle and the shooting range of the camera equipped in the above-mentioned AR service providing device are within the distance range where the above-mentioned calling vehicle and the above-mentioned AR service providing device can each shoot each other. Vehicle call service provision system.

3. In paragraph 1, The above service platform, When a vehicle is called, additional information input from the AR service providing device is transmitted to the called vehicle, so that an area related to the additional information is extracted from an image captured through a camera of the called vehicle, and a plurality of thumbnails are generated including the extracted area. Vehicle call service provision system.

4. In paragraph 1, The AR service providing device receives a selection input for one of a plurality of thumbnails using the user interface, The above service platform, Transmitting the ID and feature points of the thumbnail corresponding to the above selection input to the calling vehicle, so that the calling vehicle recognizes a candidate passenger corresponding to the transmitted thumbnail ID, and tracks the location of the recognized candidate passenger based on the feature points. Vehicle call service provision system.

5. In paragraph 4, The above AR service providing device, In response to the above selection input, the first AR image based on the location information and vehicle information of the called vehicle is changed to a second AR image based on passenger customization information and rendered on the called vehicle. Vehicle call service provision system.

6. In paragraph 1, The above service platform, In response to a non-selectable input for multiple thumbnails, a re-shooting command is transmitted to the calling vehicle, and a thumbnail corresponding to the re-shooting command is provided to the AR service providing device. Vehicle call service provision system.

7. In paragraph 4, The above service platform In response to the stopping of the called vehicle, the called vehicle performs passenger authentication by comparing the first feature corresponding to the thumbnail corresponding to the selection input with the second feature obtained from a photographed image of a candidate passenger close to the door of the called vehicle. Vehicle call service provision system.

8. In paragraph 7, The above service platform, The above-mentioned calling vehicle performs an action related to guiding a passenger to board in response to determining that the similarity between the first feature point and the second feature point satisfies a criterion or higher based on the above-mentioned comparison. Vehicle call service provision system.

9. In paragraph 7, The above service platform, In response to the fact that the calling vehicle determines that the similarity between the first feature point and the second feature point is less than a reference value based on the comparison, the calling vehicle performs an action related to passenger mismatch and transmits vehicle arrival guidance information to the AR service providing device.

10. Step of obtaining location information of the called vehicle; A step of rendering an AR image based on the location information and vehicle information of the called vehicle to the called vehicle and displaying it on a 3D map; A step of determining that the called vehicle in which the AR image is rendered has entered the boarding area; A step of requesting to generate multiple thumbnails by taking pictures including multiple candidate passengers through the camera of the above-mentioned calling vehicle; A step of providing a user interface for selecting one of the plurality of thumbnails generated above to an AR service providing device; and A step of transmitting information about a thumbnail selected through the user interface to the calling vehicle to track the location of the candidate passenger, Method of providing vehicle call service.

11. In paragraph 10, The step of determining whether the above boarding zone has been entered is as follows: A step of determining that the shooting range of the camera equipped in the above-mentioned calling vehicle and the shooting range of the camera equipped in the above-mentioned AR service providing device satisfy the distance range in which the above-mentioned calling vehicle and the above-mentioned AR service providing device can each shoot each other. Method of providing vehicle call service.

12. In paragraph 10, The step of requesting to generate the above multiple thumbnails is: A step of transmitting additional information entered in the AR service providing device to the called vehicle when calling a vehicle; A step of requesting to extract an area related to the additional information from an image captured through a camera of the calling vehicle and to generate the plurality of thumbnails including the extracted area, Method of providing vehicle call service.

13. In paragraph 10, The step of tracking the location of the above candidate passenger is: A step of transmitting the ID and feature points of the thumbnail selected through the user interface to the calling vehicle; The step of recognizing a candidate passenger corresponding to the thumbnail ID transmitted by the above-mentioned calling vehicle and tracking the location of the recognized candidate passenger based on the above-mentioned feature point is included. Method of providing vehicle call service.

14. In paragraph 10, A step of receiving a non-selectable input for a plurality of thumbnails through the above user interface; A step of transmitting a re-shooting command to the above-mentioned calling vehicle and providing a thumbnail corresponding to the re-shooting command to the AR service providing device again, Method of providing vehicle call service.

15. In paragraph 10, After the step of tracking the location of the candidate passenger, A step of recognizing the stop of the called vehicle; and The method further includes a step of performing passenger authentication by comparing a first feature corresponding to the selected thumbnail of the above-mentioned calling vehicle with a second feature obtained from a photographed image of a candidate passenger close to the door of the calling vehicle. Method of providing vehicle call service.

Images