Method for identifying a user in a vehicle, vehicle implementing the method, and computer-readable medium

Abstract

The disclosure relates to a method for identifying a user in a vehicle, a vehicle implementing the method, and a computer readable medium. A method for identifying a user in a vehicle is executed by a memory storing instructions and a processor executing the instructions. The method includes: acquiring image data of a user located in a vehicle; acquiring route data according to a request of a user; generating image estimation user data based on the image data estimation; generating route estimation user data estimated based on the route data; and generating user identification information based on the image estimated user data and the route estimated user data.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of Korean Patent Application No. 10-2024-0137634, filed on October 10, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to a method for identifying vehicle users and a vehicle for implementing the method, and more specifically, to a method for identifying vehicle users and a vehicle that uses image and route estimation information to identify vehicle users in order to provide personalized features and services to vehicle users. Background Technology

[0004] Personalized features and services can be provided by collecting driving data for each vehicle and analyzing various usage patterns. That is, even if a vehicle is used by multiple passengers, existing personalized services can be provided by identifying trends in each vehicle.

[0005] However, vehicles are often used by multiple users belonging to a specific group (such as a family). Since driving characteristics differ for passengers of different ages and genders, it is difficult to estimate the driving characteristics of each individual in the development of personalized algorithms. Because the preferences of various passengers are mixed, personalized services may not accurately reflect the preferences of a particular customer. Furthermore, when passenger information constitutes similar passenger groups composed of similar age groups, genders, residences, etc., it is difficult to further reflect the preferences of individual passengers.

[0006] Because the vehicle usage characteristics of passengers cannot be accurately estimated, a generic service that is not suitable for actual passengers is provided, and there are limitations in achieving high-performance personalized services. Summary of the Invention

[0007] The present invention aims to provide a method for identifying vehicle users and a vehicle for accurately identifying vehicle users, so as to provide personalized functions and services to vehicle users.

[0008] The technical problems to be solved by this disclosure are not limited to those described above, and those skilled in the art to which this disclosure pertains will clearly understand from the following description other technical problems not mentioned.

[0009] According to the disclosure, a method for identifying a user in a vehicle includes the steps of: providing a memory configured to store at least one instruction and a processor configured to execute the at least one instruction stored in the memory; acquiring, by the processor, image data of a user located in the vehicle; acquiring, by the processor, route data according to a request of the user; generating, by the processor, image-estimated user data estimated based on the image data; generating, by the processor, route-estimated user data estimated based on the route data; and generating, by the processor, user identification information based on the image-estimated user data and the route-estimated user data.

[0010] According to an aspect, there is provided a method for identifying a user of a vehicle, the method comprising: acquiring image data of a user riding the vehicle; acquiring route data according to a request of the user; generating image-estimated user data based on the image data; generating route-estimated user data based on the route data; and generating user identification information based on the image-estimated user data and the route-estimated user data.

[0011] According to an embodiment of the disclosure, in the method, the image-estimated user data can be generated by an image-based deep learning model, and the image-based deep learning model uses image-based matching information depending on similarity between image data of a previous user of the vehicle and the image data of the user as the image-estimated user data.

[0012] According to an embodiment of the disclosure, in the method, the image-estimated user data can include at least one of previous user probability information depending on the image-based matching information and a user identified by image inference.

[0013] According to an embodiment of the disclosure, in the method, the image data of the user can be image data including a face of the user.

[0014] According to an embodiment of the disclosure, in the method, the route-estimated user data can be output by a driving trajectory-based deep learning model, and the driving trajectory-based deep learning model can use route-based matching information depending on similarity between driving data of a previous user of the vehicle and route data of the user as the route-estimated user data.

[0015] According to an embodiment of the disclosure, in the method, the route-estimated user data can include at least one of previous user probability information depending on the route-based matching information and a user identified by driving trajectory inference.

[0016] According to an embodiment of the disclosure, in the method, the driving data and the route data can include a destination, a midway stopover, a driving trajectory, a driving time zone, and a driving time.

[0017] According to an embodiment of the disclosure, in the method, a deep learning model based on a driving trajectory can be trained through contrastive learning using driving data previously generated by driving of a plurality of users.

[0018] According to an embodiment of the disclosure, in the method, generating the user identification information can include generating the user identification information by employing an integrated model based on deep learning that takes the image estimation user data and the route estimation user data as input.

[0019] According to an embodiment of the disclosure, in the method, the method can further include, before generating the user identification information, estimating setting estimation user data based on output of vehicle setting information estimated according to a request of the user. The integrated model can employ the setting estimation user data as additional input to generate the user identification information.

[0020] According to another embodiment of the disclosure, a vehicle for implementing identification of a user includes a camera configured to obtain an image of an object inside the vehicle, a memory configured to store at least one instruction, and a processor configured to execute the at least one instruction stored in the memory, wherein, upon execution of the at least one instruction, the processor is configured to: acquire image data of a user located in the vehicle; acquire route data according to a request of the user; generate image estimation user data estimated based on the image data; generate route estimation user data estimated based on the route data; and generate user identification information based on the image estimation user data and the route estimation user data.

[0021] According to another aspect, there is provided a vehicle for implementing identification of a user of the vehicle, the vehicle including a camera obtaining an image of an object inside the vehicle, a memory storing at least one instruction, and a processor executing the at least one instruction stored in the memory. The processor is configured to: acquire image data of a user riding the vehicle; acquire route data according to a request of the user; generate image estimation user data estimated based on the image data; generate route estimation user data estimated based on the route data; and generate user identification information based on the image estimation user data and the route estimation user data.

[0022] According to the disclosure, a non-transitory computer readable medium containing program instructions executed by a processor includes program instructions to acquire image data of a user located in a vehicle, program instructions to acquire route data according to a request of the user, program instructions to generate image estimation user data estimated based on the image data, program instructions to generate route estimation user data estimated based on the route data, and program instructions to generate user identification information based on the image estimation user data and the route estimation user data.

[0023] The features briefly summarized above for this disclosure are merely exemplary aspects of the detailed description of this disclosure that follows, and are not intended to limit the scope of this disclosure.

[0024] The technical problems solved by this disclosure are not limited to those described above, and other technical problems not described herein will be clearly understood by those skilled in the art to which this disclosure pertains (hereinafter referred to as ordinary technicians) from the following description. Attached Figure Description

[0025] Figure 1 This is a diagram illustrating a vehicle communicating with another device to send and receive data.

[0026] Figure 2 This is a diagram illustrating the modules constituting a vehicle according to an embodiment of the present disclosure.

[0027] Figure 3 This is a diagram illustrating the modules constituting a server according to the present disclosure.

[0028] Figure 4 This is a diagram illustrating the user identification model.

[0029] Figure 5 This is a diagram illustrating the construction and inference of a model based on driving trajectory embedding.

[0030] Figure 6 This is a flowchart of a method for identifying vehicle users according to another embodiment of the present disclosure. Detailed Implementation

[0031] It should be understood that, as used herein, the terms “vehicle” or “of a vehicle” or other similar terms include motor vehicles in a broad sense, such as passenger cars, including sport utility vehicles (SUVs), buses, trucks, various commercial vehicles, vessels (including various boats and ships), aircraft, etc., and include hybrid vehicles, electric vehicles, plug-in hybrid electric vehicles, hydrogen-powered vehicles, and other alternative fuel vehicles (e.g., fuels derived from resources other than petroleum). As mentioned herein, a hybrid vehicle is a vehicle with two or more power sources, such as a gasoline-powered and an electric vehicle.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that when the terms “comprises” and / or “comprising” are used in this specification, they specify the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Throughout the specification, unless explicitly stated otherwise, the word “comprise” and variations such as “comprises” or “comprising” will be understood to imply the inclusion of the stated elements, but do not exclude any other elements. Furthermore, the terms “unit,” “device (-er),” “machine (-or),” and “module” described in the specification mean a unit for performing at least one function and operation, and can be implemented by hardware components or software components and combinations thereof.

[0033] Furthermore, the control logic of this disclosure can be embodied on a non-transitory computer-readable medium containing executable program instructions that are executed by a processor, controller, etc. Examples of computer-readable media include, but are not limited to, ROM, RAM, optical disc (CD)-ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. The computer-readable medium can also be distributed across a network-coupled computer system, such that the computer-readable medium is distributedly stored and executed by, for example, a telematics server or a controller area network (CAN).

[0034] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, enabling those skilled in the art to readily implement the present disclosure. However, the present disclosure can be implemented in various different ways and is not limited to the embodiments described herein.

[0035] In describing exemplary embodiments of this disclosure, well-known functions or structures will not be described in detail, as they may unnecessarily obscure the understanding of this disclosure. In the drawings, the same constituent elements are indicated by the same reference numerals, and repeated descriptions of the same elements will be omitted.

[0036] In this disclosure, when an element is simply referred to as “connected to,” “coupled to,” or “linked to” another element, it may mean that the element is “directly connected to,” “directly coupled to,” or “directly linked to” another element, or is connected to, coupled to, or linked to another element with other elements in between.

[0037] In this disclosure, unless specifically mentioned otherwise, the terms first, second, etc., are used only to distinguish one element from another and do not limit the order or importance of the elements. Thus, a first element in one embodiment may be referred to as a second element in another embodiment, and similarly, a second element in one embodiment may be referred to as a first element in another embodiment, without departing from the scope of this disclosure.

[0038] In this disclosure, elements are distinguished from each other for the purpose of clearly describing each feature, and do not necessarily mean that the elements are separate. That is, multiple elements may be integrated into a single hardware or software unit, or a single element may be distributed and formed in multiple hardware or software units. Therefore, such integrated or distributed implementations are included within the scope of this disclosure, unless otherwise stated.

[0039] In this disclosure, the elements described in the various embodiments are not necessarily essential elements, and some of them may be optional elements. Therefore, embodiments comprising a subset of the elements described in the embodiments are also included within the scope of this disclosure. Furthermore, embodiments including elements other than those described in the various embodiments are also included within the scope of this disclosure.

[0040] The advantages and features of this disclosure, as well as the ways in which they are implemented, will become apparent from the following detailed description of the embodiments, taken in conjunction with the accompanying drawings. However, embodiments may be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these embodiments are provided to complete this disclosure and to fully convey its scope to those skilled in the art.

[0041] In this disclosure, each phrase such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and each phrase such as “at least one of A, B or C” and “at least one of A, B, C or a combination thereof” can be included in any one or all possible combinations of the items listed together in the corresponding phrase.

[0042] In this disclosure, for ease of explanation, expressions of positional relationships such as “up,” “down,” “left,” and “right” are used as in this specification, and the positional relationships described in this specification can be understood in reverse if the accompanying drawings shown in this specification are reversed.

[0043] In the following text, reference will be made to Figure 1 and Figure 2 Describes a method for identifying vehicle users and a vehicle for implementing that method. Figure 1This is a diagram illustrating a vehicle communicating with another device to send and receive data.

[0044] refer to Figure 1 The vehicle 100 can be powered by either electricity or gaseous fuel. In the case of electricity, the vehicle 100 can, for example, be a purely rechargeable battery-powered vehicle powered solely by a high-voltage rechargeable battery or a gaseous fuel cell. Furthermore, the fuel cell can utilize various forms of gas capable of generating electricity, and for example, the gaseous fuel can be filled into the vehicle 100 in a liquefied state. Here, the gaseous fuel can be, for example, hydrogen. However, various gaseous fuels can be used, and this is not a limitation. In the case of fossil fuels, the vehicle 100 can be equipped with an internal combustion engine that is fuel-based (such as gasoline, diesel, or liquefied petroleum gas) and drives the actuator 114 through fuel combustion. As another example, the vehicle 100 can selectively utilize the energy of a fossil fuel-based internal combustion engine and an electric rechargeable battery to drive the actuator 114, which can be a hybrid vehicle.

[0045] Vehicle 100 can refer to a mobile device. Vehicle 100 is a ground vehicle that drives on the ground and can be a traditional passenger vehicle, commercial vehicle, or private vehicle (PBV). Vehicle 100 can be a four-wheeled vehicle, such as a bus, SUV, or light truck, or a vehicle with more than four wheels, such as a bus, large truck, container truck, or heavy vehicle. Vehicle 100 can also be a robot in the broadest sense, such as a mobile device, and the robot can move using wheels, tracks, or other motion modules. Vehicle 100 can be controlled and driven by autonomous driving, and this autonomous driving can be semi-autonomous or fully autonomous.

[0046] Simultaneously, vehicle 100 can communicate with other devices 200, 300, or other vehicles 400. Other devices may include, for example, a server 200 supporting various controls, status management, and driving functions of vehicle 100; an ITS device 300 for receiving information from an Intelligent Transportation System (ITS); and various types of user devices. Server 200 is, for example, an external device operated by the vehicle manufacturer or provided to provide various functions and services of vehicle 100, and can receive connection data from vehicle 100 and send data necessary for requests from vehicle 100. In response to requests and data sent from vehicle 100 and user devices, server 200 can send various information and software modules for controlling vehicle 100 to support driving and various services of vehicle 100.

[0047] The ITS device 300 is, for example, a roadside base station (Roadside Unit; RSU), and the ITS device 300 can exchange vehicle identification data, driving control and status data, and environmental data around the vehicle with vehicle 100 via V2I to help users drive their own vehicles or support the autonomous driving of vehicle 100. Vehicle 100 can exchange the above data with other vehicles 400 via V2V to support manual or autonomous driving.

[0048] Vehicle 100 can communicate with other vehicles or other devices based on cellular communication, in-vehicle wireless access (WAVE) communication, dedicated short-range communication (DSRC) or near-field communication, or other communication schemes.

[0049] For example, vehicle 100 can use a cellular communication network such as LTE or 5G, WiFi, or WAVE for communication with server 200, ITS device 300, and other vehicles 400. As another example, DSRC or similar technologies used in vehicle 100 can be used for communication between vehicles. The communication methods between vehicle 100, server 200, ITS device 300, other vehicles 400, and user devices are not limited to the aforementioned embodiments.

[0050] Figure 2 This is a diagram illustrating the modules constituting a vehicle according to an embodiment of the present disclosure.

[0051] The vehicle 100 may include a sensor unit 102, an operation unit 104, a display 106, an embedded device 108, and a transceiver 110.

[0052] Sensor unit 102 may include various types of detectors for sensing various states and conditions occurring in the external environment of vehicle 100, objects within vehicle 100, internal systems, user operation, and passenger space. Sensor unit 102 may include an internally guided camera 102a for acquiring images associated with objects within vehicle 100. Furthermore, sensor unit 102 includes a positioning sensor 102b for confirming the position of vehicle 100, and the positioning sensor 102a may be, for example, a GPS sensor or a GNSS sensor. Sensor unit 102 may have an environmental sensor 102c for identifying dynamic and static objects present outside vehicle 100 to obtain the behavior of those objects. Environmental sensor 102c may include at least one of an externally guided image sensor, a lidar sensor, a radar sensor, and an ultrasonic sensor. Sensor unit 102 may include wheel sensors, wheel steering sensors, posture sensors, etc., for checking longitudinal speed, lateral behavior, driving posture, etc. Sensor unit 102 may also include sensing modules for sensing various conditions not listed herein.

[0053] The operating unit 104 can be configured as a module for user navigation for driving. For example, the operating unit 104 may be a steering wheel for manual driving, an automatic or manual transmission actuator, an accelerator pedal, a brake pedal, a transmission, etc. The operating unit 104 may also include an interface for using, releasing, and selecting detailed functions of an automated driving mode requested by the user, enabling the user to use the automated driving function.

[0054] Display 106 can function as a user interface. Display 106 can indicate the operating status, control status, route / traffic information, remaining energy information, and driver requests, etc., output by processor 118 to the vehicle 100. Furthermore, display 106 can be configured as a touchscreen capable of detecting driver input to receive requests from the driver instructing processor 118.

[0055] Embedded device 108 may be a device installed inside vehicle 100 to provide functions or conveniences essential to the use of vehicle 100. Embedded device 108 is a type of non-drive electrical device other than a drive power system (such as a wheel drive unit) and may be an auxiliary device that is powered by power supply unit 112. For example, embedded device 108 includes, but is not limited to, passenger seating devices, navigation devices, in-cabin communication devices, air conditioning systems, lighting systems, content devices that provide video / audio content, autonomous driving assistance devices, and regenerative braking devices.

[0056] To facilitate user operation, the embedded device 108 may have a setting component that provides the user with settings associated with different controls and states of the device. The setting component can be configured as a hardware interface or a software interface. For example, the hardware interface can be configured as a button, a scroll wheel, a lever, etc. The software interface can be, for example, a graphical user interface displayed on a touchscreen display 106, a voice interface based on voice command recognition, etc. The device setting information specified by the user through the setting component can be stored and managed in the memory 116. Furthermore, the setting information can be managed individually for each user. In this disclosure, the setting information may be referred to as vehicle setting information.

[0057] As an example of device configuration, the seating device may have a seat setting module for adjusting the seat position to enhance user comfort. The seat setting module can be configured to adjust the seat position of at least one of the seats in the room. The seat cushion setting module may be installed near each seat cushion, or it may be supported by a software interface. The seat position may include at least one of the following: seat height, seat position relative to the front of the vehicle, seat back angle, left seat angle, and headrest height and angle.

[0058] The seat position selected by the user through the seat setting module can be stored as vehicle setting information via the user's storage request. Furthermore, multiple vehicle setting information entries related to the seat position can be managed through a function that stores information for each user identifier. The user can, for example, select a user profile associated with the identifier displayed on the display 106, or press a dedicated seat setting key / button. Therefore, the current seat setting can be maintained even if the user who is already driving the vehicle 100 does not perform the above operations. The processor 118 can consider using the currently set vehicle setting information even if there is no setting change. On the other hand, the user can automatically change the seat setting to a user-preset seat position through operation without any initial complex operation. When the above settings change, the processor 118 can recognize that the vehicle setting information is used as the changed setting.

[0059] The air conditioning system may include a heating device, a cooling device, and a ventilation device. For the settings of the air conditioning system, the user can input air conditioning settings related to the temperature, intensity, and ventilation conditions of the air conditioning system using an air conditioning panel with buttons, a scroll wheel, or a touch-screen display 106. As described above, the air conditioning settings for each user are stored in the memory 116 and can be managed as vehicle setting information. The operation and manipulation of the air conditioning settings can be described in a manner similar to seat settings.

[0060] The indoor communication device can be, for example, a wired or wireless communication setup for applications and audio using user devices in vehicle 100. Users can use the interface provided by display 106 to input Bluetooth configuration profiles and indoor communication settings related to wired and wireless connectivity options. As described above, indoor communication settings are stored in memory 116 for each user and can be managed as vehicle settings information. The operation and manipulation of the indoor communication settings can be described in a manner similar to seat settings.

[0061] The automated driving assistance system can operate using detailed automatic options input by the user. The regenerative braking system can operate, for example, at the regenerative braking level requested by the user. The aforementioned automatic option settings and regenerative braking settings can be processed through corresponding interfaces, stored in memory 116 for each user, and managed as vehicle setting information.

[0062] Embedded device 108 may include a navigation application (or device) executing on display 106. In this disclosure, the navigation application may be abbreviated as navigation. Navigation may receive a route request for a user's destination, send a request to server 200, and receive and provide route data and additional information from server 200. Route data may be driving data related to the driving trajectory on the route, expected travel time, and traffic information.

[0063] Additional information may include, for example, the passenger's intended destination based on travel time, the travel route to the intended destination, multiple recommended travel routes to the destination, and services related to the destination and stopovers. The travel route consists of GPS location tracks tracked on a map, and these GPS tracks can be processed into image format. Stopovers may be points of interest where the user previously stopped on the intended route, or points of interest based on user preferences estimated from accumulated driving patterns. Stopovers are, for example, but not limited to, rest areas along the highway, restaurants, shopping facilities, or cultural facilities. Services may include detailed information on services offered by the destination or stopovers, expected congestion, operating hours, etc.

[0064] exist Figure 2 In this disclosure, the embedded device 108 according to the present embodiment is mainly described. Since the embedded device 108 is a type of load device that consumes energy other than the actuation unit 114, the load device may also include various devices other than the embedded device 108.

[0065] Transceiver 110 can support communication with server 200, ITS device 300, surrounding vehicles 400, etc. Transceiver 110 may include modules for processing, for example, cellular communication, WAVE, DSRC communication, etc. Transceiver 110 can support communication with electronic devices carried by passengers inside vehicle 100.

[0066] The vehicle 100 may also include a power supply unit 112 and an actuation unit 114.

[0067] Power unit 112 can generate and supply power and electricity for drive and non-drive power systems such as actuation unit 114. For example, the non-drive power system may be sensor unit 102, operation unit 104, display 106, embedded device 108, transceiver 110, etc. When vehicle 100 is electrically driven, power unit 112 may be, for example, composed of an externally charged rechargeable battery, or a combination of an electric rechargeable battery and a fuel cell for charging the rechargeable battery. When vehicle 100 is fossil fuel driven, power unit 112 may be configured as an internal combustion engine. Furthermore, when vehicle 100 is a hybrid type, power unit 112 may be provided by a combination of an internal combustion engine and an electric rechargeable battery.

[0068] The actuation unit 114 includes at least one module for implementing driving operations and can perform at least one driving operation, such as longitudinal control for acceleration and deceleration and lateral control for steering, according to a user request from the operation unit 104. For this purpose, the actuation unit 114 may include multiple wheels, a drive force generation module for generating and providing driving force to the wheels or transmitting driving force, a braking module for decelerating the driving of the wheels, a steering module for implementing lateral control of the wheels, etc. When the vehicle 100 is driven by electric power, the drive force generation module is configured as a motor assembly, and the braking module may also have regenerative braking functionality.

[0069] The vehicle 100 may also include a memory 116 and a processor 118.

[0070] Memory 116 can store applications and various data used to control vehicle 100, allowing applications to be loaded, data to be read, or data to be recorded upon request by processor 118. In this disclosure, memory 116 can hold a software module for identifying users in the vehicle. This module may include a deep learning-based user identification model. Memory 116 can also hold task software modules for processing personalized functions or services for the identified user.

[0071] In the context of this disclosure, memory 116 may store image data of multiple users who have previously driven vehicle 100, driving data categorized by previous users, and vehicle setting information. Image data consists of images identifiable by user biometrics and may include, for example, the user's face. Furthermore, the image data may contain feature information from each user's image for use in a user identification model. Driving data may include, for example, destination, stopover points, driving trajectory, travel time zone, and travel time. Additionally, the driving data may contain feature information from each user's driving trajectory for use in a user identification model. Vehicle setting information may include detailed settings for the embedded device 108 and feature information based on these detailed settings.

[0072] Processor 118 can perform overall control of vehicle 100. Processor 118 can be configured to execute applications and instructions stored in memory 116. Processor 118 can execute applications that identify the vehicle user to handle user requests.

[0073] In the context of this disclosure, processor 118 can execute a user identification model in response to driving by at least one user. Processor 118 can perform processes such as acquiring image data of the user driving vehicle 100 using camera 102a and acquiring route data upon the user's request. Processor 118 can perform processes such as generating image-estimated user data based on the image data and generating route-estimated user data based on the route data. Furthermore, processor 118 can implement processes for generating user identification information based at least on the image-estimated user data and the route-estimated user data. Additionally, when the vehicle setting information of a previous user is stored in memory 116, processor 118 can generate setting-estimated user data based on the vehicle setting information and further use the generated data to process user identification information.

[0074] Here, the image estimation user data can be user identification based on the user's image data and previous users' image data. Furthermore, the image estimation user data can include various auxiliary data for user identification. For example, the image estimation user data can have probability information for identifying the user. The probability information can include matching information generated for multiple previous user categories based on the image data of the user and previous users. For example, the matching information can be probability vector data for each user category.

[0075] Route estimation user data can be user-identified based on a user's route data and driving data from previous users. Route data can be the user's driving trajectory from a navigation request to a destination or a driving trajectory estimated from a passenger's driving data. Driving data can be grouped and managed for each previous user. Furthermore, route estimation user data can include various auxiliary data for user identification. For example, route estimation user data can have probabilistic information for identifying users. Probabilistic information can include matching information generated for multiple previous user categories based on route data and driving data. For example, matching information can be probability vector data for each user category.

[0076] Setting up estimated user data can include users identified based on vehicle setting information from both the passenger and previous users, as well as auxiliary data used for identification. For example, similar to other estimated user data, setting up estimated user data can have probabilistic information for identifying users. This probabilistic information can include matching information generated for multiple previous user categories based on vehicle setting information from both the passenger and previous users. For example, the matching information could be probability vector data for each user category.

[0077] Processor 118 can process personalized functions or services for the identified user with the help of task software modules and server 200. For example, processor 118 can receive a route request input by a user of vehicle 100, send the route request to server 200, and provide the user with a response to the request from server 200. This response can be transmitted to the identified user to include customized route data and additional information.

[0078] Figure 3 This is a diagram illustrating the modules constituting a server according to the present disclosure.

[0079] As described above, server 200 can respond to requests from vehicle 100 and handle various personalized service functions for passengers. Furthermore, as an example, server 200 can train and distribute a user identification model to vehicle 100. In another example, server 200 may have a built-in trained user identification model and transmit information about passengers identified in vehicle 100 to vehicle 100 by receiving identification-related data received from vehicle 100. In this disclosure, for ease of description, a user identification model trained from server 200 is described for vehicle 100. However, embodiments of this disclosure can be applied to other examples without technical inconsistencies.

[0080] Server 200 may include communication unit 202, storage unit 204 and controller 206.

[0081] The communication unit 202 can send data to and receive data from external devices, support mutual communication with the vehicle 100 in this disclosure, and exchange data with the vehicle 100.

[0082] Storage unit 204 stores applications and various data used to operate server 200, and can load applications or read and record data according to requests from controller 206. In this disclosure, storage unit 204 can store software modules used to process route requests received from vehicle 100, such as navigation applications. Storage unit 204 can manage map information used to process route requests, vehicle driving data, demand data for congestion estimation, station information related to points of interest, and supplementary information for listing and recommending multiple points of interest.

[0083] Controller 206 can perform overall control of server 200. Server 200 can be configured to execute applications and instructions stored in storage unit 204. Controller 206 can execute navigation applications to process and respond to user requests sent from vehicle 100.

[0084] In conjunction with this disclosure, processor 118 can train a user recognition model. Figure 4This is a diagram illustrating a user identification model. The user identification model can be constructed by including an image-based deep learning model, a driving trajectory-based deep learning model, and a deep learning model based on vehicle setting information. Furthermore, the user identification model can be constructed using the output data of combined models to generate an ensemble model of the final user identification information. In this disclosure, when vehicle setting information is not collected from each vehicle 100, a deep learning model based on vehicle setting information may not be constructed.

[0085] Image-based deep learning models can Figure 4 The diagram shows a biometric-based user estimation model. The model can be trained by using image data of users collected from each vehicle 100 and identification information of each user as input and output learning data, respectively. Here, image data can be shown as including faces.

[0086] Biometric-based user estimation models can utilize convolutional neural network (CNN) algorithms. For example, the user estimation model can leverage the YOLO (One-Look-Only) family of models. Controller 206 can train the model to cluster image data associated with the faces of multiple previous users to generate latent features based on face recognition, and define user categories for the number of users. Furthermore, the biometric-based user estimation model can be trained to output image-based matching information as image-estimated user data based on the similarity between image data of previous users of vehicle 100 and image data of users riding in the vehicle. Image-estimated user data can be at least one of previous user probability information dependent on image-based matching information and users identified through image inference. The probability information can include, for example, probability vector data for each user.

[0087] Deep learning models based on driving trajectories can... Figure 4 The figure shows a user estimation model based on driving trajectory embedding. Figure 5 This is a diagram illustrating the construction and inference of a model based on driving trajectory embedding.

[0088] By using the accumulated driving data of previous users from each of the 100 vehicles and the identification information of each user as input and output learning data, respectively, a user estimation model based on driving trajectory can be trained. Here, driving data may include destination, intermediate stops, driving trajectory, driving time zone, driving time, and whether the vehicle is traveling along the navigation route.

[0089] The user estimation model based on driving trajectory can use CNN or recurrent neural network (RNN) algorithms. Controller 206 can train the model to embed driving data including at least one of driving trajectory, destination, stopover points, points of interest, and time region to output latent features, and assign user categories to driving data with similar features. The number of user categories can be as many as the number of previous users of vehicle 100. Furthermore, controller 206 can distinguish between feature-extractable and non-extractable driving trajectories, and select user-specified driving trajectories from the feature-extractable trajectories. Controller 206 can construct the model based on the selected driving trajectory to output route estimation user data. In addition to outputting user identification information, controller 206 can list multiple pieces of information related to each user's previous driving. For example, a list of details related to each user's time region driving data can be generated. This list can be used to generate additional information in response to a user's route request.

[0090] A user estimation model based on driving trajectory can be trained to output route-based matching information as route estimation user data, based on the similarity between the driving data of previous users and the route data of passengers. The route estimation user data can be at least one of previous user probability information based on route-based matching information and users inferred from driving trajectories. The probability information may include, for example, probability vector data for each user.

[0091] Furthermore, a deep learning model based on driving trajectories can be trained through contrastive learning using pre-prepared driving data from multiple users. For example, controller 206 can train the model according to predetermined criteria to increase the probability of similar driving data pairs and decrease the probability of dissimilar driving data pairs. In this way, the model can output a probability vector for each user with high accuracy.

[0092] Deep learning models based on vehicle configuration information can... Figure 4 The image shows a user estimation model based on vehicle settings.

[0093] When collecting vehicle settings from multiple users across vehicle 100, a user estimation model based on vehicle settings can be trained by using the vehicle settings information of previous users and the identification information of each user as input and output learning data, respectively. Since user profile selection readily generates vehicle settings information, this information can be collected as auxiliary data for identifying passengers.

[0094] As an example, a user estimation model based on vehicle settings can use a CNN or RNN algorithm. As another example, besides a deep learning model, a model can be constructed based on tables of mutually matched users and settings requested by the users. In this disclosure, the model is illustrated as using a deep learning model.

[0095] Controller 206 can train a vehicle-setting-based user estimation model in a manner similar to other user estimation models. Therefore, the vehicle-setting-based user estimation model can output setting-based matching information, which depends on the similarity between vehicle setting information of previous users and vehicle setting information of passengers, as setting-estimation user data. Setting-estimation user data can be at least one of previous user probability information depending on the matching information based on vehicle setting information and users identified using vehicle setting information. The probability information can include, for example, probability vector data for each user. Furthermore, the setting-estimation user data (e.g., categories of multiple users) can be matched with the outputs of biometric-based user estimation models and driving trajectory embedding-based user estimation models. Therefore, the reliability of the user identification model can be improved.

[0096] The controller 206 can train an ensemble model to combine data from the outputs of each user estimation model (i.e., image estimation user data, route estimation user data, and setup estimation data) to generate final user identification information. The ensemble model can be constructed as an algorithm, for example, based on weighted analysis or a majority-voting algorithm based on the output data of each model. A weighted analysis-based algorithm can identify a user as a passenger, for example, by applying parameters learned in the ensemble model (e.g., weighted parameters) to the user or user-specific probability vector data inferred from the corresponding user estimation data. A majority-voting algorithm can identify a user as a passenger by applying a majority vote to the user or probability information inferred from each user estimation data. Even if the setup user estimation data outputs a high-probability vector for a particular user, the ensemble model can also identify other users as passengers based on other user estimation data and parameters that have high-probability vectors for other users.

[0097] Server 200 can be used Figure 4 The loss function shown is used to train the learnable parameters of each model individually. As another example, server 200 can use a loss function applied to the entire user recognition model to train the model's parameters holistically.

[0098] In this disclosure, controller 206 is shown as consisting of a single processing module. In another example, controller 206 may be distributed across multiple processing modules, and the aforementioned processing may be performed by a distributed processing model.

[0099] In the following text, reference will be made to Figures 1 to 6The vehicle 100 is described in detail regarding user identification processing and subsequent task processing using a user identification model distributed from server 200.

[0100] Figure 6 This is a flowchart of a method for identifying a vehicle user according to another embodiment of this disclosure. In the following description, for ease of description, vehicle setting information is illustrated by example using seat setting information related to seat position settings; however, other vehicle setting information mentioned above may also be used as content described later. Furthermore, although... Figure 6 The processing is mainly performed by the processor 118, but for ease of description, the processor 118 and the vehicle 100 can be described together.

[0101] First, the processor 118 of the vehicle 100 can activate the camera 102a in response to at least one user sitting and starting the vehicle, to acquire image data of the sitting user (S105). Here, the image data can be acquired to include the user's face.

[0102] Next, the processor 118 of vehicle 100 can generate route data in response to receiving a route request from a user received by the navigation application (S110). The route request can be the user's desired destination, intermediate stops, or keywords of interest that are not specific to the destination. Keywords of interest can be text / voice messages that include the user's driving intentions. For example, keywords of interest could be nearby restaurants, cinemas, shopping malls, rest areas during driving, etc. Route data including the destination, intermediate stops, driving trajectory, driving time zone, and driving time can be generated. In another example, even without a direct request for navigation, a route request can be triggered based on the assumed expected route driven by the user.

[0103] Subsequently, the processor 118 can check whether the vehicle setting information of the embedded device 108 exists and determine whether the passenger will use the vehicle setting information (S115). For example, when the passenger selects his or her profile to change the seat setting information, or when the passenger detects that he or she will use the existing seat position according to the information without changing the seat setting information, the processor 118 can determine that the passenger will use the vehicle setting information.

[0104] When vehicle setting information exists and is used, processor 118 can use a user estimation model based on vehicle settings, which takes the vehicle setting information as input, to generate setting estimation user data (S120). Figure 3 Place Figure 5 The model and estimated user data are described in detail.

[0105] When vehicle setting information is unavailable, or when vehicle setting information is not used, processor 118 may omit generating setting estimation user data. Not using vehicle setting information may include, for example, the passenger inputting vehicle setting information that exceeds a range similar to previously stored vehicle setting information, or initializing vehicle setting information. In the following description, for ease of description, examples of generating setting estimation user data will be primarily described. When no data is generated, processor 118 may use an integrated model that takes other estimated user data besides the setting estimation user data as input to generate user identification information associated with the passenger.

[0106] Next, the processor 118 can use a biometric-based user estimation model to generate image estimation user data, which takes image data of the passenger and previous users as input (S125).

[0107] Biometric-based user estimation models can use image-based matching information, which depends on the similarity between image data of previous users and image data of passengers, as image-estimated user data. Specifically, image-estimated user data can include at least one of previous user probability information depending on image-based matching information and users identified through image inference. Details are in... Figures 3 to 5 Detailed description is provided.

[0108] The processor 118 can then use a user estimation model based on driving trajectory embedding to generate route estimation user data (S130), which takes the route data of the passenger user and the driving data of the previous user as input.

[0109] User estimation models based on driving trajectory embedding can use route-based matching information, which depends on the similarity between driving data and route data, as route estimation user data. Similar to route data, driving data can include destination, intermediate stops, driving trajectory, driving time zone, driving time, and the route taken according to navigation. Route estimation user data can include at least one of the following: previous user probability information depending on route-based matching information and users inferred from driving trajectories. Details are in... Figures 3 to 5 Detailed description is provided.

[0110] Next, the processor 118 can use the integrated model to generate user identification information based on image-estimated user data, route-estimated user data, and setting-estimated user data, and personalize and process the task of providing user identification information to the identified passenger (S135).

[0111] User identification information may include, for example, identifiers or categories of passengers. When data for multiple passengers is acquired in steps S105 to S115, user identification information can be output for each user. Tasks may include, for example, collecting travel trajectories to destinations, recommended features along the route, detailed travel data for passengers, and driving patterns. Specifically, based on a list including information related to the previous driving of passengers and similar users, travel trajectories and recommended features / presentations can be processed for personalization. Passenger-specific detailed travel data or driving patterns can also be used for other personalized services of the vehicle.

[0112] According to this disclosure, a method for identifying vehicle users and a vehicle for accurately identifying users can be provided to provide personalized functions and services to vehicle users.

[0113] Furthermore, by accurately identifying users, driving data accumulated from vehicle operations can be analyzed precisely for each user, making it easy to identify each user's driving tendencies.

[0114] Furthermore, by accurately identifying users, it is possible to more accurately predict and recommend users' desired destinations, route preferences, and points of interest.

[0115] The effects that can be obtained from this disclosure are not limited to those described above, and other effects not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains based on the following description.

[0116] Although the exemplary methods described above are presented as a series of operations for clarity of description, they are not intended to limit the order in which the steps are performed, and the steps may be performed simultaneously or in different orders as needed. To implement the method according to the invention, the described steps may also include other steps, including remaining steps in addition to some steps, or additional steps in addition to some steps.

[0117] The various embodiments of this disclosure are not a list of all possible combinations and are intended to describe representative aspects of this disclosure. The contents described in the various embodiments may be applied independently or in combination of two or more.

[0118] Furthermore, various embodiments of this disclosure can be implemented in hardware, firmware, software, or a combination thereof. When this disclosure is implemented in hardware, it can be implemented using application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, etc.

[0119] The scope of this disclosure includes software or machine-executable commands (e.g., operating systems, applications, firmware, programs, etc.) for enabling operation of methods according to various embodiments to be executed on a device or computer, and a non-transitory computer-readable medium having such software or commands stored thereon and executable on a device or computer.

Claims

1. A method for identifying a user in a vehicle, the method comprising: A memory is configured to store at least one instruction, and a processor is configured to execute the at least one instruction stored in the memory; The processor acquires image data of the user located in the vehicle; The processor obtains route data based on the user's request; The processor generates image estimation user data based on the image data. The processor generates route estimation user data based on the route data. as well as The processor generates user identification information based on the image-estimated user data and the route-estimated user data.

2. The method according to claim 1, wherein, The image-estimated user data is generated by an image-based deep learning model, and the image-based deep learning model uses image-based matching information that depends on the similarity between image data of the vehicle's previous users and the image data of the user as the image-estimated user data.

3. The method according to claim 2, wherein, The image-estimated user data includes at least one of prior user probability information dependent on the image-based matching information and users identified through image inference.

4. The method according to claim 1, wherein, The image data of the user includes the user's face.

5. The method according to claim 1, wherein, The route estimation user data is output through a deep learning model based on driving trajectory, and the deep learning model based on driving trajectory uses route-based matching information that depends on the similarity between the driving data of the vehicle's previous users and the user's route data as the route estimation user data.

6. The method according to claim 5, wherein, The route estimation user data includes at least one of prior user probability information dependent on the route-based matching information and users identified through inference from driving trajectories.

7. The method according to claim 5, wherein, The driving data and the route data include destination, intermediate stops, driving trajectory, driving time zone, and driving time.

8. The method according to claim 5, wherein, The deep learning model based on driving trajectory is trained through comparative learning using driving data previously generated by multiple users.

9. The method according to claim 1, wherein, Generating the user identification information includes generating the user identification information by using a deep learning-based ensemble model that takes the image-estimated user data and the route-estimated user data as input.

10. The method of claim 9, further comprising: Before generating the user identification information, according to the user's request, output the setting estimation user data based on the vehicle setting information; The integrated model uses the estimated user data as additional input to generate the user identification information.

11. A vehicle for identifying a user, the vehicle comprising: A camera is configured to acquire images of objects inside the vehicle; The memory is configured to store at least one instruction; as well as A processor is configured to execute the at least one instruction stored in the memory, wherein, when executing the at least one instruction, the processor is configured to: Acquire image data of the user located in the vehicle; Obtain route data according to the user's request; Generate image estimation user data based on the image data; Generate route estimation user data based on the route data; and User identification information is generated based on the image-estimated user data and the route-estimated user data.

12. The vehicle according to claim 11, wherein, The image-estimated user data is generated by an image-based deep learning model, and the image-based deep learning model uses image-based matching information that depends on the similarity between image data of the vehicle's previous users and the image data of the user as the image-estimated user data.

13. The vehicle according to claim 12, wherein, The image-estimated user data includes at least one of prior user probability information dependent on the image-based matching information and users identified through image inference.

14. The vehicle according to claim 11, wherein, The image data of the user includes image data of the user's face.

15. The vehicle according to claim 11, wherein, The route estimation user data is output by a deep learning model based on driving trajectory, and the deep learning model based on driving trajectory uses route-based matching information that depends on the similarity between the driving data of the vehicle's previous users and the user's route data as the route estimation user data.

16. The vehicle according to claim 15, wherein, The route estimation user data includes at least one of prior user probability information dependent on the route-based matching information and users identified through inference from driving trajectories.

17. The vehicle according to claim 15, wherein, The driving data and the route data include destination, intermediate stops, driving trajectory, driving time zone, and driving time.

18. The vehicle according to claim 15, wherein, The deep learning model based on driving trajectory is trained through comparative learning using driving data previously generated by multiple users.

19. The vehicle according to claim 11, wherein: Generating the user identification information includes: generating the user identification information by using a deep learning-based ensemble model that takes the image-estimated user data and the route-estimated user data as input; The processor is further configured to, before generating the user identification information, output setting estimation user data based on vehicle setting information estimation, according to the user's request; and The integrated model uses the estimated user data as additional input to generate the user identification information.

20. A non-transitory computer-readable medium comprising program instructions executable by a processor, the computer-readable medium comprising: Program instructions for acquiring image data of a user located in a vehicle; Program instructions to obtain route data according to the user's request; Program instructions for generating image estimation user data based on the image data; Program instructions for generating route estimation user data based on the route data; as well as Program instructions for generating user identification information based on the image-estimated user data and the route-estimated user data.

Images