Shared vehicle travel information management system and method that ensures driver anonymity

Abstract

The driving information management system for a shared vehicle of the present invention, which ensures the anonymity of the driver, includes a main camera installed in a vehicle and photographing the area in front of the vehicle, a sensor installed in the vehicle and detecting at least one of the vehicle's position, direction of movement, and speed, an auxiliary camera that photographs the driver inside the vehicle, a memory means, and a processor mounted on an on-board device installed in the vehicle, the processor including a driving record generation unit that generates a driving record of the vehicle based on the image captured by the main camera and the sensing data detected by the sensor, a facial feature vector extraction unit that irreversibly encodes the driver image captured by the auxiliary camera so that it cannot be restored to the original driver image, and extracts the driver's facial feature vector, and an anonymous driver driving information storage unit that matches the driving record with the driver's facial feature vector and stores it in the memory means.

Description

[Technical field]

[0001] The present invention relates to a driving information management system and method for a shared vehicle, and more particularly, to a driving information management system and method for a shared vehicle that irreversibly encodes a driver's image using machine learning to generate a facial feature vector, and manages the driving information of an anonymous driver based on the facial feature vector, thereby protecting the privacy of the driver and ensuring anonymity, while managing driving information for each driver and ensuring the anonymity of the driver, thereby making it possible to confirm whether the driver is the same driver. [Background technology]

[0002] Generally, a shared vehicle is a vehicle that is driven by two or more drivers. In addition to commercial vehicles such as buses, taxis, freight cars, and rental cars, there are various types of shared vehicles, such as corporate-owned vehicles and ride-sharing vehicles. In addition, privately owned vehicles can also have two or more drivers.

[0003] In order to efficiently manage shared vehicles, it is necessary to analyze the driving patterns of each driver. Driving patterns vary greatly from driver to driver. Analyzing diverse driving patterns is not only essential for effective management of shared vehicles, but also very useful for research into safe and autonomous driving of automobiles. At this time, it is important that driving information data of different drivers should not be mixed. If the driving information data of different drivers is mixed, a consistent driving pattern will not emerge, which may induce errors in data analysis.

[0004] US Patent Application Publication No. 2020-0110952, "SYSTEM AND METHOD FOR DETERMINING PROBABILITY THAT A VEHICLE DRIVER IS ASSOCIATED WITH A DRIVER IDENTIFIER," discloses a system for determining whether a driver matches a driver identifier in the event of an accident in order to confirm insurance settlement for a shared vehicle and the attribution of responsibility for the accident in a FMS (Fleet Management System). Korean Patent Registration No. 10-1984284, "Automated Driver Management System Using Machine Learning Model," recognizes the driver's state (such as dozing) using an internal camera and calculates the driver's driving score result in real time to guide the driver's safety. Korean Patent Registration No. 10-1122524, "Method and system for collecting driver characteristic information using synchronized data," describes a method for identifying and storing a driver by synchronizing camera images and various sensor signals. Korean Patent Publication No. 10-2020-0128285, "Method and device for opening and starting a vehicle based on facial recognition," discloses a technology that recognizes the user's face with a camera mounted on the exterior of the vehicle and allows only registered users to open and close the vehicle. Korean Patent Publication No. 10-1866768, "Driving assistance device and method considering driver characteristics," discloses a technology that selects a driver model based on an image from an internal camera and allows the driver to select settings suitable for that driver.

[0005] All of the listed conventional technologies propose a method to identify the driver from a video of the driver captured in the shared vehicle. Such a method is related to "driver Face ID" technology, and has been considered a natural part of identifying the driver in the management of shared vehicles up to now.

[0006] However, the method of photographing the driver's face and assigning an ID to them and managing them has a major blind spot in that it does not protect the driver's private life. Many drivers dislike having their facial image and ID stored along with their driving records. Furthermore, this method of managing driver IDs is highly likely to violate the Privacy Protection Act, as it exposes an individual's movements and may lead to sensitive private information being publicly leaked. In addition, if footage from inside the vehicle is stored, there is a high risk that the private lives of passengers will also be exposed. Going a step further, storing footage of the driver and the inside of the vehicle requires a lot of storage space, and managing such video data is fraught with problems such as huge installation costs, communication costs, and management costs. Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention aims to provide a system and method for managing driving information of a shared vehicle that can protect the privacy of a driver and ensure anonymity by irreversibly encoding an image of the driver to generate a facial feature vector and managing the driving information of the anonymous driver based on the facial feature vector, and also ensures the anonymity of the driver by accurately classifying and managing the driving information data of different drivers without mixing them.

[0008] Another object of the present invention is to provide a shared vehicle driving information management system and method that ensures the anonymity of the driver by comparing footage of the driver with pre-stored facial feature vectors with the driver's consent and making the driver's driving information viewable, thereby enabling a shared vehicle service provider to use the driver's driving information when analyzing driving records to provide benefits to the driver, when settling the driver's insurance premiums, and when determining the cause of liability for a traffic accident, while still maintaining the anonymity of the driver regarding the pre-stored driving information. [Means for solving the problem]

[0009] A driving information management system for a shared vehicle that ensures the anonymity of a driver according to one embodiment of the present invention includes a main camera installed in a vehicle and capturing images of the front of the vehicle, a sensor installed in the vehicle and detecting at least one of the vehicle's position, movement direction, and speed, an auxiliary camera that captures images of the driver inside the vehicle, a memory means, and a processor mounted on an on-board device installed in the vehicle, the processor including a driving record generation unit that generates a driving record of the vehicle based on images captured by the main camera and sensing data detected by the sensor, a facial feature vector extraction unit that irreversibly encodes the driver image captured by the auxiliary camera so that it cannot be restored to the original driver image, and extracts a facial feature vector of the driver, and an anonymous driver driving information storage unit that matches the driving record with the facial feature vector of the driver and stores the matched data in the memory means.

[0010] A method for managing driving information of a shared vehicle that ensures the anonymity of the driver according to one embodiment of the present invention is a method for managing driving information of a shared vehicle that is performed by an on-board device of the vehicle including a memory means and a processor, and the steps performed by the processor include: (a) a step of irreversibly encoding a driver image captured by an auxiliary camera installed inside the vehicle so that the original driver image cannot be restored, and extracting a driver's facial feature vector that indicates information about the driver's facial feature points; (b) a step of generating a driving record that includes at least one of the vehicle's current position information, driving direction information, and driving speed; and (c) a step of matching the driving record generated in step (b) with the driver's facial feature vector and storing it in the memory means.

[0011] In one embodiment of the present invention, a method for managing driving information of a shared vehicle that ensures the anonymity of the driver is provided in a method for managing driving information of a shared vehicle by communicating between an on-board device of the vehicle and a remote cloud server, the steps performed by a processor provided in the cloud server include (a) collecting from the on-board device the driving records of the vehicle and a facial feature vector of the driver that is irreversibly encoded from footage of the driver of the vehicle, and (b) classifying the driving records for each facial feature vector of the driver. Effect of the Invention

[0012] According to the shared vehicle driving information management system and method of the present invention that ensures the anonymity of the driver, an image of the driver is irreversibly encoded to generate a facial feature vector, and the driving information of the anonymous driver is managed based on the facial feature vector, thereby protecting the privacy of the driver and ensuring anonymity, and also having the effect of accurately classifying and managing the driving information data of different drivers.

[0013] In addition, according to the shared vehicle driving information management system and method of the present invention that guarantees the anonymity of the driver, by comparing video footage of the driver with pre-stored facial feature vectors with the driver's consent, the driver's driving information can be viewed, allowing the shared vehicle service provider to use the driver's driving information when analyzing driving records to provide benefits to the driver, when settling the driver's insurance premiums, or when determining the attribution of responsibility for a traffic accident, and in such cases, the driver's anonymity regarding the pre-stored driving information can be maintained. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram illustrating a shared vehicle driving information management system that ensures the anonymity of a driver according to the present invention. [Diagram 2] 1 is a block diagram illustrating a cloud server system in a shared vehicle driving information management system that ensures the anonymity of drivers according to the present invention. FIG. [Diagram 3] 4 is a flowchart illustrating a process in which driving information for each driver is stored in the present invention. [Figure 4] 4 is a flowchart illustrating a process of irreversibly extracting a facial feature vector in the present invention. [Diagram 5] FIG. 2 is a block diagram illustrating a machine learning model for extracting face feature vectors in the present invention. [Figure 6] FIG. 2 is a diagram illustrating a learning model for lossily encoding a face feature vector in the present invention. [Figure 7] FIG. 2 is a diagram illustrating a model for comparing lossy encoded facial feature vectors in the present invention. [Figure 8] FIG. 11 is a diagram showing an example in which anonymous driver driving information classification work is performed on a cloud server in the present invention. [Figure 9] 2 is a diagram illustrating a process in which a cloud server collects anonymous driver driving information in the present invention. [Figure 10] FIG. 2 is a diagram illustrating the clustering process of anonymous driver driving information in the present invention. [Figure 11] FIG. 2 is a diagram illustrating an example of how the cloud server classifies anonymous driver driving information in the present invention. [Figure 12] 1 is a diagram illustrating an example in which a cloud server in the present invention compares a user's face recognition information with anonymous driver's driving information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, specific examples of the present invention will be described with reference to the accompanying drawings. However, it should be understood that this is not intended to limit the present invention to a specific embodiment, and includes all modifications, equivalents, and alternatives included within the spirit and technical scope of the present invention.

[0016] Throughout the specification, parts having similar configurations and operations are designated by the same reference numerals. Also, the drawings accompanying the present invention are for the convenience of explanation, and the shapes and relative scales may be exaggerated or omitted.

[0017] In describing the embodiments in detail, redundant descriptions and descriptions of obvious techniques in the art are omitted. In addition, in the following description, when a part "includes" other elements, this means that elements other than the described elements can be further included, unless otherwise specified.

[0018] In addition, the terms "unit", "device", "module" and the like used in the specification refer to a unit that processes at least one function or operation, and is realized by hardware, software, or a combination of hardware and software. In addition, when a part is said to be electrically connected to another part, this includes not only the case where they are directly connected, but also the case where they are connected via another configuration in between.

[0019] Terms including ordinal numbers such as first, second, etc. may be used to describe various components, but the components are not limited by the terms. The terms are used only to distinguish one component from another. For example, a second component may be called a first component, and similarly, a first component may be called a second component, without departing from the scope of the present invention.

[0020] The present invention relates to a system and method for managing driving information of a shared vehicle that guarantees the anonymity of a driver, and provides a system and method for irreversibly encoding an image of a driver to generate a facial feature vector of the driver and managing driving information of an anonymous driver based on the facial feature vector of the driver. In the present invention, "irreversible encoding" refers to a process of encrypting or coding facial feature information and converting it into a vector form so that it cannot be decoded back to the original facial feature information. In other words, the facial feature vector of the driver that is irreversibly encoded according to the present invention cannot be restored to the original driver's face image (or facial feature information that can be inferred from such an image).

[0021] In the shared vehicle driving information management system and method of the present invention that guarantees the anonymity of the driver, the driver's captured video is only temporarily stored in a temporary storage memory, and is not stored in a permanent storage memory. In addition, the captured video of the driver may be permanently deleted from the memory means after the driver's facial feature vector is generated. Therefore, it is not possible to identify which driver the driving record is from in the on-board device of the vehicle or a remote cloud server. However, according to the present invention described below, it is possible to provide a method for later confirming whether the driving record stored together with the driver's facial feature vector is the driving record of the user (or a criminal) with the user's consent or with a warrant from a law enforcement agency.

[0022] For example, when a company providing a shared vehicle management service analyzes driving records to provide preferential treatment or incentives to drivers, when calculating insurance premiums for drivers, when determining the attribution of blame for a traffic accident, or when comparing face information provided by a law enforcement agency with the driving records of a specific driver, the pre-stored facial feature vector of the driver is compared with the user facial feature vector of a user (or a criminal instructed by a law enforcement agency's warrant) who has agreed to have his or her driving records checked, and the two vectors are constructed. Between the componentsIf the error range of both vectors is within a predetermined error range, it can be determined that the person indicated by both vectors is the same person. As a result, the present invention enables the on-board device of the vehicle or a cloud server located in a remote location to view and compare the driving records of each driver stored in the storage medium while ensuring the anonymity of many drivers.

[0023] The biggest difference between the present invention and the existing shared vehicle management system and method for managing shared vehicle driving information that guarantees the anonymity of the driver is that the system and method do not store and manage the driver's face image or the unique ID assigned to the driver. Therefore, the present invention can protect the driver's private life and guarantee anonymity while systematically managing the driving information of the shared vehicle. In addition, since the present invention does not need to store and manage the image inside the vehicle, it is possible to drastically reduce the storage space of the on-board device and greatly reduce communication costs and operating costs of the cloud server system. In addition, the present invention can ease the reluctance of the drivers to introduce the shared vehicle management service and promote the introduction and spread of the shared vehicle management service without violating the Privacy Protection Act.

[0024] Meanwhile, the shared vehicle driving information management system and method that guarantees the anonymity of the driver of the present invention is entirely performed by the on-board device of the vehicle. Also, according to the shared vehicle driving information management system and method that guarantees the anonymity of the driver of the present invention, the process of generating a driving record, extracting the facial feature vector of the driver and storing it together with the driving record is performed by the on-board device of the vehicle, and the process of managing the driving information of each vehicle (information in which the facial feature vector of the driver and the driving record are matched and stored) and comparing the facial feature vector of the user extracted from an external input image with the facial feature vector of the driver stored in advance to confirm the driving information by comparing the user and the driver is performed by a remote cloud server. Hereinafter, the shared vehicle driving information management system and method that guarantees the anonymity of the driver of the present invention will be described in detail with reference to the embodiments of the drawings.

[0025] FIG. 1 is a block diagram illustrating a shared vehicle driving information management system according to the present invention that ensures the anonymity of the driver.

[0026] 1, the shared vehicle driving information management system that ensures the anonymity of the driver of the present invention is realized by an on-board device 100 provided in a vehicle 150. Referring to FIG. 1, the vehicle 150 is provided with a GPS module 102, a sensor 104, a communication module 106, a display module 108, a main camera 112, an auxiliary camera 114, an external video input module 116, a vehicle connection module 122, a power module 124, a control module 126, a program storage memory 132, a temporary storage memory 134, a permanent storage memory 136, and a processor 200.

[0027] The GPS module 102 is a means for receiving signals sent from satellites and measuring the current position of the vehicle 150. The sensor 104 is a means for detecting the moving direction, attitude, speed, head angle, etc. of the vehicle. The sensor 104 may be composed of a plurality of sensors or a module in which a plurality of sensors are combined. For example, the sensor 104 is composed of at least one of an acceleration sensor, an angular velocity sensor, an inertial sensor, and a geomagnetic sensor, or a combination of a plurality of sensors. The sensor 104 may further include a temperature sensor, a current sensor, a voltage sensor, etc. for detecting vehicle status information. The communication module 106 is a means for the on-board device 100 to wirelessly communicate data with the cloud server system 300 described with reference to FIG. 2. The display module 108 is an output means for displaying operation-related information to the driver, displaying a destination route guidance application, and displaying a graphic user interface (GUI) or a touch user interface (TUI).

[0028] The main camera 112 is a device that is provided facing the front of the vehicle 150 and captures the road conditions in front of the vehicle. The auxiliary camera 114 is a device that is provided facing the inside of the vehicle 150 and captures the driver and the conditions inside the vehicle from inside the vehicle. The external video input module 116 is a means for inputting an external video via a wired interface or a wireless interface. For example, a user image for comparison with the driver's facial feature vector is input via a USB device. Also, a user image for comparison with the driver's facial feature vector may be input wirelessly from a remote cloud server system 300.

[0029] The vehicle connection module 122 is a means for connecting to the vehicle 150 and receiving signals related to driving information from an electronic control device in the vehicle 150. For example, the on-board device 100 of the present invention can be used to generate a driving record by receiving at least one or more signals of a vehicle speed signal, a brake signal, a turn signal signal, and an accumulated driving distance via a network inside the vehicle 150. The power supply module 124 is a means for supplying the vehicle's battery power to the components in the on-board device 100, and the control module 126 is a means for controlling the operation of the components in the on-board device 100.

[0030] Referring to FIG. 1, the on-board device 100 is provided with three memory means. The program storage memory 132 is a memory device that stores computer-readable instructions executed by the processor 200. The program storage memory 132 is a memory that can store, delete, and rewrite data even when the power is turned off. The temporary storage memory 134 is a memory device that temporarily stores data processed by the processor 200. The temporary storage memory 134 is composed of a volatile memory device in which data is volatilized when the power is turned off and can be read and written quickly, and stores the driver's captured video, intermediate processing data of the processor, data temporarily required during the operation of the processor, and the like. The permanent storage memory 136 is a memory device that permanently stores the processed data of the processor 200. The permanent storage memory 136 is composed of a non-volatile memory device in which data is stored, deleted, and rewritten even when the power is turned off, and stores the driving record and the driver's facial feature vector described below. In addition, the permanent storage memory 136 can further store vehicle status information, event information (such as a vehicle collision), a forward image when an event occurs, and the like, and is detachably provided in the on-board device 100 .

[0031] The processor 200 is a means for executing a series of processing steps of generating a driving record according to the present invention, extracting a facial feature vector of the driver, matching and storing the facial feature vector of the driver with the driving record, classifying driving information for each anonymous driver, and comparing an external input image with the facial feature vector of the driver, and is configured as a single processor or a multi-processor. The processor 200 calls the program storage memory 132 to execute processing instructions. The processor 200 includes a driving record generating unit 210, a facial feature vector extracting unit 220, an anonymous driver driving information storing unit 230, an anonymous driver driving information classifying unit 240, and a facial feature vector comparing unit 250. The configuration of the processor 200 may include machine learning, deep learning, and other artificial intelligence learning models.

[0032] The driving record generating unit 210 generates a driving record of the vehicle based on the video captured by the main camera 112 and the sensing data detected by the sensor 104. The driving record includes at least the current position information of the vehicle, the driving direction information of the vehicle, and the driving speed information of the vehicle. The driving record also includes video data of the area ahead of the vehicle captured by the main camera 112.

[0033] The facial feature vector extraction unit 220 lossily encodes the driver's image captured by the auxiliary camera 114 to extract the driver's facial feature vector. The process in which the facial feature vector extraction unit 220 extracts the driver's facial feature vector will be described in detail below with reference to FIGS. 4 to 7.

[0034] The anonymous driver's driving information storage unit 230 matches the driving record with the driver's facial feature vector and stores it in the permanent storage memory 136 of the memory means. After the driver's facial feature vector is generated, the anonymous driver's driving information storage unit 230 stores the driver's facial feature vector together with time information when the driving record was generated. The process of storing the anonymous driver's driving information will be described in detail below with reference to Figures 8 and 9.

[0035] The anonymous driver driving information classifying unit 240 classifies the driving records for each face feature vector of the driver. The anonymous driver driving information classifying unit 240 may be provided in the cloud server system 300, or may be provided in both the on-board device 100 and the cloud server system 300. The process of classifying the anonymous driver driving information will be described in detail below with reference to Figures 10 and 11.

[0036] The facial feature vector comparison unit 250 recognizes the facial feature points of the user included in the image input through the external image input module 116, and irreversibly encodes information on the facial feature points of the user to extract the facial feature vector of the user. The process of extracting the facial feature vector of the user is substantially the same as the process of extracting the facial feature vector of the driver. The facial feature vector comparison unit 250 compares the facial feature vector of the user with the facial feature vector of the driver stored in the permanent storage memory 136 of the memory means, and outputs the comparison result. The facial feature vector comparison unit 250 may also be provided in the cloud server system 300, or may be provided in both the on-board device 100 and the cloud server system 300. The process of comparing the facial feature vector of the driver and the facial feature vector of the user will be described in detail below with reference to FIG. 12.

[0037] FIG. 2 is a block diagram illustrating an example of a cloud server system in a shared vehicle driving information management system that ensures the anonymity of the driver of the present invention.

[0038] The cloud server system 300 is a server system that wirelessly communicates with the on-board device 100 of the vehicle 150 to collect, store, and manage the above-mentioned driving records and the facial feature vectors of the driver. Referring to FIG. 2, the cloud server system 300 includes a data storage server 310, a data management server 320, a data processing server 330, and an artificial intelligence server 340.

[0039] The data storage server 310 is a server that stores data received from the on-board device 100. The above-mentioned driving record and the driver's facial feature vector are matched with each other and stored in the data storage server 310. The data storage server 310 can also store general shared vehicle management information such as vehicle type information, status information, and owner information of the vehicle 150.

[0040] The data management server 320 is a server that manages data such as driving records and facial feature vectors of drivers, and manages data collection history, data storage period, data update and deletion, etc. The data processing server 330 is a server for processing data, and the artificial intelligence server 340 is a server for performing machine learning, deep learning, and other artificial intelligence learning for classifying and comparing driving information.

[0041] The artificial intelligence server 340 includes an anonymous driver driving information classifier 342 and a facial feature vector comparator 344 .

[0042] The anonymous driver driving information classifier 342 is a means that performs the same function as the anonymous driver driving information classifier 240 described in relation to the on-board device 100, and classifies the driving records for each driver's facial feature vector.

[0043] The facial feature vector comparison unit 344 is a means that performs the same function as the facial feature vector comparison unit 250 described with respect to the on-board device 100, and recognizes the user's facial feature points contained in the input image input through the input means, irreversibly encodes information regarding the user's facial feature points to extract the user's facial feature vector, compares the user's facial feature vector with the driver's facial feature vector stored in the data storage server 310, and outputs the comparison result.

[0044] Here, the input means may be a web browser 400, dedicated software, or a terminal input / output device, and is a means for inputting a video including the face of a user who has agreed to confirm his / her driver information or a criminal specified in a warrant issued by a law enforcement agency. The anonymous driver driving information classifier 342 and the face feature vector comparator 344 will be described in detail later with reference to Figs. 10 to 12.

[0045] 3 is a flow chart illustrating a process of storing driving information for each driver in the present invention. The process of generating a driving record in the driving record generating unit 210 of the on-board device 100 will be described with reference to FIG. 3.

[0046] The driving record generating unit 210 determines whether the time period is satisfied (ST310), and periodically generates driving records and stores them in the memory means (ST315). Even if the time period has not arrived, the driving record generating unit 210 determines whether an event has occurred (ST320), and stores the driving record at the time of the event occurrence in the memory means (ST325). For example, an event means an incident that requires a driving record to be kept, such as detection of an impact to the vehicle 150, or detection of a driver's drowsiness or inattention.

[0047] The driving record generating unit 210 stores the periodically generated driving records in an overwritable area of ​​the permanent storage memory 136, and stores the driving record at the time of the event occurrence in an unoverwritable area of ​​the permanent storage memory 136.

[0048] 3, the driving record generating unit 210 collects position data from the GPS module 102 and sensing data from the sensor 104 (ST330) and converts the collected data into computer-readable data (ST335). In addition, the driving record generating unit 210 receives an image of the area in front of the vehicle from the main camera 112 (ST340) and compresses the input image data (ST345).

[0049] The driving record generating unit 210 synchronizes the digitally converted position data and sensing data with the video compression data, structures them into a series of data (ST350), and generates a driving record (ST355). The generated driving record is accumulated and stored in the temporary storage memory 134, which functions as a buffer. Then, it is determined whether the driving record is based on a time period or an event occurrence, and the driving record is moved to an overwritable area or a non-overwritable area of ​​the permanent storage memory 136.

[0050] Fig. 4 is a flow chart illustrating a process of irreversibly extracting a facial feature vector in the present invention, Fig. 5 is a block diagram illustrating a machine learning model for extracting a facial feature vector in the present invention, Fig. 6 is a diagram illustrating a learning model for irreversibly encoding a facial feature vector in the present invention, and Fig. 7 is a diagram illustrating a model for comparing irreversibly encoded facial feature vectors in the present invention. The process of extracting a facial feature vector of a driver by the facial feature vector extraction unit 220 of the on-board device 100 will be described below with reference to Figs. 4 to 7.

[0051] 4, the facial feature vector extraction unit 220 performs image pre-processing (ST410) on the driver's image captured by the auxiliary camera 114 by performing at least one of image resizing and image cropping to normalize the image. Then, a primary face recognition step (ST420) is performed to recognize facial feature points (e.g., objects such as eyes, nose, and mouth) from the pre-processed image, and a secondary face recognition step (ST430) is performed to crop the image along the facial contour. Next, information on the facial feature points is lossily encoded (ST440) and the facial feature vector of the driver is output (ST490).

[0052] 4 shows a process in which the facial feature vector comparator 250 extracts a facial feature vector of a user from an external input image. In the present invention, the extraction of a facial feature vector of a user is described as being performed by the facial feature vector comparator 250, but this is merely a functional classification of the subject of processing, and is the same as the process performed by the facial feature vector extractor 220, and may be performed by the facial feature vector extractor 220.

[0053] As described above, the facial feature vector comparison unit 250 performs image preprocessing on an external input image including a user's facial image (ST450), and performs a primary face recognition step (ST460) and a secondary face recognition step (ST470), lossily encodes information about facial feature points (ST480), and then outputs the user's facial feature vector (ST490).

[0054] 5 and 6, the facial feature vector extraction unit 220 includes a plurality of deep neural network learning units (Deep Neural Networks) and can extract the facial feature vector of the driver through artificial intelligence learning. Of course, the facial feature vector of the user can also be extracted through the same artificial intelligence learning.

[0055] Referring to FIG. 5, the pre-processing unit 510 is a learning network for image pre-processing, which is a learning network for processing image normalization described in FIG. 4. The first deep neural network learning unit includes a plurality of convolution layers and is composed of an object detection unit 520 and a region proposal network (RPN) 530. The object detection unit 520 detects all objects from the pre-processed image. The region proposal network 530 calculates at least one region of interest (RoIs). The output of the region proposal network 530 is transmitted to the second deep neural network learning unit via the intermediate processing unit 540.

[0056] The second deep neural network training unit includes a second deep neural network 550 that detects objects by pooling regions of interest (RoIs), calculates a classification hierarchy of the detected objects, and recognizes facial feature points. The output of the second deep neural network 550 is transmitted to a third deep neural network training unit via a post-processing unit 560.

[0057] 6, the third deep neural network learning unit includes an encoding unit 610 and a global average pooling unit 620. The encoding unit 610 extracts and encodes facial feature information related to the recognized facial features, and the global average pooling unit 620 removes position information from the facial feature information, compresses the information into vector values, and irreversibly encodes the information to output a facial feature vector of the driver. As shown in the lower part of FIG. 6, the finally extracted facial feature vector of the driver is irreversible data that cannot be restored to a previous driver image.

[0058] Referring to FIG. 7, image 1 and image 2 are images of the same driver, and image 3 is an image of a different driver. The facial feature vector of the driver obtained by lossily encoding image 1 by the facial feature vector extraction unit 220 is represented as (3, 5, 12, 1, ...), and the facial feature vector of the driver obtained by lossily encoding image 2 is represented as (3, 6, 11, 1, ...). In the present invention, the facial feature vector of the driver is one or more The ingredients of Although shown in the form of vectors arranged in rows and columns, in the illustrated example, for purposes of understanding the invention, a vector having a single row is used. Ingredients If we compare the facial feature vectors of the drivers in Image 1 and Image 2, the first and fourth columns The ingredients are are identical, the second and third columns The ingredients are As you can see, the same rows and columns of both vectors The ingredients of Results compared to each other , between components If the error range of the two vectors is within a predetermined error range, it can be determined that the person in the original image is the same person. For example, when the two vectors are compared, Ingredients When the specific gravity is 50% or more, or when the Ingredients When the difference does not fall outside a predetermined error range, etc., the person in the original image can be defined as the same person.

[0059] On the other hand, the facial feature vector of the driver obtained by lossily encoding image 3 is represented as (21, 1, 6, 9, ...), and when compared with the facial feature vector of the driver in image 1 or image 2, The ingredients of Different , between components It can be seen that the error of also becomes large. Ingredients When the specific gravity is very small or different from each other Ingredients When the difference shows a large error margin, the person in the original image can be defined as a different person.

[0060] 8 is a diagram showing an example in which the anonymous driver's driving information classification work is performed by a cloud server in the present invention, and FIG. 9 is a diagram showing a process in which the cloud server collects anonymous driver's driving information in the present invention. The process in which the anonymous driver's driving information storage unit 230 of the on-board device 100 stores anonymous driver's driving information will be described below with reference to FIG. 8 and FIG. 9.

[0061] Referring to FIG. 8, when a driver gets into the vehicle 150, the driver is photographed by the auxiliary camera 114. Then, the facial feature vector extraction unit 220 extracts the facial feature vector of the driver as described above. The extracted facial feature vector is valid until the driver gets out of the vehicle. The anonymous driver driving information storage unit 230 stores the facial feature vector of the driver from the time the driver gets into the vehicle until the driver gets out of the vehicle in a memory means together with the driving record generated in FIG. 3. At this time, the driver ID does not exist. The driving information of the anonymous driver (the driving record stored in synchronization with the driver's facial feature vector) is classified and stored autonomously by the on-board device 100 for each driver's facial feature vector. Also, as shown in FIG. 8, the driving information of the anonymous driver may be collected by wireless communication or an SD card or the like and transmitted to a cloud server. The cloud server may also classify and store the driving information of the anonymous driver for each driver's facial feature vector.

[0062] Referring to FIG. 9, the anonymous driver driving information may include data of the vehicle number, vehicle type, driving direction, driving time, and the driver's facial feature vector. In the example of FIG. 9, it can be confirmed that the vehicle number "xxxx" was used by anonymous drivers a', b'', and a''', in the order of time. The on-board device 100 or the cloud server system 300 can classify the driving information of the anonymous driver for each anonymous driver. For example, among the driving information related to the vehicle number "xxxx", anonymous drivers a' and a''' are classified and managed as driving information by the same driver. The process of clustering and classifying the vehicle driving information for each anonymous driver will be described in detail with reference to FIGS. 10 and 11.

[0063] FIG. 10 is a diagram illustrating the clustering process of anonymous driver's driving information in the present invention, and FIG. 11 is a diagram illustrating an example of classifying anonymous driver's driving information by the cloud server in the present invention.

[0064] Referring to FIG. 10, six driver facial feature vectors are illustrated. Each point in the example is a vector value corresponding to a facial feature point drawn as a point on a two-dimensional screen. Clustering can be performed by K-means clustering based on the distance between each point. Here, "K" means the number of clusters (groups) expected to be found from the data set. "Means" means the average distance from each data to the center of the cluster to which the data belongs, and the goal of this process is to position K centroids so as to minimize this value. First, K arbitrary centroids are positioned, and each data is assigned to the closest centroid to form a temporary cluster. Next, the centroid of the cluster is updated based on the data specified as the cluster. The above process is repeated until convergence, i.e., until the centroid is no longer updated. Through this process, K clusters can be derived.

[0065] Referring to FIG. 11, the anonymous driver driving information classifier 342 of the cloud server system 300 classifies the same row and column of the driver's facial feature vector. The ingredients of Compare each other , between components The facial feature vectors of the drivers whose error range is within a predetermined error range can be defined as the facial feature vectors of the same driver, and the anonymous driver driving information can be classified. In this way, the anonymous drivers a', a'', and a''' are defined as the same driver (a), and the driving records of the drivers (anonymous drivers classified as having similar facial feature vectors) are classified into one cluster. For example, the driving information of the anonymous driver a driving vehicle numbers "xxxx" and "yyyy" is classified into one cluster. Meanwhile, such cluster and classification criteria can be similarly applied in the process of comparing the user's facial feature vector with the driver's facial feature vector.

[0066] FIG. 12 is a diagram illustrating an example in which the cloud server in the present invention compares the user's face recognition information with the anonymous driver's driving information.

[0067] 12, the facial feature vector comparison unit 344 of the cloud server system 300 (or the facial feature vector comparison unit 250 of the on-board device 100) recognizes the facial feature points of a user included in an input image input through an input means, and irreversibly encodes information about the facial feature points of the user to extract a facial feature vector of the user. The facial feature vector comparison unit 344 then compares the user's facial feature vector with the driver's facial feature vector and outputs a comparison result. At this time, the facial feature vector comparison unit 344 compares the same rows and columns of the user's facial feature vector and the driver's facial feature vector to extract a facial feature vector of the user. The ingredients of Compare , between components The person indicated by both vectors whose error range is within a predetermined error range is defined as the same person.

[0068] The facial feature vector comparison unit 344 compares both vectors in this manner, which is similar to the K-means clustering and classification described above. If the image input through the input means indicates a user c, the facial feature vector comparison unit 344 can search for driving information by anonymous drivers c', c'', and c''' from among the pre-stored anonymous driver driving information, determine that they are the same person, and can cluster and output the driving information of anonymous driver c, as shown in FIG. 12.

[0069] Based on the final clustering result, the company providing the shared vehicle management service can provide an incentive to the anonymous driver c in return for safe driving, update the insurance premium for the anonymous driver c, determine the cause of responsibility for a traffic accident caused by the anonymous driver c, and respond to a comparison of records related to the anonymous driver c upon a warrant from a law enforcement agency. At this time, the privacy and anonymity of other anonymous drivers stored in the cloud server system 300 are protected. In addition, the records of the anonymous driver c after the driving information has been temporarily compared can be stored while maintaining anonymity.

[0070] The invention disclosed above can be modified in various ways without departing from the basic concept. In other words, all the above embodiments should be interpreted as illustrative and not restrictive. Therefore, the scope of protection of the present invention should be determined by the appended claims, not the above embodiments, and when the elements limited in the appended claims are replaced with equivalents, they belong to the scope of protection of the present invention.

Claims

1. A main camera is provided in the vehicle and captures an image of the area in front of the vehicle; A sensor provided in the vehicle, the sensor detecting at least one of a position, a moving direction, and a speed of the vehicle; an auxiliary camera for photographing a driver inside the vehicle; A memory means; A processor installed in an on-board device provided in the vehicle; Including, The processor, a driving record generating unit that generates a driving record of the vehicle based on the image captured by the main camera and the sensing data detected by the sensor; a facial feature vector extraction unit that performs image preprocessing on the driver image captured by the auxiliary camera by performing at least one of image resizing and image cropping to normalize the image, recognizes facial feature points from the preprocessed image by a first deep neural network learning unit that includes a plurality of convolution layers and calculates at least one Region of Interest (RoIs) through a Region Proposal Network (RPN), and irreversibly encodes information on the facial feature points so that the information cannot be restored to the original driver image, thereby extracting a facial feature vector of the driver; and an anonymous driver driving information storage unit that matches the driving record with the face feature vector of the driver and stores the matching data in the memory means. A driving information management system for shared vehicles that ensures driver anonymity.

2. 2. The shared vehicle driving information management system that ensures the anonymity of a driver according to claim 1, wherein the sensor is at least one of a GPS module, an acceleration sensor, an angular velocity sensor, an inertial sensor, and a geomagnetic sensor.

3. 3. The shared vehicle driving information management system that ensures the anonymity of the driver as described in claim 2, wherein the driving record generating unit receives at least one of a vehicle speed signal, a brake signal, a turn signal signal, and a cumulative driving distance via a network inside the vehicle to generate the driving record.

4. 2. The driving information management system for a shared vehicle that ensures the anonymity of a driver as described in claim 1, wherein the driving record generation unit generates the driving record so as to include information on the current position of the vehicle, information on the driving direction of the vehicle, and information on the driving speed of the vehicle.

5. The system for managing driving information for a shared vehicle that ensures anonymity of a driver according to claim 4 , wherein the driving record generating unit generates the driving record by including video data captured by the main camera.

6. 2. A shared vehicle driving information management system that ensures the anonymity of a driver as described in claim 1, wherein the memory means includes a program storage memory that stores computer-readable instructions executed by the processor, a temporary storage memory that temporarily stores processed data processed by the processor, and a permanent storage memory that permanently stores the processed data.

7. 7. A driving information management system for a shared vehicle that ensures the anonymity of a driver as described in claim 6, wherein the driving record generation unit stores the periodically generated driving records in an overwritable area of ​​the permanent storage memory, and stores the driving records at the time of the occurrence of an event including impact detection in a non-overwritable area of ​​the permanent storage memory.

8. 8. The shared vehicle driving information management system that ensures the anonymity of the driver as described in claim 7, wherein the driving record generation unit reads a facial feature vector of the driver to generate an event of the driver's drowsy or inattentive state, and stores the generated event driving record in a non-overwritable area of ​​the permanent storage memory.

9. 2. The shared vehicle driving information management system according to claim 1, further comprising a second deep neural network learning unit that detects objects by pooling the regions of interest (RoIs), calculates a classification hierarchy of the detected objects, and recognizes the facial feature points.

10. A shared vehicle driving information management system that ensures the anonymity of the driver as described in claim 1, wherein the facial feature vector extraction unit irreversibly encodes and extracts the driver's facial feature vector, which is performed by a third deep neural network learning unit that removes position information from the facial feature point information, compresses the information into vector values, and irreversibly encodes the information.

11. 2. A shared vehicle driving information management system that ensures the anonymity of a driver as described in claim 1, wherein the processor permanently deletes the captured video of the driver from the memory means once the facial feature vector extraction unit has completed extraction of the driver's facial feature vector.

12. A shared vehicle driving information management system that ensures the anonymity of the driver as described in claim 1, wherein the anonymous driver driving information storage unit stores the driver's facial feature vector in the memory means together with time information when the driving record was generated after the driver's facial feature vector is generated.

13. The shared vehicle driving information management system that ensures the anonymity of the driver according to claim 1 , wherein the processor further includes an anonymous driver driving information classification unit that classifies the driving record for each facial feature vector of the driver.

14. The anonymous driver driving information classification unit compares components of the same row and column of the driver's facial feature vector with each other, and defines the driver's facial feature vector whose error range between the components is within a predetermined error range as the facial feature vector of the same driver, thereby classifying the driving record. A shared vehicle driving information management system that ensures the anonymity of the driver as described in claim 13.

15. an external video input module to which external video is input via a wired or wireless connection; 14. The shared vehicle driving information management system that ensures the anonymity of the driver as described in claim 13, further comprising: a facial feature vector comparison unit that recognizes facial feature points of a user included in the image input via the external image input module, irreversibly encodes information regarding the user's facial feature points to extract a facial feature vector of the user, compares the facial feature vector of the user with the facial feature vector of the driver stored in the memory means, and outputs a comparison result.

16. The facial feature vector comparison unit compares the components of the same row and column of the user's facial feature vector and the driver's facial feature vector, and determines that the user's facial feature vector and the driver's facial feature vector whose error range between the components is within a predetermined error range indicate that they are the same person.

17. A cloud server including an anonymous driver driving information classification unit that wirelessly communicates with the vehicle, collects, stores, and manages the driving records and the driver's facial feature vectors from a remote location, and classifies the driving records for each of the driver's facial feature vectors.

18. The anonymous driver driving information classification unit compares components of the same row and column of the driver's facial feature vector with each other, and defines the facial feature vector of a driver whose error range between the components is within a predetermined error range as the facial feature vector of the same driver, thereby classifying the driving record. A shared vehicle driving information management system that ensures the anonymity of the driver as described in claim 17.

19. 20. The shared vehicle driving information management system that ensures the anonymity of a driver as described in claim 17, wherein the cloud server further includes a facial feature vector comparison unit that recognizes facial feature points of a user included in an input image input through an input means, irreversibly encodes information regarding the facial feature points of the user to extract a facial feature vector of the user, compares the facial feature vector of the user with a facial feature vector of the driver, and outputs a comparison result.

20. 20. A shared vehicle driving information management system that ensures the anonymity of the driver as described in claim 19, wherein the facial feature vector comparison unit compares components of the same row and column of the user's facial feature vector and the driver's facial feature vector, and determines that the user's facial feature vector and the driver's facial feature vector whose error range between the components is within a predetermined error range are the same person.

21. A method for managing driving information of a shared vehicle, the method being carried out by an on-board device of the vehicle including a memory means and a processor, comprising: The steps performed by the processor include: (a) performing image pre-processing on a driver's image captured by an auxiliary camera installed inside the vehicle by performing at least one of image resizing and image cropping to normalize the image, recognizing facial feature points from the pre-processed image by a first deep neural network learning unit (Deep Neural Network) including a plurality of convolution layers and computing at least one Region of Interest (RoIs) through a Region Proposal Network (RPN), and irreversibly encoding information on the facial feature points so that the information cannot be restored to the original driver's image, thereby extracting a facial feature vector of the driver; (b) generating a driving record including at least one of current vehicle position information, driving direction information, and driving speed; (c) matching the driving record generated in step (b) with a facial feature vector of the driver and storing the matching result in the memory means. A method for managing driving information of shared vehicles that ensures the anonymity of drivers.

22. The method for managing driving information of a shared vehicle that ensures anonymity of a driver according to claim 21 , wherein step (b) generates the driving record including video data captured by a main camera of the vehicle.

23. 22. The method for managing driving information of a shared vehicle that ensures the anonymity of a driver as described in claim 21, wherein recognizing the facial feature points in step (a) is performed by a second deep neural network learning unit that pools the regions of interest (RoIs) to detect objects, calculates a classification hierarchy of the detected objects, and recognizes the facial feature points.

24. The method for managing driving information of a shared vehicle that ensures the anonymity of a driver as described in claim 21, wherein the step (a) of irreversibly encoding and extracting the facial feature vector of the driver is performed by a third deep neural network learning unit that removes position information from the facial feature point information, compresses the information into vector values, and irreversibly encodes the information.

25. 22. The method for managing driving information of a shared vehicle that ensures the anonymity of a driver according to claim 21, further comprising the step of permanently deleting the captured image of the driver from the memory means after step (a).

26. (d) inputting an external video image via a wired or wireless connection; (e) recognizing a facial feature point of the user included in the image input via the external image input module, and lossily encoding information regarding the facial feature point of the user to extract a facial feature vector of the user; (f) comparing the user's facial feature vector with the driver's facial feature vector stored in the memory means and outputting a comparison result. A method for managing driving information of a shared vehicle that ensures anonymity of a driver according to claim 21.

27. The method for managing driving information of a shared vehicle that ensures the anonymity of a driver as described in claim 26, wherein step (f) compares components of the same row and column of the user's facial feature vector and the driver's facial feature vector, and determines that the user's facial feature vector and the driver's facial feature vector whose components indicate an error range within a predetermined error range are the same person.

28. A method for managing driving information of a shared vehicle, in which an on-board device of a vehicle communicates with a remote cloud server to manage driving information of the shared vehicle, comprising: The steps performed by a processor provided in the cloud server include: (a) collecting from the on-board device a driving record of the vehicle and a facial feature vector of the driver, the facial feature vector being irreversibly encoded from a captured image of the driver of the vehicle so that the original image of the driver cannot be restored; (b) comparing components of the same row and column of the facial feature vector of the driver with each other, and classifying the driving record by defining the facial feature vector of the driver whose error range between the components is within a predetermined error range as the facial feature vector of the same driver, A method for managing driving information of shared vehicles that ensures the anonymity of drivers.

29. (c) inputting an external video; (d) recognizing a facial feature point of the user included in the external image, and lossily encoding information regarding the facial feature point of the user to extract a facial feature vector of the user; (e) comparing the facial feature vector of the user with the facial feature vector of the driver and outputting a comparison result. A method for managing driving information of a shared vehicle that ensures the anonymity of a driver according to claim 28.

30. The step (e) compares components of the same row and column of the user's facial feature vector and the driver's facial feature vector, and determines that the user's facial feature vector and the driver's facial feature vector whose error range between the components is within a predetermined error range indicate that the person indicated by the user's facial feature vector and the driver's facial feature vector is the same person.

Images