Systems and methods for handling forgotten items in an autonomous vehicle

Abstract

The present invention relates to a system and method for handling a forgotten item in an autonomous vehicle. The vehicle includes a vision sensor configured to take a first vehicle interior image and a second vehicle interior image when a passenger gets on and off the vehicle, respectively. The vehicle is configured to transmit the taken first and second vehicle interior images to a server. The server is configured to compare the first vehicle interior image with the second vehicle interior image and transmit forgotten item information to the passenger when a forgotten item in the vehicle is sensed.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2020-0026645, filed on March 3, 2020, with the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to a system and method for handling lost items in autonomous vehicles. Background Technology

[0004] With the development of autonomous driving technology, ride-sharing services such as robotaxis and purpose-built vehicles (PBVs) have emerged. Since ride-sharing services target a large number of unspecified individuals, the protection of users' personal information has become a concern. While managing lost items and maintaining vehicle cleanliness are necessary during services using vehicles accessible to numerous unspecified individuals (such as robotaxis or PBVs), these operations must be unmanned and conducted without a dedicated administrator. Summary of the Invention

[0005] This invention aims to solve the aforementioned problems in the prior art, while fully retaining the advantages achieved by the prior art.

[0006] One aspect of the present invention provides a system and method for handling lost items in autonomous vehicles, which monitors the vehicle interior to detect lost items without violating privacy rules when providing shared mobility services, and takes follow-up actions on the detected lost items.

[0007] The technical problems solved by the present invention are not limited to those described above, and any other technical problems not mentioned herein will be clearly understood by those skilled in the art through the following description.

[0008] According to one aspect of the present invention, a system for handling lost items in an autonomous vehicle includes: a vehicle and a server, the vehicle including a vision sensor configured to capture a first interior image and a second interior image of the vehicle when a passenger gets on and off the vehicle, respectively, and to send the captured first interior image and second interior image of the vehicle to the server; the server is configured to compare the first interior image of the vehicle with the second interior image of the vehicle, and to send information about the lost item to a user terminal associated with the passenger when a lost item is detected in the vehicle.

[0009] The vehicle can be configured to determine a time point before or after the passenger gets on the vehicle as a first photographing time point and a time point after or before the passenger gets off the vehicle as a second photographing time point based on whether the passenger agrees.

[0010] The vehicle can be configured to blur or delete an area in the first vehicle interior image that matches a ride position of the other passenger in a state in which the other passenger rides the vehicle at the first photographing time point.

[0011] The vehicle can be configured to blur or delete an area in the second vehicle interior image that matches a ride position of the other passenger in a state in which the other passenger rides the vehicle at the second photographing time point.

[0012] The vehicle can be configured to determine whether the vehicle has arrived at the call location based on a vehicle location.

[0013] The vehicle can be configured to determine whether the vehicle has arrived at the call location further based on at least one of a vehicle speed and / or a brake state information.

[0014] The vehicle can be configured to sense a time point before the passenger gets on the vehicle based on a door state information.

[0015] The vehicle can be configured to sense the passenger based on information provided by at least one of a weight sensor, a vision sensor, a camera, an ultrasonic sensor, and / or a radar.

[0016] The vehicle can be configured to determine the end of the service for the passenger based on completion of a service fee payment and a crew member location information.

[0017] The vehicle can be configured to sense a time point after the passenger gets off the vehicle based on at least one of a door state information, an external button signal, information provided by a weight sensor, information provided by an ultrasonic sensor, information provided by a radar, and / or information provided by a vision sensor.

[0018] The server can be configured to transmit lost article information to a business operator when a lost article in the vehicle is sensed.

[0019] The lost article in the vehicle can be at least one of a passenger's belongings, garbage, and / or waste.

[0020] According to an aspect of the present application, a method for processing a forgotten item in an autonomous vehicle, comprising the steps of: a first step of receiving, by a server, a first vehicle interior image and a second vehicle interior image photographed by a vision sensor at the time of boarding and alighting of a passenger, respectively; a second step of comparing, by the server, the first vehicle interior image with the second vehicle interior image to sense a forgotten item in the vehicle; and a third step of transmitting, by the server, forgotten item information to a user terminal related to the passenger.

[0021] The first step can include determining a time point before boarding or after boarding as a first photographing time point based on whether the passenger agrees to be photographed, and determining a time point before alighting or after alighting as a second photographing time point based on whether the passenger agrees to be photographed.

[0022] The first step can include blurring or deleting an area in the first vehicle interior image that matches a boarding position of another passenger when the other passenger is present in the vehicle at the first photographing time point.

[0023] The first step can include blurring or deleting an area in the second vehicle interior image that matches a boarding position of another passenger in a state in which the other passenger boards the vehicle at the second photographing time point.

[0024] The first step can include determining, by the vehicle, whether the vehicle has arrived at a call location based on a vehicle location.

[0025] The first step can further include determining, by the vehicle, whether the vehicle has arrived at a call location based further on at least one of a vehicle speed and / or a brake state information.

[0026] The first step can include sensing, by the vehicle, a time point before boarding of the passenger based on a door state information.

[0027] The first step can include sensing, by the vehicle, the passenger based on information provided by at least one of a weight sensor, a vision sensor, a camera, an ultrasonic sensor, and / or a radar.

[0028] The first step can include determining, by the vehicle, an end of service of the passenger based on completion of payment of a service fee and a passenger position information.

[0029] The first step can further include sensing, by the vehicle, a time point after alighting of the passenger based on at least one of a door state information, an external button signal, information provided by a weight sensor, information provided by an ultrasonic sensor, information provided by a radar, and / or information provided by a vision sensor.

[0030] The first step can further include transmitting, by the server, the lost item information to a business operator upon sensing the lost item in the vehicle.

[0031] The lost item in the vehicle can be at least one of a passenger's belongings, trash, and / or waste. BRIEF DESCRIPTION OF DRAWINGS

[0032] The above and other objects, features and advantages of the present application will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0033] Figure 1 is a configuration diagram illustrating a system for processing a lost item in an autonomous vehicle according to an embodiment of the present application;

[0034] Figure 2 is Figure 1 a block diagram of a vehicle shown;

[0035] Figure 3 is a flowchart illustrating a method for processing a lost item in an autonomous vehicle according to an embodiment of the present application;

[0036] Figure 4 and Figure 5 are exemplary views for describing a processing method of an image related to the present application;

[0037] Figure 6 is a block diagram of a computing system that performs a method for processing a lost item in an autonomous vehicle according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] Some embodiments of the present application will be described below in detail with reference to the accompanying drawings. In adding reference numerals to components of each drawing, it should be noted that the same or equivalent components, even if shown on other drawings, are designated by the same reference numerals. Also, in describing embodiments of the present application, when it is determined that a detailed description of related known configurations or functions will hinder the understanding of embodiments of the present application, it will be omitted.

[0039] In describing components of embodiments according to the present application, terms such as first, second, A, B, (a), (b), etc. can be used. These terms are used only to distinguish one component from another component, and the terms do not limit the nature, order or sequence of the components. Unless otherwise defined, terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0040] Figure 1 is a configuration diagram illustrating a system for processing a forgotten item in an autonomous vehicle according to an embodiment of the present application. Further, Figure 2 is Figure 1 a block diagram of a vehicle. In the present embodiment, a case where a shared mobility service is provided using an autonomous vehicle (e.g., a robotic taxi, a PBV, etc.) is described as an example.

[0041] The system for processing a forgotten item in an autonomous vehicle includes a user terminal 100, a vehicle 200, and a server 300.

[0042] The user terminal 100 is a device owned by a user (e.g., a person who wants to take a ride, a person who makes a reservation, a passenger, an owner, and an occupant) and capable of wireless or wired communication. The user terminal 100 can be implemented as a device such as a smartphone, a tablet, a personal data assistant (PDA), a mobile multimedia player (PMP), or a notebook computer. Although not shown in the drawing, such a user terminal 100 includes a communication module, a user input module, an output module, a processor, and a memory.

[0043] The user terminal 100 can make a reservation for use of a shared mobility service (i.e., use of a shared vehicle) on the server 300 through a specific software or application provided by a service provider. That is, the user terminal 100 requests a ride reservation (vehicle call) to the server 300 based on a user's input. When requesting a ride reservation, the user terminal 100 transmits ride reservation information including a ride location (e.g., a pickup location and a departure point), a destination (e.g., a drop-off location), a ride time, information about a reservation person (e.g., a name, a contact address, etc.), the number of people taking a ride, whether the user agrees to be photographed or payment information (including a payment plan or a payment method), etc.

[0044] When the user (a person who made a reservation) gets on / off the vehicle 200, the user terminal 100 can transmit user identification information (e.g., a user ID, a phone number, an email address, payment scheme information, or a terminal unique number, etc.) to the vehicle 200. The vehicle 200 uses the user identification information to identify the passenger who gets on / off.

[0045] The vehicle 200 (a shared vehicle that can be used by numerous unspecified persons) can be implemented as an autonomous taxi or a private bus vehicle (PBV). Although only one vehicle is illustrated in FIG. 1, two or more shared vehicles can exist. Referring to FIG. 2, the vehicle 200 includes a position measuring device 210 (or a position sensing device), a detector 220, a passenger sensor 230, a communication device 240, a vision sensor 250, a storage device 260, and a processor 270. Figure 1 Figure 2 The vehicle 200 (a shared vehicle that can be used by numerous unspecified persons) can be implemented as an autonomous taxi or a private bus vehicle (PBV). Although only one vehicle is illustrated in FIG. 1, two or more shared vehicles can exist. Referring to FIG. 2, the vehicle 200 includes a position measuring device 210 (or a position sensing device), a detector 220, a passenger sensor 230, a communication device 240, a vision sensor 250, a storage device 260, and a processor 270.

[0046] The position measuring device 210 measures or senses the current position of the vehicle 200. The position measuring device 210 measures the vehicle position using at least one of positioning techniques such as a global positioning system (GPS), dead reckoning (DR), differential GPS (DGPS), or carrier phase differential GPS (CDP GPS).

[0047] The detector 220 obtains vehicle information from an electronic control unit (ECU) or a sensor mounted to the vehicle. The vehicle information can include vehicle speed, brake status information, or door status information (e.g., whether a door opening button is manipulated or whether a door handle is manipulated). The sensor can be implemented as a speed sensor, a brake pedal position sensor, a door sensor, a weight sensor, or a vision sensor. The electronic control unit can be implemented as a door system of the vehicle, an electronic stability control (ESC), a transaction control system (TCS), or an anti-lock brake system (ABS).

[0048] The passenger sensor 230 senses a passenger who gets on the vehicle 200 using a passenger sensing technique using a weight sensor, a vision sensor (e.g., a camera), an ultrasonic sensor, or a radar. The passenger sensor 230 can obtain a seating position of a passenger (i.e., an occupant) who gets on the vehicle or the number of passengers. The passenger sensor 230 senses a passenger (a user) who gets on / off the vehicle 200. In addition, the passenger sensor 230 can sense a seat change of a passenger in the vehicle 200.​

[0049] The communication device 240 can enable the vehicle 200 to perform wireless communication with the user terminal 100 or the server 300. The communication device 240 can employ a communication technology such as Wi-Fi, Bluetooth, near field communication (NFC), long term evolution (LTE), international mobile telecommunication (IMT)-2020, Ethernet, or vehicle to everything (V2X).

[0050] The vision sensor 250 is installed inside (indoors) the vehicle 200 and photographs a preset passenger boarding space. The vision sensor 250 can be implemented as at least one of a charge-coupled device (CCD) image sensor, a complementary metal-oxide semiconductor (CMOS) image sensor, a charge priming device (CPD) image sensor, or a charge injection device (CID) image sensor. The vision sensor 250 can obtain a complete image of the inside (indoors) of the vehicle through an image sensor separately installed at the front of the inside of the vehicle 200, or obtain an image of the inside of the vehicle for each seat through each image sensor installed for each seat. When the image sensor is installed for each seat, each lens located at the front of each image sensor is installed to face the matched seat. The vision sensor 250 can include an image processor that performs image processing such as noise removal, color reproduction, file compression, image quality adjustment, saturation adjustment, or the like on an image obtained through the image sensor.

[0051] The storage device 260 can be installed at the processor 270 and store software programmed to perform a preset operation. The storage device 260 can store map data, and the storage device 260 can store an image obtained by the vision sensor 250. The storage device 260 can be implemented as at least one of a storage medium (recording medium) such as at least one of a flash memory, a hard disk, a secure digital card (SD card), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an electrically erasable programmable ROM (EEPROM), an erasable programmable ROM (EPROM), or a register.

[0052] The processor 270 controls overall operations (e.g., autonomous driving) of the vehicle 200. The processor 270 can be implemented as at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a microcontroller, or a microprocessor.

[0053] The processor 270 receives vehicle call information through the communication device 240. The vehicle call information includes a call location (a call place), a drop-off location, a pickup time, a number of people to be picked up, whether the user agrees to be photographed, payment information, or information about a person who makes a reservation. The processor 270 can determine a photographing time point based on whether the user agrees to be photographed in the vehicle call information. In other words, when the person who makes a reservation (the user) agrees to be photographed, the processor 270 can determine a time point immediately after getting on the vehicle as a photographing time point when getting on the vehicle (a first photographing time point), and determine a time point immediately before getting off the vehicle as a photographing time point when getting off the vehicle (a second photographing time point). In addition, when the person who makes a reservation (the user) does not agree to be photographed, the processor 270 can determine a time point immediately before getting on the vehicle as the first photographing time point, and determine a time point immediately after getting off the vehicle as the second photographing time point. The time point immediately before or after an event can mean that the time point is within a predetermined time period before or after the event. The predetermined time period can be preset by a manufacturer of the vehicle, or can be set by an administrator of the vehicle.

[0054] The processor 270 sets the call location included in the vehicle call information as a destination, and searches for a driving route to the destination based on a vehicle location measured by the location measuring device 210 and map data stored in the storage 260. The processor 270 controls autonomous driving of the vehicle 200 along the found driving route. While the vehicle 200 is autonomously driven, the processor 270 compares the vehicle location measured by the location measuring device 210 with the call location. The processor 270 determines whether the vehicle 200 has arrived at the call location based on a comparison result of the vehicle location and the call location. In addition, when the vehicle 200 arrives at the call location, the processor 270 determines whether the vehicle 200 has arrived at the call location by determining whether the vehicle 200 is parked based on vehicle speed or brake state information. When the vehicle 200 is parked at the call location, the processor 270 can determine whether the person who makes a reservation is around the vehicle 200 through wireless communication with the user terminal 100.

[0055] When it is identified that the person making the reservation is around the vehicle 200, the processor 270 determines whether the passenger is before getting on. Based on manipulation information of a door handle or a door opening button (i.e., door state information), the processor 270 can sense a time point immediately before the passenger gets on (before the passenger gets on). Also, the processor 270 can determine whether the passenger is before getting on using the passenger information obtained by the passenger sensor 230. When the passenger is before getting on, the processor 270 operates the vision sensor 250 to photograph an image of the inside of the vehicle 200 (first vehicle interior image). When there is another occupant (passenger) in the vehicle 200 before the passenger gets on, the processor 270 occludes, blurs, or deletes a region in the first vehicle interior image that matches a ride position of the corresponding other occupant. When the passenger agrees to be photographed, the processor 270 obtains the first vehicle interior image through the vision sensor 250 after the passenger gets on the vehicle 200. In this regard, the processor 270 selectively obtains an image of a seating space (ride space) of the passenger who gets on (first vehicle interior image). In other words, when there is another passenger in the vehicle 200 in addition to the passenger who gets on, the processor 270 can protect the personal information of the other passenger by blurring or deleting a region in the first vehicle interior image photographed by the vision sensor 250 that matches a ride position of the corresponding other passenger. The processor 270 transmits the first vehicle interior image to the server 300 through the communication device 240.

[0056] When the passenger gets on the vehicle 200, the processor 270 identifies a ride position of the passenger or the number of passengers who get on through the passenger sensor 230. For example, based on information provided by at least one of a weight sensor, a vision sensor, a camera, an ultrasonic sensor, or a radar, the processor 270 can sense the passenger who gets on the vehicle 200. The processor 270 can determine whether the passenger who gets on is the person making the reservation (caller). For example, the processor 270 can receive user identification information from the user terminal 100 through Bluetooth, NFC, etc., and compare the received user identification information with information about the person making the reservation in the vehicle call information to identify the occupant.

[0057] When the passenger has gotten on the vehicle 200, the processor 270 sets a travel route by setting a drop-off position as a destination based on the vehicle call information. Also, the processor 270 controls the vehicle 200 to automatically travel to the destination along the set travel route. The processor 270 can use one of known automatic driving technologies.

[0058] When the vehicle 200 arrives at the destination, the processor 270 performs a service use end procedure of the user (the passenger who gets off). Based on the vehicle position measured by the position measuring device 210, the map data stored in the storage device 260, and the destination information of the vehicle call information, the processor 270 determines whether the vehicle 200 has arrived at the destination. In addition, the processor 270 determines whether the vehicle 200 is parked based on the vehicle speed or the brake state information. The processor 270 can identify information about the position of the occupant in the vehicle 200 through the passenger sensor 230 to determine whether the passenger has gotten off and identify the position (seat) where the passenger gets off or the number of people who get off. In other words, based on whether the destination has been reached, whether the vehicle is parked, the occupant position information, or whether the payment of the service fee is completed, the processor 270 can determine the end of the service use.

[0059] Upon arrival at the destination, upon completion of the payment of the service fee, at a second photographing time point based on whether the passenger who gets off agrees to be photographed, the processor 270 photographs an internal image of the vehicle 200 (a second vehicle internal image) by operating the vision sensor 250. When the passenger who gets off agrees to be photographed, the processor 270 obtains the second vehicle internal image through the vision sensor 250 immediately before the passenger gets off. When the passenger who gets off does not agree to be photographed, the processor 270 obtains the second vehicle internal image through the vision sensor 250 after the passenger gets off. Based on at least one of the door state information, the external button signal, the information provided by the weight sensor, the information provided by the ultrasonic sensor, the information provided by the radar, and the information provided by the vision sensor, the processor 270 can sense a time point immediately after the passenger gets off. When photographing the second vehicle internal image, if there is another passenger in the vehicle 200 in addition to the passenger who gets off, the processor 270 can protect the personal information of the other passenger by blurring or deleting an area in the image photographed by the vision sensor 250 that matches the ride position of the corresponding other passenger. The processor 270 transmits the photographed second vehicle internal image to the server 300.

[0060] When a request for a ride reservation of the vehicle 200 (i.e., use of the shared mobility service) is received from the user terminal 100, the server 300 allocates the vehicle 200, generates vehicle allocation information based on the allocation of the vehicle, and transmits the generated vehicle allocation information to the user terminal 100. The user terminal 100 displays the vehicle allocation information on the display screen so that the user can identify the vehicle allocation information. The vehicle allocation information includes the license plate number of the allocated vehicle, the vehicle model, etc. The server 300 generates vehicle call information based on the ride reservation information received from the user terminal 100 and transmits the vehicle call information to the allocated vehicle 200.

[0061] When the first and second vehicle interior images transmitted by the vehicle 200 are received, the server 300 stores the first and second vehicle interior images in a database (DB). The vehicle 200 can transmit the first and second vehicle interior images immediately after the time points at which the first and second vehicle interior images are respectively photographed, or can transmit the first and second vehicle interior images simultaneously after being photographed.

[0062] The server 300 compares and analyzes the first and second vehicle interior images to determine (sense) whether there is a forgotten item in the vehicle. In this regard, the forgotten item can be a passenger's belongings, trash, or garbage. When the sensed forgotten item is the belongings of a passenger who has gotten off, the server 300 transmits forgotten item information including a situation in which the forgotten item is sensed, a kind of the forgotten item, a picture of the forgotten item, or a method of finding the forgotten item, etc. to the user terminal 100 of the passenger who has just gotten off. At this time, the server 300 can also transmit the forgotten item information to the service operator. In addition, the server 300 can register the forgotten item information in a lost and found place. Only when the sensed forgotten item is trash or garbage, the server 300 can transmit a notification requesting to dispose of the trash or garbage to the service operator.

[0063] Figure 3 FIG. 1 is a flowchart illustrating a method for monitoring in a vehicle according to an embodiment of the present application.

[0064] Referring to Figure 3 , the vehicle 200 receives a vehicle call transmitted from the server 300 (S110). The user terminal 100 can request the server 300 to share the use of a travel service based on a user input. In other words, the user can connect to the server 300 through the user terminal 100 to make a ride reservation for a shared vehicle. When making a ride reservation for a shared vehicle, the user terminal 100 transmits ride reservation information to the server 300. The server 300 allocates the vehicle 200 based on the request (i.e., the ride reservation information) of the user terminal 100, and transmits vehicle allocation information (e.g., a license plate number, a vehicle model, etc.) to the user terminal 100. In addition, the server 300 generates vehicle call information based on the ride reservation information, and transmits the vehicle call information to the allocated vehicle 200.

[0065] When the call position of the user (the person to be picked up) is reached (S120), the vehicle 200 photographs a first vehicle interior image using the vision sensor 250 when the user gets on the vehicle 200 (S130). The vehicle 200 reaches the call position and then stops, and subsequently photographs the first vehicle interior image at a first photographing time point. The vehicle 200 can determine the first photographing time point through the detector 220 and the passenger sensor 230. When there is another passenger in the vehicle 200, the vehicle 200 can protect the personal information of the other passenger by blurring or deleting a region matching the corresponding other passenger in the first vehicle interior image photographed by the vision sensor 250. In addition, when the passenger gets on the vehicle 200, the vehicle 200 can determine whether the corresponding passenger is the person who made the reservation through wireless communication with the user terminal 100 of the passenger.

[0066] The vehicle 200 transmits the photographed first vehicle interior image to the server 300 (S140). The server 300 receives the first vehicle interior image and stores the received first vehicle interior image in a database (DB) (S145).

[0067] When the passenger gets on (rides in) the vehicle 200, the vehicle 200 travels along a travel route to a destination (S150). When the passenger gets on the vehicle 200, the vehicle 200 searches for a travel route to the destination based on the position measuring device 210 and map data, and automatically travels along the found travel route.

[0068] When the destination is reached (S160), the vehicle 200 photographs a second vehicle interior image (S170). The vehicle 200 can determine whether the destination has been reached using the position measuring device 210 and map data. In addition, the vehicle 200 can determine whether the vehicle is stopped based on vehicle speed or brake information. When the stop position corresponds to the destination, the vehicle 200 determines that the destination has been reached. When the destination is reached, the vehicle 200 photographs the second vehicle interior image at a second photographing time point. The vehicle 200 can determine the second photographing time point through the detector 220 and the passenger sensor 230. When there is another passenger in the vehicle 200 in addition to the passenger who gets off, the vehicle 200 blurs or deletes a region matching the corresponding other passenger in the second vehicle interior image photographed by the vision sensor 250.

[0069] The vehicle 200 transmits the photographed second vehicle interior image to the server 300 (S180). The server 300 receives the second vehicle interior image and stores the received second vehicle interior image in a database (DB) (S185). In the present embodiment, it is described that the vehicle 200 transmits the first vehicle interior image and the second vehicle interior image to the server 300 immediately when the first vehicle interior image and the second vehicle interior image are obtained, but the present application is not limited thereto. The vehicle 200 can be implemented to transmit the first vehicle interior image and the second vehicle interior image to the server 300 simultaneously after both the first vehicle interior image and the second vehicle interior image are obtained.

[0070] The server 300 compares and analyzes the first vehicle interior image with the second vehicle interior image to determine (sense) whether there is a forgotten item in the vehicle (S190). In this regard, the forgotten item can be a passenger's belongings, a waste, or a litter.

[0071] When the forgotten item in the vehicle is sensed, the server 300 transmits forgotten item information to the user terminal 100 of the passenger who gets off (S200). When the sensed forgotten item is the belongings of the passenger who gets off, the server 300 transmits the forgotten item information including the situation in which the forgotten item is sensed, the kind of the forgotten item, a picture of the forgotten item, or a method of finding the forgotten item, etc. to the user terminal 100 of the passenger who gets off (the owner of the forgotten item). In addition, the server 300 can register the forgotten item information in a lost and found place such as a DB. When the sensed forgotten item is a waste or a litter, the server 300 can transmit a notification requesting to process the waste or the litter to a service operator. In the present embodiment, it is described that the notification is transmitted to the service operator only when the forgotten item in the vehicle is a waste or a litter. However, it can also be implemented to transmit the forgotten item information to the service operator and the user terminal 100 of the passenger who gets off simultaneously when the forgotten item in the vehicle (i.e., at least one of the passenger's belongings, the waste, and the litter) is sensed.

[0072] Figure 4 and Figure 5 are exemplary views for describing an image processing method related to the present application. The present embodiment describes a method for photographing and processing (screening) a vehicle interior image to protect another passenger's personal information in the case where another passenger is present in the vehicle when the passenger gets off.

[0073] Reference Figure 4In a state in which six passengers (i.e., passenger 1, passenger 2, passenger 3, passenger 4, passenger 5, and passenger 6) are riding in the vehicle 200, when the passenger 6 gets off the vehicle 200, the vehicle 200 photographs an inside image of the vehicle through the vision sensor 250. In this regard, the vision sensor 250 can block the passenger 1, the passenger 2, the passenger 3, the passenger 4, and the passenger 5, thereby selecting only the space information of the passenger 6's ride position.

[0074] Referring to Figure 5 In a state in which six passengers are riding in the vehicle 200, when three passengers (i.e., the passenger 2, the passenger 5, and the passenger 6) get off the vehicle 200, the vehicle 200 determines whether there are other passengers in the vehicle. When there are other passengers in the vehicle, the vehicle 200 identifies the ride positions of the corresponding passengers (i.e., the passenger 1, the passenger 3, and the passenger 4). The vehicle 200 blocks the areas corresponding to the ride positions of the other passengers (the other passengers being identified from the inside image of the vehicle obtained through the vision sensor 250).

[0075] Figure 6 is a block diagram illustrating a computing system that performs a method for handling a forgotten item in an autonomous vehicle according to an embodiment of the present application.

[0076] Referring to Figure 6 The computing system 1000 can include at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage 1600, and a network interface 1700, which are connected through a bus 1200.

[0077] The processor 1100 can be a central processing unit (CPU) or a semiconductor device that processes commands (the commands being stored in the memory 1300 or the storage 1600). The memory 1300 and the storage 1600 can include various types of volatile or non-volatile storage media. For example, the memory 1300 can include a ROM (Read Only Memory) 1310 and a RAM (Random Access Memory) 1320.

[0078] Accordingly, the operations of the method or algorithm described with respect to the embodiments of the present application can be embodied in hardware, in software, or in a combination of hardware and software. The software can reside in a storage medium (e.g., a memory such as a RAM, a flash memory, a ROM, an EPROM, an EEPROM, a register, a hard disk, a removable disk, a CD-ROM, etc.). The storage medium can be coupled to the processor 1100 such that the processor 1100 can read information from, and write information to, the storage medium. In another embodiment, the storage medium can be integral to the processor 1100. The processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). The ASIC can reside in a user terminal. In another embodiment, the processor and the storage medium can reside as discrete components in a user terminal.

[0079] The above description is merely illustrative of the technical idea of the present application, and those skilled in the art can make various modifications and changes without departing from the essential characteristics of the present application. Therefore, the embodiments disclosed in the present application are not intended to limit the technical idea of the present application, but to explain the present application, and the scope of the technical idea of the present application is not limited by the embodiments. The scope of the present application should be interpreted as including the scope of the appended claims, and all technical ideas falling within the scope of the claims should be understood to be included in the scope of the present application.

[0080] According to the present application, the vehicle interior images are obtained before the passenger gets on the vehicle and after the passenger gets off the vehicle and compared with each other, so that the forgotten article in the vehicle can be sensed by monitoring the vehicle interior, and violation of the privacy rule is prevented.

[0081] In addition, according to the present application, when the passenger gets on or off the vehicle, if there is another occupant in the vehicle, the area where the other occupant is located is blocked, so that the personal information of the other occupant can be protected.

[0082] In addition, according to the present application, when the forgotten article in the vehicle is sensed, the information of the forgotten article is provided to the owner of the sensed forgotten article. Therefore, the forgotten article can be handled so that the shared mobility service can be operated without a person.

[0083] Although the present application has been described with reference to the example embodiments and the accompanying drawings, the present application is not limited thereto, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application claimed by the appended claims.

Claims

1. A system for processing a forgotten item in an autonomous vehicle, the system comprising: a vehicle including a vision sensor configured to photograph a first vehicle interior image and a second vehicle interior image at a first photographing time point and a second photographing time point, respectively, the vehicle configured to transmit the photographed first vehicle interior image and the second vehicle interior image to a server; and a server configured to compare the first vehicle interior image with the second vehicle interior image, and transmit forgotten item information to a user terminal associated with a passenger when a forgotten item in the vehicle is sensed; wherein the vehicle is configured to: determine a time point immediately after getting on as the first photographing time point and a time point immediately before getting off as the second photographing time point when the passenger agrees to photographing; determine a time point immediately before getting on as the first photographing time point and a time point immediately after getting off as the second photographing time point when the passenger does not agree to photographing; determine the time point immediately before getting on based on door status information; determine the time point immediately after getting off based on at least one of door status information, an external button signal, information provided by a weight sensor, information provided by an ultrasonic sensor, information provided by a radar, and information provided by the vision sensor. the vehicle is configured to blur or delete an area in the first vehicle interior image that matches a ride position of another passenger when there is another passenger riding in the vehicle at the first photographing time point.

2. The system for handling a forgotten item in an autonomous vehicle of claim 1, wherein, the vehicle is configured to blur or delete an area in the second vehicle interior image that matches a ride position of another passenger when there is another passenger riding in the vehicle at the second photographing time point.

3. The system for handling a forgotten item in an autonomous vehicle of claim 1, wherein, the vehicle is configured to determine whether the vehicle has arrived at a call location based on a vehicle location.

4. The system for handling a forgotten item in an autonomous vehicle of claim 1, wherein, the vehicle is configured to determine whether the vehicle has arrived at a call location further based on at least one of a vehicle speed and / or a brake status information.

5. The system for handling a forgotten item in an autonomous vehicle of claim 4, wherein, the vehicle is configured to sense a passenger based on information provided by at least one of a weight sensor, a vision sensor, a camera, an ultrasonic sensor, and / or a radar.

6. The system for handling a forgotten item in an autonomous vehicle of claim 1, wherein, the vehicle is configured to determine an end of service for a passenger based on completion of a service fee payment and a passenger position information.

7. The system for handling a forgotten item in an autonomous vehicle of claim 1, wherein, the server is configured to transmit forgotten item information to a business operator when a forgotten item in the vehicle is sensed.

8. The system for handling a forgotten item in an autonomous vehicle of claim 1, wherein, the forgotten item in the vehicle is at least one of a passenger's belongings, trash, and / or waste.

9. The system for handling a forgotten item in an autonomous vehicle of claim 8, wherein, 10.A method for processing a forgotten item in an autonomous vehicle, the method comprising: a first step of receiving, by a server, a first vehicle interior image and a second vehicle interior image photographed by a vision sensor at a first photographing time point and a second photographing time point, respectively; a second step of comparing, by the server, the first vehicle interior image with the second vehicle interior image to sense a forgotten item in the vehicle; a third step of transmitting, by the server, forgotten item information to a user terminal associated with a passenger; ​ wherein, when the passenger agrees to the photographing, a time point immediately after the boarding is determined as the first photographing time point, and a time point immediately before the alighting is determined as the second photographing time point; when the passenger does not agree to the photographing, a time point immediately before the boarding is determined as the first photographing time point, and a time point immediately after the alighting is determined as the second photographing time point; determining the time point immediately before the boarding based on the door status information; determining the time point immediately after the alighting based on at least one of the door status information, the external button signal, information provided by the weight sensor, information provided by the ultrasonic sensor, information provided by the radar, and information provided by the vision sensor.

11. The method of claim 10, wherein, The first step includes: when there is another passenger in the vehicle at the first photographing time point, blurring or deleting a region in the first vehicle interior image that matches a boarding position of the other passenger.

12. The method of claim 10, wherein, The first step includes: when there is another passenger in the vehicle at the second photographing time point, blurring or deleting a region in the second vehicle interior image that matches a boarding position of the other passenger.

13. The method of claim 10, wherein, The first step includes: determining, by the vehicle, whether the vehicle has arrived at the call location based on the vehicle position.

14. The method of claim 13, wherein, The first step further includes: further determining, by the vehicle, whether the vehicle has arrived at the call location based on at least one of the vehicle speed and / or the brake status information.

15. The method of claim 10, wherein, The first step includes: sensing, by the vehicle, the passenger based on information provided by at least one of the weight sensor, the vision sensor, the camera, the ultrasonic sensor, and / or the radar.

16. The method of claim 10, wherein, The first step includes: determining, by the vehicle, the end of the service for the passenger based on the completion of the service fee payment and the crew position information.

17. The method of claim 10, wherein, The first step further includes: when a forgotten item in the vehicle is sensed, transmitting, by the server, the forgotten item information to the business operator.

18. The method of claim 17, wherein, The forgotten item in the vehicle is at least one of the passenger's belongings, garbage, and / or waste.

Images