System and method for sharing vehicle cleanliness detection

Abstract

Systems and methods for shared vehicle cleanliness detection. A method of operating a vehicle having a passenger compartment with a plurality of interior sections includes generating image data for each of the plurality of interior sections with an imaging device positioned within the passenger compartment. The method also includes processing the generated image data with a controller operably connected to the imaging device to generate score data including a plurality of scores, each of the plurality of scores corresponding to one of the plurality of interior sections, and merging the generated scores into a single vehicle cleanliness score. The method further includes generating availability data indicating that the vehicle is available or unavailable based on a comparison of the cleanliness score to a cleanliness threshold, and removing the vehicle from service based on the availability data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of shared mobility, and more particularly to using computer vision to determine the relative cleanliness of a vehicle passenger compartment. BACKGROUND

[0002] Shared mobility refers to the shared use of vehicles, bicycles, or other modes of transportation. In the case of shared vehicles, vehicles are provided to users on an as-needed basis. For example, according to a round-trip model, a user selects a vehicle from an initial location, drives the vehicle to a destination, and then returns the vehicle to the initial location. According to a one-way model, a user selects a vehicle from an initial location, drives the vehicle to a destination, and then leaves the vehicle at a location at the destination. There are many other models for shared vehicle use.

[0003] Vehicles in a shared vehicle program are exposed to many users in use. Some users leave the vehicle interior in a substantially clean state, while other users leave the vehicle interior in a dirty state, either accidentally or intentionally. Moreover, even when used by conscientious customers, shared vehicle interiors become progressively dirtier over time, in the same way that non-shared vehicle (i.e., personal vehicle) interiors become progressively dirty or soiled.

[0004] When the operator of a personal vehicle notices that the vehicle interior has become unacceptably dirty, the operator arranges for the vehicle to be cleaned or cleans the vehicle interior themselves. However, for shared vehicles, the operator / user of the vehicle is generally not responsible for ensuring that the vehicle is in a clean state. Instead, agents of a mobility company are responsible for ensuring that the vehicle interior is in a clean state. Problematically, agents of a mobility company are remote from the shared vehicles, resulting in in-person visual inspections of the vehicles being both time-consuming and expensive.

[0005] Due at least to the foregoing reasons, there is a need for further development in the field of shared mobility to optimize the process used by mobility providers in determining the cleanliness of shared vehicle interiors. SUMMARY

[0006] According to an exemplary embodiment of the present disclosure, a method is for operating a vehicle, the vehicle including a passenger compartment having a plurality of interior sections. The method includes generating image data for each of the plurality of interior sections with an imaging device located within the passenger compartment. The method further includes processing the generated image data with a controller operably connected to the imaging device to generate score data including a plurality of scores, each of the plurality of scores corresponding to one of the plurality of interior sections, and merging the generated scores into a single vehicle cleanliness score. The method also includes generating availability data indicating that the vehicle is available or unavailable based on a comparison of the cleanliness score to a cleanliness threshold, and removing the vehicle from service when the availability data indicates that the vehicle is unavailable.

[0007] According to another example embodiment of the present disclosure, an apparatus is for monitoring a passenger compartment of a vehicle, the passenger compartment comprising a plurality of interior sections. The apparatus comprises an imaging device and a controller. The imaging device is located within the passenger compartment and is configured to generate image data for each of the plurality of interior sections. The controller is operably connected to the imaging device. The controller is configured to process the generated image data to generate score data comprising a plurality of scores, each of the plurality of scores corresponding to one of the plurality of interior sections, and to combine the generated scores into a single vehicle cleanliness score. The controller is further configured to generate availability data indicating that the vehicle is available or unavailable based on a comparison of the cleanliness score to a cleanliness threshold, and to remove the vehicle from a shared vehicle service when the availability data indicates that the vehicle is unavailable. BRIEF DESCRIPTION OF DRAWINGS

[0008] The above features and advantages and other features that will be become apparent to one of ordinary skill in the art from the following detailed description and the accompanying drawings.

[0009] Figure 1 is a block diagram of a shared vehicle mobility system comprising a vehicle having an apparatus for automatically monitoring vehicle cleanliness and a base station;

[0010] Figure 2 depicts a passenger compartment of a vehicle imaged by an imaging device of the apparatus of Figure 1 ;

[0011] Figure 3 depicts the apparatus installed in the vehicle of Figure 1 ;

[0012] Figure 4 is a plot of an occlusion metric data generated by the system of Figure 3 ;

[0013] Figure 5A depicts a driver seat and a driver seat section of the vehicle of Figure 1 that is free of any non-vehicle elements;

[0014] Figure 5B depicts a driver seat and a driver seat section of the vehicle of Figure 1 that includes a first non-vehicle element;

[0015] Figure 5C depicts a driver seat and a driver seat section of the vehicle of Figure 1 that includes a plurality of non-vehicle elements;

[0016] Figure 6 is a plot of a cleanliness score data generated by the system of Figure 1The graph of weighted occlusion metric data generated by the system;

[0017] Figure 7 It is by Figure 1 The system generates graphs of the tracked cleanliness score data and change data; and

[0018] Figure 8 It describes the operation. Figure 1 A flowchart of an exemplary method for a shared vehicle mobility system. Detailed Implementation

[0019] For the purpose of promoting an understanding of the principles of this disclosure, reference will now be made to the embodiments shown in the accompanying drawings and the embodiments described in the following written description. It should be understood that this is not intended to limit the scope of the disclosure. It should also be understood that this disclosure includes any changes and modifications to the illustrated embodiments, and includes broader applications of the principles of this disclosure that would commonly occur to those skilled in the art to which this disclosure pertains.

[0020] Various aspects of this disclosure are disclosed in the accompanying specification. Alternative embodiments and equivalents of the invention can be devised without departing from the spirit or scope of the invention. It should be noted that any discussion herein with reference to “one embodiment,” “embodiment,” “exemplary embodiment,” etc., indicates that the described embodiment may include specific features, structures, or characteristics, and not necessarily every embodiment includes such specific features, structures, or characteristics. Furthermore, references to the foregoing do not necessarily include references to the same embodiments. Finally, whether explicitly described or not, those skilled in the art will readily understand that each of the specific features, structures, or characteristics of a given embodiment can be utilized in combination with or in combination with any other embodiments discussed herein.

[0021] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0022] The terms “comprising,” “including,” “having,” etc., used in relation to embodiments of this disclosure are synonymous.

[0023] like Figure 1As shown, the shared vehicle mobility system 100 includes a vehicle 104 and a base station 108 in electronic wireless communication with the Internet 112. The vehicle 104 includes a device 116 for automatically monitoring the cleanliness of the interior of the vehicle 104. The base station 108 processes data generated by the device 116 to determine when the vehicle 104 should be removed from service for inspection or cleaning. The device 116 also determines which of a plurality of users of the vehicle 104 is responsible for a significant change in the cleanliness level so that the ride company can take appropriate action, such as assessing a cleaning fee against the user. Each element of the shared mobility system 100 is described below.

[0024] In one embodiment, the exemplary vehicle 104 is a shared vehicle 104, which can be either passenger-controlled (i.e., from level zero to level four autonomous control) or fully autonomously controlled (i.e., level five autonomous control). In other embodiments, the vehicle 104 is a rental vehicle, a shuttle, a limousine, a company vehicle, a valet vehicle, a taxi, a personal vehicle, or a robotic taxi. The vehicle 104 is any vehicle that carries a person. The vehicle 104 can be included in a fleet of shared vehicles 104, which includes a plurality of vehicles 104.

[0025] In Figure 2 the vehicle 104 is part of a car sharing program and is at least partially controlled by a driver. In Figure 2 the device 116 is not shown because Figure 2 the view provided is generated by the imaging device 238 of the device 116. The vehicle 104 includes an interior vehicle passenger compartment 120 having a driver seat 124, a passenger seat 128, a back seat 132, a center console 136, a driver floor 140, a passenger floor 144, an instrument panel 148, a driver door 152, a passenger door 156, and a steering wheel 160. The driver seat 124 includes a seat bottom 164 and a seat back 168. The driver seat 124 is movable within the passenger compartment 120 relative to the device 116 to accommodate drivers, users, and operators of different sizes. The passenger seat 128 includes a seat bottom 172 and a seat back 176. The passenger seat 128 is also movable within the passenger compartment 120 relative to the device 116 to accommodate users and operators of different sizes. The back seat 132 is generally in a fixed position relative to the device 116.

[0026] The driver floor 140 is in front of the seat bottom 164 of the driver seat 124 and is the area where the driver places his feet while operating the vehicle 104. The passenger floor 144 is in front of the seat bottom 172 of the passenger seat 128 and is the area where the passenger places his feet while riding in the vehicle 104.

[0027] A center console 136 is located between the driver's seat 124 and the passenger seat 128. The center console 136 provides a space for a user to rest their arm while seated in the passenger compartment. Furthermore, in some embodiments, the center console 136 includes or is located near at least one cup holder 188. An exemplary passenger compartment 120 includes two cup holders 188.

[0028] The dashboard 148 is located at the front of the vehicle 104 and includes instruments and controls for operating and / or interacting with the vehicle 104. A steering wheel 160 extends from the dashboard 148 and is used to control the direction of movement of the vehicle 104. The driver's door 152 is located near the driver's seat 124, while the passenger door 156 is located near the passenger seat 128.

[0029] like Figure 3 As shown, device 116 includes a housing 190 located in the overhead console 194 of vehicle 104, near the rearview mirror 198 of vehicle 104. Specifically, in one embodiment, device 116 is adapted to the eyeglasses frame 202 of vehicle 104 by removing the cover (not shown) of eyeglasses frame 202 and inserting the housing 190 of device 116 into a corresponding opening in eyeglasses frame 202. In one embodiment, no electrical connection is made between device 116 and vehicle 104, such that device 116 is electrically isolated from vehicle 104 and operates independently of the powertrain of vehicle 104.

[0030] Refer again Figure 1 The device 116 includes at least one visible light device 230, at least one infrared light device 234, an imaging device 238, a memory 242, and a transceiver 246, each operatively connected to the controller 250. In one embodiment, the visible light device 230 is a white light-emitting diode (“LED”) configured to emit light in the visible spectrum. In another embodiment, the visible light device 230 is any illumination device that emits light in the visible spectrum, such as an incandescent light bulb or a fluorescent light source.

[0031] In one embodiment, the infrared light device 234 is an infrared (“IR”) LED configured to emit light in the infrared spectrum. In another embodiment, the IR light device 234 is any lighting device that emits light in the IR spectrum, such as an incandescent bulb or a fluorescent light source.

[0032] Imaging device 238 is a digital imaging device or camera configured to generate image data 254 of the passenger compartment 120 of vehicle 104. An exemplary field of view of imaging device 238 is... Figure 2The field of view of imaging device 238 includes at least driver seat 124, passenger seat 128, rear seat 132, center console 136, driver floor 140, and passenger floor 144. The field of view of imaging device 238 can also include at least a portion of instrument panel 148, driver door 152, passenger door 156, and steering wheel 160. In the exemplary embodiment, only one imaging device 238 is shown. In other embodiments, device 116 includes at least two imaging devices 238. For example, a first imaging device 238 generates image data 254 of driver seat 124, passenger seat 128, center console 136, and cup holder 188; while a second imaging device 238 generates image data 254 of rear seat 132. A vehicle 100 having three rows of seats can include a third imaging device 238 to generate image data 254 of the third row of seats (not shown). The image data 254 generated by each imaging device 238 is stored as image data 254.

[0033] Transceiver 246 (also referred to as a wireless transmitter and receiver) is configured to wirelessly transmit data from vehicle 104 to another electronic device, and wirelessly receive data from another electronic device, e.g., via Internet 112. An exemplary electronic device with which transceiver 246 is in electronic communication is transceiver 292 of base station 108. Thus, transceiver 246 operably connects vehicle 104 to Internet 112 and other electronic devices, such as base station 108. In other embodiments, transceiver 246 transmits and receives data using a cellular network, a wireless local area network ("Wi-Fi"), a personal area network, and / or any other wireless network. Thus, transceiver 246 is compatible with any desired wireless communication standard or protocol, including but not limited to near field communication ("NFC"), IEEE 802.11, IEEE 802.15.1 ("Bluetooth®"), Global System for Mobiles ("GSM"), and Code Division Multiple Access ("CDMA").

[0034] Controller 250 is configured to execute program instruction data 258 in order to operate device 116. Controller 250 is provided as at least one microcontroller and / or microprocessor.

[0035] Memory 242 is configured to store at least image data 254, program instruction data 258, internal section data 268, occlusion metric data 272, weighting scale data 276, weighted occlusion metric data 278, cleanliness score data 280, and hazard condition data 284. Memory 242 is also referred to herein as a non-transitory computer readable medium.

[0036] The device 116 also includes a battery 288 that is operatively connected to at least the visible light source 230, the IR light source 234, the imaging device 238, the memory 242, the transceiver 246, and the controller 250 to provide power thereto. The battery 288 is rechargeable and can be provided as a lithium-based battery or any other battery type. Additionally or alternatively, the device 116 receives power from the powertrain system of the vehicle 104.

[0037] The base station 108 of the system 100 includes a transceiver 292 and a memory 296 that are operatively connected to a computer system 298. The transceiver 292 of the base station 108 is substantially identical to the transceiver 246 of the device 116. In particular, the transceiver 292, also referred to as a wireless transmitter and receiver, is configured to wirelessly transmit data from the base station 108 to another electronic device and to wirelessly receive data from another electronic device, e.g., via the Internet 112. An exemplary electronic device with which the transceiver 292 is in electronic communication is the transceiver 246 of the device 116. Thus, the transceiver 292 operatively connects the base station 108 to the Internet 112 and other electronic devices, such as the device 116. In other embodiments, the transceiver 292 transmits and receives data using a cellular network, a wireless local area network (“Wi-Fi”), a personal area network, and / or any other wireless network. Thus, the transceiver 292 is compatible with any desired wireless communication standard or protocol, including but not limited to near-field communication (“NFC”), IEEE 802.11, IEEE 802.15.1 (“Bluetooth®”), Global System for Mobile (“GSM”), and Code Division Multiple Access (“CDMA”).

[0038] The memory 296 of the base station 108 is an electronic storage device that is configured to store at least tracked score data 302, change data 306, and availability data 310. The memory 296 is also referred to herein as a non-transitory computer readable medium.

[0039] The computer system 298 of the base station 108 is, for example, a personal computer that is operatively connected to the Internet 112 through the transceiver 292 to receive electronic data from the device 116. The computer system 298 can include a monitor for rendering data and enabling an operator of the computer system 298 to implement changes to a fleet of vehicles 104. For example, the computer system 298 can enable the operator to identify a vehicle 104 as being in an “in-use” status, can enable the operator to identify a vehicle 104 as being in an “out-of-use” status, and can enable the operator to identify a vehicle 104 as being scheduled for cleaning.

[0040] Referring again to Figure 2The controller 250 processes image data 254 generated by the imaging device 238 to generate section data 268. The section data 268 includes portions of the image data 254 that correspond to the plurality of interior sections 318, 322, 326, 330, 334, 338 of the passenger compartment 120. For example, the section data 268 includes: driver seat section data 268 that includes image data 254 corresponding to the driver seat section 318 that includes the driver seat 124; passenger seat section data 268 that corresponds to the passenger seat section 322 that includes the passenger seat 128; back seat section data 268 that corresponds to the back seat section 326 that includes the back seat 132; console section data 268 that corresponds to the console section 330 that includes the center console 136; driver floor section data 268 that corresponds to the driver floor section 334 that includes the driver floor 140; and passenger floor section data 268 that corresponds to the passenger floor section 338 that includes the passenger floor 144. The section data 268 includes most and / or all areas of the passenger compartment 120 where a user can contact the vehicle 104, and most and / or all areas of the vehicle 104 that a user can contact with their hands.

[0041] In one embodiment, the section data 268 includes cup holder section data 268 that includes image data 254 corresponding to a cup holder section 340 that includes the cup holder 188. In embodiments that include the cup holder section 340, the cup holder section 340 is excluded from the console section 330. In embodiments that do not include the cup holder section 340, image data 254 of the cup holder 188 is included in the console section 330. It is beneficial to include the cup holder 188 in the section data 268 because the cup holder 188 is often a disposal point for trash and other non-vehicle elements 344 left in the passenger compartment 120.

[0042] The section data 268 is generated by the controller 250 of the device 116 based on the image data 254 using computer vision and image processing techniques. For example, in one embodiment, the controller 250 uses a semantic segmentation approach in which each pixel of the image data 254 is processed and provided with a class label that corresponds to one of the interior sections 318, 322, 326, 330, 334, 338. The controller 250 can perform semantic segmentation on the image data 254 using a neural network.

[0043] Device 116 is configured to accurately identify the position, size, and configuration of segments 318, 322, 326, 330, 334, and 338 for each position of movable elements of vehicle 104. For example, driver's seat 124 is movable to the foremost and rearmost positions of seat bottom 164, and seat back 168 is tilted from an upright position to a fully tilted position. For each position of driver's seat 124, controller 250 is configured to identify image data 254 corresponding to driver's seat segment 318 and store image data 254 as segment data 268. Controller 250 is similarly configured to identify the position of segments 318, 322, 326, 330, 334, and 338 for each additional movable element of vehicle 104, including passenger seat 128. Furthermore, controller 250 is configured to determine the position, size, and configuration of driver floor segment 334 and passenger floor segment 338, respectively, in response to movement of seat bottom 164 and seat bottom 172.

[0044] The controller 250 of device 116 generates occlusion measurement data 272 based on image data 254. Occlusion measurement data 272 is an exemplary metric used to generate a cleanliness score data 280. The exemplary occlusion measurement data 272 is... Figure 4 The data is drawn over a period of approximately thirty-five days. To generate occlusion measurement data 272, the controller 250 processes segment data 268 to identify pixels in segment data 268 corresponding to vehicle 104 and pixels in segment data 268 corresponding to non-vehicle elements 344. Figure 2 , Figure 5B and Figure 5C The ratio or percentage of non-vehicle element pixels to vehicle pixels is stored as occlusion measurement data 272 for segments 318, 322, 326, 330, 334, and 338. Occlusion measurement data 272 represents the percentage or ratio of the corresponding internal segments occluded by non-vehicle elements 344. The term "occlusion" is used because non-vehicle elements 344 occlude, hide, mask, block, or otherwise alter the appearance of a portion of vehicle 104, as imaged by imaging device 238. High values ​​of occlusion measurement data 272 correspond to a "dirty" passenger compartment 120 that is occluded by non-vehicle elements, while low values ​​of occlusion measurement data 272 correspond to a "clean" passenger compartment 120 that is less occluded or not occluded by non-vehicle elements 344. In other embodiments, device 116 uses any other measure to determine the cleanliness of vehicle 104 by directly processing segment data 268 and / or image data 254. In addition, multiple metrics can be combined to obtain occlusion metric data272.

[0045] As used herein, the term "non-vehicle elements" 344 includes any object or substance that is not a vehicle 104. Exemplary non-vehicle elements 344 include a cell phone, headphones, clothing, jewelry, computer equipment, toys, money, glasses, food, trash, wrapping paper, a cup, a bottle, a straw, broken glass. In addition, non-vehicle elements 344 can also be liquids, grease, dirt, grime, ash, snow, and bodily fluids, which can appear in image data 254 as stains or discolorations on the surfaces of vehicle 104.

[0046] In Figure 5A , controller 250 has identified the pixels of image data 254 that correspond to driver seat section 318, and are stored as section data 268. In addition, controller 250 has processed section data 268 of driver seat section 318 to determine that all of those pixels correspond to vehicle 104, and none of those pixels correspond to non-vehicle elements 344. As such, in Figure 5A , the occlusion metric data 272 for the driver seat section is 0%, because in Figure 5A , none of the parts of vehicle 104 are occluded by non-vehicle elements 344. However, in Figure 5B , controller 250 has processed section data 268 of driver seat section 318 to determine that some of the pixels correspond to vehicle 104, while other pixels correspond to non-vehicle elements 344, such as a folder, a wallet, a purse, trash, or a piece of paper. Controller 250 divides the portion of section data 268 that corresponds to non-vehicle elements 344 by the portion of section data 268 that corresponds to vehicle 104, to arrive at an occlusion metric of 5% in Figure 5B . In Figure 5C , a different non-vehicle element 344 is included, and that non-vehicle element 344 represents a liquid stain or clothing left on driver seat 124, which has discolored driver seat 124 as detected by controller 250 processing section data 268. In Figure 5C , even more of section 318 is occluded by non-vehicle elements 344, and the occlusion metric is calculated by controller to be 40%.

[0047] In Figure 4 , the occlusion metric data 272 is plotted for the five exemplary sections 318, 322, 326, 330, 334, 338 in Figure 2 . For example, on day 14, occlusion metric data 272 indicates that driver floor section 334 is occluded by about 18%, passenger seat section 322 is occluded by about 8.5%, center console section 330 is occluded by about 5%, and passenger floor section 338 is occluded by about 3%. Driver seat section 318 is not substantially occluded (i.e., is occluded by 0%). Later on Figure 4On the 14th day, the passenger compartment 120 is cleaned and the obscuration metric data 272 indicates that the sections 318, 322, 326, 330, 334, 338 are obscured by 0%, as expected in a cleaned vehicle 104 absent non-vehicle elements 344.

[0048] In some embodiments, the controller 250 is configured to weight the obscuration metric data 272 according to weighting scale data 276. In Figure 6 An exemplary weighted obscuration metric data 278 is plotted in FIG. 14. In one embodiment, the weighted obscuration metric data 278 is weighted based on the relative importance of the sections 318, 322, 326, 330, 334, 338 to the user experience. For example, most users prefer that the touch areas of the vehicle 104 are clean to facilitate touching, and accept some uncleanliness in portions of the vehicle 104 that do not require physical contact. The following weighting scale data 276 is based on this approach, and includes the following weighting for each section 318, 322, 326, 330, 334, 338: driver seat 30%, passenger seat 25%, driver floor 10%, passenger floor 10%, center console 15%, and back seat 10%. In Figure 6 On the 14th day, the weighted values include the driver seat section 318 with a weight of 0 (0 * 0.30), the passenger seat section 322 with a weight of 2.1 (8.5 * 0.25), the driver floor section 334 with a weight of 1.8 (18 * 0.10), the passenger floor section 338 with a weight of 0.3 (3 * 0.1), and the center console section 330 with a weight of 0.75 (5 * 0.15). It should be noted that the weights and percentages provided here are exemplary only, and can be adjusted based on customer and / or vehicle requirements.

[0049] The weighting scale data 276 weights the obscuration data 272 most heavily for the driver seat section 318, because the user will touch this portion of the vehicle 104. If the driver seat 124 is particularly dirty, the user can not even accept the vehicle 104 for use. The weight for the obscuration data 272 for the back seat section 326 is less, because the user is unlikely to touch the back seat 132 unless the user plans to transport back seat passengers.

[0050] In one embodiment, the weighted obscuration metric data 278 includes a plurality of "scores," and is also referred to herein as "score data." Thus, the weighted obscuration metric data 278 includes a score corresponding to each of the sections 318, 322, 326, 330, 334, 338. In Figure 6On the 14th day in the example, the scores include: the driver seat section 318 has a score of 0, the passenger seat section 322 has a score of 2.1, the driver floor section 334 has a score of 1.8, the passenger floor section 338 has a score of 0.3, and the center console section 330 has a score of 0.75.

[0051] Again considering Figure 6 On the 14th day in the example, after weighting the occlusion metric data 272 according to the weighting proportion data 276, the resulting weighted occlusion metric data 278 results in the passenger seat section 322 being prioritized over the driver floor section 334. This is evidenced by the peak of the weighted occlusion metric data 278 corresponding to the passenger seat section 322 being higher than the peak of the weighted occlusion metric data 278 corresponding to the driver floor section 334. The driver of the vehicle 104 is more likely to touch the passenger seat 128 with his hands than the driver floor 140 with his hands, and thus the weighting proportion data 276 takes this into account to improve the user experience. Figure 6

[0052] The values of the weighting proportion data 276 can be fully adjusted and configured by the device 116 and / or the base station 108. In particular, in at least one embodiment, the weighting proportion data 276 is seasonally and regionally adjusted. For example, in the winter months of a northern climate (i.e., a seasonal time period), the driver floor 140 and the passenger floor 144 are typically soiled with salt, mud, and snow. Whereas in the summer months (i.e., another seasonal time period), the driver floor 140 and the passenger floor 144 are typically clean and / or not as soiled. The device 116 and / or the base station 108 are configured to apply a lower weight to the driver floor 140 and the passenger floor 144 in the winter months than in the summer months, so as to not prioritize the importance of a clean driver floor 140 and a clean passenger floor 144 as much in the winter months. This is because users are more willing to accept a driver floor 140 covered in snow and salt in the winter than a dirty driver floor 140 in the summer. In one embodiment, in the summer months, the driver floor section 334 is weighted at 15% and the passenger floor section 338 is weighted at 12%, and in the winter months, the driver floor section 334 is weighted at 10% and the passenger floor section 338 is weighted at 8%.

[0053] ​The weighted proportion data 276 can also be adjusted and configured based on the number of users occupying the vehicle 104. For example, when the vehicle 104 is used by only the driver, the device 116 prioritizes or weights the driver seat section 318 by 30% and the passenger seat section 322 by 20%. The passenger seat section 322 is lower than the driver seat section 318 because no user is occupying the passenger seat 128. Whereas, when both the driver seat 124 and the passenger seat 128 are occupied, the device 116 prioritizes the driver seat by 30% and the passenger seat by 30% because both are occupied and in contact with a user. When a back seat passenger is present in the back seat 132, the same method is used to increase the weight of the back seat section 326.

[0054] In another embodiment, the weighted proportion data 276 is updated and adjusted based on customer feedback data. For example, when a user ends a ride using the vehicle 104, the user provides feedback to indicate whether the vehicle 104 was clean enough. In one embodiment, the feedback is provided through an "App" or application running on the user's phone. The application can ask the user to rate the cleanliness of each section 318, 322, 326, 330, 334, 338 on a scale of, for example, one to five. If the user regularly provides low scores to the passenger seat section 322, the weighted proportion data 276 of the passenger seat section 322 can be increased to prioritize the cleanliness of the passenger seat 128 more highly based on customer feedback. Whereas, if the user regularly provides high scores to the back seat section 326, the weight data 276 of the back seat section 326 can be decreased to deprioritize the cleanliness of the back seat in favor of other sections 318, 322, 330, 334, 338 of the vehicle 104.

[0055] With reference to Figure 7 The controller 250 generates vehicle cleanliness score data 280 based on the combination of the weighted occlusion metric data 278. In particular, the "scores" of each section 318, 322, 326, 330, 334, 338 of the weighted occlusion metric data 278 are combined (i.e., summed in one embodiment) to arrive at a single numerical value representative of the overall vehicle cleanliness score of the passenger compartment 120 and stored as the cleanliness score data 280. For example, again considering the 14thday, the cleanliness score data 280 is 4.95 (2.1 + 1.8 + 0.75 + 0.3 + 0.0). A higher score in the cleanliness score data 280 corresponds to a dirtier vehicle 104, whereas a lower score in the cleanliness score data 280 corresponds to a cleaner vehicle 104. When the vehicle 104 is completely clean, the corresponding cleanliness score data 280 is equal to 0.0.

[0056] In one embodiment, daily cleanliness score data 280 is generated by the controller 250 of device 116 and then sent to base station 108 as tracked score data 302 for long-term storage. Therefore, base station 108 tracks the cleanliness score data 280 over time. The tracked score data 302 includes cleanliness score data 280 over a predetermined time period (e.g., one month or one year). The tracked score data 302 may also include image data 254 and segment data 268 for each corresponding score of the score data 280.

[0057] like Figure 7 As shown, in one embodiment, the computer system 298 of base station 108 compares cleanliness score data 280 with a cleanliness threshold 356. The cleanliness threshold 356, for example, represents a level of cleanliness / uncleanliness at or above which vehicle 104 can be removed from service for cleaning. Figure 7 In this process, when the cleanliness score is below the cleanliness threshold of 356, vehicle 104 will not be cleaned because it is relatively clean. However, when the cleanliness score is equal to or higher than the cleanliness threshold of 356, vehicle 104 will be cleaned. Cleaning can take place on the same day as the schedule or shortly thereafter.

[0058] The level of the cleanliness threshold 356 can be adjusted according to the season and the area in which the vehicle 104 operates. For example, as mentioned above, during the winter months in northern climates, the driver's floor 140 and passenger floor 144 are typically soiled with salt, mud, and snow. During the summer months, the driver's floor 140 and passenger floor 144 are generally clean and / or generally less soiled. The cleanliness threshold 356 can be increased during the winter months to address the increased likelihood that the vehicle 104 will become soiled due to weather-related events. Alternatively, instead of increasing the cleanliness threshold 356 throughout the winter, it can be increased based on weather forecasts. For example, if a winter storm is predicted to hit a specific area, the vehicle 104 operating in that area can be evaluated with a larger cleanliness threshold 356 than the one used to evaluate vehicles 104 operating outside areas predicted to receive winter storms.

[0059] Furthermore, the cleanliness threshold 356 can be adjusted based on the area in which the vehicle 104 typically operates. For example, the base station 108 can set a higher cleanliness threshold 356 for the vehicle 104 operating in another environment compared to the cleanliness threshold 356 for the vehicle 104 operating in one environment.

[0060] The level of the cleanliness threshold 356 can be set based on any other factors, including the relative preferences of users in one city versus users in another city. For example, it can be determined that users of vehicles 104 in a first city accept a greater degree of uncleanness than users of vehicles 104 in a second city. The cleanliness threshold 356 can be adjusted differently and independently for each city.

[0061] The base station 108 processes the tracked score data 302 to generate change data 306 and availability data 310. Exemplary change data 306 is plotted in Figure 7 and is the percent change in cleanliness score data 280 compared to at least one previous cleanliness score data 280. Typically, each use or service run of a vehicle 104 produces a data point of cleanliness score data 280, which is why Figure 7 there are multiple cleanliness scores for certain days in FIG. 3. The change data 306 represents the percent change in the cleanliness score of the vehicle from one user to the next or from one vehicle use to the next by the same customer. For example, the cleanliness score data 380 for five consecutive uses of a vehicle 104 is 0.10, 0.15, 1.20, 1.25, and 2.0. The change data 306 starting after the second use is 50% ((0.15-0.10) / 0.10), 700% ((1.20-0.15) / 0.15), 4% ((1.25-1.20) / 1.20), and 60% ((2.0-1.25) / l.25), respectively.

[0062] The change data 306 is useful for determining use events of the vehicle 104 that result in significant additional uncleanness within the passenger compartment 120. In most aspects, it is assumed that the vehicle 104 will gradually become unclean, as shown by the cleanliness score data 280 from day 4 to day 9 in Figure 7 . After each use from day 4 to day 9, there is a small increase in uncleanness (i.e., more obscuration) and it is expected, and the change data 306 from day 4 to day 9 is relatively flat. However, on day 10, the cleanliness score data 280 peaks with an increase of about 2.5 points and causes a corresponding peak in the change data 306. If desired, the user of the vehicle 104 on day 10 can be identified by the base station 108 and that user can be rated as being responsible for significantly soiling the vehicle 104, as detected automatically by the system 100.

[0063] In one embodiment, the change data 306 is compared to a change threshold 364. An increase in the change data 306 above the change threshold 364 indicates that the user greatly contributed to the soiling of the passenger compartment 120 of the vehicle 104, while when the change data 306 is positive and below the change threshold 364, it indicates that the user only slightly increased the soiling of the passenger compartment 120. When the change data 306 is negative, it indicates that the user cleaned the passenger compartment 120 by, for example, removing non-vehicle elements 344 from the passenger compartment 120.

[0064] The availability data 310 generated by the computer system 298 corresponds, for example, to the availability of the vehicle 104 for providing a shared vehicle service. When the vehicle 104 is not cleaned or not serviced, the vehicle 104 is identified in the availability data 310 as available for providing a shared vehicle service. When the vehicle 104 is cleaned or serviced, the vehicle 104 is identified in the availability data 310 as not available for providing a shared vehicle service. The availability data 310 enables the mobility service provider to quickly determine the status of the vehicle 104 as available or not available for a shared vehicle service.

[0065] Referring again to Figure 1 In some embodiments, the device 116 generates hazard condition data 284 based on the image data 254. In addition to processing the image data 254 to generate the occlusion data 272, the controller 250 processes the image data 254 to detect the presence of any hazardous conditions present in the passenger compartment 120. If a hazardous condition is detected by the device 116, the corresponding image data 254 is stored as hazard condition data 284 and uploaded or transmitted to the base station 108, which configures the availability data 310 to identify the vehicle 104 as "not available". In this way, the system 100 automatically prevents users from encountering hazards that are automatically detected within the passenger compartment 120.

[0066] The controller 250 uses object detection, image processing, and computer vision techniques to identify hazardous non-vehicle elements 344. Other identifiable hazardous non-vehicle elements 344 include changes to the passenger compartment 120 such as cuts, scratches, scrapes, holes, and broken glass, all of which change the appearance of the passenger compartment 120 in the image data 254. In addition, in some embodiments, identifiable hazardous non-vehicle elements 344 include biological hazards such as blood, vomit, urine, and feces. The hazard condition data 284 is stored in the memory 242 and transmitted to the base station 108 for processing. When the system 100 detects a hazardous non-vehicle element 344, the availability data 310 is typically updated to identify the vehicle 104 as "not available".

[0067] Referring to Figure 8A method 800 for operating the system 100 is illustrated by the flowchart. The method 800 uses the device 116 to automatically detect the cleanliness of the vehicle 104 and update the availability data 310 accordingly. The method 800 is different from other methods of detecting forgotten or left behind items within the passenger compartment 120. Specifically, when users utilize the service of the shared vehicle 104, some users return the vehicle 104 in the same condition, with no trash and other non-vehicle elements 344 in the passenger compartment 120, while other users leave the passenger compartment 120 in a dirty condition, with many non-vehicle elements 344 in the passenger compartment 120. The method 800 determines the extent of non-vehicle elements 344 (i.e., typically trash and debris) that are present in the passenger compartment 120 at the end of a use event of the vehicle 104. Thus, the method 800 tends to ensure that the vehicle 104 is in a clean condition that is acceptable to the next user. This method 800 is described in greater detail below.

[0068] At block 804, the method 800 includes determining whether the vehicle 104 has ended a use event. Exemplary use events include a user completing a one-way or two-way trip with the vehicle 104, parking the vehicle 104, turning off the vehicle 104, and exiting the passenger compartment 120. If the use event has not ended, the method 800 loops back and continues to monitor whether the use event has ended. When the device 116 determines that the use event has ended, the method 800 moves to block 808. In one embodiment, the device 116 determines that the use event has ended by processing the image data 254 to determine that no one is in the passenger compartment 120. In other embodiments, the base station 108 sends an electrical signal to the device 116 to indicate that the use event has ended and that the device 116 should determine the cleanliness of the passenger compartment 120.

[0069] At block 808 of the method 800, the device 116 is configured to image the passenger compartment 120 of the vehicle 104 and generate image data 254. The image data 254 is stored to the memory 242 of the device 116 and represents the condition of the vehicle 104 at the end of the use event as released by the user. Essentially, at the end of each use of the vehicle, the device 116 "takes a picture" of the passenger compartment 120 and stores the picture as image data 254 in the memory 242. The image data 254 includes data for each interior section 318, 322, 326, 330, 334, 338.

[0070] Next, at block 812, the controller 250 of the device 116 is configured to process the image data 254 to search for any dangerous non-vehicle elements 344. As described above, if a dangerous non-vehicle element 344 is identified, the controller 250 stores at least a portion of the image data 254 as dangerous condition data 284.

[0071] At block 816 of the method 800, if the device 116 identifies a potentially dangerous non-vehicle element 344, the vehicle 104 is removed from service at block 820. In particular, at block 820, the system 100 uses the transceivers 246, 292 to transmit the hazard condition data 284 to the base station 108. When the base station 108 receives the transmitted hazard condition data 284, the computer system 298 updates the availability data 310 to indicate that the vehicle 104 is not available for service, thereby preventing other users from encountering the dangerous or potentially dangerous non-vehicle element 344.

[0072] Further, at block 820, if the vehicle 104 is a fully autonomous vehicle (i.e., a level 5 autonomous vehicle), instructions are transmitted to the vehicle 104 to cause the vehicle 104 to drive itself to a safe location for disposal of the dangerous non-vehicle element 344. If the vehicle 104 is not a fully autonomous vehicle, a human can use the computer system 298 to review a presentation of the hazard condition data 284 and determine an appropriate course of action, such as: (i) dispatching an agent to the vehicle 104 to dispose of the dangerous non-vehicle element 344, (ii) scheduling a cleaning event for the vehicle 104, (iii) dismissing the hazard condition data 284 as a false positive, and putting the vehicle 104 back into service by updating the availability data 310 accordingly.

[0073] At block 816 of the method 800, if no dangerous non-vehicle element 344 is detected in the image data 254, the device 116 generates the cleanliness score data 280 and stores the cleanliness score data 280 in the memory 242.

[0074] Next, at block 828, the cleanliness score data 280 and image data 254 are transmitted to the base station 108 for storage in the memory 296 and further processing by the computer system 298. In particular, at block 832, the cleanliness score data 280 is compared to other cleanliness scores of the tracked score data 302 and the change data 306 is updated accordingly. If, at block 832, the computer system 298 determines that the cleanliness score data 280 is above the cleanliness threshold 356, the vehicle 104 is removed or scheduled to be removed from service, as identified at block 820. The vehicle 104 is removed from service because the most recent cleanliness score of the cleanliness score data 280 indicates that the vehicle 104 has become unacceptably dirty and unsuitable in terms of cleanliness for the next user. In one embodiment, when transitioning from block 832 to block 820, if the vehicle 104 is a fully autonomous vehicle 104, instructions are sent to the vehicle 104 to drive itself to a location for cleaning and removal of the one or more non-vehicle elements 344. If the vehicle 104 is not a fully autonomous vehicle, a human can review the corresponding image data 254 and determine an appropriate course of action, for example: (i) dispatch an agent to the vehicle 104 to clean and remove the dangerous non-vehicle elements 344, (ii) schedule a cleaning event for the vehicle 104, (iii) treat the cleanliness score data 280 and corresponding image data 254 as a false positive and put the vehicle 104 back into service by updating the availability data 310 accordingly.

[0075] Further, at block 832, the computer system 298 processes the change data 306 to determine whether a refund to the user is appropriate to pay for the cost of cleaning the passenger compartment 120. In particular and with reference to Figure 7 , assume that normal use of the vehicle 104 will result in the passenger compartment 120 becoming progressively dirtier. In Figure 7 , the example cleanliness scores of the passenger compartment 120 representing the expected progressive dirtiness are shown on days 4-9. In Figure 7 , the cleanliness score reaches or exceeds the cleanliness threshold 356 on day 8, and the system 100 determines that cleaning of the passenger compartment 120 should be performed or scheduled. However, the change data 306 on day 8 indicates that the user has contributed very little to the increase in uncleanliness. Thus, the user of the vehicle 104 that pushed the cleanliness score above the cleanliness threshold 356 is not charged any additional cleaning fees or surcharges because that user has only contributed a small increase in the level of dirtiness of the passenger compartment 120. Again, consider Figure 7On day 14, the change data 306 indicates that the user caused a substantial increase (i.e., over 2000%) in the level of dirtiness of the passenger compartment 120. On day 14, the cleanliness score data 280 is well above the cleanliness threshold 356 and the change data 306 is well above the change threshold 364. On day 14, the user, by one's own effort, changed the passenger compartment 120 from a "clean" state to a "dirty" state. Accordingly, the base station 108 can identify the user on day 14 and assess a cleaning fee for the user causing an unexpected and unacceptable level of dirtiness of the passenger compartment 120. Additionally, on day 14, the base station 108 can schedule the interior space of the passenger compartment 120 to be cleaned so that the vehicle 104 is ready for service again.

[0076] While most users dirty the passenger compartment 120 to some extent, some users clean the passenger compartment 120 at the end of a use event. The base station 108 is configured to identify when a user cleans the passenger compartment 120 by detecting negative change data 306. The negative change data 306 indicates that the passenger compartment 120 is cleaner (i.e., less obstructed) at the end of a use event than at the beginning of the use event. When the base station 108 detects that a user has cleaned the passenger compartment 120, the user can be identified to receive a benefit such as a discounted fee to use the vehicle 104.

[0077] If at block 832, the computer system 298 determines that the cleanliness score data 280 does not exceed the cleanliness threshold 356, the method 800 moves to block 836 and the vehicle 104 remains in service. The availability data 310 can be updated or checked at block 836 to ensure that the vehicle 104 is identified as available and ready with a clean passenger compartment 120 for further use events.

[0078] While the disclosure has been illustrated and described in detail in the drawings and foregoing description, the same is to be considered as illustrative and not restrictive in character, it being understood that only the preferred embodiments have been presented and that all changes, modifications and further applications that come within the spirit of the disclosure are desired to be protected.

Claims

1. A method for operating a vehicle comprising a passenger cabin having a plurality of interior sections, the method comprising: generating image data for each of the plurality of interior sections with an imaging device positioned within the passenger cabin; processing the generated image data with a controller operably connected to the imaging device to generate score data comprising a plurality of scores, each of the plurality of scores corresponding to one of the plurality of interior sections; consolidating the generated scores into a single vehicle cleanliness score; generating availability data indicating that the vehicle is available or unavailable based on a comparison of the cleanliness score to a cleanliness threshold; and removing the vehicle from service when the availability data indicates that the vehicle is unavailable; the cleanliness score comprises a first cleanliness score and a second cleanliness score, the method further comprising: generating percentage change data according to a percentage change in the second cleanliness score compared to the first cleanliness score; wherein the first cleanliness score is associated with a first user and the second cleanliness score is associated with a second user, the method further comprising: evaluating a penalty to the second user when the percentage change data exceeds a percentage change threshold; in processing the generated image data to generate the plurality of scores, the controller is further configured to: (i) determine an occlusion metric for each of the plurality of interior sections to derive a plurality of occlusion metrics, and (ii) convert each of the occlusion metrics to a respective score of the plurality of scores by weighting each occlusion metric according to a weighting proportion, wherein each of the occlusion metrics is a percentage of the respective interior section that is occluded by a non-vehicle element.

2. The method of claim 1, wherein: the plurality of interior sections comprises at least a driver seat section, a passenger seat section, a driver floor section, a passenger floor section, a center console section, and a back seat section, and the weighting proportion applies the following weights to the plurality of interior sections: a weight of 30% to the driver seat section, a weight of 25% to the passenger seat section, a weight of 10% to the driver floor section, a weight of 10% to the passenger floor section, a weight of 15% to the center console section, a weight of 10% to the back seat section.

3. The method of claim 1, wherein, the weighting proportion is a first weighting proportion, and converting each of the occlusion metrics to the respective score comprises: weighting each occlusion metric according to the first weighting proportion during a first seasonal time period, and weighting each occlusion metric according to a second weighting proportion during a second seasonal time period different from the first seasonal time period.

4. The method of claim 3, wherein: the plurality of interior sections comprises at least a driver seat section, a passenger seat section, a driver floor section, a passenger floor section, a center console section, and a back seat section, the first weighting proportion applies the following weights to the plurality of interior sections: a weight of 30% to the driver seat section, a weight of 25% to the passenger seat section, a weight of 10% to the driver floor section, a weight of 10% to the passenger floor section, a weight of 15% to the center console section, a weight of 10% to the back seat section. The back seat section has a weight of 10% and The second weighting proportion applies the following weights to the plurality of interior sections: The driver seat section has a weight of 40%, The passenger seat section has a weight of 30%, The driver floor section has a weight of 5%, The passenger floor section has a weight of 5%, The center console section has a weight of 15%, The back seat section has a weight of 5%.

5. The method of claim 1, further comprising: processing the generated image data to generate hazardous item data corresponding to a hazardous item in the passenger compartment, removing the vehicle from service based on the hazardous item data.

6. An apparatus for monitoring a passenger compartment of a vehicle, the passenger compartment comprising a plurality of interior sections, the apparatus comprising: an imaging device positioned within the passenger compartment and configured to generate image data for each of the plurality of interior sections; a controller operably connected to the imaging device and configured to: process the generated image data to generate score data comprising a plurality of scores, each of the plurality of scores corresponding to one of the plurality of interior sections, merge the generated scores into a single vehicle cleanliness score, generate availability data indicating that the vehicle is available or unavailable based on a comparison of the cleanliness score to a cleanliness threshold; and remove the vehicle from a shared vehicle service when the availability data indicates that the vehicle is unavailable; wherein the cleanliness score comprises a first cleanliness score and a second cleanliness score, and a percent change data is generated from a percent change in the second cleanliness score compared to the first cleanliness score; wherein the first cleanliness score is associated with a first user and the second cleanliness score is associated with a second user, and a penalty is assessed to the second user when the percent change data exceeds a percent change threshold; in processing the generated image data to generate the plurality of scores, the controller is further configured to: (i) determine an occlusion metric for each of the plurality of interior sections to derive a plurality of occlusion metrics, and (ii) convert each of the occlusion metrics to a respective score of the plurality of scores by weighting each occlusion metric according to a weighting proportion, wherein each of the occlusion metrics is a percentage of the respective interior section that is occluded by a non-vehicle element.

7. The apparatus of claim 6, further comprising: a housing positioned within a partitioned space of the passenger compartment, wherein the imaging device and the controller are positioned within the housing.

8. The apparatus of claim 7, wherein, The partitioned space is positioned in an overhead console of the vehicle.

9. The apparatus of claim 6, further comprising: at least one visible light illumination device operably connected to the controller and configured to illuminate the passenger compartment with light in a visible light spectrum; and an infrared illumination device operably connected to the controller and configured to illuminate the passenger compartment with light in an infrared spectrum. The controller and the imaging device are electrically isolated from an electrical system of the vehicle.

10. The apparatus of claim 6, wherein, 11. The apparatus of claim 6, further comprising: a transceiver operably connected to the controller and configured to wirelessly transmit at least the availability data to a remote base station. ​

Images