Systems for detecting objects thrown from vehicles

Abstract

A method for detecting an object thrown from a vehicle passenger compartment includes: generating window image data of a window opening using a camera mounted on the vehicle; and processing the window image data using a controller operably connected to the camera to generate thrown object data corresponding to at least one object thrown from the passenger compartment through the window opening. The method further includes using the controller to correlate the thrown object data with occupant data of a vehicle occupant.

Description

Technical Field

[0001] This disclosure relates to the fields of vehicles and computer vision, and particularly to the use of computer vision to identify objects, such as waste and garbage, thrown from vehicles. Background Technology

[0002] Ride-hailing and ride-sharing services are becoming increasingly popular as a mode of transportation in urban environments. Furthermore, related to autonomous vehicle technology, self-driving taxis and shuttle buses will emerge in the near future. The prospects of ride-sharing services and shared autonomous vehicles have brought new business models and corresponding challenges.

[0003] One of the challenges of maintaining a fleet of shared vehicles that are both human-driven and autonomously controlled is keeping the vehicles clean and free of waste left by previous passengers. For example, some shared vehicles use cameras or other systems to identify waste left in the vehicle's cabin by passengers. If a passenger leaves the vehicle dirty enough, a service fee can be assessed to be charged to the passenger for cleaning the vehicle.

[0004] To avoid service fees when using shared vehicles, most riders clean themselves by disposing of trash in appropriate containers, such as trash cans. However, other riders employ different methods to avoid service fee assessments. In particular, some riders throw waste and debris from the vehicle while it is in motion. This method keeps the vehicle's cabin clean and debris-free, but leads to the unintended consequence of increased litter along roads and in the community.

[0005] The current business models used to keep shared vehicles clean incentivize some riders to dispose of waste in unintended ways. Therefore, further development in the shared and autonomous vehicle sectors is expected to contribute to environmental protection efforts, thereby saving community cleaning costs and / or providing cleaner communities and environments overall. Summary of the Invention

[0006] According to an exemplary embodiment of this disclosure, a method for detecting an object thrown from a vehicle passenger compartment includes: generating window image data of a window opening using a camera mounted on the vehicle; and processing the window image data using a controller operatively connected to the camera to generate thrown object data corresponding to at least one object thrown from the passenger compartment through the window opening. The method further includes associating the thrown object data with occupant data of a vehicle occupant using the controller.

[0007] According to another exemplary embodiment of this disclosure, a method for detecting an object thrown from a vehicle cabin includes: generating window image data of a window opening when a corresponding movable window panel is in an open position; and processing the window image data to generate thrown object data corresponding to an object thrown from the cabin through the window opening. The method further includes generating littering event data based on the generated thrown object data and associating the littering event data with a vehicle operator.

[0008] According to another exemplary embodiment of this disclosure, a system for detecting an object thrown from a vehicle passenger compartment includes a camera and a controller. The camera is mounted on the vehicle and configured to generate window image data of a window opening. The controller is operatively connected to the camera to receive the window image data. The controller is configured to (i) process the window image data to generate thrown object data corresponding to at least one object thrown from the passenger compartment through the window opening, and (ii) associate the thrown object data with occupant data of the vehicle occupants. Attached Figure Description

[0009] The above-described features and advantages, as well as other features and advantages, will become more apparent to those skilled in the art from the following detailed description and accompanying drawings, in which:

[0010] Figure 1 This is a top view block diagram of an exemplary embodiment of a vehicle, which includes a system for detecting objects ejected from the vehicle's passenger compartment as disclosed herein.

[0011] Figure 2 This is a side view block diagram, which illustrates... Figure 1 The vehicle and its systems are illustrated, and an occupant throws an object from the vehicle through the windshield panel with the window open while the vehicle is in motion.

[0012] Figure 3 It is a diagram. Figure 1 Block diagrams of vehicles and systems; and

[0013] Figure 4 It is a graphical operation Figure 1 The flowchart illustrates an exemplary method for a system to detect objects thrown from a vehicle's passenger compartment. Detailed Implementation

[0014] For the purpose of promoting an understanding of the principles of this disclosure, reference will now be made to embodiments illustrated in the accompanying drawings and described in the following written description. It should be understood that this is not intended to limit the scope of this disclosure. It should also be understood that this disclosure includes any changes and modifications to the illustrated embodiments, and includes further applications of the principles of this disclosure that would normally occur to those skilled in the art to which this disclosure pertains.

[0015] Various aspects of this disclosure are described in the accompanying description. Alternative embodiments and equivalents of this disclosure may be devised without departing from the spirit or scope thereof. It should be noted that any discussion herein with reference to “an embodiment,” “an embodiment,” “an exemplary embodiment,” etc., indicates that the described embodiment may include specific features, structures, or characteristics, and such specific features, structures, or characteristics are not necessarily included in every embodiment. Furthermore, references to the foregoing do not necessarily include references to the same embodiment. Finally, whether explicitly described or not, those skilled in the art will readily understand that each specific feature, structure, or characteristic of a given embodiment may be used in relation to or in combination with features, structures, or characteristics of any other embodiment discussed herein.

[0016] For the purpose of publication, the phrase “A and / or B” means (A), (B), or (A and B). For the purpose of publication, the phrase “A, B, and / or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0017] The terms “comprising,” “including,” “having,” etc., used in respect of embodiments of this disclosure are synonymous.

[0018] like Figure 1 As shown, vehicle 100 includes system 104 configured to detect when an occupant 108 of vehicle 100 throws an object 112 from vehicle 100 while vehicle 100 is in motion. When vehicle 100 is in motion, system 104 detects when occupant 108 passes through window opening 116 using computer vision. Figure 2 The system 104 moves items from inside vehicle 100 to outside vehicle 100 to determine when occupant 108 litters. In one embodiment, the system 104 displays a message to occupant 108 of vehicle 100 to instruct occupant 108 to stop littering. The system 104 also associates object 112 with the identity of occupant 108, thereby enabling the company operating vehicle 100 to track occupant 108 in an appropriate manner, such as by assessing service fees for occupant or by preventing occupant 108 from reusing vehicle 100. The litter sensing system 104 provides occupant 108 of vehicle 100 with an incentive to dispose of waste (i.e., object 112) in an appropriate container rather than throwing object 112 on or near the road to avoid service fees. Each aspect of system 104 and method 400 of operating system 104 are described herein. Figure 4 ).

[0019] Reference Figure 1 and Figure 2The illustration shows an exemplary vehicle 100 including system 104. In one embodiment, vehicle 100 is a shared vehicle 100, which can be controlled by occupant 108 (i.e., level 0 to level 4 autonomous control) or fully autonomous (i.e., level 5 autonomous control). In other embodiments, vehicle 100 is a taxi, shuttle bus, limousine, company vehicle, delivery vehicle, taxi, or personal vehicle. Vehicle 100 is any vehicle carrying human occupant 108.

[0020] Vehicle 100 includes chassis 120, which has doors 124 and window panels 128 defining an interior passenger compartment 132. (As...) Figure 2 As shown, the left front door 124 defines a window opening 116, wherein the movable window panel 128 can be positioned in an open window state and a closed window state. Figure 2 In the present embodiment, the left front window panel 128 is in the open window state, and the left rear window panel 128 of the left rear door 124 is in the closed window state. In some embodiments, the vehicle 100 includes an additional window opening 116 and a corresponding additional window panel 128, which is located in the roof of the vehicle 100 and is referred to as a sunroof or movable roof. The sunroof and / or movable roof is included in the window opening 116 and the window panel 128.

[0021] like Figure 1 As shown, vehicle 100 also includes occupant seats 140 located within passenger compartment 132, an instrument panel 144 including a display screen 148, at least one speaker 152, a center console 156, side mirror housings 160, and an electronic control unit (“ECU”) 164. In the exemplary vehicle 100, four occupant seats 140 are included. In other embodiments, vehicle 100 may include any desired number of occupant seats 140 in any configuration.

[0022] The dashboard 144 is located at the front of the vehicle 100 and includes instruments and controls for operating the vehicle 100 and / or interacting with the vehicle 100.

[0023] Continue to refer Figure 1In one embodiment, the display screen 148 is integrated into the dashboard 144. The display screen 148 displays vehicle data to the occupants 108 of the vehicle 100. In an occupant-controlled vehicle 100, the display screen 148 displays vehicle speed, remaining battery power, navigation information (from the Global Positioning System (“GPS”), and other messages to the driver 108 and other occupants 108 of the vehicle 100. In an autonomous vehicle 100, the display screen 148 displays information to the occupants 108 of the vehicle 100, such as remaining time to reach the destination and instructions for interacting with the vehicle 100. The display screen 148 can be configured to display any desired information to the occupants 108 of the vehicle 100. In one embodiment, the display screen 148 is provided as a liquid crystal display or any other information display technology.

[0024] The at least one speaker 152 is located at least partially in the cabin 132. The speaker 152 generates sound to provide auditory information and entertainment to the occupants 108.

[0025] The central console 156 is located between the first two occupant seats 140. The central console 156 may include space for accommodating occupant items such as smartphones, wallets, and / or beverage containers (i.e., object 112).

[0026] The side mirror housing 160 is mounted on the vehicle 100 and includes a mirror (not shown) positioned to provide a view along the side of the vehicle 100 and to the rear. In some embodiments, such as the fully autonomous vehicle 100, the side mirror housing 160 includes a camera (such as the mirror camera 208 described below) instead of a mirror.

[0027] ECU 164 is configured to control the electronic components of vehicle 100. For example, ECU 164 is configured to control an internal combustion engine (not shown) and / or a traction motor (not shown) to cause desired movement of vehicle 100. In one embodiment, ECU 164 is connected to at least one window status sensor 170 that monitors the state of movable window panels 128, and at least one door status sensor 174 that monitors the state of doors 124. ECU 164 is also connected to a speed sensor 178 that monitors the speed of movement of vehicle 100. ECU 164 generates data corresponding to the position (i.e., open or closed) of each movable window panel 128, the state (i.e., open or closed) of each door 124, and the speed of vehicle 100. In one embodiment, ECU 164 is electrically connected to window status sensor 170, door status sensor 174, and speed sensor 178 via a CAN bus (Controller Area Network bus).

[0028] like Figure 1 and Figure 3As shown, a waste sensing system 104 is mounted on vehicle 100 and includes a cabin camera 204, a mirror camera 208, a transceiver 212, and a memory 216, each operatively connected to a controller 220. The cabin camera 204, mounted on vehicle 100 within cabin 132, is a digital imaging device configured to generate cabin image data 228. The cabin camera 204 is positioned to observe the behavior and movement of occupant 108 and generates cabin image data 228 based on the behavior and movement of occupant 108. The cabin camera 204 may be configured as a visible light camera, a thermal camera, and / or an infrared camera.

[0029] Forward cabin camera 204 (i.e., Figure 1 The cabin camera 204, which is closest to the instrument panel 144, has a limited field of view of 232. Figure 1 The field of view 232 includes at least the entire front passenger seat 140, dashboard 144, center console 156, and at least a portion of the interior floor 236 of the vehicle 100. A front cabin camera 204 is mounted on the dashboard 144 in one embodiment and on the interior headliner structure of the vehicle 100 in another embodiment. The front cabin camera 204 is mounted in either location to position at least the entire front passenger seat 140, dashboard 144, center console 156, and at least a portion of the floor 236 within the field of view 232. Images generated by the front cabin camera 204 within the field of view 232 are included in cabin image data 228.

[0030] Rear cabin camera 204 (i.e., in) Figure 1 The cabin camera 204, positioned furthest from the instrument panel 144, has a limited field of view of 240. Figure 1 The field of view 240 includes at least the entire rear passenger seat 140 and at least a portion of the floor 236. In one embodiment, a rear cabin camera 204 is mounted on the interior ceiling structure of the cabin 132. The rear cabin camera 204 is mounted in any position to position at least the entire rear passenger seat 140 and at least a portion of the floor 236 within the field of view 240. Images within the field of view 240 generated by the rear cabin camera 240 are included in cabin image data 228.

[0031] In some embodiments, the waste sensing system 104 includes only one cabin camera 204 positioned to generate cabin image data 228 encompassing the entire cabin 236 including all occupant seats 140. In other embodiments, the waste sensing system 104 includes three or more cabin cameras 204. For example, in a bus, minivan, or other vehicle with three or more rows of seats 140, each row of seats 140 may include a corresponding cabin camera 204. In yet another embodiment, each individual seat 140 of vehicle 100 is monitored by a separate cabin camera 204. The waste sensing system 104 includes any number of cabin cameras 204 that can be used to monitor the occupants 108 and / or seats 140 of the corresponding vehicle 100.

[0032] The mirror camera 208 is a digital imaging device configured to generate window image data 244. In one embodiment, the mirror camera 208 is a panoramic camera positioned to generate window image data 244 of object 112 when object 112 is thrown out of window opening 116. The left mirror camera 208 is mounted on the left mirror housing 160, and the right mirror camera 208 is mounted on the opposite right mirror housing 160. The mirror camera 208 may be configured as a visible light camera, a thermal camera, and / or an infrared camera.

[0033] like Figure 1 and Figure 2 As shown, the left-side mirror camera 208 defines a field of view 248 encompassing the left side of vehicle 100, including the entire window opening 116 of the front door 124 and the entire window opening 116 of the rear door 124. The right-side mirror camera 208 defines a field of view 252 encompassing the right side of vehicle 100, including the entirety of the corresponding other window opening 116. In at least one embodiment, the fields of view 248, 252 of the mirror cameras 208 also extend to the roof 256 of vehicle 100. Figure 2 The image extends above the vehicle 100 and onto the ground on which it is positioned. Since the field of view 248, 252 of the mirror camera 208 includes the entirety of each window opening 116, any object 112 passing through one of the window openings 116 is detected by at least one of the mirror cameras 208. The image within the field of view 248, 252 of the mirror camera 208 is included in window image data 244.

[0034] Transceiver 212, also referred to as a wireless transmitter and receiver, is configured to, for example, wirelessly transmit data from vehicle 100 to another electronic device (not shown) and wirelessly receive data from another electronic device via the Internet. Thus, transceiver 212 operatively connects vehicle 100 to the Internet and other electronic devices. In other embodiments, transceiver 212 uses cellular networks, wireless local area networks (“Wi-Fi”), personal area networks, and / or any other wireless networks to transmit and receive data. Therefore, transceiver 212 is compatible with any desired wireless communication standards or protocols, including but not limited to Near Field Communication (“NFC”), IEEE 802.11, IEEE 802.15.1 (“Bluetooth®”), Global System for Mobile Communications (“GSM”), and Code Division Multiple Access (“CDMA”).

[0035] Memory 216 is an electronic storage device configured to store at least cabin image data 228, window image data 244, and program instructions 260 for operating the waste sensing system 104. Memory 216 is also referred to herein as a non-transient computer-readable medium.

[0036] The memory 216 is also configured to store window status data 264, door status data 268, and speed data 270. Window status data 264 is generated by the controller 220 based on signals received from the ECU 164, which is electrically connected to the window status sensor 170. Window status data 164 indicates which window panel 128 is in the open window state and which is in the closed window state at any given time. Alternatively or additionally, window status data 264 is generated by the ECU 164 and transmitted to the controller 220 for storage in the memory 216.

[0037] Door status data 268 is generated by controller 220 based on signals received from ECU 164, which is electrically connected to door status sensor 174. Door status data 268 indicates which door 124 is in the open state and which door 124 is in the closed state at any given time. Alternatively or additionally, door status data 268 is generated by ECU 164 and transmitted to controller 220 for storage in memory 216.

[0038] Speed ​​data 270 corresponds to the ground speed of vehicle 100. In one embodiment, speed data 270 is generated by controller 220 based on signals from speed sensor 178 (such as signals received by controller 220 from ECU 164). In another embodiment, ECU 164 generates speed data 270 and transmits it to controller 220 for storage in memory 216. In some embodiments, speed data 270 also includes data corresponding to the driving state of vehicle 100. For example, driving state may include driving mode and non-driving mode, and the corresponding information is included in speed data 270. Speed ​​data 270 may indicate that vehicle 100 is traveling at zero mph in parking mode, traveling at ten mph in driving mode, or stationary in driving mode.

[0039] A driving mode is a driving state of vehicle 100, in which vehicle 100 can be moved by an engine and / or an electric motor. In a passenger vehicle 100, a driving mode corresponds to placing the transmission in "D" for forward movement or in "R" for backward movement. In an autonomous vehicle 100, a driving mode corresponds to any driving state of vehicle 100, in which vehicle 100 can be moved using the engine and / or electric motor. Driving modes contrast with non-driving modes, which are often referred to as neutral mode "N" or parking mode "P". Vehicle 100 cannot be driven (manually or autonomously) in non-driving modes.

[0040] The memory 216 also stores data on thrown objects 272 and data on littering incidents 276. The controller 220 generates the thrown object data 272 based on window image data 244. The thrown object data 272 includes image data of an object 112 thrown, tossed, or falling from the passenger cabin 132 through a window opening 116. Figure 2 As shown, occupant 108 has ejected object 112 from cabin 132 through window opening 116. Object 112 is within the field of view 248 of mirror camera 208, and the corresponding image data of object 112 is included in window image data 244. Based on the processing performed by controller 220, data corresponding to the ejected object 112 is identified, separated, extracted, and / or classified in window image data 244 and stored separately in memory 216 as ejected object data 272. That is, window image data 244 includes all image data generated by mirror camera 208, and ejected object data 272 includes window image data 244, which includes images of the ejected object 112 as automatically identified by controller 220.

[0041] The littering incident data 276 stored in memory 216 includes at least a portion of the thrown object data 272 and at least one of the date, time, and location at which the thrown object 112 was thrown from vehicle 100. This location is determined, for example, by the vehicle 100's onboard GPS (not shown). When presented on a suitable display (such as a computer monitor), the littering incident data 276 is beneficial for human analysis to determine whether a littering incident actually occurred or whether the claimed object 112 is road debris or other non-litter elements.

[0042] In addition to the above, memory 216 also stores occupant data 280, throwing motion data 284, and incoming object data 288. Occupant data 280 includes at least one of the occupant 108's name, address, and account number. In another embodiment, occupant data 280 includes seat position data based on the position of each occupant seat 140, and occupant data 280 does not include occupant 108's identification data. Instead, vehicle sensing system 104 transmits occupant data 280 to, for example, a fleet operator of vehicle 100, and the fleet operator associates occupant data 280 with occupant 108's personal identification data. Therefore, in at least some embodiments, vehicle sensing system 104 does not store occupant 108's personal identification data.

[0043] Throwing motion data 284 is identified, separated, exported, and / or classified by controller 220 from cabin image data 228, and corresponds to an image of occupant 108's arm or hand moving toward window opening 116 during a throwing or tossing motion. Specifically, throwing motion data 284 is part of cabin image data 228, which has been identified by controller 220 as including image data corresponding to hand or arm movements made by occupant 108 consistent with throwing object 112 out of window opening 116. That is, cabin image data 228 includes all image data generated by cabin camera 204, and throwing motion data 284 includes cabin image data 228, which includes images of arm and hand movements toward window opening 116 as automatically identified by controller 220. Identifying and / or generating throwing motion data 284 is sometimes a precursor to littering incidents, as occupant 108 may be moving toward window opening 116 with an object 112 about to be thrown.

[0044] The incoming object data 288 is another part of the cabin image data 228 corresponding to the object 112 that moves into the cabin 132. For example, when occupant 108 enters vehicle 100 with a beverage container (represented by object 112), cabin camera 204 generates corresponding cabin image data 228. According to method 400 described below, controller 220 processes cabin image data 228 to identify, separate, extract, and / or classify image data corresponding to the element (i.e., object 112) brought into cabin 132.

[0045] The controller 220 of the vehicle sensing system 104 is configured to execute program instruction data 260 to operate the sensing system 104. The controller 220 is provided as at least one microcontroller and / or microprocessor. In one embodiment, the controller 220 is electrically connected to the ECU 164 via a CAN bus to receive one or more of window status data 264, door status data 268, and speed data 270.

[0046] In operation, system 104 is configured to implement the... Figure 4 The flowchart illustrates method 400. In this exemplary embodiment, vehicle 100 is a shared autonomous vehicle configured to autonomously drive to the location of occupant 108 and then autonomously transport occupant 108 to the desired location using a public road network upon occupant 108 entering vehicle 100. For example, occupant 108 may use a smartphone application (i.e., an "app") to utilize the services of vehicle 100. Occupant 108 is also referred to herein as a passenger, user, operator, or person. In other embodiments, vehicle 100 is any type of passenger vehicle as described above and may be controlled by occupant 108.

[0047] In block 404 of method 400, vehicle 100 is stationary, and waste sensing system 104 uses cabin camera 204 to detect objects 112 entering and leaving vehicle 100 through open doors 124. System 104 determines that vehicle 100 is stationary based on speed data 270. Furthermore, system 104 determines which door 124 is open (i.e., in the open door state) based on door status data 268. In one embodiment, system 104 generates entry object data 288 only when at least one door 124 is in the open door state. When all doors 124 are closed, system 104 does not generate entry object data 288. System 104 utilizes cabin camera 204 to generate cabin image data 228, which includes images of occupant 108 and objects 112 entering cabin 132. Occupant 108 places objects 112 in, for example, a cup holder; in this example, object 112 is a beverage container or coffee cup. Object 112 is in the field of view 232 of the front cabin camera 204.

[0048] Controller 220 processes cabin image data 228 to generate incoming object data 288, which in this example corresponds to object 112. In one embodiment, controller 220 uses computer vision object detection technology to identify common objects in cabin image data 228. Incoming object data 288 is a representation of each object 112 detected by sensing system 104 as being brought into cabin 132 through open door 124. In one embodiment, system 104 distinguishes object 112 from occupant 108 and identifies anything that is neither occupant 108 nor vehicle 100 as an incoming object 112.

[0049] At frame 404, system 104 can detect an object 112 leaving vehicle 100 through open door 124 based on cabin image data 228. When vehicle 100 is in non-driving mode (and stationary) and object 112 is detected as having moved out of the field of view 232, 240 of cabin camera 204 from cabin 132 through open door 124 or window opening 116, object 112 is removed from incoming object data 288. System 104 does not treat the movement of object 112 through open door 124 as a littering event and does not cause littering event data 276 to be generated. This is because when an object moves through open door 124, it is more likely that occupant 108 is unloading object 112 from vehicle 100, rather than littering or improperly disposing of object 112.

[0050] Next, in block 408 of method 400, controller 220 determines whether vehicle 100 is in driving mode and / or is moving. If controller 220 determines that vehicle 100 is in driving mode, method 400 continues to block 412. System 104 determines the state of vehicle 100 and whether vehicle 100 is moving or stationary based on speed data 270.

[0051] At box 412, system 104 determines, based on window status data 264, whether any movable window panel 128, including the movable sunroof and / or movable roof window panel 128, is in an open window state. If system 104 determines that all window panels 128 are closed, method 400 returns to box 408 to monitor the state of vehicle 100 based on speed data 270. System 104 determines that no littering event will occur if vehicle 100 is in driving mode and all window panels 128 are in the closed window state, because there is no opening for throwing objects 112 through them. If system 104 determines that at least one window panel 128 is in the open window state, method 400 proceeds to box 416.

[0052] At frame 416, system 104 monitors the position of object 112 and the behavior and movement of occupant 108, as littering is potentially possible. That is, at frame 416, vehicle 100 is in driving mode and at least one window panel 128 is open. Therefore, there is an opening through which occupant 108 can throw object 112 from vehicle 100, and occupant 108 is unlikely to load or unload vehicle 100 while the vehicle is in driving mode. Object 112 represents any element within vehicle 100 that could be thrown out of window opening 116, and includes personal belongings, trash, food, etc.

[0053] In one embodiment, at block 416, controller 220 compares window image data 244 with incoming object data 288 to determine ejected object data 272. Specifically, controller 220 stores image data of each object 112 brought into cabin 132 by processing cabin image data 228 and by generating incoming object data 288. Then, at block 416, controller 220 processes window image data 244 for data corresponding to the object 112 in the incoming object data 288. If any object 112 of the incoming object data 288 is detected in window image data 244, controller 220 determines that the corresponding object 112 has been ejected from cabin 132 through window opening 116.

[0054] For example, when object 112 is loaded into cabin 132, controller 220 generates incoming object data 288 corresponding to an image of object 112. Then, when vehicle 100 is in driving mode, controller 220 uses computer vision technology to process window image data 244 to search for a corresponding image of object 112 in window image data 244. If controller 220 identifies a match between object 112 in window image data 244 and object 112 in incoming object data 288, ejected object data 272 is generated, and ejected object data 272 includes an image of object 112 ejected from vehicle 100 from window image data 244. In one embodiment, for object 112 to appear and be detected in window image data 244, it must have already been ejected from vehicle 100, because the field of view 248, 252 of mirror camera 208 is only located outside cabin 132 of vehicle 100 and does not include any part of cabin 132.

[0055] In another embodiment, controller 220 processes only window image data 244 to generate ejected object data 272, without generating incoming object data 288. Therefore, in some embodiments, incoming object data 288 is not required. Window image data 244 is generated only when system 104 detects that window panel 128 is in an open window state. In this embodiment, controller 220 uses computer vision technology to detect object 112 in the corresponding fields of view 248, 252 of mirror camera 208. Many of the images generated as window image data 244 are static images of the side of vehicle 100. Therefore, in order to detect object 112 in window image data 244, controller 228 may employ a subtraction method to identify the difference between a side image of vehicle 100 in which object 112 is not present in fields of view 248, 252 and a side image of vehicle 100 in which object 112 is present in at least one of fields of view 248, 252.

[0056] In addition, additional or alternative image processing techniques can be used to identify the object 112 in the window image data 244. For example, the controller 220 processes the window image data 244 of any element that leaves a track along the side of the vehicle 100 from front to back, such as when the object 112 is ejected from the vehicle 100. When such image data is detected, the corresponding window image data 244 is saved to the ejected object data 272.

[0057] In one embodiment, the controller 220 is additionally or alternatively configured to identify objects 112 detected in window image data 244 as bouncing on the road or falling from vehicle 100 and impacting the road as thrown object data 272. Such traces in window image data 244 have a downward trajectory along the exterior of door 124 and indicate a thrown object 112.

[0058] When processing window image data 244, in at least one embodiment, controller 220 is configured to ignore image data corresponding to a hand or arm of occupant 108 resting on window opening 116. This method reduces false alarms in littering incident data 276.

[0059] Additionally or alternatively, at frame 416, system 104 uses throwing motion data 284 as a trigger to begin generating and processing window image data 244. For example, when object 112 is thrown from cabin 132, occupant 108 catches object 112 and moves object 112 toward window opening 116 by throwing or tossing motion. Figure 1The movement of occupant 108 and object 112 is depicted in the image. Controller 220 detects the throwing motion as throwing motion data 284 from cabin image data 228. Upon detecting throwing motion data 284, controller 220 begins generating window image data 244 because the thrown object 112 may be about to appear. In this way, system 104 does not generate window image data 244 discontinuously, but only after detecting throwing motion data 284 and when at least one window panel 128 is in the open window state. This method also tends to reduce the amount of false alarm littering incident data 276.

[0060] In another embodiment at frame 416, controller 220 processes cabin image data 228 to detect whether any object 112 is being moved toward window opening 116. Therefore, instead of or supplementing the detection of throwing motion data 284, controller 228 monitors and determines the position of each object 112 within cabin 132. When any object 112 is determined to have moved sufficiently toward or near window opening 116, then controller 220 begins generating window image data 244, as the thrown object 112 may be about to appear. In one embodiment, controller 220 begins generating window image data 244 when an object 112 detected in cabin image data 228 is moving within a distance threshold from window opening 116. For example, the distance threshold is from 2 inches to 12 inches.

[0061] In yet another embodiment at block 416 of method 400, controller 220 uses only one of the plurality of mirror cameras 20 to generate window image data 244. For example, if only the left front window panel 128 is in the open window state (e.g. Figure 2 As shown), the controller 220 generates window image data 244 using only the left mirror camera 208 and does not generate window image data 244 using the right mirror camera 208, because the object 112 cannot be thrown out from the right side of the vehicle 100 since all window panels 128 are in the closed window state.

[0062] Moving to block 420 of method 400, controller 220 determines whether littering has occurred and generates littering event data 276. Specifically, system 104 generates littering event data 276 by associating the object data 272 to be thrown with at least one of the time, data, and location of vehicle 100. Furthermore, littering event data 276 includes occupant data 280, which may include the operator's account, name, residential address, email address, and / or workplace. Therefore, when littering event data 276 is generated, controller 220 associates the object data 272 to occupant data 280. In some embodiments, frames of cabin image data 228 are included in littering event data 276 to further connect the thrown object 112 to a specific occupant 108 included in cabin image data 228. Specifically, to identify occupant 108, images of the occupant's face may be stored and cataloged in littering event data 276. In another embodiment, occupant data 280 does not include personal identification data of occupant 108, and occupant data 280 only includes vehicle 100 status data, such as, for example, seat position data.

[0063] Also at box 420, in one embodiment, system 104 uses display screen 148 and / or speaker 152 to warn occupant 108 that littering has been detected. When littering event data 276 is generated, controller 220 sends the thrown object data 272 and / or littering event data 276 to display screen 148 for display to occupant 108. For example, display screen 148 shows occupant 108 an image of the thrown object 112 and may broadcast a message on speaker 152 requesting occupant 108 to stop all littering activities. Display screen 148 and / or audio message may further inform occupant 108 that they will face a service charge due to the throwing of object 112 from vehicle 100. Display screen 148 and / or audio message may additionally or alternatively inform occupant 108 that they can no longer use vehicle 100 for future transport needs in order to prevent occupant 108, who is known to have littered, from littering again. In a further embodiment, when litter is detected, the account of passenger 108 may be revoked or modified until appropriate steps have been taken to correct the detected littering incident, such as paying a service fee, cleaning up the litter, and / or performing a certain number of community service hours.

[0064] In one embodiment, littering event data 276 is transmitted by transceiver 212 to a remote location for processing and storage. For example, when littering event data 276 is received at a remote location, a human analyst can review thrown object data 272 to determine whether actual littering has occurred or whether littering event data 276 is based on false detection.

[0065] Still referencing Figure 4 If, at box 408, controller 220 determines that vehicle 100 is not in driving mode (i.e., in non-driving mode), method 400 proceeds to box 424. At box 424, system 104 again determines, based on window status data 264, whether any movable window panels 128 (including movable sunroof and / or movable roof window panels 128) are in the open window state. If system 104 determines that all window panels 128 are in the closed window state, method 400 returns to box 404 to use cabin camera 204 to detect objects 112 entering and leaving vehicle 100 through open doors 124 (if any). If system 104 determines in box 424 that at least one window panel 128 is in the open window state, method 400 proceeds to box 428.

[0066] At frame 428, controller 220 processes window image data 244 to identify an object 112 thrown through window opening 116 and contacting the ground on which vehicle 100 is positioned as litter. The corresponding window image data 244 is stored as littering event data 276 as described above. Thereafter, method 400 continues to frame 408 to monitor whether vehicle 100 is in driving mode.

[0067] The processing steps occurring at box 428 illustrate numerous use cases, including drive-through services, automated teller machines (“ATMs”), and the transfer of items to a person outside vehicle 100, and are used to prevent false garbage detection by system 104. Specifically, in the drive-through service use case, occupant 108 of vehicle 100 makes a payment to a person outside vehicle 100 through window opening 116, and in exchange, receives an item from the person outside vehicle 100 through window opening 116. Neither the payment nor the received item touches the ground on which vehicle 100 is located, and therefore, the movement of the payment and the received item is not littering and should not be detected as littering by system 104. Similarly, when using a trash can located outside vehicle 100, occupant 108 moves an object through window opening 116 and discards the object into the trash can. The discarded object does not touch the ground and is not littering. However, in a littering incident, occupant 108 passes or throws an object 112 through window opening 116, and the object 112 falls to the ground. The thrown object 112 falls from window opening 116 and impacts the ground. Therefore, controller 220 is configured to process window image data 244 to detect data corresponding to an object 112 passed through window opening 116 and then falling to the ground, as such data is very likely to correspond to a littering incident. The corresponding window image data 244 is included in littering incident data 276.

[0068] Before or during the processing step of block 428, method 400 may also include the processing step of block 416 to monitor the position of the detected object 112 and / or monitor the behavior of the occupants 108 inside the vehicle 100.

[0069] Although this disclosure has been detailed and described in the accompanying drawings and the foregoing description, it should be considered illustrative in nature and not restrictive. It should be understood that only preferred embodiments have been presented, and all variations, modifications, and further applications within the spirit of this disclosure are intended to be protected.

Claims

1. A method for detecting an object ejected from the passenger compartment of a vehicle, comprising: Window image data of the vehicle's window openings are generated using a first camera installed on the exterior of the passenger cabin of the vehicle. The window image data is processed using a controller operably connected to the first camera to generate ejected object data corresponding to at least one object thrown out of the cabin through the window opening. and The controller is used to associate the data of the thrown object with the occupant data of the vehicle occupants. The method further includes: The state of a movable window panel that can move within the window opening is detected using at least one window state sensor of the vehicle, and the detected state includes an open window state and a closed window state. The passenger cabin image data of the vehicle is generated using a second camera installed in the passenger cabin, the passenger cabin image data being based at least in part on the movement of the occupants in the passenger cabin; The controller processes the cabin image data to detect throwing motion data corresponding to the movement of the occupant's arm or hand toward the window opening; and Window image data is generated only when the movable window panel is in an open window state and throwing motion data is detected.

2. The method according to claim 1, wherein, The first camera is mounted directly on the side mirror housing of the vehicle.

3. The method according to claim 1, further comprising: The controller uses the generated data on thrown objects and occupants to generate data on littering incidents. The littering incident data includes image data of the thrown object and at least one of the date, time, and location at which the object was thrown from the vehicle.

4. The method according to claim 1, wherein, The method further includes: The second camera is used to generate data on incoming objects, the data corresponding to objects placed in the cabin by the occupants; and When at least one object is ejected from the cabin, the controller identifies the incoming object in the window image data.

5. The method of claim 4, further comprising: The state of at least one door of the vehicle is detected, including an open door state and a closed door state; and Data on the entering object is generated only when at least one door is in the open state.

6. The method according to claim 1, wherein, The ejected object data includes image data of at least one object.

7. The method of claim 6, further comprising: Assess the service fees for the occupants of the vehicle.

8. A method for detecting an object thrown from the passenger compartment of a vehicle, comprising: When the corresponding movable window panel is in the open position, a window image data of the window opening is generated using a first camera installed outside the passenger cabin; Process window image data to generate ejected object data corresponding to an object thrown from the passenger cabin through the window opening; Data on littering incidents is generated based on the generated data of the thrown objects. as well as The littering incident data is correlated with the occupants of the vehicle. The method further includes: The state of the movable window panel is detected, including the open window state corresponding to the open position and the closed window state corresponding to the closed position of the movable window panel. A second camera installed inside the passenger cabin generates passenger cabin image data, which is based at least in part on the movement of occupants within the passenger cabin. Processing the cabin image data to detect throwing motion data corresponding to the movement of an occupant's arm or hand toward the window opening; and Window image data is generated only when the movable window panel is in an open window state and throwing motion data is detected.

9. The method according to claim 8, wherein, The first camera is mounted directly on the side mirror housing of the vehicle.

10. A system for detecting an object ejected from the passenger compartment of a vehicle, comprising: A first camera installed on the exterior of the passenger compartment of the vehicle and configured to generate window image data of the window openings; A controller operably connected to the camera to receive the window image data, the controller being configured to i) process the window image data to generate ejected object data corresponding to at least one object ejected from the passenger compartment through the window opening, and ii) associate the ejected object data with occupant data of the vehicle occupants. as well as A second camera, installed inside the passenger cabin and configured to generate passenger cabin image data, the passenger cabin image data being based at least in part on the movement of the occupants within the passenger cabin. In this embodiment, at least one window status sensor of the vehicle is configured to detect the status of a movable window panel that can move within the window opening, the detected status including an open window status and a closed window status, and The controller is further configured to i) process the cabin image data to detect throwing motion data corresponding to the movement of an occupant's arm or hand toward the window opening, and ii) generate the window image data only when the movable window panel is in the open window state and the throwing motion data is detected.

11. The system according to claim 10, wherein, The first camera is mounted directly on the side mirror housing of the vehicle.

12. The system according to claim 10, wherein: The controller is configured to generate data on the thrown object and send the data to the vehicle's display screen. The display screen is operatively connected to the controller and configured to display messages to the occupants based on data of the ejected object.

13. The system according to claim 12, wherein, The message instructs passengers to stop littering.

Images