Vehicle system and method for determining anomalies inside vehicle

Abstract

The present disclosure relates to a vehicle system for determining an abnormality inside a vehicle, the system comprising: a line-of-sight monitoring device configured to monitor a line-of-sight space origin and a line-of-sight direction of a vehicle occupant with respect to the vehicle; and a processing circuit operably connected to the sight line monitoring device, the processing circuit being configured to determine a region of interest based on the monitored origin of the sight line space and the sight line direction, and to determine whether the region of interest should be considered to be abnormal according to the stored reference information of the interior of the vehicle. The disclosure also relates to a method and a computer program product for determining anomalies inside a vehicle.

Description

Technical Field

[0001] The present disclosure relates to a vehicle system for determining an anomaly inside a vehicle, a method for determining an anomaly inside a vehicle, and a computer program product. Background Art

[0002] Today, some vehicles are used by multiple different vehicle occupants, such as rental vehicles, and also, for example, taxi vehicles that transport different vehicle occupants throughout the day. In the future, it is expected that there will be more driverless vehicles, such as autonomous taxi vehicles and autonomous shuttle vehicles that transport different vehicle occupants.

[0003] Vehicles used by many different vehicle occupants are subject to wear and tear of the vehicle passenger compartment. One problem is tracking the wear and tear. Since there is no driver to track the wear and tear, it is particularly problematic to track the wear and tear of the passenger compartment of an autonomous vehicle.

[0004] Today, the wear and tear must be tracked by individuals through regular inspections. Those inspections are costly and time-consuming because the inspections take time and the use of the vehicle is suspended. For autonomous vehicles, the time required for inspections is even longer because the vehicle needs to go to a repair shop for inspection.

[0005] For today's ordinary vehicles, such as taxi vehicles and family vehicles, there is also an expectation of tracking wear and tear and minimizing the vehicle inspection time.

[0006] This is also the expectation for inspecting the wear and tear of vehicles used by different professionals, such as transport vehicles, police vehicles, and also passenger buses, commuter trains, shuttle buses, elevators, etc. Summary of the Invention

[0007] Therefore, there is a need to minimize wear and tear inspections in order to minimize the time when the vehicle is not in use. The inventors have recognized that humans are very good at noticing anomalies, i.e., things that are not suitable. The inventors have recognized that this information should be used to notice problems inside the vehicle.

[0008] The object of the present disclosure is to mitigate, alleviate, or eliminate one or more of the above-mentioned deficiencies and drawbacks in the prior art and to solve at least the above problems.

[0009] According to a first aspect, there is provided a vehicle system for determining an anomaly inside a vehicle. The system includes: a line-of-sight monitoring device configured to monitor the line-of-sight space origin and line-of-sight direction of a vehicle occupant relative to the vehicle; a processing circuit operably connected to the line-of-sight monitoring device, the processing circuit being configured to: determine a region of interest based on the monitored line-of-sight space origin and line-of-sight direction, and determine whether the region of interest should be regarded as an anomaly according to the stored reference information inside the vehicle.

[0010] The advantage of this first aspect is that it is possible to determine an abnormality inside the vehicle without the need for another person to check, minimizing or even eliminating the need for another person to check, but rather using the line-of-sight spatial origin and line-of-sight direction of the vehicle occupant to determine the abnormality inside the vehicle.

[0011] According to some embodiments, there is provided a vehicle system for determining an abnormality inside a vehicle. The system includes: a line-of-sight monitoring device configured to monitor the line-of-sight spatial origin and line-of-sight direction of a vehicle occupant relative to the vehicle; a processing circuit operably connected to the line-of-sight monitoring device, the processing circuit being configured to: determine a region of interest based on the monitored line-of-sight spatial origin and line-of-sight direction, and determine whether the region of interest should be considered an abnormality compared to other regions of interest inside the vehicle, as defined in the stored vehicle interior reference information, that are known to include prominent features, where the abnormality is defined as the vehicle interior deviating from a particular characteristic that can be considered expected.

[0012] According to some embodiments, the processing circuit is further configured to: in response to determining that the region of interest is abnormal according to the stored vehicle interior reference information, initiate a remedial measure to handle the determined abnormality.

[0013] The advantage of this embodiment is that it is possible to initiate an inspection and / or remedial measure only when considered necessary, which minimizes the downtime when the vehicle cannot be used, thus saving time.

[0014] According to some embodiments, the initiation of the remedial measure includes transmitting a message with information about the abnormality to a vehicle interior cleaning service provider.

[0015] The advantage of this embodiment is that it is possible to remotely notify the interior cleaning service provider via the transmitted message and only when considered necessary, which minimizes the downtime when the vehicle cannot be used, thus saving time.

[0016] According to some embodiments, the monitoring of the line-of-sight spatial origin and line-of-sight direction is performed while the vehicle is operating to provide a transportation service for taking the vehicle occupant to a destination.

[0017] The advantage of this embodiment is that the monitoring of the line-of-sight spatial origin and line-of-sight direction is performed only when there is a vehicle occupant in the vehicle, in a manner of determining a region of interest based on each vehicle occupant when the vehicle is operating to provide a transportation service.

[0018] According to some embodiments, the determination of the region of interest is based on: when the cumulative time that the vehicle occupant gazes at a specific line-of-sight spatial origin and line-of-sight direction exceeds a predefined threshold time value, and / or when the number of times the vehicle occupant gazes in the specific direction within a predetermined time period exceeds a predefined threshold number value.

[0019] The advantage of this embodiment is that the region of interest can be determined more precisely based on the cumulatively obtained data indicating the region of interest.

[0020] According to some embodiments, the processing circuit is further configured to determine a heatmap of the region of interest based on at least a first region of interest, so as to visually present at least the first region of interest to the user via a user interface of the system, in order to facilitate inspection of potential abnormalities inside the vehicle.

[0021] The advantage of this embodiment is that the region of interest can be visually presented (e.g., in the form of a three-dimensional heatmap) onto an image representation of the interior of the vehicle via the user interface.

[0022] According to some embodiments, the processing circuit is further configured to: recognize the facial expression of the vehicle occupant when gazing at the region of interest; and determine a predefined feeling associated with the region of interest based on the recognized facial expression of the vehicle occupant and based on a predefined general facial expression of a human associated with a predetermined feeling.

[0023] The advantage of this embodiment is that by combining the region of interest information with information on how the vehicle occupant feels when gazing at the region of interest, determining whether the region of interest should be regarded as an abnormality can also include: determining whether the abnormality is positive or negative, such as interesting or disgusting.

[0024] According to a second aspect, a method for determining an abnormality inside a vehicle is provided. The method includes the following steps: monitoring the line-of-sight spatial origin and line-of-sight direction of a vehicle occupant, determining a region of interest based on the monitored line-of-sight spatial origin and line-of-sight direction, and determining whether the region of interest should be regarded as an abnormality according to the stored reference information of the interior of the vehicle.

[0025] The advantage of this second aspect is that the abnormality inside the vehicle can be determined without the need for another person to check, minimizing or even eliminating the need for another person to check, but using the line-of-sight spatial origin and line-of-sight direction of the vehicle occupant to determine the abnormality inside the vehicle.

[0026] According to some embodiments, a method for determining an anomaly inside a vehicle is provided, the method comprising the steps of: monitoring a line-of-sight spatial origin and a line-of-sight direction of a vehicle occupant; determining a region of interest based on the monitored line-of-sight spatial origin and line-of-sight direction; and determining whether the region of interest should be considered an anomaly as compared to other regions of interest inside the vehicle, as defined in the stored vehicle interior reference information, which are known to include prominent features, wherein the anomaly is defined as the vehicle interior deviating from a specific characteristic that can be considered expected.

[0027] According to some embodiments, in response to determining that the region of interest is anomalous according to the stored reference information of the vehicle interior, a remedial measure is initiated to handle the determined anomaly.

[0028] The advantage of this embodiment is that inspection and / or remedial measures can be initiated only when considered necessary, which minimizes the downtime when the vehicle is out of use, thus saving time.

[0029] According to some embodiments, the initiation of the remedial measure includes transmitting a message with information about the anomaly to a vehicle interior cleaning service provider.

[0030] The advantage of this embodiment is that the interior cleaning service provider can be remotely notified via the transmitted message and only when considered necessary, which minimizes the downtime when the vehicle is out of use, thus saving time.

[0031] According to some embodiments, the monitoring of the line-of-sight spatial origin and line-of-sight direction is performed while the vehicle is operating to provide a transportation service for taking the vehicle occupant to a destination.

[0032] The advantage of this embodiment is that the monitoring of the line-of-sight spatial origin and line-of-sight direction is performed only when there is a vehicle occupant in the vehicle, in a manner of determining a region of interest based on each vehicle occupant when the vehicle is operating to provide a transportation service.

[0033] According to some embodiments, the determination of the region of interest is based on: when the cumulative time that the vehicle occupant gazes in a specific line-of-sight direction exceeds a predefined threshold time value, and / or when the number of times the vehicle occupant gazes in the specific direction within a predetermined time period exceeds a predefined threshold number value.

[0034] The advantage of this embodiment is that the region of interest can be determined more precisely based on the cumulatively obtained data indicating the region of interest.

[0035] According to some embodiments, the method further comprises the step of determining a region of interest heatmap based on at least a first region of interest to visually present the at least first region of interest to a user via a user interface of the system, so as to facilitate inspection of potential anomalies inside the vehicle.

[0036] The advantage of this embodiment is that the region of interest can be visually presented (e.g., in the form of a three-dimensional heatmap) via the user interface onto an image representation of the interior of the vehicle.

[0037] According to some embodiments, the method further comprises the steps of recognizing a facial expression of a vehicle occupant when gazing at the region of interest; and determining a predefined feeling associated with the region of interest based on the recognized facial expression of the vehicle occupant and based on predefined universal facial expressions of humans associated with a predefined feeling.

[0038] The advantage of this embodiment is that by combining the region of interest information with information on how the vehicle occupant feels when gazing at the region of interest, determining whether the region of interest should be regarded as an anomaly may further include determining whether the anomaly is positive or negative, e.g., disgusting or interesting.

[0039] According to a third aspect, there is provided a computer program product comprising a non-transitory computer-readable medium having a computer program including program instructions thereon, the computer program being loadable into a processing circuit and configured to, when run by the processing circuit, perform the method.

[0040] The effects and features of the second and third aspects are to a large extent similar to the above-mentioned effects and features related to the first aspect. Embodiments related to the first aspect are to a large extent compatible with the second and third aspects.

[0041] The present disclosure will become apparent from the following detailed description. The detailed description and specific examples disclose only the preferred embodiments of the present disclosure by way of illustration. Those skilled in the art will understand from the guidance of the detailed description that changes and modifications can be made within the scope of the present disclosure.

[0042] Accordingly, it should be understood that the disclosure herein is not limited to the specific components of the described devices or the steps of the described methods, as such devices and methods may vary. It should be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. It should be noted that, as used in the specification and the appended claims, unless the context clearly dictates otherwise, the articles "a," "an," "the," and "said" are intended to denote the presence of one or more elements. Thus, for example, reference to "a unit" or "the unit" may include a number of devices and the like. In addition, the words "comprising," "consisting of," "including," and similar words do not exclude other elements or steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and additional objects, features, and advantages of the present disclosure will be more fully understood from the following illustrative and non-limiting detailed description of exemplary embodiments of the present disclosure when taken in conjunction with the accompanying drawings.

[0044] Figure 1 An example vehicle system for determining an abnormality inside a vehicle is shown in accordance with an embodiment of the present disclosure.

[0045] Figure 2 An example vehicle system for determining an abnormality inside an autonomous vehicle is shown in accordance with an embodiment of the present disclosure.

[0046] Figure 3 A flowchart of method steps according to a second aspect of the present disclosure is shown.

[0047] Figure 4 A computer program product according to a third aspect of the present disclosure is shown. DETAILED DESCRIPTION

[0048] The present disclosure will now be described with reference to the accompanying drawings, in which preferred exemplary embodiments of the present disclosure are shown. However, the present disclosure may be embodied in other forms and should not be construed as limited to the embodiments disclosed herein. The disclosed embodiments are provided to fully convey the scope of the present disclosure to those skilled in the art.

[0049] The inventors have recognized that humans are very good at noticing abnormalities, i.e., things that are not appropriate. The inventors have recognized that this information is to be used in order to notice problems inside a vehicle.

[0050] Figure 1 An example vehicle system 100 for determining an abnormality inside a vehicle 1 is shown in accordance with an embodiment of the present disclosure.

[0051] A first aspect of the present disclosure shows a vehicle system 100 for determining an abnormality inside a vehicle 1. The system 100 includes line-of-sight monitoring devices 40a, 40b configured to monitor the line-of-sight space origin and the line-of-sight direction GD of a vehicle occupant VO relative to the vehicle 1.

[0052] According to some embodiments, the line-of-sight space origin and the line-of-sight direction GD of the vehicle occupant VO are obtained, wherein the line-of-sight monitoring devices 40a, 40b are configured to track the eyes of the vehicle occupant VO.

[0053] According to some embodiments, the line-of-sight space origin and the line-of-sight direction GD of the vehicle occupant VO are obtained, wherein the line-of-sight monitoring devices 40a, 40b are configured to track the face of the vehicle occupant VO.

[0054] According to some embodiments, the line-of-sight monitoring devices 40a, 40b are configured to obtain an image of the vehicle occupant VO and obtain the line-of-sight space origin and the line-of-sight direction GD of the vehicle occupant VO, wherein image processing is used to process the image of the vehicle occupant VO in order to determine the line-of-sight space origin and the line-of-sight direction GD of the vehicle occupant VO.

[0055] According to some embodiments, the line-of-sight space origin and the line-of-sight direction GD of the vehicle occupant VO relative to the vehicle 1 are obtained by configuring the line-of-sight monitoring devices 40a, 40b to be installed at a predetermined position on the vehicle 1.

[0056] According to some embodiments, the line-of-sight space origin and the line-of-sight direction GD of the vehicle occupant VO relative to the vehicle 1 are obtained by: configuring the line-of-sight monitoring devices 40a, 40b to be installed at a predetermined position on the vehicle related to the predetermined seating position of the vehicle occupant VO in the vehicle 1.

[0057] According to some embodiments, the line-of-sight space origin and the line-of-sight direction GD of the vehicle occupant VO relative to the vehicle 1 are obtained, wherein the line-of-sight monitoring devices 40a, 40b are configured to determine the position of the eyes of the vehicle occupant VO relative to the monitoring devices 40a, 40b and / or the vehicle 1.

[0058] According to some embodiments, the line-of-sight space origin and the line-of-sight direction GD of the vehicle occupant VO relative to the vehicle 1 are obtained, wherein the line-of-sight monitoring devices 40a, 40b are configured to be calibrated when installed in the vehicle 1 to determine the position of the eyes of the vehicle occupant VO relative to the monitoring devices 40a, 40b and / or the vehicle 1 when the vehicle occupant VO is sitting at the predetermined seating position for the vehicle occupant VO in the vehicle 1.

[0059] According to some embodiments, the line-of-sight monitoring devices 40a, 40b are camera devices configured to perform line-of-sight tracking to monitor the line-of-sight spatial origin and the line-of-sight direction GD of a vehicle occupant VO. According to some embodiments, the line-of-sight monitoring devices 40a, 40b are camera devices configured to perform head tracking to monitor the line-of-sight spatial origin of a vehicle occupant VO. According to some embodiments, the line-of-sight monitoring devices 40a, 40b are camera devices configured to perform head tracking to monitor the line-of-sight direction GD of a vehicle occupant VO. According to some embodiments, the line-of-sight and head tracking are performed by performing image analysis on an image of the vehicle occupant VO. According to some embodiments, the line-of-sight monitoring devices 40a, 40b are further configured to obtain an image of the vehicle occupant VO.

[0060] Figure 1 An example of how the line-of-sight monitoring devices 40a, 40b are arranged in the vehicle 1 is shown. In this example, the first line-of-sight monitoring device 40a is arranged at the front window facing the vehicle occupant, while the second line-of-sight monitoring device 40b is arranged at the front seat facing the vehicle occupant in the rear seat. In Figure 1 the example illustration in, the vehicle occupant VO is sitting in the rear seat of the vehicle, adjacent to the second vehicle occupant 2VO.

[0061] The system 100 further includes processing circuits 102a, 102b, 102c.

[0062] According to some embodiments, the processing circuit 102a is the processing circuit of an in-vehicle computer, as Figure 1 shown.

[0063] According to some embodiments, the processing circuit 102b is included in an electronic device connectable to the vehicle system 100 via a wireless communication network 50, as Figure 1 shown.

[0064] According to some embodiments, the processing circuit 102c is included in a remote electronic device 500 connectable to the vehicle system 100 via a wireless communication network 50, as Figure 1 shown.

[0065] According to some embodiments, the vehicle system 100 further includes memories 101a, 101b, 101c configured to store data.

[0066] According to some embodiments, the memory 101a is the memory of an in-vehicle computer, as Figure 1 shown.

[0067] According to some embodiments, the memory 101b is included in an electronic device connectable to the vehicle system 100 via a wireless communication network 50, as Figure 1 shown.

[0068] According to some embodiments, the memory 101c is included in a remote electronic device 500 that is connectable to the vehicle system 100 via a wireless communication network 50, as Figure 1 shown.

[0069] According to some embodiments, as Figure 1 shown, the wireless communication network 50 is a standardized wireless wide area network, such as Global System for Mobile Communications (GSM), Extended GSM, General Packet Radio Service (GPRS), Enhanced Data Rates for GSM Evolution (EDGE), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), Narrowband Internet of Things (Narrowband-IoT), 5G, Worldwide Interoperability for Microwave Access (WiMAX), or Ultra Mobile Broadband (UMB), or similar networks.

[0070] According to some embodiments, the wireless communication network 50 is a standardized wireless local area network, such as Wireless Local Area Network (WLAN), Bluetooth TM , ZigBee, Ultra-Wideband, Radio Frequency Identification (RFID), or similar networks.

[0071] According to some embodiments, the wireless communication network 50 can also be a combination of a local area network and a wide area network.

[0072] According to some embodiments, the wireless communication network 50 is defined by the general Internet protocol.

[0073] The processing circuits 102a, 102b, 102c are operably connected to the line-of-sight monitoring devices 40a, 40b, configured to determine a region of interest ROI based on the monitored line-of-sight spatial origin and line-of-sight direction GD, and to determine whether the region of interest ROI should be considered abnormal according to the stored reference information inside the vehicle.

[0074] According to some embodiments, an abnormality is defined as a deviation from a specific characteristic that can be considered normal / expected inside the vehicle.

[0075] In Figure 1 the example illustration, the vehicle occupant VO is looking at a dirty spot on the passenger seat. The line-of-sight spatial origin and line-of-sight direction GD are shown in Figure 1 by an arrow from the eyes of the vehicle occupant VO to the passenger seat, at the region of interest ROI. In this example, based on the reference information of the passenger seat of vehicle 1 stored, the region of interest ROI is considered abnormal due to the incorrect coloring of the passenger seat at the region of interest ROI. However, an abnormality can also be an object, such as an item forgotten by a previous vehicle occupant. For example, the item can be a bag, clothes, or a mobile phone, etc.

[0076] The advantage of this first aspect is that it is possible to determine an anomaly inside the vehicle 1 without the need for another person to check, minimizing or even eliminating the need for another person to check, but rather using the origin of the line-of-sight space and the line-of-sight direction of the vehicle occupant VO to determine the anomaly inside the vehicle 1.

[0077] According to some embodiments, a vehicle system 100 for determining an anomaly inside a vehicle 1 is provided. The system 100 includes: line-of-sight monitoring devices 40a, 40b configured to monitor the origin of the line-of-sight space and the line-of-sight direction GD of a vehicle occupant VO relative to the vehicle 1; processing circuits 102a, 102b, 102c operatively connected to the line-of-sight monitoring devices 40a, 40b, the processing circuits being configured to: determine a region of interest ROI based on the monitored origin of the line-of-sight space and the line-of-sight direction GD, and determine whether the region of interest ROI should be considered an anomaly compared to other regions of interest inside the vehicle that are defined in the stored vehicle interior reference information and are known to include prominent features, wherein the anomaly is defined as a deviation from a particular characteristic that can be considered expected inside the vehicle.

[0078] According to some embodiments, the line-of-sight monitoring devices 40a, 40b are configured to obtain an image of the interior of the vehicle.

[0079] According to some embodiments, the line-of-sight monitoring devices 40a, 40b are configured to obtain an image of the region of interest ROI.

[0080] According to some embodiments, the system 100 further includes interior cameras 80a, 80b configured to obtain an image of the interior of the vehicle.

[0081] According to some embodiments, the interior cameras 80a, 80b are configured to obtain an image of the region of interest ROI. The advantage of the interior cameras 80a, 80b is that the line-of-sight monitoring devices 40a, 40b can be used only to monitor the origin of the line-of-sight space and the line-of-sight direction GD of the vehicle occupant VO relative to the vehicle 1, while the interior cameras 80a, 80b can be used only to monitor the interior of the vehicle.

[0082] According to some embodiments, the stored reference information inside the vehicle 1 is an image of the stored reference information of the vehicle compartment part inside the vehicle 1.

[0083] According to some embodiments, the stored reference information image is at least one of the following: a predefined image, or an image captured by one or more cameras before the vehicle occupant enters the vehicle 1 or before the vehicle 1 reaches the pick-up location or near the pick-up location.

[0084] According to some embodiments, the vehicle compartment part is a component of the vehicle 1. In an example, the component is a seat component, a door component, or a floor mat component, etc.

[0085] According to some embodiments, the stored reference information of the vehicle interior is a stored reference information image of the vehicle interior. According to some embodiments, the stored reference information image of the vehicle interior is a three-dimensional model of the interior of vehicle 1.

[0086] According to some embodiments, the processing circuits 102a, 102b, 102c are further configured to: in response to determining that the region of interest is abnormal according to the stored reference information of the vehicle interior, initiate a remedial measure to handle the determined abnormality.

[0087] The advantage of this embodiment is that the inspection and / or remedial measures can be initiated only when considered necessary, which minimizes the downtime when the vehicle is not in use, thus saving time.

[0088] According to some embodiments, the initiation of the remedial measure includes transmitting a message with information about the abnormality to a vehicle interior cleaning service provider.

[0089] The advantage of this embodiment is that the interior cleaning service provider can be remotely notified via the transmitted message, and only when considered necessary, which minimizes the downtime when the vehicle is not in use, thus saving time. According to some embodiments, transmitting the message further includes location information data defining the geographical location of vehicle 1, so as to notify the vehicle interior cleaning service providers near vehicle 1.

[0090] According to some embodiments, while the vehicle is operating to provide a transportation service for taking the vehicle occupants to a destination, the monitoring of the origin of the line-of-sight space and the line-of-sight direction is performed.

[0091] The advantage of this embodiment is that the monitoring of the origin of the line-of-sight space and the line-of-sight direction is performed only when there are vehicle occupants in the vehicle, in a manner of determining the region of interest based on each vehicle occupant when the vehicle is operating to provide a transportation service.

[0092] According to some embodiments, the determination of the region of interest ROI is based on: when the cumulative time that the vehicle occupant VO gazes at a specific origin of the line-of-sight space and line-of-sight direction GD exceeds a predefined threshold time value, and / or when the number of times that the vehicle occupant VO gazes in the specific direction GD within a predetermined time period exceeds a predefined threshold number value.

[0093] The advantage of this embodiment is that the region of interest can be determined more precisely based on the cumulatively obtained data indicating the region of interest ROI.

[0094] According to some embodiments, the processing circuits 102a, 102b, 102c are further configured to: determine a region of interest heat map ROI HM based on at least a first region of interest ROI, to visualize at least the first region of interest 1 ROI to a user via the user interfaces 800a, 800b, 800c of the system 100, so as to facilitate inspection of potential anomalies inside the vehicle 1.

[0095] According to some embodiments, the user interface 800a is a user interface of an in-vehicle computer, as Figure 1 shown. According to some embodiments, the user interface 800b is a user interface of an electronic device connectable to the vehicle system 100 via a wireless communication network 50, as Figure 1 shown. According to some embodiments, the user interface 800c is a user interface of a remote portable electronic device connectable to the vehicle system 100 via a wireless communication network 50, as Figure 1 shown.

[0096] The advantage of this embodiment is that the region of interest ROI can be visually presented (e.g., in the form of a three-dimensional region of interest heat map ROI HM map) via the user interfaces 800a, 800b, 800c onto an image representation inside the vehicle 1. In Figure 1 the illustrated example, a portable electronic device is used to visually present a region of interest heat map ROI HM of at least a first region of interest ROI via the user interface 800c of the portable electronic device. In the example, a provider of vehicle interior cleaning services can, via the user interface 800c of the portable electronic device, use the visual presentation of the region of interest heat map ROI HM to prepare for inspection and cleaning before entering the vehicle.

[0097] According to some embodiments, the processing circuits 102a, 102b, 102c are further configured to: recognize a facial expression of the vehicle occupant VO when gazing at the region of interest ROI, and determine a predefined feeling associated with the region of interest ROI based on the recognized facial expression of the vehicle occupant VO and based on a predefined universal facial expression of a human associated with a predefined feeling.

[0098] The advantage of this embodiment is that by combining the region of interest information with information on how the vehicle occupant feels when gazing at the region of interest, determining whether the region of interest should be regarded as an anomaly can further include: determining whether the anomaly is positive or negative, e.g., interesting or repulsive.

[0099] In an example, this can be an indication of the urgency of inspecting the vehicle. For example, if the region of interest (ROI) is associated with a smile (i.e., being interesting), the urgency of inspecting vehicle 1 is lower. However, if the ROI is associated with a facial expression of disgust, vehicle 1 needs to be inspected urgently.

[0100] According to some embodiments, the processing circuits 102a, 102b, 102c are further configured to: filter out a predetermined origin of the line-of-sight space and the line-of-sight direction, wherein the origin of the line-of-sight space and the line-of-sight direction GD of the vehicle occupant (VO) are determined to be irrelevant to the determination of the region of interest (ROI).

[0101] According to some embodiments, at least a first predefined origin of the line-of-sight space and the line-of-sight direction GD are excluded from the determination process of the region of interest (ROI). According to some embodiments, the at least first predefined origin of the line-of-sight space and the line-of-sight direction GD excluded from the determination process include any one of the following: the origin of the line-of-sight space and the line-of-sight direction to a second vehicle occupant (2VO), the origin of the line-of-sight space and the line-of-sight direction to the dashboard, the origin of the line-of-sight space and the line-of-sight direction to the head-up display, and the origin of the line-of-sight space and the line-of-sight direction through any window. For example, more examples of when to exclude at least the first predefined fixation space origin and the fixation direction GD from the determination process of the region of interest (ROI) may include: eye-catching features such as a screen, a fancy vent, or other components, or the color / shape of a part of the interior, which should not be regarded as abnormal phenomena to be processed. Therefore, such eye-catching features should be regarded as part of the reference. In addition, the sound-generating components or regions inside the vehicle can attract the attention of the vehicle occupant, where he / she can fixate on the direction of the sound. The knowledge of such sound-generating components / regions can also be part of the reference and be excluded from the determination process of the region of interest (ROI).

[0102] According to some embodiments, the processing circuits 102a, 102b, 102c are further configured to: determine the origin of the line-of-sight space and the line-of-sight direction GD of the vehicle occupant (VO); determine the time when the vehicle occupant (VO) is fixating on a specific direction; determine the number of times the vehicle occupant (VO) is fixating on a specific direction GD within a predetermined time period; and identify a region in the origin of the line-of-sight space and the line-of-sight direction GD of the vehicle occupant (VO) as the region of interest (ROI) based on the cumulative time that the vehicle occupant (VO) is fixating on a specific direction and / or the number of times the vehicle occupant (VO) is fixating on a specific direction within a predetermined time period.

[0103] The advantage of this embodiment is that based on the cumulatively obtained data indicating the region of interest (ROI), the region of interest can be determined more precisely.

[0104] According to some embodiments, the processing circuits 102a, 102b, 102c are further configured to: determine the origin and direction of the line of sight of the second vehicle occupant 2VO; determine the time when the second vehicle occupant 2VO is looking in a specific direction; determine the number of times the second vehicle occupant 2VO is looking in a specific direction within a predetermined time period; and identify the region in the origin and direction of the line of sight GD of the vehicle occupant VO that corresponds to the region in the origin and direction of the line of sight of the second vehicle occupant 2VO, and based on the same region that both the vehicle occupant VO and the second vehicle occupant 2VO are looking at, and based on the cumulative time that the vehicle occupant VO and the second vehicle occupant 2VO are looking in a specific direction and / or the number of times the vehicle occupant VO and the second vehicle occupant 2VO are looking in a specific direction within a predetermined time period, identify the region as the region of interest ROI.

[0105] The advantage of this embodiment is that two different vehicle occupants have looked at the same region for a certain time and / or a certain number of times, which gives a more accurate way to determine the region of interest ROI.

[0106] Figure 2 An example vehicle system 100 for determining anomalies inside an autonomous vehicle according to an embodiment of the present disclosure is shown. In Figure 2 there is no driver, but there are two vehicle occupants. In Figure 2 the example illustration in, the vehicle occupant VO is sitting in the back seat of vehicle 1, adjacent to the second vehicle occupant 2VO.

[0107] In Figure 2 the illustration, the first line of sight monitoring device 40a is configured to monitor the origin and direction of the line of sight GD of the vehicle occupant VO relative to vehicle 1, and the second line of sight monitoring device 40b is configured to monitor the origin and direction of the line of sight of the second vehicle occupant 2VO relative to vehicle 1. Figure 2 The illustration in also shows a first interior camera 80a configured to obtain an image of the interior of vehicle 1 and a second interior camera 80b configured to obtain an image of the interior of vehicle 1.

[0108] According to some embodiments, when it is determined that both the vehicle occupant VO and the second vehicle occupant 2VO are looking at the same region of interest ROI, the determination of the region of interest ROI is regarded as an anomaly. In Figure 2 the illustration, both the vehicle occupant VO and the second vehicle occupant 2VO are looking at the upper left part of the passenger seat as the region of interest ROI, and then it is determined that this is regarded as an anomaly.

[0109] According to some embodiments, the determination of the region of interest (ROI) is timestamped, and when it is determined that multiple predefined different vehicle occupants identify the same region as the region of interest (ROI) within a certain time period, the determined region of interest (ROI) should be regarded as an anomaly.

[0110] Figure 3 A flowchart showing the method steps according to the second aspect of the present disclosure is presented.

[0111] The second aspect of the present disclosure shows a method for determining an anomaly inside a vehicle 1. The method includes step S1: monitoring the origin of the line-of-sight space and the line-of-sight direction GD of the vehicle occupant VO; step S2: determining the region of interest (ROI) based on the monitored origin of the line-of-sight space and the line-of-sight direction GD; and step S6: determining whether the region of interest (ROI) should be regarded as an anomaly according to the stored reference information inside the vehicle.

[0112] The advantage of this second aspect is that it is possible to determine the anomaly inside the vehicle 1 without the need for another person to check, minimizing or even eliminating the need for another person to check, but using the origin of the line-of-sight space and the line-of-sight direction of the vehicle occupant VO to determine the anomaly inside the vehicle 1.

[0113] According to some embodiments, a method for determining an anomaly inside a vehicle 1 is provided. The method includes step S1: monitoring the origin of the line-of-sight space and the line-of-sight direction GD of the vehicle occupant VO; step S2: determining the region of interest (ROI) based on the monitored origin of the line-of-sight space and the line-of-sight direction GD; and step S6: determining whether the region of interest (ROI) should be regarded as an anomaly compared to other regions of interest inside the vehicle that are defined in the stored reference information inside the vehicle and are known to include prominent features, where the anomaly is defined as deviating from a specific characteristic that can be considered expected inside the vehicle.

[0114] According to some embodiments, the method includes: in response to determining that the region of interest is anomalous according to the stored reference information inside the vehicle, initiating a remedial measure to handle the determined anomaly.

[0115] The advantage of this embodiment is that the inspection and / or remedial measure can be initiated only when considered necessary, which minimizes the downtime when the vehicle cannot be used, thus saving time.

[0116] According to some embodiments, the initiation of the remedial measure includes transmitting a message with information about the anomaly to a vehicle interior cleaning service provider.

[0117] The advantage of this embodiment is that the interior cleaning service provider can be remotely notified via the transmitted message, and only when considered necessary, which minimizes the downtime when the vehicle cannot be used, thus saving time.

[0118] According to some embodiments, while the vehicle is operating to provide a transportation service for taking the vehicle occupants to a destination, the monitoring of the origin of the sight space and the sight direction is performed.

[0119] The advantage of this embodiment is that the monitoring of the origin of the sight space and the sight direction is only performed when there are vehicle occupants in the vehicle, in a manner of determining the region of interest based on each vehicle occupant when the vehicle is operating to provide a transportation service.

[0120] According to some embodiments, the determination of the region of interest ROI is based on: when the cumulative time that the vehicle occupant VO gazes in a specific sight direction GD exceeds a predefined threshold time value, and / or when the number of times that the vehicle occupant VO gazes in the specific direction GD within a predetermined time period exceeds a predefined threshold number value.

[0121] The advantage of this embodiment is that based on the cumulatively obtained data indicating the region of interest ROI, the region of interest can be determined more precisely.

[0122] According to some embodiments, the method further includes step S3: determining a region of interest heat map ROI HM based on at least a first region of interest ROI, so as to visually present the at least first region of interest 1 ROI to the user via the user interfaces 800a, 800b, 800c of the system 100, for easy inspection of potential abnormalities inside the vehicle 1.

[0123] The advantage of this embodiment is that the region of interest can be visually presented (for example, in the form of a three-dimensional heat map) onto the image representation inside the vehicle via the user interface.

[0124] According to some embodiments, the method further includes step S4: recognizing the facial expression of the vehicle occupant VO when gazing at the region of interest ROI; and step S5: determining a predefined feeling associated with the region of interest ROI based on the recognized facial expression of the vehicle occupant VO and based on a predefined general facial expression of a human associated with a predetermined feeling.

[0125] The advantage of this embodiment is that by combining the region of interest information with information on how the vehicle occupant feels when gazing at the region of interest, determining whether the region of interest should be regarded as abnormal can also include: determining whether the abnormality is positive or negative, such as disgusting or interesting.

[0126] Figure 4 A computer program product according to a third aspect of the present disclosure is shown.

[0127] A third aspect of the present disclosure shows a computer program product. The second aspect includes a non-transitory computer-readable medium having a computer program including program instructions, the computer program being loadable into processing circuits 102a, 102b, 102c and configured to cause the execution of the method when the computer program is run by the processing circuits 102a, 102b, 102c.

[0128] Those skilled in the art will recognize that the present disclosure is not limited to the above preferred embodiments. Those skilled in the art will also recognize that modifications and variations can be made within the scope of the appended claims. Additionally, by studying the drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement changes to the disclosed embodiments when practicing the claimed disclosure.

Claims

1. A vehicle system (100) for determining an anomaly inside a vehicle (1), the system (100) comprising: Line-of-sight monitoring devices (40a, 40b) configured to monitor the line-of-sight spatial origin and line-of-sight direction (GD) of a vehicle occupant (VO) relative to the vehicle (1); Processing circuits (102a, 102b, 102c) operatively connected to the line-of-sight monitoring devices (40a, 40b), the processing circuits being configured to: - Determine a region of interest (ROI) based on the monitored line-of-sight spatial origin and line-of-sight direction (GD); And - Determine whether the region of interest (ROI) should be considered an anomaly according to reference information inside the vehicle stored.

2. The system (100) according to any one of the preceding claims, wherein, The processing circuits (102a, 102b, 102c) are further configured to: - In response to determining that the region of interest is an anomaly according to the reference information inside the vehicle stored, initiate a remedial measure to handle the determined anomaly.

3. The system (100) according to claim 2, wherein, The initiation of the remedial measure includes transmitting a message with information about the anomaly to a vehicle interior cleaning service provider.

4. The system (100) according to any one of the preceding claims, wherein, The monitoring of the line-of-sight spatial origin and line-of-sight direction is performed while the vehicle is operating to provide a transportation service that takes the vehicle occupant to a destination.

5. The system (100) according to any one of the preceding claims, wherein, The determination of the region of interest (ROI) is based on: when the cumulative time that the vehicle occupant (VO) gazes at a specific line-of-sight spatial origin and line-of-sight direction (GD) exceeds a predefined threshold time value, and / or when the number of times the vehicle occupant (VO) gazes in the specific direction (GD) within a predetermined time period exceeds a predefined threshold number value.

6. The system (100) according to any one of the preceding claims, wherein, The processing circuits (102a, 102b, 102c) are further configured to: - Determine a region-of-interest heat map (ROIHM) based on at least a first region of interest (ROI) to visually present at least the first region of interest (1ROI) to a user via a user interface (800a, 800b, 800c) of the system (100) for easy inspection of potential anomalies inside the vehicle (1).

7. The system (100) according to any one of the preceding claims, wherein, The processing circuits (102a, 102b, 102c) are further configured to: - Identify the facial expression of the vehicle occupant (VO) when gazing at the region of interest (ROI); and - Determine a predefined feeling associated with the region of interest (ROI) based on the identified facial expression of the vehicle occupant (VO) and based on predefined general facial expressions of humans associated with a predetermined feeling.

8. A method for determining an anomaly inside a vehicle (1), the method comprising: -(S1) Monitoring the line-of-sight spatial origin and line-of-sight direction (GD) of a vehicle occupant (VO), -(S2) Determining a region of interest (ROI) based on the monitored line-of-sight spatial origin and line-of-sight direction (GD), and -(S6) Determining whether the region of interest (ROI) should be considered an anomaly according to the reference information inside the vehicle stored.

9. The method according to claim 8, comprising: In response to determining that the region of interest is an anomaly according to the reference information inside the vehicle stored, initiate a remedial measure to handle the determined anomaly.

10. The method according to claim 9, wherein, Initiation of the remedial measure includes transmitting a message with information about the anomaly to a vehicle interior cleaning service provider.

11. According to the method according to any one of claims 8-10, wherein, Monitoring of the line-of-sight space origin and line-of-sight direction is performed while the vehicle is operating to provide a transportation service for taking the vehicle occupants to a destination.

12. The method according to any one of claims 8-11, wherein, Determination of the region of interest (ROI) is based on: when the cumulative time that the vehicle occupant (VO) gazes in a specific direction (GD) exceeds a predefined threshold time value, and / or when the number of times the vehicle occupant (VO) gazes in the specific direction (GD) within a predetermined time period exceeds a predefined threshold number value.

13. The method according to any one of claims 8 - 12, further comprising: -(S3) Determining a region of interest heat map (ROIHM) based on at least a first region of interest (ROI) to visually present the at least first region of interest (1ROI) to a user via a user interface (800a, 800b, 800c) of the system (100) for easy inspection of potential anomalies inside the vehicle (1).

14. The method according to any one of claims 8 - 13, further comprising: -(S4) Identifying the facial expression of the vehicle occupant (VO) when gazing at the region of interest (ROI); and -(S5) Determining a predefined feeling associated with the region of interest (ROI) based on the identified facial expression of the vehicle occupant (VO) and based on predefined universal facial expressions of humans associated with predetermined feelings.

15. A computer program product (500) comprising a non-transitory computer-readable medium having a computer program including program instructions thereon, the computer program being loadable into a processing circuit (102a, 102b, 102c) and configured to: when the computer program is run by the processing circuit (102a, 102b, 102c), perform the method according to any one of claims 8 to 14.

Images