Anomaly Detection System and Anomaly Detection Method

Abstract

This abnormality detection system (1) comprises a plurality of applications for respectively detecting an abnormality on the basis of image data from a camera (25). When the plurality of applications are executed, a detection unit (41) analyzes the respective image data and detects abnormalities. A transmission unit (43) transmits the analysis result obtained through analysis by the detection unit (41) and at least image data used when an abnormality is detected to a cloud (5). A storage unit (61) stores the analysis result and the image data transmitted from the transmission unit (23). The abnormality detection system (1) reports the analysis result to a report target from at least one of an on-vehicle device 3 and the cloud 5.

Description

Cross - reference to related applications

[0001] This international application claims priority based on Japanese Patent Application No. 2022 - 073548, filed with the Japan Patent Office on April 27, 2022, and incorporates by reference the entire contents of Japanese Patent Application No. 2022 - 073548 into this international application.

Technical Field

[0002] This disclosure relates to a technique for detecting abnormalities such as dirt on seats in a vehicle interior.

Background Art

[0003] Conventionally, as a technique for detecting dirt in a vehicle interior, there is known a technique of arranging a camera in the vehicle interior, photographing the vehicle interior with the camera, and detecting dirt by analyzing the photographed image (see, for example, Patent Document 1).

[0004] In this conventional technique, the image photographed by the camera is transmitted to the cloud side, and the cloud side analyzes the image to detect dirt and the like.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

[0006] As a result of the inventors' detailed examination, the following problems were found with the conventional technology.

[0007] Specifically, when photographing the vehicle interior with a camera and detecting dirt and the like based on the photographed image, it is conceivable to perform some processing on the vehicle side and also some processing on the cloud side, but the examination of what kind of processing and the like should be appropriately performed on each side is not sufficient.

[0008] For example, when applying the above-described conventional technology to a service that shares vehicles (e.g., carsharing), sufficient consideration has not been given to what processes should be performed on the vehicle side or the cloud side to be preferable for the carsharing operator or the user who uses the vehicle (e.g., whether the convenience is high).

[0009] One aspect of the present disclosure desirably provides a technology that is preferable for an operator or a user when communicating between a vehicle and a cloud and performing processing related to an abnormality such as dirt in the vehicle interior.

[0010] [1] An abnormality detection system (1) according to one aspect of the present disclosure includes a cloud (5) that collects data of a vehicle (9), and an in-vehicle device (3) communicably connected to the cloud, and is an abnormality detection system that detects an abnormality in the vehicle interior.

[0011] This abnormality detection system includes a plurality of applications configured to detect respective target abnormalities based on image data from a camera (25) that has photographed the vehicle interior.

[0012] The in-vehicle device includes a detection unit (41) and a transmission unit (43). The detection unit is configured to analyze the image data and detect an abnormality when respectively executing the plurality of applications. The transmission unit is configured to transmit the analysis result analyzed by the detection unit and the image data used at least when an abnormality is detected to the cloud.

[0013] The cloud includes a storage unit (61). The storage unit is configured to store the analysis result and the image data transmitted from the transmission unit.

[0014] Furthermore, the abnormality detection system is configured to notify the analysis result to a notification target from at least one of the in-vehicle device and the cloud.

[0015] With such a configuration, in the anomaly detection system according to one aspect of the present disclosure, when communicating between a vehicle and the cloud to perform processing related to anomalies such as dirt inside the vehicle cabin, for example, it is possible to provide a technology that is favorable (for example, highly convenient) for business operators engaged in car-sharing or users who use the vehicle.

[0016] Specifically, in the anomaly detection system according to one aspect of the present disclosure, in the in-vehicle device, when each of a plurality of applications is executed, it analyzes image data to detect anomalies, and transmits the analysis result obtained by the detection unit and at least the image data (i.e., predetermined image data) used when an anomaly is detected to the cloud. On the other hand, in the cloud, it stores the analysis result and the predetermined image data transmitted from the transmission unit. Then, it notifies the analysis result from the in-vehicle device or the cloud to notification targets such as business operators and users.

[0017] In this way, in the anomaly detection system according to one aspect of the present disclosure, it is possible to detect anomalies such as dirt based on image data obtained by photographing the interior of the vehicle. Further, by transmitting the analysis result such as the anomaly detection result and the predetermined image data to the cloud, the cloud can store the analysis result and the image data.

[0018] Thereby, the analysis result and the image data (for example, the image data serving as the basis for the anomaly) can be reliably stored, so that the basis for taking corresponding actions according to the analysis result at a later date becomes reliable. Further, since the analysis result is notified to business operators and users, the business operators and users who have received the notification can take appropriate actions according to the content of the notification.

[0019] [2] An anomaly detection system (1) according to another aspect of the present disclosure includes a cloud (5) that collects data of a vehicle (9) and an in-vehicle device (3) communicably connected to the cloud, and is an anomaly detection system that detects anomalies inside the vehicle cabin.

[0020] This anomaly detection system includes a first application and a second application configured to detect respective target anomalies based on image data from a camera that captures the interior of a vehicle. The anomalies detected by the first application have a higher urgency to notify when an anomaly is detected than the anomalies detected by the second application.

[0021] The in-vehicle device includes a first detection unit (121), a first transmission unit (123), and a second transmission unit (125).

[0022] The first detection unit is configured to analyze the image data to detect anomalies when implementing the first application.

[0023] The first transmission unit is configured to transmit the analysis result analyzed by the first detection unit to the cloud, and also transmit at least the image data used when an anomaly is detected to the cloud.

[0024] The second transmission unit is configured to transmit the image data to the cloud when implementing the second application.

[0025] The cloud includes a first storage unit (131), a second detection unit (133), and a second storage unit (135).

[0026] The first storage unit is configured to store the analysis result and the image data transmitted from the first transmission unit when implementing the first application.

[0027] The second detection unit is configured to analyze the image data transmitted from the second transmission unit to detect anomalies when implementing the second application.

[0028] The second memory unit is configured to store the image data transmitted from the second transmission unit and the analysis result analyzed by the second detection unit.

[0029] Furthermore, the abnormality detection system is configured to notify the analysis result to the notification target from at least one of the in-vehicle device and the cloud.

[0030] With such a configuration, in the abnormality detection system according to another aspect of the present disclosure, when communicating between the vehicle and the cloud to perform processing related to abnormalities such as dirt in the vehicle interior, for example, it is possible to provide a technology that is preferable for business operators who perform car sharing and users who use the vehicle.

[0031] Moreover, in this abnormality detection system, when an abnormality with a high urgency (i.e., priority) of notification is detected, it is immediately notified to the notification target, so that the notification target such as the business operator or the user can take corresponding measures for the notification content.

[0032] [3] An abnormality detection system according to an aspect of the present disclosure includes a cloud (5) that collects data of a vehicle (9), and an in-vehicle device (3) that is communicably connected to the cloud and is communicably connected to a relay device (23) that relays frames flowing through the vehicle network, and is an abnormality detection system (1) that detects abnormalities in the vehicle interior.

[0033] The in-vehicle device includes an in-vehicle communication unit (45), a detection unit (41), and a transmission unit (43).

[0034] The in-vehicle communication unit is configured to communicate with electronic control devices (32, 36) connected to the vehicle network via a relay device. The detection unit is configured to analyze image data from a camera (25) that captures the interior of the vehicle to detect abnormalities. The transmission unit is configured to transmit the analysis result analyzed by the detection unit to the cloud and transmit at least the image data used when an abnormality is detected to the cloud.

[0035] Furthermore, the in-vehicle device is configured to notify the analysis result analyzed by the detection unit to the outside of the vehicle.

[0036] With such a configuration, in the abnormality detection system according to one aspect of the present disclosure, when communicating between the vehicle and the cloud to perform processing related to abnormalities such as dirt in the vehicle interior, for example, it is preferable (for example, highly convenient) for a business operator providing car-sharing or a user using the vehicle. technology can be provided.

[0037] [4] An abnormality detection method according to one aspect of the present disclosure is an abnormality detection method capable of communicating between an in-vehicle device (3) mounted on a vehicle (9) and a cloud (5) and detecting an abnormality in the vehicle interior.

[0038] In this abnormality detection method, a plurality of applications configured to detect respective target abnormalities are used based on image data from a camera (25) that has photographed the interior of the vehicle.

[0039] In the in-vehicle device, when each of the plurality of applications is executed, the image data is analyzed to detect an abnormality, the analyzed result is transmitted to the cloud, and at least the image data used when an abnormality is detected is transmitted to the cloud. In the cloud, the transmitted analysis result and image data are stored.

[0040] Furthermore, in this abnormality detection method, the analysis result is notified to the notification target from at least one of the in-vehicle device and the cloud.

[0041] With such a configuration, in the abnormality detection method according to one aspect of the present disclosure, when communicating between the vehicle and the cloud to perform processing related to abnormalities such as dirt in the vehicle interior, for example, it is preferable for a business operator providing car-sharing or a user using the vehicle. technology can be provided.

[0042] [5]The abnormal detection method according to another aspect of the present disclosure is an abnormal detection method capable of communicating between an in-vehicle device (3) mounted on a vehicle (9) and a cloud (5) and detecting an abnormality inside the vehicle cabin.

[0043] In this abnormal detection method, a first application and a second application configured to detect a target abnormality are used based on image data from a camera (25) that captures the interior of the vehicle cabin. The abnormality detected by the first application has a higher urgency to notify when the abnormality is detected than the abnormality detected by the second application.

[0044] In the in-vehicle device, when implementing the first application, the image data is analyzed to detect an abnormality, the analyzed analysis result is transmitted to the cloud, and at least the image data used when an abnormality is detected is transmitted to the cloud. When implementing the second application, the image data is transmitted to the cloud.

[0045] In the cloud, when implementing the first application, the transmitted analysis result and image data are stored. When implementing the second application, the transmitted image data is analyzed to detect an abnormality, and the transmitted image data and the analyzed analysis result are stored.

[0046] Furthermore, in this abnormal detection method, the analysis result is notified to the notification target from at least one of the in-vehicle device and the cloud.

[0047] With such a configuration, in the abnormal detection method according to another aspect of the present disclosure, when communicating between the vehicle and the cloud to perform processing related to an abnormality such as dirt inside the vehicle cabin, for example, a preferable technology can be provided for a business operator performing car sharing or a user using the vehicle.

[0048] [6] Another aspect of the abnormality detection method of the present disclosure is an abnormality detection method for detecting an abnormality in a vehicle interior, using a cloud (5) that collects data of a vehicle (9), and an in-vehicle device (3) that is communicably connected to the cloud and is communicably connected to a relay device (23) that relays frames flowing through the network of the vehicle.

[0049] In this abnormality detection method, the in-vehicle device communicates with electronic control devices (32, 36) connected to the vehicle network via the relay device, analyzes image data from a camera (25) that captures the vehicle interior to detect an abnormality, transmits the analyzed result to the cloud, and also transmits at least the image data used when an abnormality is detected to the cloud. Further, in this cloud, the transmitted analysis result is notified to the notification target.

[0050] With such a configuration, in the abnormality detection method of another aspect of the present disclosure, when communicating between the vehicle and the cloud to perform processing related to an abnormality such as dirt in the vehicle interior, for example, it is possible to provide a preferable technology for an operator performing carsharing or a user using the vehicle.

[0051] Further, the reference numerals in parentheses described in this column and the claims indicate the correspondence relationship with the specific means described in the embodiments described later as one aspect, and do not limit the technical scope of the present disclosure.

Brief Description of Drawings

[0052]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0053] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings.

[0054] [1. First Embodiment] In this first embodiment, as an example of a mobility IoT system, an abnormality detection system for detecting abnormalities such as stains in the interior of a vehicle (for example, an automobile) will be described. Note that IoT is an abbreviation for Internet of Things.

[0055] [1-1. Overall Configuration] First, the overall configuration of the abnormality detection system 1 of this first embodiment will be described based on FIG. 1.

[0056] As shown in FIG. 1, the abnormality detection system 1 includes an in-vehicle device 3, a cloud 5, and a service providing server 7. Note that a server that manages the operations of the cloud 5 and the like is referred to as a management server.

[0057] In FIG. 1, for the sake of convenience, only the in-vehicle device 3 is described, but the abnormality detection system 1 may include, for example, a plurality of in-vehicle devices 3, and the plurality of in-vehicle devices 3 may be mounted on different vehicles 9, respectively.

[0058] The in-vehicle device 3 can communicate wirelessly with the cloud 5 and the mobile terminal 15 via the communication device 11 mounted on the vehicle 9. The detailed configurations of the in-vehicle device 3 and the vehicle 9 will be described later.

[0059] The cloud 5 can communicate with the in-vehicle device 3, the service providing server 7, and the mobile terminal 15 via the communication unit 13. The communication unit 13 can communicate wirelessly with the in-vehicle device 3 and the mobile terminal 15. The cloud 5 can collect data of the vehicle 9 from the in-vehicle device 3 via the communication device 11 and the communication unit 13. The detailed configuration of the cloud 5 will be described later.

[0060] The service providing server 7 can communicate with the cloud 5. The service providing server 7 is, for example, a server installed to provide services such as managing the operation of the vehicle 9. The abnormality detection system 1 may include a plurality of service providing servers 7 with different service contents.

[0061] The mobile terminal 15 is, for example, a mobile terminal (i.e., an information terminal) owned by a car-sharing operator. Examples of the mobile terminal 15 include a smartphone, a tablet terminal, a notebook PC, etc. In addition to the mobile terminal 15, a desktop-type computer may be used.

[0062] As will be described later, it may be possible to communicate wirelessly with the user's mobile terminal 16 (see, for example, FIG. 3) via the communication device 11 of the in-vehicle device 3. Also, it may be possible to communicate wirelessly with the user's mobile terminal 16 via the communication unit 13 of the cloud 5.

[0063] Hereinafter, each configuration will be described in detail.

[0064] [1-2. Configuration on the Vehicle Side] Next, the configuration on the vehicle 9 side will be described with reference to FIGS. 1 to 4.

[0065] As shown in FIG. 1, in addition to the in-vehicle device 3, the vehicle 9 includes a sensor 21, a vehicle ECU 23, a camera 25, a lighting device 27, a communication device 11, and an alert device 29.

[0066] The sensor 21 is a detection device that detects the state of the vehicle 9. Examples of this sensor 21 include various sensors that detect states such as the engine being turned on or off, the start or stop of driving, the vehicle speed, the shift position, the presence or absence of seating on the seat 40 (see, for example, FIG. 3), the opening and closing of the door, and the locking (i.e., locking) or unlocking (i.e., unlocking) of the door.

[0067] The vehicle ECU 23 is an electronic control unit (i.e., ECU) connected to the sensor 21. In this vehicle ECU 23, a signal from the sensor 21 is received and the signal is processed as necessary. Further, a signal (i.e., information) obtained from the sensor 21 is transmitted from the vehicle ECU 23 to the in-vehicle device 3 via a communication line.

[0068] As shown in FIG. 3, the camera 25 is one or more in-vehicle cameras arranged in the vehicle interior for photographing the vehicle interior, and for example, an infrared camera is used. Note that a digital camera such as a CCD camera may also be used. Note that a color image can be adopted as the photographed image.

[0069] Examples of the mounting position of the camera 25 include the upper part of the windshield, the vicinity of the rearview mirror, and the ceiling. The photographing range of the camera 25 is set to include a range where objects are likely to be placed or a range where dirt is likely to adhere in the vehicle interior. Specifically, the photographing range of the camera 25 is set to include, for example, some or all of the driver's seat, the passenger seat, each seat 40 of the rear seats (e.g., the seat surface and the backrest portion of the seat 40), the dashboard, and the inner surface of the door.

[0070] In addition, a plurality of cameras 25 may be arranged so that the imaging target can be imaged from different angles to enable detection of three-dimensional objects.

[0071] As shown in FIG. 3, the lighting device 27 is a light that lights up to illuminate the interior of the vehicle when the interior of the vehicle is imaged by the camera 25. For example, a light that irradiates infrared rays, an LED light, or the like can be adopted.

[0072] The communication device 11 is a communication device capable of wireless communication with the communication unit 13 of the cloud 5, the mobile terminals 15 and 16. Through this communication device 11, image data, analysis results of the image data, and the like are transmitted from the vehicle 9 to the cloud 5. As will be described later, the in-vehicle device 3 can be controlled by a signal from the mobile terminal 15.

[0073] The alert device 29 is a device that issues a warning to users of the vehicle 9 or the like by means of an electronic sound, voice, or the like. As this alert device 29, a speaker or the like can be adopted.

[0074] <In-vehicle network> Here, the configuration regarding the network in the vehicle 9 connected to the in-vehicle device 3 will be described. The in-vehicle device 3 can be communicably connected to the vehicle ECU 23 or the like by being retrofitted to the network in the vehicle 9.

[0075] As shown in FIG. 2, in the vehicle 9, the vehicle ECU 23 includes a CPU 24 and a memory 26 such as a ROM 26a and a RAM 26b as a configuration for performing various arithmetic processes.

[0076] The vehicle ECU 23 is connected to a plurality of ECUs 32 and an off-vehicle communication device 34 that communicates with the outside of the vehicle through an in-vehicle communication network 30 or the like that performs in-vehicle communication. Each ECU 32 is communicably connected to another ECU 36.

[0077] The vehicle ECU 23 can achieve coordinated control for the entire vehicle 9 by integrating a plurality of ECUs 32. Each ECU 32 is provided for each domain classified by the functions in the vehicle 9, and can mainly execute the control of a plurality of ECUs 36 existing within that domain. The domains are, for example, the power train, the body, the chassis, the cockpit, etc. Note that the ECU 36 is, for example, an ECU that controls sensors and actuators.

[0078] Note that the network within the vehicle 9 is used to transmit and receive frames containing various information among each component within the vehicle 9 (for example, the in-vehicle device 3, ECUs 23, 32, 36, the vehicle external communication device 34, etc.). Examples of this network include the in-vehicle communication network 30, etc. Also, the in-vehicle device 3 is connected to the vehicle battery 50, and the power supply is shared with other electrical components within the vehicle 9.

[0079] <In-vehicle device> Next, the in-vehicle device 3 will be described in detail.

[0080] The in-vehicle device 3 includes a control unit 31 and a storage unit 33. The control unit 31 includes a CPU 35 and a semiconductor memory such as a RAM or a ROM (hereinafter referred to as the memory 37). Note that the control unit 31 is constituted by, for example, a microcomputer or the like.

[0081] The functions of the control unit 31 are realized by the CPU 35 executing a program stored in a non-transitory tangible recording medium (that is, the memory 37). Also, when this program is executed, a method corresponding to the program is executed.

[0082] Note that the method for realizing the various functions of the control unit 31 is not limited to software, and for some or all of its elements, one or a plurality of hardware may be used for realization. For example, when the above functions are realized by an electronic circuit which is hardware, the electronic circuit may be realized by a digital circuit including a number of logic circuits, an analog circuit, or a combination thereof.

[0083] Also, in the first embodiment, the memory 37 stores a plurality of applications (i.e., programs) configured to detect respective target abnormalities based on image data from the camera 25 that captures the interior of the vehicle 9.

[0084] For example, as will be described in detail later, programs for detecting dirt on the seat 40, programs for detecting forgotten items on the seat 40, etc. are stored.

[0085] The storage unit 33 is a storage that can store information. In this storage unit 33, for example, information (i.e., image data) of an image captured by the camera 25 can be stored. Also, the result of analyzing the image (i.e., the analysis result) can be stored. Note that examples of the storage unit 33 include a hard disk drive (i.e., HDD) and a solid state drive (i.e., SSD).

[0086] <Functional Configuration of the Control Unit> Here, the functional configuration of the control unit 31 will be described.

[0087] As shown in FIG. 4, the control unit 31 of the vehicle 9 functionally includes a detection unit 41, a transmission unit 43, and an in-vehicle communication unit 45.

[0088] When the detection unit 41 executes the plurality of programs (i.e., applications) respectively, it acquires and analyzes the image data (i.e., the image data of the pre-image and the post-image described later) captured by the camera 25 respectively, and is configured to detect abnormalities in the vehicle interior such as dirt and forgotten items on the seat 40.

[0089] The transmission unit 43 is configured to drive the communication device 11 to transmit the analysis result that is the detection result of the abnormality (for example, the analysis result indicating that an abnormality has been detected) and at least the image data used when the abnormality is detected (for example, the image data of the pre-image and the post-image) to the cloud 5.

[0090] Note that the analysis results may be notified from the vehicle 9 to the business operator's mobile terminal 15 and the user's mobile terminal 16.

[0091] The in-vehicle communication unit 45 is configured to communicate with the vehicle ECU 23 and to communicate with other ECUs 32 and 36 via the vehicle ECU 23.

[0092] [1-3. Configuration on the Cloud Side] Next, the configuration on the cloud 5 side will be described based on FIGS. 1, 3, and 5.

[0093] The cloud 5 includes a control unit 51, a communication unit 13, and a storage unit 53. The control unit 51 includes a CPU 55 and a semiconductor memory such as a RAM or a ROM (hereinafter referred to as a memory 57 which is a non-transitory physical recording medium). The configuration and functions of the control unit 51 are basically the same as those of the control unit 31 of the vehicle 9, and are realized by the CPU 55 executing a program stored in the memory 57. Further, when this program is executed, a method corresponding to the program is executed.

[0094] The communication unit 13 can perform wireless communication with the communication device 11 and the mobile terminal 15. For example, in the cloud 5, the analysis results and image data transmitted from the vehicle 9 can be received via the communication device 11 and the communication unit 13.

[0095] The storage unit 53 is a storage that stores the same information as the storage unit 33 of the vehicle 9, and can store the analysis results and image data received from the vehicle 9.

[0096] Note that the cloud 5 configured as described above can collect the data of the vehicle 9 transmitted via the communication device 11 from each of the plurality of in-vehicle devices 3. Further, the cloud 5 can store the collected data in the storage unit 53 for each vehicle 9.

[0097] In addition, the cloud 5 creates a digital twin based on the data of the vehicle 9 stored in the storage unit 53. The digital twin is normalized index data. The service providing server 7 can acquire the data of a predetermined vehicle stored in the storage unit 53 by using the index data obtained from the digital twin. The service providing server 7 determines the control content of the vehicle 9 and transmits an instruction corresponding to the control content to the cloud 5. The cloud 5 transmits the control content to the vehicle 9 based on the instruction.

[0098] <Functional Configuration of the Control Unit> Here, the functional configuration of the control unit 51 will be described.

[0099] As shown in FIG. 5, the control unit 51 of the cloud 5 functionally includes a storage unit 61.

[0100] The storage unit 61 is configured to store, for example, the analysis result and the image data transmitted from the communication device 11 of the vehicle 9 in the storage unit 53.

[0101] Note that the analysis result stored in the storage unit 53 is notified from the cloud 5 to the business operator's mobile terminal 15 via the communication unit 13, but it may also be notified to the user's mobile terminal 16.

[0102] [1-4. Overall Operation] Next, based on FIG. 3 and the like, the overall operation of the abnormality detection system 1 will be described by taking the case of car sharing as an example.

[0103] <Operation Before Using the Vehicle> (1) A user who uses the vehicle 9 through car sharing usually registers and obtains an IC card (not shown) used when using the vehicle 9.

[0104] (2) When the user intends to use the vehicle 9, the user reserves the use of the vehicle 9 in advance using a mobile terminal 16 such as a smartphone.

[0105] (3) Next, when using the vehicle 9, the user goes to the location where the vehicle 9 is parked at the reserved time. Then, the user can unlock the door by holding the IC card up to a reader (not shown) provided on the window or the like of the vehicle 9.

[0106] (4) Also, for example, the interior of the vehicle can be photographed by the camera 25 at the timing when the door lock is released by holding the IC card up to the reader (i.e., the pre-image can be acquired).

[0107] (5) Next, the user opens the door and gets in the vehicle, and obtains the key (not shown) of the vehicle 9 stored in the glove box or the like in the vehicle interior. When obtaining the key, it is input that the vehicle 9 is in use by a switch or the like. Then, the vehicle 9 is started by turning on the engine of the vehicle 9 using the key.

[0108] <Operations after using the vehicle> (1) When the use of the vehicle 9 ends and the vehicle 9 is stopped, the key is returned to a predetermined position in the glove box. When returning the key, it is input that the use of the vehicle 9 has ended by a switch or the like.

[0109] (2) Next, the user opens the door of the vehicle 9, gets out, and closes the door.

[0110] (3) Next, the door is locked by holding the IC card up to the reader of the vehicle 9. Thereby, the use of the vehicle 9 is completed.

[0111] (4) And, for example, the interior of the vehicle can be photographed by the camera 25 at the timing when the door is locked by holding the IC card up to the reader (i.e., the post-image can be acquired).

[0112] Note that here, examples of automatically photographing the interior of the vehicle at each of the above timings before getting in and after getting out have been given. However, for example, the interior of the vehicle may be photographed according to a command from the operator (e.g., by remote operation via the Internet or the like).

[0113] <Other determination methods before boarding and after alighting> As a method for determining before boarding (i.e., before use) and after alighting (i.e., after use), as described above, a method of using door unlocking or door locking with an IC card can be considered. That is, a method can be considered in which it is before boarding when the door is unlocked with the IC card, and it is after alighting when the door is locked with the IC card.

[0114] In addition, if it is before boarding, the interior of the vehicle before boarding can be photographed to obtain a front image, and if it is after alighting, the interior of the vehicle after alighting can be photographed to obtain a rear image.

[0115] In addition, various other methods can be considered.

[0116] For example, as described later, when the door is opened and closed (i.e., when the door is opened and then closed), if there is no person in the vehicle interior, it may be determined that it is after alighting. Note that when the door is opened and closed and there is a person in the vehicle interior, it may be determined that the person is in the vehicle.

[0117] Also, for example, when the door is unlocked and it is confirmed by, for example, a seating sensor or the like that there is no person in the vehicle, it may be determined that it is before boarding. Furthermore, for example, when it is detected by a seating sensor that the state has changed from a seated state to a non-seated state and then the door is locked, it may be determined that it is after alighting.

[0118] Furthermore, since the date and time when the user uses the vehicle 9 is reserved, before the predetermined time of use start, for example, the interior of the vehicle can be automatically photographed by a control signal from the cloud 5 to obtain a front image.

[0119] Note that in this first embodiment, the in-vehicle device 3 is in a state where power is supplied and it can operate until at least the front image and the rear image are acquired, an abnormality is detected, and the analysis result and the image data are transmitted to the cloud 5.

[0120] <Post-Use Processing of Vehicle> (1) In the vehicle 9 (i.e., the in-vehicle device 3), when the use of the vehicle 9 is completed, based on the above-described image taken before boarding (i.e., the pre-image) and the image taken after getting off (i.e., the post-image), an abnormality in the vehicle interior (e.g., dirt on the seat 40 or an item left on the seat 40) is detected. The method for detecting this abnormality will be described in detail later.

[0121] (2) Next, by analyzing the data of both images (i.e., the image data), it is determined whether there is an abnormality in the vehicle interior. Then, this analysis result and the image data used for the analysis are transmitted from the vehicle 9 to the cloud 5.

[0122] As this analysis result, there are cases where an abnormality is detected (i.e., the abnormal case) and cases where no abnormality is detected (i.e., the normal case). It is desirable to transmit the analysis results of both the abnormal case and the normal case to the cloud 5. Note that, only in the abnormal case, "notification that an abnormality has been detected" may be transmitted to the cloud 5.

[0123] Also, as for the case of transmitting the image data to the cloud 5, it is conceivable to transmit only in the abnormal case the image data (i.e., the pre-image and the post-image) used for detecting the abnormality. Note that, also in the normal case, the image data may be transmitted to the cloud 5.

[0124] Note that, regarding whether to transmit the analysis result and the image data only when there is an abnormality or whether to transmit the analysis result and the image data regardless of whether it is abnormal or normal, the same applies to the second embodiment.

[0125] (3) In the cloud 5, the analysis result (e.g., the presence or absence of dirt or an item left behind) and the image data transmitted from the vehicle 9 are stored in the storage unit 53.

[0126] (4) Also, in the cloud 5, for example, when an abnormality such as dirt or an item left behind is detected, the analysis result is transmitted to the business operator's mobile terminal 15. Note that, even when no abnormality is detected, the analysis result may be transmitted to the mobile terminal 15.

[0127] Further, the analysis result may be transmitted to the user's mobile terminal 16, for example, when there is an item left behind. In that case, an alert device 29 such as a speaker may be used to notify the user that there is an item left behind.

[0128] [1-5. Method for Detecting Abnormality] Next, a method for detecting an abnormality in the vehicle interior (for example, dirt on the seat 40 or an item left on the seat 40) will be described.

[0129] Since the process of detecting dirt on the seat 40 is different from the process of detecting an item left on the seat 40, for example, an application (i.e., a dirt detection application) is used to detect dirt on the seat 40, and another application (i.e., an item left behind detection application) is used to detect an item left on the seat 40.

[0130] In the first embodiment, a pre-image taken before getting in the vehicle and a post-image taken after getting out of the vehicle are compared using a dirt detection application and an item left behind detection application, respectively, and an abnormality such as dirt on the seat 40 or an item left behind is detected from the difference between the pre-image and the post-image.

[0131] For example, a difference is taken between the pre-image and the post-image (i.e., a difference image is obtained), and an abnormality is detected based on the difference. When taking the difference, in order to suppress false detection due to a difference in brightness (i.e., luminance), the luminance of the pre-image and the post-image is adjusted so that the difference can be accurately detected. For example, well-known gamma correction is applied to the pre-image and the post-image or to one of the images, and the two images to be compared are adjusted to a certain gamma value (i.e., adjusted to have the same luminance value). Thereby, the accuracy of foreign object detection can be improved.

[0132] For example, when an infrared camera is used as the camera 25, differences such as the state of dirt on the imaging target are clearly shown in the image. Therefore, for example, when there is dirt on the sheet 40, a difference image corresponding to the dirt (that is, an image including a difference region corresponding to the dirt) can be obtained between the pre-image and the post-image taken by the infrared camera. Thus, when a difference image is obtained from the pre-image and the post-image, it can be determined that there is dirt on the sheet 40. That is, when there is a difference in the image between the pre-image and the post-image (that is, when there is a difference region), it can be determined that there is an abnormality such as dirt.

[0133] Also, when the same imaging target is imaged with a plurality of cameras 25 having different arrangements (that is, imaging positions), as is well known, a three-dimensional object can be detected. Therefore, when there is a three-dimensional object on the sheet 40, it can be determined that it is an item left behind.

[0134] In addition, information obtained by well-known machine learning may be applied to the captured image to detect dirt, items left behind, and the like.

[0135] [1-6. Control Processing] Next, the control processing performed by the abnormality detection system 1 will be described based on FIGS. 6 and 7.

[0136] Note that this control processing includes processing performed by the control unit 31 of the vehicle 9 and processing performed by the control unit 51 of the cloud 5.

[0137] <Dirt Detection Processing> This dirt detection processing is processing performed in the application for dirt detection.

[0138] As shown in FIG. 5, in step (hereinafter, S) 100, before boarding, the interior of the vehicle is photographed with the camera 25 (for example, an infrared camera), and an image of the interior of the vehicle (that is, a pre-image) is acquired. For example, when the door is unlocked with an IC card, it is considered to be before boarding, so the interior of the vehicle is photographed at that timing to acquire the pre-image. When photographing, the lighting device 27 is turned on to illuminate the interior of the vehicle.

[0139] In the subsequent S110, it is determined whether there is an instruction to photograph the interior of the vehicle before boarding from the business operator's mobile terminal 15. If an affirmative determination is made here, the process returns to S100 to photograph the interior of the vehicle before boarding. On the other hand, if a negative determination is made, the process proceeds to S120. When returning to S100, if the pre-image has already been acquired, two pre-images with different photographing times are acquired, and either image can be used as the pre-image.

[0140] In S120, it is determined whether there is an instruction to photograph the interior of the vehicle after boarding from the business operator's mobile terminal 15. If an affirmative determination is made here, the process proceeds to S160. On the other hand, if a negative determination is made, the process proceeds to S130.

[0141] In S130, it is determined whether the door of the vehicle 9 has been opened or closed based on the signal of a sensor such as a door switch. If it is determined here that the door has been opened or closed, the process proceeds to S140. On the other hand, if a negative determination is made, the process returns to S110.

[0142] In S140, since the door has been opened or closed, there is a possibility that a person has boarded. Therefore, extraction of a person (that is, detecting a person) is performed inside the vehicle. For example, the interior of the vehicle is photographed with the camera 25, and a person is extracted by analyzing the image data (that is, well-known image recognition). Also, extraction of a person may be performed by a well-known seating sensor that detects that a person is seated on the seat 40, a temperature sensor that detects a person's body temperature, or the like.

[0143] In subsequent S150, it is determined whether a person exists in the vehicle interior according to the result of the process of extracting the person in S140. If an affirmative determination is made here, the process returns to S110, while if a negative determination is made, it proceeds to S160. When returning to S110, it will wait until the door is opened and closed again.

[0144] In S160, a post-image after closing the door is acquired. That is, after the door is opened and closed, since no person has boarded, it is regarded as the state where the door is closed when getting off (i.e., getting off), the interior of the vehicle is photographed, and an image after getting off (i.e., a post-image) is acquired. When photographing, the lighting device 27 is turned on to illuminate the interior of the vehicle.

[0145] In subsequent S170, a process of detecting dirt attached to the seat 40 etc. is performed.

[0146] Specifically, as shown in the above "abnormality detection method", when detecting dirt on the seat 40 etc., for example, the difference between the pre-image and the post-image by an infrared camera can be taken, and dirt can be detected based on that difference. That is, when there is dirt on the seat 40, a difference image corresponding to the dirt can be obtained between the pre-image and the post-image. Therefore, when such a difference image is obtained, it can be determined that there is dirt on the seat 40.

[0147] In subsequent S180, the detection result of the dirt (i.e., the analysis result) is transmitted to the cloud 5. Here, only when there is dirt, the analysis result may be transmitted, but even when there is no dirt, the analysis result may be transmitted. Also, when transmitting the analysis result, for example, when there is dirt, the image data of the pre-image and the post-image used for detecting the dirt is transmitted to the cloud 5. Also, the analysis result and the image data are stored in the storage unit 53.

[0148] In subsequent S190, it is determined whether dirt exists based on the analysis result transmitted from the vehicle 9 to the cloud 5. If an affirmative determination is made here, it proceeds to S195, while if a negative determination is made, this process is once terminated.

[0149] In the case of S195, since there is dirt on the sheet 40 or the like, this fact (i.e., the analysis result indicating that dirt has been detected) is transmitted to the business operator's mobile terminal 15 (i.e., the analysis result is notified to the business operator), and this process is temporarily terminated. At this time, the analysis result may be transmitted to the user's mobile terminal 16 (i.e., the analysis result is notified to the user), or the user may be notified by the alert device 29.

[0150] Note that in this process, the processes of S100 to S180 are performed by the vehicle 9, and the processes of S190 and S195 are performed by the cloud 5.

[0151] In addition to the above process, the result of the dirt detection in S170 (for example, the analysis result indicating that dirt has been detected) may be notified to the business operator's mobile terminal 15 or the user's mobile terminal 16 before the process of S180. In that case, the determination in S190 and the notification in S195 can be omitted.

[0152] <Forgotten item detection process> This forgotten item detection process is a process implemented in the application for forgotten item detection.

[0153] In this forgotten item detection process, the front image and the rear image used in the dirt detection process are used to detect forgotten items. Note that in this forgotten item detection process as well, similar to the dirt detection process, the process of acquiring the front image and the rear image by taking pictures may be performed.

[0154] As shown in FIG. 6, in S200, it is determined whether the front image and the rear image have been acquired in the dirt detection process. If an affirmative determination is made here, the process proceeds to S210, while if a negative determination is made, this process is temporarily terminated.

[0155] In S210, a process of detecting forgotten items is performed.

[0156] Specifically, as shown in the "abnormality detection method" above, when detecting forgotten items on the sheet 40 or the like, an image captured by a camera 25 such as an infrared camera can be used. For example, by capturing the same imaging target (for example, the same sheet 40) with a plurality of cameras 25, a three-dimensional object (that is, a forgotten item) on the sheet 40 can be detected.

[0157] In the subsequent S220, the detection result of the forgotten item (that is, the analysis result) is transmitted to the cloud 5. Here, only when there is a forgotten item, the analysis result may be transmitted, but even when there is no forgotten item, the analysis result may be transmitted. Also, when transmitting the analysis result, for example, when there is a forgotten item, the image data of the pre-image and the post-image used for detecting the forgotten item is transmitted to the cloud 5. Note that the analysis result and the image data are stored in the storage unit 53.

[0158] In the subsequent S230, based on the analysis result transmitted from the vehicle 9 to the cloud 5, it is determined whether there is a forgotten item. If an affirmative determination is made here, the process proceeds to S240, while if a negative determination is made, this process is terminated once.

[0159] In S240, since there is a forgotten item on the sheet 40 or the like, this fact (that is, the analysis result indicating that a forgotten item has been detected) is transmitted to the business operator's mobile terminal 15 (that is, the analysis result is notified to the business operator), and this process is terminated once. At that time, the analysis result may be transmitted to the user's mobile terminal 16 (that is, the analysis result is notified to the user), or the user may be notified by the alert device 29.

[0160] Note that in this process, the processes of S200 to S220 are performed by the vehicle 9, and the processes of S230 and S240 are performed by the cloud 5.

[0161] In addition to the above process, the result of detecting the forgotten item in S210 (for example, the analysis result indicating that a forgotten item has been detected) may be notified to the business operator's mobile terminal 15 or the user's mobile terminal 16 before the process of S220. In that case, the determination in S230 and the notification in S240 can be omitted.

[0162] Note that, in principle, when the ignition 52 (e.g., the ignition switch shown in FIG. 1) of the vehicle 9 is turned off, the operation (i.e., startup) of the in-vehicle device 3 ends. That is, it becomes a power-off state in which the supply of power from the vehicle battery 50 is cut off. However, when the above-described applications for dirt detection and forgotten item detection are operating, until the processing of each application is completed (i.e., until each step of the flowchart in FIG. 6 or FIG. 7 ends), even if the ignition 52 is turned off midway, the startup of the in-vehicle device 3 is maintained and the processing is executed. Then, when the processing is completed until the end of each flowchart, the power is turned off. Note that when both applications are implemented, when the processing is completed until the end of both flowcharts, the power is turned off.

[0163] Note that, regarding the applications such as the second embodiment, the power-off operation is the same.

[0164] [1-7. Effects] According to this embodiment, the following effects can be obtained.

[0165] (1a) In this first embodiment, since abnormalities such as dirt and forgotten items in the vehicle interior can be suitably detected, it is a preferable technique for, for example, business operators who perform car sharing and users who use the vehicle 9.

[0166] Specifically, in the in-vehicle device 3, when implementing a plurality of applications respectively, each analyzes image data to detect an abnormality, and transmits the analysis result obtained by the detection unit 41 and at least the image data (i.e., predetermined image data) used when an abnormality is detected to the cloud 5. On the other hand, in the cloud 5, the analysis result and the predetermined image data transmitted from the transmission unit 43 are stored. Then, the analysis result is notified to the business operator or the user from the in-vehicle device 3 or the cloud 5.

[0167] Thus, in the first embodiment, the in-vehicle device 3 can detect abnormalities such as dirt based on the image data obtained by photographing the interior of the vehicle 9. Further, by transmitting the analysis result such as the abnormality detection result and the predetermined image data to the cloud 5, the cloud 5 can store the analysis result and the image data.

[0168] As a result, the analysis result and the image data (for example, the image data serving as the basis for the abnormality) can be reliably stored, so that the basis for taking corresponding measures according to the analysis result at a later date is ensured. Further, since the analysis result is notified to the business operator or the user, the business operator or the user who has received the notification can take appropriate measures according to the content of the notification. Note that by performing abnormality detection with the in-vehicle device 3, there is an advantage that the occurrence of an abnormality can be promptly notified to the user or the like as necessary.

[0169] (1b) In the first embodiment, it is possible to detect dirt on the seat 40 and items left on the seat 40.

[0170] (1c) In the first embodiment, since an infrared camera can be used as the camera 25, abnormalities such as those of the seat 40 can be easily detected from the image of the infrared camera.

[0171] (1d) In the first embodiment, when photographing is performed with the camera 25, the lighting device 27 that illuminates the object to be photographed is turned on, so that a clear image can be obtained. Therefore, abnormalities such as those of the seat 40 can be easily detected from the image.

[0172] (1e) In the first embodiment, based on the difference between the image data of the pre-image (i.e., the pre-image data) obtained by photographing the interior of the vehicle with the camera 25 before the user boards the vehicle 9 and the image data of the post-image (i.e., the post-image data) obtained by photographing the interior of the vehicle with the camera 25 after the user gets off the vehicle 9, abnormalities such as those of the seat 40 can be easily detected.

[0173] (1f) In the first embodiment, when detecting an abnormality based on the difference between the previous image data and the subsequent image data, brightness adjustment is performed on the previous image data and the subsequent image data, so that false detection of an abnormality due to the difference in brightness between the previous image and the subsequent image can be suppressed.

[0174] (1g) In the first embodiment, dirt and omissions can be discriminated.

[0175] (1h) In the first embodiment, when an instruction to capture an image inside the vehicle is received from outside the vehicle 9 (for example, from a business operator), the camera 25 can perform the capture. Then, an abnormality can be detected based on the image data obtained by the capture.

[0176] [1-8. Corresponding relationship] Next, the relationship between the first embodiment and the present disclosure will be described.

[0177] The vehicle 9 corresponds to a vehicle, the cloud 5 corresponds to a cloud, the in-vehicle device 3 corresponds to an in-vehicle device, the abnormality detection system 1 corresponds to an abnormality detection system, the camera 25 corresponds to a camera, the detection unit 41 corresponds to a detection unit, the transmission unit 43 corresponds to a transmission unit, the storage unit 61 corresponds to a storage unit, and the vehicle ECU 23 corresponds to a relay device.

[0178] [1-9. Modification example] Next, a modification example of the first embodiment will be described.

[0179] As a configuration on the cloud 5 side, the configuration shown in FIG. 8, which is managed by the cloud 5 (that is, the management server), can be adopted. That is, a known cloud service may be used to record the analysis result by the database 71 and record the image data by the file server 101.

[0180] Specifically, as the database 71, a configuration including a control unit 73 having a CPU 91 and a memory 93 and a communication unit 75 can be adopted, and the analysis result transmitted from the management server to the database 71 is stored in the storage unit 77.

[0181] Also, as the file server 101, a configuration including a control unit 103 having a CPU 111 and a memory 113 and a communication unit 105 can be adopted, and the image data transmitted from the management server to the file server 101 is stored in the storage unit 107.

[0182] [2. Second Embodiment] Since the basic configuration of the second embodiment is the same as that of the first embodiment, the differences from the first embodiment will be mainly described below. Note that the same reference numerals as those in the first embodiment indicate the same configuration, and reference is made to the previous description.

[0183] Since the hardware configuration of this second embodiment is the same as that of the first embodiment, the description thereof is omitted.

[0184] The abnormality detection system 1 of this second embodiment includes a first application and a second application configured to detect respective target abnormalities based on the image data from the camera 25 that has photographed the interior of the vehicle 9. Further, the abnormality detected by the first application has a higher urgency level for notification when the abnormality is detected than the abnormality detected by the second application.

[0185] [2-1. Functional Configuration] As shown in FIG. 9, in this second embodiment, the control unit of the in-vehicle device 3 functionally includes a first detection unit 121, a first transmission unit 123, and a second transmission unit 125.

[0186] The first detection unit 121 is configured to analyze the image data to detect an abnormality when implementing the first application.

[0187] The first transmission unit 123 is configured to transmit the analysis result analyzed by the first detection unit 121 to the cloud 5 and transmit at least the image data used when an abnormality is detected to the cloud 5.

[0188] When the second application is executed, the second transmission unit 125 is configured to transmit image data to the cloud 5.

[0189] Also, as shown in FIG. 10, the control unit 51 of the cloud 5 includes a first storage unit 131, a second detection unit 133, and a second storage unit 135.

[0190] The first storage unit 131 is configured to store the analysis result and the image data transmitted from the first transmission unit 123 when the first application is executed.

[0191] When the second application is executed, the second detection unit 133 is configured to analyze the image data transmitted from the second transmission unit 125 and detect an abnormality.

[0192] The second storage unit 135 is configured to store the image data transmitted from the second transmission unit 125 and the analysis result analyzed by the second detection unit 133.

[0193] Also, the analysis result is configured to be notified from at least one of the in-vehicle device 3 and the cloud 5 to at least one of the business operator's mobile terminal 15 and the user's mobile terminal 16.

[0194] [2-2. Control Process] <Biological Detection Process> This biological detection process is a process with a high notification urgency (i.e., priority) (i.e., a process by the first application).

[0195] As shown in FIG. 11, in S300 to S360 of the second embodiment, the same processes as S100 to S360 of the first embodiment are performed.

[0196] Subsequently, at 370, the in-vehicle device 3 performs biometric detection processing. This biometric detection processing is processing for detecting living things such as children like babies, the elderly, and pets (i.e., living bodies).

[0197] As a method for detecting a living body, first, abnormalities are detected from what based on the difference between the above-described previous image and the subsequent image. That is, if there is a difference area corresponding to the difference in the images, it is determined that there is some abnormality. And, as a method for detecting a baby, a pet, etc. for an object for which an abnormality has been detected, for example, a well-known image recognition process can be mentioned. Also, at that time, it is conceivable to detect the temperature of the object to improve the detection accuracy. Further, the detection accuracy can also be improved by adopting the method for detecting a three-dimensional object described above.

[0198] Subsequently, at S380, the detection result of the living body is transmitted to the cloud 5. Here, only when a living body is detected, the analysis result may be transmitted, but even when a living body is not detected, the analysis result may be transmitted. Also, when transmitting the analysis result (for example, when a living body is detected), the image data of the previous image and the subsequent image used for the detection of the living body is transmitted to the cloud 5. Note that the analysis result and the image data are stored in the storage unit 53.

[0199] Subsequently, at S390, based on the analysis result transmitted from the vehicle 9 to the cloud 5, it is determined whether or not a living body exists. If an affirmative determination is made here, the process proceeds to S395, while if a negative determination is made, this process is once terminated.

[0200] At S395, since a living body exists on the seat 40 or the like, this fact (i.e., the analysis result indicating that a living body has been detected) is transmitted to the business operator's mobile terminal 15 and also to the user's mobile terminal 16, and this process is once terminated. In this case, it is preferable to promptly notify the user presumed to be near the vehicle 9 using the alert device 29.

[0201] Note that in this process, the processes of S300 to S380 are performed in the vehicle 9, and the processes of S390 and S395 are performed in the cloud 5.

[0202] In addition to the above processing, the result of detecting the living body in S370 (for example, the analysis result indicating that a living body has been detected) may be notified to the business operator's mobile terminal 15 and the user's mobile terminal 16 before the processing in S380. In that case, the determination in S390 and the notification in S395 can be omitted.

[0203] <Stain detection processing> This stain detection processing is a process with a lower notification priority than the living body detection processing (i.e., a process by the second application). Note that instead of the stain detection processing, the above-mentioned forgotten item detection processing may be performed.

[0204] This stain detection processing performs stain detection processing using the pre-image and the post-image used in the living body detection processing. Note that in this stain detection processing as well, similar to the stain detection processing of the first embodiment, a process of capturing and acquiring the pre-image and the post-image may be performed.

[0205] As shown in FIG. 12, in S400, it is determined whether a pre-image and a post-image have been acquired by the first application. If an affirmative determination is made here, the process proceeds to S410, while if a negative determination is made, this process is terminated once.

[0206] In the subsequent S410, the image data of the pre-image and the post-image is transmitted to the cloud 5. The image data is stored in the storage unit 53.

[0207] In the subsequent S420, at the cloud 5, a process of detecting stains is performed in the same manner as in the first embodiment. The analysis result is stored in the storage unit 53.

[0208] In the subsequent S430, based on the analysis result of S420, it is determined whether stains are present. If an affirmative determination is made here, the process proceeds to S440, while if a negative determination is made, this process is terminated once.

[0209] In S440, since there is dirt on the seat 40 or the like, this fact (i.e., the analysis result indicating that dirt has been detected) is transmitted to the business operator's mobile terminal 15 (i.e., the analysis result is notified to the business operator), and this process is temporarily terminated. At this time, the analysis result may be transmitted to the user's mobile terminal 16 (i.e., the analysis result is notified to the user). Alternatively, the user may be notified by the alert device 29.

[0210] Note that in this process, the processes of S400 and S410 are performed by the vehicle 9, and the processes of S420 to S440 are performed by the cloud 5.

[0211] In the second embodiment, the same effects as those of the first embodiment are achieved. Furthermore, in the second embodiment, the process of detecting a living body such as a baby or a pet is promptly performed after getting off the vehicle, and when a baby or a pet is detected, the user and the business operator are promptly notified, so there is an effect of high safety.

[0212] Note that the following examples are given as modifications of the second embodiment.

[0213] Specifically, as a process with a high degree of notification urgency, instead of the living body detection process, a forgotten item detection process similar to that of the first embodiment can be adopted. In this case, instead of the living body detection process of S370, a forgotten item detection process such as S210 can be adopted, and instead of the process of determining the presence of a living body in S390, a process of determining the presence of a forgotten item such as S230 can be adopted.

[0214] [3. Other Embodiments] As described above, the embodiments of the present disclosure have been described. Needless to say, the present disclosure is not limited to the above embodiments and can take various forms.

[0215] (3a) The present disclosure can be applied to services in which a vehicle is shared by a plurality of users. For example, it can be applied to car-sharing services and rental car services.

[0216] (3b) As multiple applications, two or more applications can be adopted.

[0217] (3c) Abnormalities inside the vehicle cabin include dirt, forgotten items, damaged parts, the presence of a living body after getting off the vehicle, etc. Locations of abnormalities include seats, locations other than seats (for example, doors, windows, floors, dashboards), etc.

[0218] (3d) Regarding detection methods for dirt, forgotten items, and living bodies, various methods other than the detection methods described above can be adopted. For example, since differences in materials of the object to be photographed can be understood from the images of an infrared camera, it is possible to discriminate whether it is, for example, a seat or an object such as paper, clothes, or a bag outside the seat (i.e., a forgotten item) based on the difference between the previous image and the subsequent image.

[0219] (3e) As the image data transmitted from the vehicle side to the cloud side, when an abnormality is detected, the image data used for detecting the abnormality (for example, the image data of the previous image and the subsequent image) is included. However, even when no abnormality is detected, the image data may be transmitted for confirmation.

[0220] (3f) The abnormality detection system and the abnormality detection method described in the present disclosure may be realized by a dedicated computer provided by configuring a processor and a memory programmed to execute one or more functions embodied by a computer program.

[0221] Alternatively, the abnormality detection system and the abnormality detection method described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits.

[0222] Alternatively, the anomaly detection system and method described in the present disclosure may be implemented by one or more dedicated computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured by one or more hardware logic circuits.

[0223] Further, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions executable by a computer. The method for realizing the functions of each part included in the anomaly detection system does not necessarily include software, and all of its functions may be realized using one or more hardware.

[0224] (3g) In addition to the above-described anomaly detection system, the present disclosure can also be realized in various forms such as a program for operating the computer of the anomaly detection system, a non-transitory tangible recording medium such as a semiconductor memory storing this program, a control method, and the like.

[0225] (3h) In each of the above embodiments, a plurality of functions of one component may be realized by a plurality of components, or one function of one component may be realized by a plurality of components. Also, a plurality of functions of a plurality of components may be realized by one component, or one function realized by a plurality of components may be realized by one component. Further, a part of the configuration of each of the above embodiments may be omitted. Also, at least a part of the configuration of each of the above embodiments may be added to or replaced with the configuration of another embodiment.

[0226] [4. Technical Ideas Disclosed in this Specification] [Item 1] An anomaly detection system (1) that includes a cloud (5) that collects data of a vehicle (9) and an in-vehicle device (3) communicably connected to the cloud, and that detects anomalies inside the vehicle cabin. Based on the image data from the camera (25) that has photographed the interior of the vehicle, it is provided with a plurality of applications configured to detect the respective target abnormalities. The in-vehicle device When each of the plurality of applications is executed, a detection unit (41) configured to analyze the image data to detect the abnormality, A transmission unit (43) configured to transmit the analysis result analyzed by the detection unit to the cloud and to transmit at least the image data used when the abnormality is detected to the cloud, and is provided with The cloud is provided with a storage unit (61) configured to store the analysis result and the image data transmitted from the transmission unit. An abnormality detection system configured to notify the analysis result to a notification target from at least one of the in-vehicle device and the cloud. An abnormality detection system. [Item 2] An abnormality detection system (1) including a cloud (5) that collects data of a vehicle (9) and an in-vehicle device (3) communicably connected to the cloud, and detecting an abnormality in the interior of the vehicle, Based on the image data from the camera that has photographed the interior of the vehicle, it is provided with a first application and a second application configured to detect the respective target abnormalities. The abnormality detected by the first application has a higher urgency to be notified when the abnormality is detected than the abnormality detected by the second application. The in-vehicle device When the first application is executed, a first detection unit (121) configured to analyze the image data to detect the abnormality, A first transmission unit (123) configured to transmit the analysis result analyzed by the first detection unit to the cloud and to transmit at least the image data used when the abnormality is detected to the cloud. When the second application is executed, a second transmission unit (125) configured to transmit the image data to the cloud; and comprising the cloud When the first application is executed, a first storage unit (131) configured to store the analysis result transmitted from the first transmission unit and the image data; When the second application is executed, a second detection unit (133) configured to analyze the image data transmitted from the second transmission unit to detect the abnormality; a second storage unit (135) configured to store the image data transmitted from the second transmission unit and the analysis result analyzed by the second detection unit; and comprising An abnormality detection system configured to notify the analysis result to a notification target from at least one of the in-vehicle device and the cloud. Abnormality detection system. [Item 3] A cloud (5) that collects data of a vehicle (9), and an in-vehicle device (3) that is communicably connected to the cloud and is also communicably connected to a relay device (23) that relays frames flowing through a network (30) of the vehicle, the abnormality detection system (1) for detecting an abnormality in the vehicle interior, wherein the in-vehicle device an in-vehicle communication unit (45) configured to communicate with electronic control devices (32, 36) connected to the network of the vehicle via the relay device; a detection unit (41) configured to analyze image data from a camera (25) that has photographed the vehicle interior to detect the abnormality; a transmission unit (43) configured to transmit the analysis result analyzed by the detection unit to the cloud and also transmit the image data used at least when the abnormality is detected to the cloud; and comprising configured to notify the analysis result analyzed by the detection unit to the outside of the vehicle Abnormality detection system [Item 4] The abnormality detection system according to any one of Items 1 to 3, wherein the abnormality is an abnormality related to the seat including at least dirt on the seat (40) or an item left on the seat. Abnormality detection system [Item 5] The abnormality detection system according to any one of Items 1 to 4, wherein the camera (25) is an infrared camera. Abnormality detection system [Item 6] The abnormality detection system according to any one of Items 1 to 5, configured to turn on illumination for illuminating an object to be photographed when photographing is performed by the camera. Abnormality detection system [Item 7] The abnormality detection system according to any one of Items 1 to 6, configured to detect the abnormality based on a difference between pre-image data obtained by photographing the interior of the vehicle before a user boards the vehicle with the camera and post-image data obtained by photographing the interior of the vehicle after the user gets off the vehicle with the camera. Abnormality detection system [Item 8] The abnormality detection system according to Item 7, configured to adjust the brightness of the pre-image data and the post-image data when detecting the abnormality based on the difference between the pre-image data and the post-image data. Abnormality detection system [Item 9] The abnormality detection system according to any one of Items 1 to 8, configured to discriminate between dirt and an item left behind as the abnormality. Abnormality detection system [Item 10] An abnormality detection system according to any one of Items 1 to 9, configured to perform imaging by the camera and detect the abnormality based on the image data obtained by the imaging when an instruction to perform the imaging from outside the vehicle is received. Abnormality detection system. [Item 11] An abnormality detection system according to any one of Items 1 to 10, wherein the in-vehicle device is connected to a vehicle battery (50) and is configured to terminate the activation of the camera and terminate its own activation when the ignition (52) of the vehicle is turned off. Abnormality detection system. [Item 12] An abnormality detection system according to Item 11, configured not to terminate the activation of the camera and the activation of its own device until the analysis result is notified outside the vehicle when the abnormality is detected after the ignition of the vehicle is turned off. Abnormality detection system. [Item 13] An abnormality detection method for detecting an abnormality in a vehicle interior, wherein communication is possible between an in-vehicle device (3) mounted on a vehicle (9) and a cloud (5), using a plurality of applications configured to detect the respective target abnormalities based on image data from a camera (25) that has imaged the vehicle interior, wherein in the in-vehicle device, when each of the plurality of applications is executed, the image data is analyzed to detect the abnormality, the analyzed analysis result is transmitted to the cloud, and at least the image data used when the abnormality is detected is transmitted to the cloud, wherein in the cloud, the transmitted analysis result and the image data are stored, and further, the analysis result is notified to a notification target from at least one of the in-vehicle device and the cloud. Abnormality detection method. [Item 14] An abnormality detection method that enables communication between an in-vehicle device (3) mounted on a vehicle (9) and a cloud (5) and detects an abnormality in the vehicle interior, using a first application and a second application configured to detect the respective target abnormalities based on image data from a camera (25) that has photographed the vehicle interior, and the abnormality detected by the first application has a higher urgency to notify when the abnormality is detected than the abnormality detected by the second application, In the in-vehicle device, when implementing the first application, analyzing the image data to detect the abnormality, transmitting the analyzed analysis result to the cloud, and transmitting at least the image data used when the abnormality is detected to the cloud, when implementing the second application, transmitting the image data to the cloud, In the cloud, when implementing the first application, storing the transmitted analysis result and the image data, when implementing the second application, analyzing the transmitted image data to detect the abnormality, and storing the transmitted image data and the analyzed analysis result, Furthermore, notifying the analysis result to a notification target from at least one of the in-vehicle device and the cloud, Abnormality detection method. [Item 15] An abnormality detection method that uses a cloud (5) that collects data of a vehicle (9) and an in-vehicle device (3) that is communicably connected to the cloud and is also communicably connected to a relay device (23) that relays frames flowing through the network (30) of the vehicle to detect an abnormality in the vehicle interior, The in-vehicle device is, Communicate with the electronic control devices (32, 36) connected to the network of the vehicle via the relay device, analyze the image data from the camera (25) that has photographed the interior of the vehicle to detect the abnormality, transmit the analyzed analysis result to the cloud, and transmit at least the image data used when the abnormality is detected to the cloud, and notify the analyzed analysis result outside the vehicle. Abnormality detection method.

Claims

1. An abnormality detection system (1) comprising a cloud (5) that collects data of a vehicle (9) and an in-vehicle device (3) communicably connected to the cloud, for detecting an abnormality in a vehicle interior, comprising a first application and a second application configured to detect the respective target abnormalities based on image data from a camera that has photographed the vehicle interior, wherein the abnormality detected by the first application has a higher urgency to be notified when the abnormality is detected than the abnormality detected by the second application, the in-vehicle device is a first detection unit (121) configured to analyze the image data and detect the abnormality when the first application is executed, a first transmission unit (123) configured to transmit the analysis result analyzed by the first detection unit to the cloud and transmit at least the image data used when the abnormality is detected to the cloud, a second transmission unit (125) configured to transmit the image data to the cloud when the second application is executed, and the cloud is a first storage unit (131) configured to store the analysis result and the image data transmitted from the first transmission unit when the first application is executed, a second detection unit (133) configured to analyze the image data transmitted from the second transmission unit and detect the abnormality when the second application is executed, a second storage unit (135) configured to store the image data transmitted from the second transmission unit and the analysis result analyzed by the second detection unit, and configured to notify the analysis result to a notification target from at least one of the in-vehicle device and the cloud, abnormality detection system.

2. The abnormality detection system according to Claim 1, wherein when the abnormality is detected by the analysis in the in-vehicle device, the analysis result is configured to be notified from the in-vehicle device to the notification target, abnormality detection system.

3. The abnormality detection system according to Claim 1, wherein the abnormality is at least an abnormality related to the seat (40) including dirt on the seat or an item left on the seat. Abnormality detection system.

4. The abnormality detection system according to claim 1, wherein the camera (25) is an infrared camera. Abnormality detection system.

5. The abnormality detection system according to claim 1, wherein when shooting is performed with the camera, lighting for illuminating the object to be shot is configured to be turned on. Abnormality detection system.

6. The abnormality detection system according to claim 1, configured to detect the abnormality based on a difference between pre-image data obtained by shooting the interior of the vehicle with the camera before a user boards the vehicle and post-image data obtained by shooting the interior of the vehicle with the camera after the user gets off the vehicle. Abnormality detection system.

7. The abnormality detection system according to claim 6, wherein when detecting the abnormality based on a difference between the pre-image data and the post-image data, the brightness of the pre-image data and the post-image data is adjusted. Abnormality detection system.

8. The abnormality detection system according to claim 1, configured to discriminate between dirt and forgotten items as the abnormality. Abnormality detection system.

9. The abnormality detection system according to claim 1, wherein when an instruction to perform shooting from outside the vehicle is received, shooting is performed by the camera, and the abnormality is detected based on the image data obtained by the shooting. Abnormality detection system.

10. The abnormality detection system according to claim 1, wherein the in-vehicle device is connected to a vehicle battery (50), and when the ignition (52) of the vehicle is turned off, the operation of the camera is terminated and the operation of the in-vehicle device itself is terminated. Abnormality detection system.

11. The abnormality detection system according to claim 10, wherein when the abnormality is detected after the ignition of the vehicle is turned off, the operation of the camera and the operation of the in-vehicle device itself are not terminated until the analysis result is notified outside the vehicle. Abnormality detection system.

12. An abnormality detection method that enables communication between an in-vehicle device (3) mounted on a vehicle (9) and a cloud (5) and detects an abnormality in the vehicle interior. Based on the image data from the camera (25) that captures the interior of the vehicle, the first application and the second application, which are each configured to detect the target abnormality, are used. The abnormality detected by the first application has a higher urgency to notify when the abnormality is detected than the abnormality detected by the second application. In the in-vehicle device, When the first application is executed, the image data is analyzed to detect the abnormality, and the analyzed result is transmitted to the cloud. At the same time, at least the image data used when the abnormality is detected is transmitted to the cloud. When the second application is executed, the image data is transmitted to the cloud. In the cloud, When the first application is executed, the transmitted analysis result and the image data are stored. When the second application is executed, the transmitted image data is analyzed to detect the abnormality, and the transmitted image data and the analyzed result are stored. Furthermore, the analysis result is notified to the notification target from at least one of the in-vehicle device and the cloud. Abnormality detection method.

Images