Abnormality detection system and abnormality detection method

Abstract

An abnormality detection system (1) is provided with a plurality of applications that detect an abnormality on the basis of image data from a camera (25). When a plurality of applications are implemented, the detection means (41) analyzes each of the image data and detects an abnormality. The transmission unit (43) transmits, to the cloud (5), the analysis result analyzed by the detection unit (41) and the image data used at least when the abnormality is detected. A storage unit (61) stores the analysis result and the image data transmitted from the transmission unit (23). The abnormality detection system (1) reports the analysis result to a report target from at least one of the in-vehicle device (3) and the cloud (5).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present international application claims priority based on Japanese Patent Application No. 2022-073548 filed with the Japan Patent Office on April 27, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a technology for detecting abnormalities such as dirt on seats and the like in a vehicle interior. Background Art

[0004] Conventionally, as a technology for detecting dirt in a vehicle interior, there is known a technology in which a camera is arranged in the vehicle interior, the camera takes an image of the vehicle interior, and the captured image is analyzed to detect dirt (for example, refer to Patent Document 1).

[0005] In this conventional technology, an image captured by a camera is transmitted to a cloud side, and the image is analyzed on the cloud side to detect dirt and the like.

[0006] Patent Document 1: Japanese Patent Application Publication No. 2022-014372

[0007] As a result of detailed studies by the inventors, the following problems have been found with respect to the prior art.

[0008] Specifically, when using a camera to capture the interior of a vehicle and detecting dirt based on the captured image, it is considered to perform certain processing on the vehicle side and also on the cloud side, but what kind of processing is appropriate on each side has not been fully studied.

[0009] For example, when the above-mentioned conventional technologies are applied to shared vehicle services (e.g., car sharing), there is insufficient research on what kind of processing is performed on the vehicle side and the cloud side, and what is preferred (e.g., high convenience) for car sharing operators and users of the vehicles. Summary of the invention

[0010] One aspect of the present disclosure is to provide a technology that is preferable for operators and users when a vehicle and a cloud are communicated to perform processing related to abnormalities such as dirt in a vehicle interior.

[0011] [1] An abnormality detection system (1) of a technical solution disclosed herein comprises a cloud (5) for collecting data of a vehicle (9), and a vehicle-mounted device (3) connected to the cloud so as to be able to communicate, and the abnormality detection system (1) detects abnormalities in a vehicle cabin.

[0012] The abnormality detection system includes a plurality of application programs, each of which is configured to detect a target abnormality based on image data from a camera (25) that captures an image of the interior of a vehicle.

[0013] The vehicle-mounted device includes a detection unit (41) and a transmission unit (43). The detection unit is configured to analyze image data and detect abnormalities when multiple applications are respectively implemented. The transmission unit is configured to transmit the analysis results analyzed by the detection unit and the image data used when at least an abnormality is detected to the cloud.

[0014] The cloud includes a storage unit (61) configured to store the analysis result and image data transmitted from the transmission unit.

[0015] Furthermore, the abnormality detection system is configured to report the analysis result to a reporting target from at least one of the vehicle-mounted device and the cloud.

[0016] Through such a structure, in an abnormality detection system of a technical solution of the present invention, when communication is performed between the vehicle and the cloud to perform processing related to abnormalities such as dirt in the vehicle interior, for example, a technology that is preferred (for example, highly convenient) for operators who share cars and users who use the vehicles is provided.

[0017] Specifically, in an abnormality detection system of a technical solution of the present disclosure, in a vehicle-mounted device, when multiple applications are respectively implemented, image data is analyzed to detect abnormalities, and the analysis results obtained by the detection unit and the image data used when at least an abnormality is detected (i.e., the specified image data) are sent to the cloud. On the other hand, the cloud stores the analysis results and the specified image data sent from the sending unit. Moreover, the above-mentioned analysis results are reported from the vehicle-mounted device and the cloud to the reporting object such as the operator and the user.

[0018] Thus, in an abnormality detection system of a technical solution of the present disclosure, it is possible to detect abnormalities such as dirt based on image data obtained by photographing the interior of a vehicle. In addition, by sending analysis results such as abnormality detection results and prescribed image data to the cloud, the analysis results and image data can be stored in the cloud.

[0019] Thus, the analysis results and image data (for example, image data that is the basis for abnormality) can be reliably stored, so that the basis for taking measures corresponding to the analysis results in the future becomes reliable. In addition, the analysis results are reported to the operator and the user, so the operator and the user who receive the report can take appropriate measures according to the content of the report.

[0020] [2] Another technical solution of the present disclosure is an abnormality detection system (1) comprising a cloud (5) for collecting data of a vehicle (9), and a vehicle-mounted device (3) connected to the cloud so as to be able to communicate, and the abnormality detection system (1) detects abnormalities in a vehicle cabin.

[0021] The abnormality detection system includes a first application and a second application, and the first application and the second application are configured to detect the abnormality of the purpose based on the image data from the camera that takes pictures of the interior of the vehicle. The abnormality detected by the first application has a higher urgency reported when the abnormality is detected than the abnormality detected by the second application.

[0022] The vehicle-mounted device includes a first detection unit (121), a first transmission unit (123), and a second transmission unit (125).

[0023] The first detection unit is configured to analyze the image data and detect abnormality when the first application is executed.

[0024] The first transmitting unit is configured to transmit the analysis result analyzed by the first detecting unit to the cloud, and to transmit at least image data used when abnormality is detected to the cloud.

[0025] The second transmission unit is configured to transmit the image data to the cloud when the second application is executed.

[0026] The cloud includes a first storage unit (131), a second detection unit (133), and a second storage unit (135).

[0027] The first storage unit is configured to store the analysis result and the image data transmitted from the first transmission unit when the first application is executed.

[0028] The second detection unit is configured to analyze the image data transmitted from the second transmission unit and detect abnormality when the second application is executed.

[0029] The second storage unit is configured to store the image data transmitted from the second transmission unit and the analysis result analyzed by the second detection unit.

[0030] Furthermore, the abnormality detection system is configured to report the analysis result to a reporting target from at least one of the vehicle-mounted device and the cloud.

[0031] Through such a structure, in the abnormality detection system of another technical solution of the present disclosure, when communication is performed between the vehicle and the cloud to perform processing related to abnormalities such as dirt in the vehicle interior, it is possible to provide a technology that is preferred for, for example, car sharing operators and users of the vehicles.

[0032] Furthermore, in this abnormality detection system, when an abnormality with high urgency (i.e., priority) to be reported is detected, it is promptly reported to the reporting target, thereby achieving an effect that the reporting target such as the operator and the user can take measures according to the content of the report.

[0033] [3] An abnormality detection system (1) of a technical solution disclosed in the present invention comprises a cloud (5) for collecting data of a vehicle (9), and a vehicle-mounted device (3), wherein the vehicle-mounted device (3) is connected to the cloud so as to be able to communicate, and is connected to a relay device (23) for relaying each frame of a network flowing through the vehicle so as to be able to communicate, and the abnormality detection system (1) detects abnormalities in a vehicle cabin.

[0034] The vehicle-mounted device includes an in-vehicle communication unit (45), a detection unit (41), and a transmission unit (43).

[0035] The in-vehicle communication unit is configured to communicate with an electronic control unit (32, 36) connected to a network of the vehicle via a relay device. The detection unit is configured to analyze image data from a camera (25) that captures the interior of the vehicle to detect abnormalities. The transmission unit is configured to transmit the analysis result analyzed by the detection unit to the cloud, and at least transmit the image data used when the abnormality is detected to the cloud.

[0036] Furthermore, the vehicle-mounted device is configured to report the analysis result obtained by the detection unit to the outside of the vehicle.

[0037] Through such a structure, in an abnormality detection system of a technical solution of the present invention, when communication is performed between the vehicle and the cloud to perform processing related to abnormalities such as dirt in the vehicle interior, for example, a technology that is preferred (for example, highly convenient) for car-sharing operators and users of the vehicles can be provided.

[0038] [4] In an abnormality detection method of a technical solution of the present disclosure, communication can be performed between a vehicle-mounted device (3) mounted on a vehicle (9) and a cloud (5) to detect abnormalities in the vehicle interior.

[0039] In the abnormality detection method, a plurality of application programs are used, and the plurality of application programs are configured to detect target abnormalities based on image data from a camera (25) that captures an image of a vehicle interior.

[0040] In the vehicle-mounted device, when multiple applications are respectively implemented, image data is analyzed to detect abnormalities, and the analyzed results are sent to the cloud, and at least the image data used when the abnormality is detected is sent to the cloud. The sent analysis results and image data are stored in the cloud.

[0041] Furthermore, in the abnormality detection method, the analysis result is reported to the reporting target from at least one of the vehicle-mounted device and the cloud.

[0042] Through such a structure, in an abnormality detection method of a technical solution of the present invention, when communication is performed between the vehicle and the cloud to perform processing related to abnormalities such as dirt in the vehicle interior, for example, a technology that is preferred (for example, highly convenient) for operators who share cars and users of the vehicles can be provided.

[0043] [5] In another technical solution of the abnormality detection method disclosed in the present invention, communication can be performed between a vehicle-mounted device (3) mounted on a vehicle (9) and a cloud (5) to detect abnormalities in the vehicle interior.

[0044] In the abnormality detection method, a first application and a second application are used, and the first application and the second application are configured to detect the target abnormality based on image data from a camera (25) that takes pictures of the interior of a vehicle, and the abnormality detected by the first application is reported with a higher urgency when the abnormality is detected than the above-mentioned abnormality detected by the second application.

[0045] In the vehicle-mounted device, when the first application is implemented, the image data is analyzed to detect abnormalities, the analyzed results are sent to the cloud, and at least the image data used when the abnormality is detected is sent to the cloud. When the second application is implemented, the image data is sent to the cloud.

[0046] In the cloud, when the first application is executed, the sent analysis results and image data are stored. When the second application is executed, the sent image data is analyzed to detect abnormalities, and the sent image data and the analyzed analysis results are stored.

[0047] Furthermore, in the abnormality detection method, the analysis result is reported to the reporting target from at least one of the vehicle-mounted device and the cloud.

[0048] Through such a structure, in the abnormality detection method of another technical solution of the present invention, when communication is performed between the vehicle and the cloud to perform processing related to abnormalities such as dirt in the vehicle interior, for example, it is possible to provide a technology that is preferred (for example, highly convenient) for car-sharing operators and vehicle users.

[0049] [6] Another technical solution of the present invention is an abnormality detection method that uses a cloud (5) that collects data on a vehicle (9) and a vehicle-mounted device (3) to detect abnormalities in a vehicle cabin, wherein the vehicle-mounted device (3) is connected to the cloud to enable communication, and is connected to a relay device (23) that relays each frame of a network flowing through the vehicle to enable communication.

[0050] In the abnormality detection method, the vehicle-mounted device communicates with an electronic control device (32, 36) connected to a vehicle network via a relay device, analyzes image data from a camera (25) that captures the interior of the vehicle to detect abnormalities, sends the analyzed results to a cloud, and at least sends the image data used when the abnormality is detected to the cloud. In addition, the sent analysis results are reported to a reporting target in the cloud.

[0051] Through such a structure, in the abnormality detection method of another technical solution of the present invention, when communication is performed between the vehicle and the cloud to perform processing related to abnormalities such as dirt in the vehicle interior, for example, it is possible to provide a technology that is preferred (for example, highly convenient) for car-sharing operators and vehicle users.

[0052] In addition, the reference numerals in parentheses in this column and in the claims indicate the correspondence relationship with the specific means described in the embodiment described later as one claim, and do not limit the technical scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is an explanatory diagram showing the overall configuration of the abnormality detection system according to the first embodiment.

[0054] Figure 2 This is a block diagram showing a hardware configuration mounted on the vehicle according to the first embodiment.

[0055] Figure 3 It is an explanatory diagram showing a method of utilizing the abnormality detection system according to the first embodiment.

[0056] Figure 4 This is a block diagram functionally showing the control unit of the vehicle-mounted device according to the first embodiment.

[0057] Figure 5 This is a block diagram functionally showing the cloud control unit of the first embodiment.

[0058] Figure 6 1 is a flowchart showing the contamination detection process according to the first embodiment.

[0059] Figure 7 This is a flowchart showing the forgotten object detection process according to the first embodiment.

[0060] Figure 8 It is a block diagram showing the structure of a modified example.

[0061] Fig. 9 This is a block diagram functionally showing a control unit of the vehicle-mounted device according to the second embodiment.

[0062] Fig.10 This is a block diagram functionally showing the cloud control unit of the second embodiment.

[0063] Fig.11 This is a flowchart showing the biological body detection process according to the second embodiment.

[0064] Fig.12 : is a flowchart showing the contamination detection process according to the second embodiment. DETAILED DESCRIPTION

[0065] Below, while referring to the attached Figure 1 While describing exemplary embodiments of the present disclosure.

[0066] [1. First embodiment]

[0067] In the first embodiment, as an example of a mobile IoT system, an abnormality detection system for detecting abnormalities such as dirt in the interior of a vehicle (for example, a car) will be described. IoT is an abbreviation for Internet of Things.

[0068] [1-1. Overall structure]

[0069] First, based on Figure 1 The overall configuration of the abnormality detection system 1 according to the first embodiment will be described.

[0070] like Figure 1 As shown, the abnormality detection system 1 includes an in-vehicle device 3, a cloud 5, and a service providing server 7. In addition, a server that manages the operation of the cloud 5 and the like is referred to as a management server.

[0071] exist Figure 1 For the sake of convenience, only one vehicle-mounted device 3 is described, but the abnormality detection system 1 may include, for example, a plurality of vehicle-mounted devices 3 , and the plurality of vehicle-mounted devices 3 may be respectively mounted on different vehicles 9 .

[0072] The in-vehicle device 3 can perform wireless communication with the cloud 5 and the mobile terminal 15 via a communication device 11 mounted on the vehicle 9. The detailed configurations of the in-vehicle device 3 and the vehicle 9 will be described later.

[0073] The cloud 5 can communicate with the vehicle-mounted device 3, the service providing server 7, and the mobile terminal 15 via the communication unit 13. In addition, the communication unit 13 can communicate with the vehicle-mounted device 3 and the mobile terminal 15 by wireless. The cloud 5 can collect data of the vehicle 9 from the vehicle-mounted device 3 via the communication device 11 and the communication unit 13. In addition, the detailed structure of the cloud 5 will be described later.

[0074] The service providing server 7 can communicate with the cloud 5. The service providing server 7 is, for example, a server provided to provide a service for managing the operation of the vehicle 9. In addition, the abnormality detection system 1 may include a plurality of service providing servers 7 with different service contents.

[0075] The mobile terminal 15 is, for example, a mobile terminal (ie, an information terminal) owned by a car-sharing operator. Examples of the mobile terminal 15 include a smartphone, a tablet terminal, and a notebook PC. In addition to the mobile terminal 15, a desktop computer may also be used.

[0076] In addition, as described later, the communication device 11 of the vehicle-mounted device 3 may be able to communicate with the user's mobile terminal 16 (for example, see Figure 3 ) for communication. In addition, it is also possible to communicate with the user's mobile terminal 16 via wireless via the communication unit 13 of the cloud 5.

[0077] Hereinafter, each structure will be described in detail.

[0078] [1-2. Structure on the vehicle side]

[0079] Next, based on Figure 1 to Figure 4 The structure on the vehicle 9 side will be described.

[0080] like Figure 1 As shown, the vehicle 9 includes a sensor 21 , a vehicle ECU 23 , a camera 25 , a lighting device 27 , a communication device 11 , and an alarm device 29 in addition to the vehicle-mounted device 3 .

[0081] The sensor 21 is a detection device for detecting the state of the vehicle 9. Examples of the sensor 21 include engine on / off, start of travel, stop, vehicle speed, shift position, seat 40 (for example, see Figure 3 )'s presence or absence of a seat, the opening and closing of a door, and the locking (i.e., locking) and unlocking (i.e., unlocking) of a door.

[0082] The vehicle ECU 23 is an electronic control unit (i.e., ECU) connected to the sensor 21. The vehicle ECU 23 receives a signal from the sensor 21 and processes the signal as needed. In addition, the signal (i.e., information) obtained from the sensor 21 is sent from the vehicle ECU 23 to the vehicle-mounted device 3 via a communication line.

[0083] like Figure 3 As shown, the camera 25 is one or more vehicle-mounted cameras arranged in the vehicle interior for photographing the vehicle interior, and for example, an infrared camera is used. In addition, a digital camera such as a CCD camera can also be used. In addition, a color image can be used as the image photographed.

[0084] As the installation position of the camera 25, the upper part of the windshield, the vicinity of the rearview mirror in the vehicle, the ceiling, etc. are listed. As the shooting range of the camera 25, it is set to include a range with a high possibility of objects placed in the vehicle interior, and a range with a high possibility of dirt adhesion. Specifically, the shooting range of the camera 25 is set to include, for example, a part or all of each seat 40 of the driver's seat, the front passenger seat, and the rear seat (for example, the seat surface and backrest portion of the seat 40), the dashboard, the inner side of the door, etc.

[0085] Furthermore, in order to detect a three-dimensional object, a plurality of cameras 25 may be arranged to capture images of the object from different angles.

[0086] like Figure 3 As shown, the lighting device 27 is a lamp that is turned on to illuminate the interior of the vehicle when the camera 25 captures an image of the interior of the vehicle, and can be, for example, a lamp that emits infrared rays, an LED lamp, or the like.

[0087] The communication device 11 is a communication device capable of wireless communication with the communication unit 13 of the cloud 5 and the mobile terminals 15 and 16. Image data, analysis results of image data, etc. are sent from the vehicle 9 to the cloud 5 through the communication device 11. In addition, as described later, the vehicle-mounted device 3 can be controlled by a signal from the mobile terminal 15.

[0088] The alarm device 29 is a device that generates a warning, such as an electronic sound or voice, to the user of the vehicle 9. As the alarm device 29, a speaker or the like can be used.

[0089] <In-vehicle network>

[0090] Here, the configuration related to the network in the vehicle 9 connected to the vehicle-mounted device 3 will be described. The vehicle-mounted device 3 can be connected to the network in the vehicle 9 afterward so as to be able to communicate with the vehicle ECU 23 and the like.

[0091] like Figure 2As shown, in the vehicle 9 , the vehicle ECU 23 includes a CPU 24 , and a memory 26 such as a ROM 26 a and a RAM 26 b as a configuration for performing various calculation processes.

[0092] The vehicle ECU 23 is connected to a plurality of ECUs 32 and an external communication device 34 for communicating with the outside of the vehicle via an internal communication network 30 for communicating within the vehicle. Each ECU 32 is connected to other ECUs 36 for communication.

[0093] The vehicle ECU 23 can realize the overall coordinated control of the vehicle 9 by collectively controlling the plurality of ECUs 32. Each ECU 32 is provided, for example, for each domain divided by functions in the vehicle 9, and can mainly perform control of the plurality of ECUs 36 existing in the domain. The domain is, for example, a powertrain, a vehicle body, a chassis, a cockpit, etc. In addition, the ECU 36 is, for example, an ECU that controls sensors and actuators.

[0094] In addition, the network in the vehicle 9 is used to send and receive frames containing various information between various structures in the vehicle 9 (for example, the vehicle-mounted device 3, ECUs 23, 32, 36, and the vehicle-external communication device 34). As such a network, for example, the vehicle-mounted communication network 30 is listed. In addition, the vehicle-mounted device 3 is connected to the vehicle battery 50 and shares a power source with other electrical structures in the vehicle 9.

[0095] <In-vehicle device>

[0096] Next, the vehicle-mounted device 3 will be described in detail.

[0097] The vehicle-mounted device 3 includes a control unit 31 and a storage unit 33. The control unit 31 includes a CPU 35 and a semiconductor memory such as a RAM or a ROM (hereinafter referred to as a memory 37). The control unit 31 is constituted by, for example, a microcomputer.

[0098] The functions of the control unit 31 are realized by the CPU 35 executing a program stored in a non-migrated physical recording medium (ie, the memory 37). In addition, by executing the program, a method corresponding to the program is executed.

[0099] In addition, the method of realizing various functions of the control unit 31 is not limited to software, and some or all of the elements may be realized by using one or more hardware. For example, when the above functions are realized by an electronic circuit as hardware, the electronic circuit may be realized by a digital circuit including a plurality of logic circuits, or an analog circuit, or a combination thereof.

[0100] In the first embodiment, a plurality of application programs (ie, programs) are stored in the memory 37 , each of which is configured to detect a target abnormality based on image data from the camera 25 that captures the interior of the vehicle 9 .

[0101] For example, as described in detail later, a program for detecting dirt on the seat 40 , a program for detecting a forgotten object on the seat 40 , and the like are stored.

[0102] The storage unit 33 is a memory capable of storing information. For example, information of an image captured by the camera 25 (i.e., image data) can be stored in the storage unit 33. In addition, the result of analyzing the image (i.e., analysis result) can be stored. In addition, as the storage unit 33, for example, a hard disk drive (i.e., HDD) and a solid state hard disk drive (i.e., SSD) are listed.

[0103] <Functional structure of the control unit>

[0104] Here, the functional configuration of the control unit 31 will be described.

[0105] like Figure 4 As shown, the control unit 31 of the vehicle 9 functionally includes a detection unit 41 , a transmission unit 43 , and an in-vehicle communication unit 45 .

[0106] The detection unit 41 is configured to respectively acquire and analyze the image data captured by the camera 25 (i.e., the image data of the front image and the rear image described later) when the above-mentioned multiple programs (i.e., applications) are respectively implemented, and detect abnormalities in the vehicle interior such as dirt on the seat 40 and forgotten objects.

[0107] The sending unit 43 is configured to drive the communication machine 11 and send the detection result of the abnormality, i.e., the analysis result (for example, the analysis result of the content such as the detection of the abnormality) and the image data used when at least the abnormality is detected (for example, the image data of the front image and the rear image) to the cloud 5.

[0108] Furthermore, the analysis result may be reported from the vehicle 9 to the operator's mobile terminal 15 or the user's mobile terminal 16 .

[0109] The in-vehicle communication unit 45 is configured to communicate with the vehicle ECU 23 and to communicate with other ECUs 32 and 36 via the vehicle ECU 23 .

[0110] [1-3. Cloud side structure]

[0111] Next, based on Figure 1 , Figure 3 , Figure 5 The structure on the cloud 5 side will be described.

[0112] The cloud 5 includes a control unit 51, a communication unit 13, and a storage unit 53. The control unit 51 includes a CPU 55 and a semiconductor memory such as a RAM or a ROM (hereinafter referred to as a non-migrating physical recording medium, i.e., a memory 57). The structure and function of the control unit 51 are basically the same as those of the control unit 31 of the vehicle 9, and are implemented by the CPU 55 executing a program stored in the memory 57. In addition, by executing the program, a method corresponding to the program is executed.

[0113] The communication unit 13 can perform wireless communication with the communication device 11 and the mobile terminal 15. For example, the cloud 5 can receive analysis results and image data transmitted from the vehicle 9 via the communication device 11 and the communication unit 13.

[0114] The storage unit 53 is a memory storing the same information as the storage unit 33 of the vehicle 9 , and can store analysis results and image data received from the vehicle 9 .

[0115] The cloud 5 having the above configuration can collect data of the vehicle 9 transmitted from each of the plurality of vehicle-mounted devices 3 via the communication device 11 . The cloud 5 can store the collected data in the storage unit 53 for each vehicle 9 .

[0116] In addition, the cloud 5 creates a digital twin based on the data of the vehicle 9 stored in the storage unit 53. The digital twin is standardized index data. The service provider server 7 can use the index data obtained from the digital twin to obtain the data of a specified vehicle stored in the storage unit 53. The service provider server 7 determines the control content of the vehicle 9 and sends an instruction corresponding to the control content to the cloud 5. The cloud 5 sends the control content to the vehicle 9 based on the instruction.

[0117] <Functional structure of the control unit>

[0118] Here, the functional structure of the control unit 51 is described.

[0119] like Figure 5 As shown, the control unit 51 of the cloud 5 functionally includes a storage unit 61 .

[0120] The storage unit 61 is configured to store the analysis result and the image data transmitted from the communication device 11 of the vehicle 9 in, for example, the storage unit 53 .

[0121] Furthermore, the analysis result stored in the storage unit 53 is reported from the cloud 5 to the operator's mobile terminal 15 via the communication unit 13 , but may also be reported to the user's mobile terminal 16 .

[0122] [1-4. Overall action]

[0123] Next, based on Figure 3 The overall operation of the abnormality detection system 1 will be described by taking a case where car sharing is performed as an example.

[0124] <Actions before using the vehicle>

[0125] (1) A user who uses the vehicle 9 through car sharing usually registers and obtains an IC card (not shown) used when using the vehicle 9 .

[0126] (2) When the user wants to use the vehicle 9 , the user makes a reservation for use of the vehicle 9 in advance using a mobile terminal 16 such as a smartphone.

[0127] (3) Next, when using the vehicle 9, the user goes to the place where the vehicle 9 is parked at the scheduled time. Then, the user can unlock the door by placing the IC card on a reader (not shown) provided on the window of the vehicle 9 or the like.

[0128] (4) In addition, for example, when the IC card is placed on the reader and the door is unlocked, the interior of the vehicle can be photographed by the camera 25 (that is, a front image can be acquired).

[0129] (5) Next, the user opens the door and gets in the vehicle, and for example, obtains the key (not shown) of the vehicle 9 stored in the glove box in the vehicle. When obtaining the key, the user inputs that the vehicle 9 is in use by switching on the switch, etc. Then, the engine of the vehicle 9 is started by the key, and the vehicle 9 starts.

[0130] <Operation after using the vehicle>

[0131] (1) When the use of the vehicle 9 is finished and the vehicle 9 is stopped, the key is returned to a predetermined position in the glove box. When the key is returned, the fact that the use of the vehicle 9 has been finished is inputted by a switch or the like.

[0132] (2) Next, the user opens the door of the vehicle 9 and gets out, and then closes the door.

[0133] (3) Next, the IC card is placed on the reader of the vehicle 9 to lock the vehicle door. Thus, the use of the vehicle 9 is completed.

[0134] (4) Furthermore, for example, when the IC card is placed on the reader and the door is locked, the interior of the vehicle can be photographed by the camera 25 (ie, a rear image can be acquired).

[0135] In addition, here, the example of automatically photographing the interior of the vehicle at the above-mentioned timings before boarding and after getting off the vehicle is listed, but the interior of the vehicle may also be photographed according to instructions from the operator (for example, through remote operation via a network, etc.).

[0136] <Other judgment methods before boarding and after getting off the bus>

[0137] As a method for determining before boarding (i.e., before use) and after getting off the vehicle (i.e., after use), as described above, a method of unlocking and locking the vehicle door using an IC card is considered. That is, when unlocking the vehicle door using an IC card, the method before boarding is considered, and when locking the vehicle door using an IC card, the method after getting off the vehicle is considered.

[0138] Furthermore, if it is before getting on the vehicle, the interior of the vehicle before getting on the vehicle can be photographed to obtain a front image, and if it is after getting off the vehicle, the interior of the vehicle after getting off the vehicle can be photographed to obtain a rear image.

[0139] In addition, various other methods are conceivable.

[0140] For example, as described later, it may be that when the door is opened and closed (that is, closed after the door is opened), when there is no one in the vehicle, it is judged to be after getting off. In addition, it may be that when the door is opened and closed, when there is someone in the vehicle, it is judged to be in the vehicle.

[0141] In addition, for example, when the door is unlocked, if it is confirmed by the seat sensor that there is no one in the car, it can be judged as before getting on the car. In addition, for example, after the seat sensor detects a change from a seated state to a non-seated state, when the door is locked, it can be judged as after getting off the car.

[0142] Furthermore, since the time for the user to use the vehicle 9 is reserved, the interior of the vehicle may be automatically photographed before the predetermined time for starting the use, for example, based on a control signal from the cloud 5 , to acquire a front image.

[0143] In the first embodiment, the vehicle-mounted device 3 is powered and operable until at least the front image and the rear image are acquired, abnormality is detected, and the analysis result and image data are transmitted to the cloud 5 .

[0144] <Post-use treatment of vehicles>

[0145] (1) In the vehicle 9 (i.e., the vehicle-mounted device 3), when the use of the vehicle 9 ends, an abnormality in the vehicle interior (e.g., dirt on the seat 40, or a forgotten object on the seat 40) is detected based on the image captured before boarding the vehicle (i.e., the front image) and the image captured after getting off the vehicle (i.e., the rear image). The method for detecting the abnormality will be described in detail later.

[0146] (2) Next, the data of the two images (ie, image data) are analyzed to determine whether or not there is an abnormality in the vehicle interior. The analysis result and the image data used in the analysis are transmitted from the vehicle 9 to the cloud 5 .

[0147] As the analysis result, there are cases where an abnormality is detected (i.e., abnormal case) and cases where an abnormality is not detected (i.e., normal case), but it is preferable to send the analysis results of both the abnormal case and the normal case to the cloud 5. In addition, it is also possible to send "the content of abnormality detection" to the cloud 5 only in the case of an abnormality.

[0148] In addition, as a case where image data is transmitted to the cloud 5, it is considered that image data (ie, the front image and the back image) used for detecting abnormality is transmitted only in the case of abnormality. In addition, image data may also be transmitted to the cloud 5 in the normal case.

[0149] The second embodiment is also the same as to whether the analysis result and image data are transmitted only when an abnormality occurs or whether the analysis result and image data are transmitted regardless of an abnormality or a normal state.

[0150] (3) In the cloud 5 , the analysis results (for example, the presence or absence of dirt or forgotten objects) and the image data transmitted from the vehicle 9 are stored in the storage unit 53 .

[0151] (4) In the cloud 5, when an abnormality such as dirt or a forgotten object is detected, the analysis result is sent to the operator's mobile terminal 15. In addition, when no abnormality is detected, the analysis result may be sent to the mobile terminal 15.

[0152] In addition, the analysis result may be sent to the user's mobile terminal 16, for example, when there is a forgotten item. At this time, an alarm device 29 such as a speaker may be used to notify the user that there is a forgotten item.

[0153] [1-5. Abnormality detection method]

[0154] Next, a method of detecting abnormality in the vehicle interior (for example, dirt on the seat 40 or an object left on the seat 40 ) will be described.

[0155] The process of detecting dirt on the seat 40 is different from the process of detecting a forgotten object on the seat 40. Therefore, for example, the dirt on the seat 40 is detected by a certain application (i.e., the dirt detection application), and the forgotten object on the seat 40 is detected by another application (i.e., the forgotten object detection application).

[0156] In this first embodiment, a dirt detection application and a forgotten object detection application are used respectively to compare the front image taken before boarding the vehicle with the rear image taken after getting off the vehicle, and based on the difference between the front image and the rear image, abnormalities such as dirt on the seat 40 or forgotten objects are detected.

[0157] For example, the difference between the front image and the rear image is obtained (i.e., the differential image is obtained), and the abnormality is detected based on the difference. In addition, in the case of obtaining the difference, in order to suppress the false detection caused by the difference in brightness (i.e., brightness), the brightness of the image and the rear image is adjusted, and the difference can be accurately detected. For example, the known gamma correction is performed on the front image and the rear image or on the image of one side, and the two images to be compared are adjusted to a certain gamma value (i.e., adjusted to the same brightness value). In this way, the accuracy of foreign body detection can be improved.

[0158] For example, when an infrared camera is used as the camera 25, the difference in the state of dirt of the photographed object and the like clearly appears in the image. Therefore, for example, when there is dirt on the seat 40, a differential image corresponding to the dirt (i.e., an image including a differential area corresponding to the dirt) is obtained in the front image and the rear image captured by the infrared camera. In this way, when a differential image is obtained based on the front image and the rear image, it can be determined that there is dirt on the seat 40. That is, when there is an image difference between the front image and the rear image (i.e., when there is a differential area), it can be determined that there is an abnormality such as dirt.

[0159] Furthermore, when the same object is photographed by a plurality of cameras 25 arranged differently (ie, at photographing positions), a three-dimensional object can be detected as known. Therefore, when a three-dimensional object is present on the seat 40, it can be determined to be a forgotten object.

[0160] Furthermore, dirt, forgotten objects, and the like may be detected by applying information obtained through known machine learning to the captured image.

[0161] [1-6. Control Process]

[0162] Next, based on Figure 6 as well as Figure 7 The control process performed by the abnormality detection system 1 will be described.

[0163] It should be noted that this control process includes a process performed by the control unit 31 of the vehicle 9 and a process performed by the control unit 51 of the cloud 5 .

[0164] <Dirt detection processing>

[0165] This contamination detection process is a process performed in a contamination detection application program.

[0166] like Figure 5 As shown, in step (hereinafter, S) 100, before boarding, the camera 25 (for example, an infrared camera) takes a picture of the interior of the vehicle, and acquires an image of the interior of the vehicle (i.e., a front image). For example, when the door is unlocked by an IC card, it is considered to be before boarding, so the interior of the vehicle is photographed at this timing to acquire a front image. In addition, when photographing, the lighting device 27 is turned on to illuminate the interior of the vehicle.

[0167] In the next S110, it is determined whether there is an instruction to photograph the interior of the vehicle before boarding from the operator's mobile terminal 15. Here, if the determination is positive, the process returns to S100 and photographs the interior of the vehicle before boarding. On the other hand, if the determination is negative, the process proceeds to S120. In addition, when returning to S100, if the front image has been acquired, two front images with different shooting times are acquired, but any image can be used as the front image.

[0168] In S120, it is determined whether there is an instruction to take a picture of the interior of the vehicle after boarding from the operator's mobile terminal 15. If the determination is affirmative, the process proceeds to S160, while if the determination is negative, the process proceeds to S130.

[0169] In S130, it is determined whether the door of the vehicle 9 is opened or closed based on the signal of the sensor such as the door switch. Here, if it is determined that the door is opened or closed, the process proceeds to S140, while if a negative determination is made, the process returns to S110.

[0170] In S140, since the door is opened and closed, a person may be in the vehicle. Therefore, the person is extracted (i.e., detected) in the vehicle. For example, the vehicle interior is photographed by the camera 25, and the person is extracted by analyzing the image data (i.e., known image recognition). Alternatively, the person may be extracted by a known seat sensor that detects the person sitting on the seat 40, a temperature sensor that detects the person's body temperature, or the like.

[0171] In the next S150, it is determined whether there is a person in the vehicle cabin based on the result of the human extraction process in S140. Here, if a positive determination is made, the process returns to S110, while if a negative determination is made, the process proceeds to S160. In addition, when returning to S110, the process waits until the vehicle door is opened and closed again.

[0172] In S160, a rear image after the door is closed is acquired. That is, after the door is opened and closed, the person does not get on the vehicle, so it is considered that the door is closed when getting off the vehicle (i.e., getting off the vehicle), and the interior of the vehicle is photographed to acquire an image after getting off the vehicle (i.e., the rear image). In addition, when photographing, the lighting device 27 is turned on to illuminate the interior of the vehicle.

[0173] In the next S170 , a process of detecting dirt adhering to the seat 40 and the like is performed.

[0174] Specifically, as shown in the above-mentioned "abnormal detection method", when detecting dirt on the seat 40, for example, the difference between the front image and the rear image based on the infrared camera can be obtained, and the dirt can be detected based on the difference. That is, when there is dirt on the seat 40, a difference image corresponding to the dirt is obtained in the front image and the rear image. Therefore, when such a difference image is obtained, it can be determined that there is dirt on the seat 40.

[0175] In the next S180, the detection result of dirt (i.e., the analysis result) is sent to the cloud 5. In addition, here, the analysis result may be sent only when there is dirt, or it may be sent even when there is no dirt. In addition, when sending the analysis result, for example, when there is dirt, the image data of the front image and the rear image used for the detection of the dirt are sent to the cloud 5. In addition, the analysis result and the image data are stored in the storage unit 53.

[0176] In the next S190, it is determined whether or not there is dirt based on the analysis result transmitted from the vehicle 9 to the cloud 5. Here, if an affirmative determination is made, the process proceeds to S195, while if a negative determination is made, the present process is temporarily terminated.

[0177] In S195, since there is dirt in the seat 40, etc., this situation (i.e., the analysis result of the content of detecting dirt) is sent to the operator's mobile terminal 15 (i.e., the analysis result is reported to the operator), and the present process is temporarily terminated. In addition, at this time, the above-mentioned analysis result can also be sent to the user's mobile terminal 16 (i.e., the analysis result is reported to the user), or reported to the user through the alarm device 29.

[0178] In this process, the processes of S100 to S180 are performed by the vehicle 9 , and the processes of S190 and S195 are performed by the cloud 5 .

[0179] In addition to the above process, the result of the dirt detection in S170 (e.g., the analysis result of the content of the dirt detection) may be reported to the operator's mobile terminal 15 or the user's mobile terminal 16 before the process in S180. In this case, the determination in S190 and the report in S195 may be omitted.

[0180] <Forgotten Item Detection and Processing>

[0181] The forgotten object detection process is a process performed in a forgotten object detection application.

[0182] In this forgotten object detection process, the front image and the rear image used in the above-mentioned dirt detection process are used to detect the forgotten object. In addition, in this forgotten object detection process, the front image and the rear image can also be obtained by shooting, just like the above-mentioned dirt detection process.

[0183] like Figure 6 As shown, in S200, it is determined whether the front image and the rear image are acquired in the above-mentioned dirt detection process. Here, if the determination is affirmative, the process proceeds to S210, while if the determination is negative, the present process is temporarily terminated.

[0184] In S210, a process of detecting a forgotten object is performed.

[0185] Specifically, as shown in the above-mentioned "abnormal detection method", when detecting a forgotten object on the seat 40, an image captured by a camera 25 such as an infrared camera can be used. For example, by using multiple cameras 25 to capture the same object (for example, the same seat 40), a three-dimensional object (that is, a forgotten object) located on the seat 40 can be detected.

[0186] In the next S220, the detection result of the forgotten object (i.e., the analysis result) is sent to the cloud 5. In addition, here, the analysis result can be sent only when the forgotten object exists, but the analysis result can also be sent when the forgotten object does not exist. In addition, when the analysis result is sent, for example, when the forgotten object exists, the image data of the front image and the rear image used for the detection of the forgotten object are sent to the cloud 5. In addition, the analysis result and the image data are stored in the storage unit 53.

[0187] In the next S230, it is determined whether there is a forgotten item based on the analysis result transmitted from the vehicle 9 to the cloud 5. Here, if an affirmative determination is made, the process proceeds to S240, while if a negative determination is made, the present process is temporarily terminated.

[0188] In S240, since there is a forgotten object in the seat 40, etc., this situation (i.e., the analysis result of the content of the detected forgotten object) is sent to the operator's mobile terminal 15 (i.e., the analysis result is reported to the operator), and this process is temporarily terminated. In addition, at this time, the above-mentioned analysis result can also be sent to the user's mobile terminal 16 (i.e., the analysis result is reported to the user), or reported to the user through the alarm device 29.

[0189] In this process, the processes of S200 to S220 are performed by the vehicle 9 , and the processes of S230 and S240 are performed by the cloud 5 .

[0190] In addition to the above processing, the result of the detection of the forgotten object in S210 (for example, the analysis result of the content of the detected forgotten object) may be reported to the operator's mobile terminal 15 and the user's mobile terminal 16 before the processing in S220. In this case, the determination in S230 and the report in S240 may be omitted.

[0191] In addition, in principle, the vehicle-mounted device 3 can be turned on if the ignition device 52 of the vehicle 9 (for example, Figure 1 When the ignition switch described in the description is turned off, the operation (i.e., starting) of the vehicle-mounted device 3 ends. That is, the power supply from the vehicle battery 50 is cut off. However, when the above-mentioned dirt detection and forgotten object detection applications are operated, the processing of each application is completed (i.e., until the Figure 6 or Figure 7 The process is performed by maintaining the startup of the vehicle-mounted device 3 even if the ignition device 52 is turned off during the process. The power supply is turned off when the process is completed to the end of each flowchart. In addition, when two applications are implemented, the power supply is turned off when the process is completed to the end of both flowcharts.

[0192] In addition, the above-mentioned power-off operation is also the same for the application program of the second embodiment and the like.

[0193] [1-7. Effect]

[0194] According to this embodiment, the following effects are obtained.

[0195] (1a) In the first embodiment, abnormalities such as dirt and forgotten objects in the vehicle interior can be appropriately detected, and therefore, this is a technique suitable for, for example, a car-sharing operator or a user of the vehicle 9 .

[0196] Specifically, in the vehicle-mounted device 3, when a plurality of applications are respectively implemented, the image data is analyzed to detect abnormalities, and the analysis results obtained by the detection unit 41 and the image data used when at least an abnormality is detected (i.e., predetermined image data) are sent to the cloud 5. On the other hand, the analysis results and predetermined image data sent from the sending unit 43 are stored in the cloud 5. Furthermore, the above-mentioned analysis results are reported from the vehicle-mounted device 3 and the cloud 5 to the operator and the user.

[0197] Thus, in the first embodiment, the vehicle-mounted device 3 can detect abnormalities such as dirt based on image data obtained by photographing the interior of the vehicle 9. In addition, by sending analysis results such as abnormal detection results and specified image data to the cloud 5, the analysis results and image data can be stored in the cloud 5.

[0198] Thus, the analysis results and image data (for example, image data that is the basis of abnormality) can be reliably stored, so that the basis for taking measures corresponding to the analysis results in the future becomes reliable. In addition, since the analysis results are reported to the operator and the user, the operator and the user who receive the report can take appropriate measures according to the content of the report. In addition, there is an advantage that by performing abnormality detection by the vehicle-mounted device 3, the occurrence of abnormality can be quickly reported to the user, etc. as needed.

[0199] (1b) In the first embodiment, dirt on the seat 40 and objects left on the seat 40 can be detected.

[0200] (1c) In the first embodiment, an infrared camera can be used as the camera 25. Therefore, an abnormality of the seat 40 or the like can be easily detected based on an image of the infrared camera.

[0201] (1d) In the first embodiment, when the camera 25 is used to take an image, the lighting device 27 that illuminates the object of the image is turned on, so that a clear image can be obtained. Therefore, an abnormality of the seat 40 or the like can be easily detected from the image.

[0202] (1e) In the first embodiment, based on the difference between image data of a front image captured by the camera 25 while the user is inside the vehicle before getting on the vehicle 9 (i.e., front image data) and image data of a rear image captured by the camera 25 while the user is inside the vehicle after getting off the vehicle 9 (i.e., rear image data), abnormalities of the seat 40, etc. can be easily detected.

[0203] (1f) In the first embodiment, when an abnormality is detected based on the difference between the front image data and the rear image data, the brightness of the front image data and the rear image data is adjusted, thereby suppressing the false detection of the abnormality caused by the difference in brightness between the front image and the rear image.

[0204] (1g) In the first embodiment, it is possible to distinguish between dirt and forgotten objects.

[0205] (1h) In the first embodiment, when an instruction to photograph the interior of the vehicle is received from outside the vehicle 9 (eg, from a driver), photographing can be performed by the camera 25. Then, abnormality can be detected based on image data obtained by photographing.

[0206] [1-8. Correspondence]

[0207] Next, the relationship between the first embodiment and the present disclosure will be described.

[0208] The vehicle 9 corresponds to the vehicle, the cloud 5 corresponds to the cloud, the on-board device 3 corresponds to the on-board device, the abnormality detection system 1 corresponds to the abnormality detection system, the camera 25 corresponds to the camera, the detection unit 41 corresponds to the detection unit, the sending unit 43 corresponds to the sending unit, the storage unit 61 corresponds to the storage unit, and the vehicle ECU 23 corresponds to the relay device.

[0209] [1-9. Modifications]

[0210] Next, a modification of the first embodiment will be described.

[0211] As the configuration on the cloud 5 side, it is possible to adopt a system managed by the cloud 5 (i.e., the management server) Figure 8 That is, the analysis results may be recorded in the database 71 and the image data may be recorded in the file server 101 by using a known cloud service.

[0212] Specifically, the database 71 may include a control unit 73 having a CPU 91 and a memory 93 , and a communication unit 75 , and the analysis result transmitted from the management server to the database 71 is stored in the storage unit 77 .

[0213] The file server 101 may include a control unit 103 having a CPU 111 , a memory 113 , and a communication unit 105 , and the image data transmitted from the management server to the file server 101 may be stored in the storage unit 107 .

[0214] [2. Second Embodiment]

[0215] The basic structure of the second embodiment is the same as that of the first embodiment, so the following mainly describes the differences from the first embodiment. In addition, the same reference numerals as those of the first embodiment represent the same structures, and the previous description is referred to.

[0216] The hardware configuration of the second embodiment is the same as that of the first embodiment, and therefore the description thereof will be omitted.

[0217] The abnormality detection system 1 of the second embodiment includes a first application and a second application, each of which detects a target abnormality based on image data from a camera 25 that captures the interior of a vehicle 9. In addition, the abnormality detected by the first application has a higher urgency reported when the abnormality is detected than the abnormality detected by the second application.

[0218] [2-1. Functional structure]

[0219] like Fig. 9As shown, in the second embodiment, the control unit of the vehicle-mounted device 3 functionally includes a first detection unit 121 , a first transmission unit 123 , and a second transmission unit 125 .

[0220] The first detection unit 121 is configured to analyze the image data and detect abnormality when the first application is executed.

[0221] The first sending unit 123 is configured to send the analysis result analyzed by the first detecting unit 121 to the cloud 5 , and to send the image data used at least when an abnormality is detected to the cloud 5 .

[0222] The second transmission unit 125 is configured to transmit the image data to the cloud 5 when the second application is executed.

[0223] In addition, if Fig.10 As shown, the control unit 51 of the cloud 5 includes a first storage unit 131 , a second detection unit 133 , and a second storage unit 135 .

[0224] The first storage unit 131 is configured to store the analysis result and the image data transmitted from the first transmission unit 123 when the first application is executed.

[0225] The second detection unit 133 is configured to analyze the image data transmitted from the second transmission unit 125 and detect abnormality when the second application is executed.

[0226] The second storage unit 135 is configured to store the image data transmitted from the second transmission unit 125 and the analysis result analyzed by the second detection unit 133 .

[0227] Furthermore, the configuration is such that the analysis result is reported from at least one of the vehicle-mounted device 3 and the cloud 5 to at least one of the operator's mobile terminal 15 and the user's mobile terminal 16 .

[0228] [2-2. Control Process]

[0229] <Biological Detection Process>

[0230] This biological body detection process is a process having a high urgency (ie, priority) of the report (ie, a process performed by the first application).

[0231] like Fig.11 As shown, in S300 to S360 of the second embodiment, the same processing as S100 to S360 of the first embodiment is performed.

[0232] In the next step 370, the vehicle-mounted device 3 performs a biological body detection process. The biological body detection process is a process for detecting a living biological body (ie, a biological body) such as a child such as an infant, an elderly person, or a pet.

[0233] As a method for detecting a biological body, first, some abnormality is detected based on the difference between the front image and the rear image. That is, when there is a difference area corresponding to the difference of the image, it is judged that some abnormality exists. In addition, for the object detected with abnormality, for example, a method of detecting infants, pets, etc. through well-known image recognition processing is listed. In addition, at this time, the temperature of the detected object is considered to improve the detection accuracy. In addition, the method of detecting the above-mentioned three-dimensional object can also be used to further improve the detection accuracy.

[0234] In the next S380, the detection result of the organism is sent to the cloud 5. In addition, here, the analysis result may be sent only when the organism is detected, or the analysis result may be sent even when the organism is not detected. In addition, when the analysis result is sent (for example, when the organism is detected), the image data of the front image and the rear image used for the detection of the organism are sent to the cloud 5. In addition, the analysis result and the image data are stored in the storage unit 53.

[0235] In the next S390, it is determined whether a living body exists based on the analysis result transmitted from the vehicle 9 to the cloud 5. Here, if an affirmative determination is made, the process proceeds to S395, while if a negative determination is made, the present process is temporarily terminated.

[0236] In S395, since there is a living body in the seat 40, etc., the content (i.e., the analysis result of the content of detecting the living body) is sent to the operator's mobile terminal 15 and also to the user's mobile terminal 16, temporarily terminating the present process. In addition, at this time, it is preferred to use the alarm device 29 to quickly report to the user who is presumed to be near the vehicle 9.

[0237] In this process, the processes of S300 to S380 are performed by the vehicle 9 , and the processes of S390 and S395 are performed by the cloud 5 .

[0238] In addition to the above processing, the results of the biological detection in S370 (for example, the analysis results of the content of the detected biological body) may be reported to the operator's mobile terminal 15 and the user's mobile terminal 16 before the processing in S380. In this case, the determination in S390 and the report in S395 can be omitted.

[0239] <Dirt detection processing>

[0240] The dirt detection process is a process with a lower reporting priority than the biological body detection process (ie, a process by the second application program). In addition, the above-mentioned forgotten object detection process may be implemented instead of the dirt detection process.

[0241] This dirt detection process uses the front image and the back image used in the biological body detection process to perform the dirt detection process. In addition, in this dirt detection process, the process of capturing and acquiring the front image and the back image may be performed similarly to the dirt detection process of the first embodiment.

[0242] like Fig.12 As shown, in S400, the first application determines whether the front image and the rear image are acquired. If a positive determination is made, the process proceeds to S410, while if a negative determination is made, the process is temporarily terminated.

[0243] In the next S410 , the image data of the front image and the rear image are transmitted to the cloud 5 . The image data is stored in the storage unit 53 .

[0244] In the next step S420 , the cloud 5 performs a process of detecting dirt using the same method as in the first embodiment. The analysis result is stored in the storage unit 53 .

[0245] In the next S430, it is determined whether there is dirt based on the analysis result of S420. Here, if the determination is affirmative, the process proceeds to S440, while if the determination is negative, the present process is temporarily terminated.

[0246] In S440, since there is dirt in the seat 40, etc., the content (i.e., the analysis result of the content that detects dirt) is sent to the operator's mobile terminal 15 (i.e., the analysis result is reported to the operator), and the present process is temporarily terminated. In addition, at this time, the above-mentioned analysis result can also be sent to the user's mobile terminal 16 (i.e., the analysis result is reported to the user). In addition, it can also be reported to the user through the alarm device 29.

[0247] In this process, the processes of S400 and S410 are performed by the vehicle 9 , and the processes of S420 to S440 are performed by the cloud 5 .

[0248] In the second embodiment, the same effect as the first embodiment is achieved. In addition, in the second embodiment, the process of detecting a living body such as an infant or a pet is quickly implemented after getting off the vehicle, and when an infant or a pet is detected, it is quickly reported to the user or the operator, thereby having the effect of high safety.

[0249] In addition, the following examples are given as modified examples of the second embodiment.

[0250] Specifically, as a process of reporting high urgency, the same forgotten object detection process as in the first embodiment can be adopted instead of the above-mentioned biological body detection process. At this time, the above-mentioned biological body detection process of S370 can be replaced by the above-mentioned forgotten object detection process of S210, and the above-mentioned biological body existence determination process of S390 can be replaced by the above-mentioned forgotten object existence determination process of S230.

[0251] [3. Other embodiments]

[0252] As mentioned above, although embodiment of this disclosure was described, this disclosure is not limited to the said embodiment, It is needless to say that various forms can be adopted.

[0253] (3a) The present disclosure can be applied to a service in which a plurality of users share a vehicle, for example, a car sharing service or a car rental service.

[0254] (3b) As a plurality of application programs, two or more application programs can be used.

[0255] (3c) Examples of abnormalities in the vehicle interior include dirt, forgotten items, damaged parts, and the presence of living organisms after exiting the vehicle. Examples of abnormal locations include seats and locations other than seats (e.g., doors, windows, floors, and dashboards).

[0256] (3d) Regarding the detection method of dirt, forgotten objects, and living organisms, various methods can be used in addition to the above-mentioned detection methods. For example, based on the image of an infrared camera, the difference in the material of the photographed object can be known, so it is possible to distinguish, for example, whether it is a seat or an object other than a seat, such as paper, clothing, or a bag (i.e., a forgotten object), based on the difference between the front image and the back image.

[0257] (3e) As image data sent from the vehicle side to the cloud side, image data used for detecting abnormalities when an abnormality is detected (for example, image data of front images and rear images) are listed, but even when no abnormality is detected, the above-mentioned image data can be sent for confirmation.

[0258] (3f) The anomaly detection system and anomaly detection method described in the present disclosure may also be implemented by a special-purpose computer provided by a processor and a memory programmed to execute one or more functions embodied by a computer program.

[0259] Alternatively, the abnormality detection system and the abnormality detection method described in the present disclosure may be implemented by a dedicated computer provided by a processor composed of one or more dedicated hardware logic circuits.

[0260] Alternatively, the anomaly detection system and anomaly detection method described in the present disclosure may also be implemented by one or more special-purpose computers, which are composed of a combination of a processor programmed to perform one or more functions and a memory and a processor composed of one or more hardware logic circuits.

[0261] In addition, the computer program may be stored in a computer-readable non-transferable tangible recording medium as an instruction executed by the computer. The method of realizing the functions of each unit included in the abnormality detection system does not necessarily need to include software, and all functions may be realized using one or more hardware.

[0262] (3g) In addition to the above-mentioned abnormality detection system, the present disclosure can be implemented in various forms such as a program for making a computer of the abnormality detection system function, a non-migrating tangible recording medium such as a semiconductor memory recording the program, and a control method.

[0263] (3h) It is also possible to implement multiple functions of one component in each of the above embodiments by multiple components, or to implement one function of one component by multiple components. In addition, it is also possible to implement multiple functions of multiple components by one component, or to implement one function implemented by multiple components by one component. In addition, a part of the structure of each of the above embodiments may be omitted. In addition, it is also possible to add or replace at least a part of the structure of each of the above embodiments relative to the structure of other embodiments.

[0264] [4. Technical ideas disclosed in this specification]

[0265] [Project 1]

[0266] An abnormality detection system (1) comprises a cloud (5) for collecting data of a vehicle (9), and a vehicle-mounted device (3) connected to the cloud so as to be able to communicate, wherein the abnormality detection system (1) detects abnormalities in a vehicle cabin, wherein:

[0267] A plurality of application programs are provided, each of which is configured to detect the above-mentioned abnormality as a target based on image data from a camera (25) that captures the above-mentioned vehicle interior,

[0268] The above-mentioned vehicle-mounted device has:

[0269] A detection unit (41) configured to analyze the image data and detect the abnormality when the plurality of application programs are respectively executed; and

[0270] a sending unit (43) configured to send the analysis result analyzed by the detection unit to the cloud, and at least send the image data used when the abnormality is detected to the cloud,

[0271] The cloud includes a storage unit (61) configured to store the analysis result and the image data transmitted from the transmission unit.

[0272] The analysis result is reported to a reporting target from at least one of the vehicle-mounted device and the cloud.

[0273] [Project 2]

[0274] An abnormality detection system (1) comprises a cloud (5) for collecting data of a vehicle (9), and a vehicle-mounted device (3) connected to the cloud so as to be able to communicate, wherein the abnormality detection system (1) detects abnormalities in a vehicle cabin, wherein:

[0275] The system comprises a first application and a second application, wherein the first application and the second application are configured to detect the abnormality as a target, based on image data from a camera that captures the interior of the vehicle, respectively.

[0276] The anomaly detected by the first application program is reported with a higher urgency when the anomaly is detected than the anomaly detected by the second application program.

[0277] The above-mentioned vehicle-mounted device has:

[0278] A first detection unit (121) is configured to analyze the image data and detect the abnormality when the first application is executed;

[0279] A first sending unit (123) is configured to send the analysis result analyzed by the first detection unit to the cloud, and at least send the image data used when the abnormality is detected to the cloud; and

[0280] The second sending unit (125) is configured to send the image data to the cloud when the second application is executed.

[0281] The above cloud has:

[0282] A first storage unit (131) configured to store the analysis result and the image data sent from the first sending unit when the first application is executed;

[0283] a second detection unit (133) configured to analyze the image data sent from the second sending unit to detect the abnormality when the second application is executed; and

[0284] The second storage unit (135) is configured to store the image data transmitted from the second transmission unit and the analysis result analyzed by the second detection unit.

[0285] The analysis result is reported to a reporting target from at least one of the vehicle-mounted device and the cloud.

[0286] [Item 3]

[0287] An abnormality detection system (1) comprises a cloud (5) for collecting data of a vehicle (9), and an onboard device (3), the onboard device (3) being connected to the cloud so as to be communicable, and being connected to a relay device (23) for relaying each frame of a network (30) flowing through the vehicle so as to be communicable, the abnormality detection system (1) detecting an abnormality in a vehicle cabin, wherein:

[0288] The above-mentioned vehicle-mounted device has:

[0289] An in-vehicle communication unit (45) is configured to communicate with an electronic control device (32, 36) connected to the network of the vehicle via the relay device;

[0290] A detection unit (41) configured to analyze image data from a camera (25) that captures the interior of the vehicle to detect the abnormality; and

[0291] a sending unit (43) configured to send the analysis result analyzed by the detection unit to the cloud, and at least send the image data used when the abnormality is detected to the cloud,

[0292] The vehicle is configured to report the analysis result obtained by the detection unit to the outside of the vehicle.

[0293] [Item 4]

[0294] An anomaly detection system according to any one of items 1 to 3, wherein:

[0295] The abnormality is an abnormality related to the seat including at least dirt on the seat (40) or a forgotten object on the seat.

[0296] [Item 5]

[0297] An anomaly detection system according to any one of items 1 to 4, wherein:

[0298] The camera (25) is an infrared camera.

[0299] [Item 6]

[0300] An anomaly detection system according to any one of items 1 to 5, wherein:

[0301] When photographing with the camera, the lighting for illuminating the photographed object is turned on.

[0302] [Item 7]

[0303] An anomaly detection system according to any one of items 1 to 6, wherein:

[0304] The abnormality is detected based on a difference between front image data obtained by photographing the interior of the vehicle before the user gets on the vehicle using the camera and rear image data obtained by photographing the interior of the vehicle after the user gets off the vehicle using the camera.

[0305] [Item 8]

[0306] According to the anomaly detection system described in item 7, wherein:

[0307] When the abnormality is detected based on the difference between the front image data and the rear image data, the brightness of the front image data and the rear image data is adjusted.

[0308] [Item 9]

[0309] An anomaly detection system according to any one of items 1 to 8, wherein:

[0310] The device is configured to discriminate dirt and forgotten items as the above-mentioned abnormality.

[0311] [Item 10]

[0312] An anomaly detection system according to any one of items 1 to 9, wherein:

[0313] When an instruction to perform the imaging is received from outside the vehicle, the camera performs imaging, and the abnormality is detected based on image data obtained by the imaging.

[0314] [Item 11]

[0315] An anomaly detection system according to any one of items 1 to 10, wherein:

[0316] The vehicle-mounted device is connected to a vehicle battery (50), and when an ignition device (52) of the vehicle is turned off, the activation of the camera is terminated, and the activation of the vehicle-mounted device itself is also terminated.

[0317] [Item 12]

[0318] According to the anomaly detection system described in item 11,

[0319] After the ignition of the vehicle is turned off, when the abnormality is detected, activation of the camera and activation of the vehicle-mounted device itself are not terminated until the analysis result is reported to the outside of the vehicle.

[0320] [Item 13]

[0321] An abnormality detection method is provided, wherein a vehicle-mounted device (3) mounted on a vehicle (9) and a cloud (5) can communicate with each other to detect abnormalities in a vehicle interior, wherein:

[0322] Using a plurality of application programs, the plurality of application programs are configured to respectively detect the above-mentioned abnormalities as the target based on image data from a camera (25) that captures the above-mentioned vehicle interior,

[0323] In the vehicle-mounted device, when the plurality of applications are respectively implemented, the image data is analyzed to detect the abnormality, the analyzed results are sent to the cloud, and at least the image data used when the abnormality is detected is sent to the cloud.

[0324] The cloud stores the analysis results and the image data sent thereto.

[0325] Furthermore, the analysis result is reported to a reporting target from at least one of the vehicle-mounted device and the cloud.

[0326] [Item 14]

[0327] An abnormality detection method is provided, wherein a vehicle-mounted device (3) mounted on a vehicle (9) and a cloud (5) can communicate with each other to detect abnormalities in a vehicle interior, wherein:

[0328] Using a first application and a second application, the first application and the second application are configured to detect the abnormality as the target based on image data from a camera (25) that captures the interior of the vehicle, and the abnormality detected by the first application has a higher urgency level reported when the abnormality is detected than the abnormality detected by the second application,

[0329] In the above-mentioned vehicle-mounted device,

[0330] In the case of implementing the first application, the image data is analyzed to detect the abnormality, the analyzed result is sent to the cloud, and at least the image data used in the case of detecting the abnormality is sent to the cloud,

[0331] In the case of implementing the second application, the image data is sent to the cloud.

[0332] In the above cloud,

[0333] When the first application is implemented, the analysis result and the image data sent are stored.

[0334] When the second application is implemented, the image data sent is analyzed to detect the abnormality, the image data sent and the analyzed result are stored,

[0335] Furthermore, the analysis result is reported to a reporting target from at least one of the vehicle-mounted device and the cloud.

[0336] [Item 15]

[0337] An abnormality detection method detects abnormalities in a vehicle cabin using a cloud (5) for collecting data of a vehicle (9) and an onboard device (3), wherein the onboard device (3) is connected to the cloud so as to be communicable and is connected to a relay device (23) for relaying frames of a network (30) flowing through the vehicle so as to be communicable, wherein:

[0338] The vehicle-mounted device communicates with an electronic control device (32, 36) connected to the network of the vehicle via the relay device, analyzes image data from a camera (25) that photographs the interior of the vehicle to detect the abnormality, sends the analyzed results to the cloud, and at least sends the image data used when the abnormality is detected to the cloud, and reports the analyzed results to the outside of the vehicle.

Claims

1. An abnormality detection system (1), comprising a cloud (5) for collecting data of a vehicle (9), and a vehicle-mounted device (3) connected to the cloud so as to be able to communicate, wherein the abnormality detection system (1) detects abnormalities in a vehicle cabin, wherein: A plurality of application programs are provided, each of which is configured to detect the abnormality that is the target based on image data from a camera (25) that captures the interior of the vehicle. The vehicle-mounted device comprises: A detection unit (41) configured to analyze the image data respectively and detect the abnormality when the plurality of application programs are respectively implemented; as well as a sending unit (43) configured to send the analysis result analyzed by the detection unit to the cloud, and to send at least the image data used when the abnormality is detected to the cloud, The cloud comprises a storage unit (61) configured to store the analysis result and the image data transmitted from the transmission unit. The analysis result is reported to a reporting target from at least one of the vehicle-mounted device and the cloud.

2. An abnormality detection system (1), comprising a cloud (5) for collecting data of a vehicle (9), and a vehicle-mounted device (3) connected to the cloud so as to be able to communicate, wherein the abnormality detection system (1) detects abnormalities in a vehicle cabin, wherein: A first application and a second application are provided, wherein the first application and the second application are configured to detect the abnormality as a target, based on image data from a camera that captures the interior of the vehicle, respectively. The anomaly detected by the first application is reported with a higher urgency when the anomaly is detected than the anomaly detected by the second application, The vehicle-mounted device comprises: A first detection unit (121) configured to analyze the image data and detect the abnormality when the first application is implemented; A first sending unit (123) configured to send the analysis result analyzed by the first detection unit to the cloud, and at least send the image data used when the abnormality is detected to the cloud; as well as A second sending unit (125) is configured to send the image data to the cloud when the second application is executed. The cloud has: a first storage unit (131) configured to store the analysis result and the image data sent from the first sending unit when the first application is executed; a second detection unit (133) configured to analyze the image data sent from the second sending unit to detect the abnormality when the second application is executed; as well as A second storage unit (135) is configured to store the image data transmitted from the second transmission unit and the analysis result analyzed by the second detection unit, The analysis result is reported to a reporting target from at least one of the vehicle-mounted device and the cloud.

3. An abnormality detection system (1), comprising a cloud (5) for collecting data of a vehicle (9), and an onboard device (3), the onboard device (3) being connected to the cloud so as to be communicable, and being connected to a relay device (23) for relaying each frame of a network (30) flowing through the vehicle so as to be communicable, the abnormality detection system (1) detecting an abnormality in a vehicle cabin, wherein: The vehicle-mounted device comprises: An in-vehicle communication unit (45) configured to communicate with an electronic control device (32, 36) connected to the network of the vehicle via the relay device; A detection unit (41) configured to analyze image data from a camera (25) that captures the interior of the vehicle to detect the abnormality; as well as a sending unit (43) configured to send the analysis result analyzed by the detection unit to the cloud, and to send at least the image data used when the abnormality is detected to the cloud, The vehicle is configured to report an analysis result obtained by the detection unit to the outside of the vehicle.

4. The abnormality detection system according to any one of claims 1 to 3, wherein: The abnormality is an abnormality related to the seat including at least dirt on the seat (40) or a forgotten object on the seat.

5. The abnormality detection system according to any one of claims 1 to 3, wherein: The camera (25) is an infrared camera.

6. The abnormality detection system according to any one of claims 1 to 3, wherein: When photographing with the camera, the lighting for illuminating the photographed object is turned on.

7. The abnormality detection system according to any one of claims 1 to 3, wherein: The abnormality is detected based on a difference between front image data obtained by photographing the interior of the vehicle before the user gets on the vehicle using the camera and rear image data obtained by photographing the interior of the vehicle after the user gets off the vehicle using the camera.

8. The abnormality detection system according to claim 7, wherein: When the abnormality is detected based on the difference between the front image data and the rear image data, the brightness of the front image data and the rear image data is adjusted.

9. The abnormality detection system according to any one of claims 1 to 3, wherein: The device is configured to discriminate dirt and forgotten objects as the abnormality.

10. The abnormality detection system according to any one of claims 1 to 3, wherein: When an instruction to perform the imaging is received from outside the vehicle, the camera performs imaging, and the abnormality is detected based on image data obtained by the imaging.

11. The abnormality detection system according to any one of claims 1 to 3, wherein: The vehicle-mounted device is connected to a vehicle battery (50), and when an ignition device (52) of the vehicle is turned off, the activation of the camera is terminated, and the activation of the vehicle-mounted device itself is also terminated.

12. The abnormality detection system according to claim 11, wherein: After the ignition of the vehicle is turned off, if the abnormality is detected, activation of the camera and activation of the vehicle-mounted device itself are not terminated until the analysis result is reported to the outside of the vehicle.

13. An abnormality detection method, wherein a vehicle-mounted device (3) mounted on a vehicle (9) and a cloud (5) can communicate with each other to detect abnormalities in a vehicle interior, wherein: Using a plurality of application programs, the plurality of application programs are configured to respectively detect the abnormality that is the target based on image data from a camera (25) that captures the interior of the vehicle, In the vehicle-mounted device, when the plurality of applications are respectively implemented, the image data is analyzed to detect the abnormality, the analyzed results are sent to the cloud, and at least the image data used when the abnormality is detected is sent to the cloud, storing the sent analysis result and the image data in the cloud, Furthermore, the analysis result is reported to a reporting target from at least one of the vehicle-mounted device and the cloud.

14. An abnormality detection method, wherein a vehicle-mounted device (3) mounted on a vehicle (9) and a cloud (5) can communicate with each other to detect abnormalities in a vehicle interior, wherein: Using a first application and a second application, the first application and the second application are configured to respectively detect the abnormality that is the purpose based on image data from a camera (25) that captures the interior of the vehicle, and the abnormality detected by the first application has a higher urgency level reported when the abnormality is detected than the abnormality detected by the second application, In the vehicle-mounted device, In the case of implementing the first application, the image data is analyzed to detect the abnormality, the analyzed result is sent to the cloud, and at least the image data used in the case of detecting the abnormality is sent to the cloud, In case of implementing the second application, sending the image data to the cloud, In the cloud, When the first application is implemented, the sent analysis result and the image data are stored, When the second application is implemented, the transmitted image data is analyzed to detect the abnormality, the transmitted image data and the analyzed result are stored, Furthermore, the analysis result is reported to a reporting target from at least one of the vehicle-mounted device and the cloud.

15. An abnormality detection method, using a cloud (5) for collecting data of a vehicle (9), and an onboard device (3) to detect abnormalities in a vehicle cabin, the onboard device (3) being connected to the cloud so as to be communicable, and connected to a relay device (23) for relaying frames of a network (30) flowing through the vehicle so as to be communicable, wherein: The vehicle-mounted device communicates with an electronic control device (32, 36) connected to the network of the vehicle via the relay device, analyzes image data from a camera (25) that photographs the interior of the vehicle to detect the abnormality, sends the analyzed analysis results to the cloud, and at least sends the image data used when the abnormality is detected to the cloud, and reports the analyzed analysis results to the outside of the vehicle.

Images