Method, in-vehicle device, and program

Abstract

We will improve the technology for detecting objects near vehicle doors. [Solution] The method performed by the in-vehicle device 10 includes setting a first area and a second area in front of the door inside the vehicle in the video captured by the camera unit 16, extracting passengers and objects from the video using image analysis technology, and stopping the opening and closing operation of the door when at least one of a passenger in the first area and an object in the second area that is captured in the video is detected.

Description

Technical Field

[0001] The present disclosure relates to a method, an in-vehicle device, and a program.

Background Art

[0002] Conventionally, there has been a technology for detecting an object near a vehicle door. For example, Patent Document 1 discloses providing a door pinch detection area and a gap detection area near the door of a railway vehicle.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Near the door, since the behaviors of people and objects are different, it is necessary to detect them separately. Also, when light passing through the window glass included in the vehicle door is misdetected as an object, etc., the opening and closing operation of the door stops each time, causing annoyance. Thus, there has been room for improvement in the technology for detecting an object near a vehicle door.

Means for Solving the Problems

[0005] The method according to the present disclosure is a method executed by an in-vehicle device including a photographing unit that photographs the interior of a vehicle and a control unit, wherein the control unit sets a first area and a second area on the front side of a door in the interior of the vehicle in the video photographed by the photographing unit, extracts passengers and objects from the video using image analysis technology, and stops the opening and closing operation of the door when at least one of the passengers present in the first area and the objects present in the second area included in the video is detected.

[0006] The in-vehicle device according to this disclosure comprises a camera unit for photographing the interior of a vehicle and a control unit, wherein the control unit sets a first area and a second area in front of the door of the interior of the vehicle in the image captured by the camera unit, extracts passengers and objects from the image using image analysis technology, and stops the opening and closing operation of the door when it detects at least one of a passenger in the first area and an object in the second area that is captured in the image.

[0007] The program relating to this disclosure causes a computer, which is an in-vehicle device comprising a camera unit for photographing the interior of a vehicle and a control unit, to perform operations including setting a first area and a second area in front of the door inside the vehicle's interior in the image captured by the camera unit, extracting passengers and objects from the image using image analysis technology, and stopping the opening and closing operation of the door when at least one of a passenger in the first area and an object in the second area that is captured in the image is detected. [Effects of the Invention]

[0008] This disclosure makes it possible to improve the technology for detecting objects near vehicle doors. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows the schematic configuration of the in-vehicle device according to this embodiment. [Figure 2] This flowchart shows an example of the operation of the in-vehicle device according to the first embodiment. [Figure 3] This is a diagram illustrating the video footage acquired by the in-vehicle device. [Figure 4] This flowchart shows an example of the operation of the in-vehicle device according to the second embodiment. [Figure 5A] This is a diagram illustrating the video footage acquired by the in-vehicle device. [Figure 5B] This is a diagram illustrating the video footage acquired by the in-vehicle device. [Modes for carrying out the invention]

[0010] (First Embodiment) Hereinafter, a first embodiment of this disclosure will be described with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0011] Referring to Figure 1, an overview of the in-vehicle device 10 according to this embodiment will be described. The in-vehicle device 10 is a computer mounted in a vehicle.

[0012] Vehicles include, for example, route buses or tour buses. Vehicles are any type of automobile, such as gasoline cars, diesel cars, hydrogen cars, HEVs, PHEVs, BEVs, or FCEVs. "HEV" is an abbreviation for hybrid electric vehicle. "PHEV" is an abbreviation for plug-in hybrid electric vehicle. "BEV" is an abbreviation for battery electric vehicle. "FCEV" is an abbreviation for fuel cell electric vehicle. Vehicles may be driven by a driver or their operation may be automated to any level. The level of automation may be, for example, one of levels 1 to 5 in the SAE classification. "SAE" is an abbreviation for Society of Automotive Engineers. Vehicles may also be MaaS-dedicated vehicles. "MaaS" is an abbreviation for Mobility as a Service. Vehicles are equipped with control units that control the operation of various parts of the vehicle, including doors.

[0013] First, an overview of this embodiment will be described, and details will be described later. The in-vehicle device 10 sets a first area and a second area in front of the door inside the vehicle in the video footage of the vehicle's interior. The in-vehicle device 10 extracts passengers and objects from the video footage using image analysis technology. When the in-vehicle device 10 detects at least one of a passenger in the first area and an object in the second area that is captured in the video footage using image analysis technology, it stops the opening and closing operation of the door.

[0014] The objects include luggage, strollers, or suitcases carried by passengers. According to this embodiment, passengers and objects are detected separately in the first and second areas. As described below, by detecting objects in the second area, which has a smaller area than the first area for passenger detection, the possibility of non-objects being mistakenly detected as objects, such as light entering through a window, is reduced. Therefore, it is possible to safely stop the opening and closing operation only when a passenger or object is actually present near the door, while avoiding the inconvenience caused by frequent stopping of the door opening and closing operation or frequent notifications indicating that an object has been detected. Thus, it is possible to improve the technology for detecting objects near vehicle doors.

[0015] Referring to Figure 1, the configuration of the in-vehicle device 10 according to this embodiment will be described. The in-vehicle device 10 comprises a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, an output unit 15, and an imaging unit 16. Each unit is connected to the others so as to be able to communicate with each other via an in-vehicle network such as CAN (Controller Area Network) or a dedicated line.

[0016] The control unit 11 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The dedicated circuit is, for example, an FPGA or ASIC. "FPGA" is an abbreviation for field-programmable gate array. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 11 may include an ECU. "ECU" is an abbreviation for Electronic Control Unit. The control unit 11 controls each part of the in-vehicle device 10 and executes processes related to the operation of the in-vehicle device 10.

[0017] The storage unit 12 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or at least two combinations thereof. The semiconductor memory is, for example, RAM or ROM. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read-only memory. The RAM is, for example, SRAM or DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read-only memory. The storage unit 12 functions, for example, as a main memory, auxiliary memory, or cache memory. The storage unit 12 stores information used for the operation of the in-vehicle device 10 and information obtained by the operation of the in-vehicle device 10. The information stored in the storage unit 12 may be updatable with information obtained from the network via the communication unit 13.

[0018] The communication unit 13 includes at least one communication interface. The communication interface is, for example, an interface corresponding to a mobile communication standard such as LTE, 4G standard, or 5G standard, an interface corresponding to a short-range wireless communication standard such as Bluetooth (registered trademark), or a LAN interface. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 13 receives information used for the operation of the in-vehicle device 10 via a network, and transmits information obtained by the operation of the in-vehicle device 10 via the network.

[0019] The network includes the Internet, at least one WAN, at least one MAN, or any combination thereof. "WAN" is an abbreviation for wide area network. "MAN" is an abbreviation for metropolitan area network. The network may include at least one wireless network, at least one optical network, or any combination thereof. The wireless network is, for example, an ad hoc network, a cellular network, a wireless LAN, a satellite communication network, or a terrestrial microwave network. "LAN" is an abbreviation for local area network.

[0020] The input unit 14 includes at least one input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen provided integrally with a display, or a microphone. The input unit 14 receives an operation for inputting data used for the operation of the vehicle and the in-vehicle device 10. Instead of being provided in the in-vehicle device 10, the input unit 14 may be connected to the in-vehicle device 10 as an external input device. As a connection method, for example, any method such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used. "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface.

[0021] The output unit 15 includes at least one output interface. The output interface is, for example, a display or a speaker. The display is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescence. The output unit 15 outputs data obtained by the operation of the vehicle and the in-vehicle device 10. Instead of being provided in the in-vehicle device 10, the output unit 15 may be connected to the in-vehicle device 10 as an external output device. As a connection method, for example, any method such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0022] The imaging unit 16 is a camera that images the interior of the vehicle. The imaging unit 16 outputs the captured video to the control unit 11. The imaging unit 16 is provided, for example, on the ceiling of the vehicle and can capture a video in which the vehicle door and the floor surface in front of the door are reflected.

[0023] The functions of the in-vehicle device 10 are realized by executing the program according to this embodiment on the processor, which acts as the control unit 11. In other words, the functions of the in-vehicle device 10 are realized by software. The program causes the computer to perform the operations of the in-vehicle device 10, thereby causing the computer to function as the in-vehicle device 10. That is, the computer functions as the in-vehicle device 10 by performing the operations of the in-vehicle device 10 according to the program.

[0024] The program can be stored on a non-temporary computer-readable medium. Examples of non-temporary computer-readable mediums include flash memory, magnetic recording devices, optical discs, magneto-optical recording media, or ROM. The program can be distributed, for example, by selling, transferring, or lending portable media such as SD cards, DVDs, or CD-ROMs containing the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may also be distributed by storing it in server storage and transferring it from the server to other computers. The program may also be provided as a program product.

[0025] A computer, for example, stores a program stored on a portable medium or a program transferred from a server in its main memory. Then, the computer reads the program stored in the main memory with its processor and executes the processing according to the read program. The computer may also read the program directly from the portable medium and execute the processing according to the program. The computer may also execute the processing according to the received program sequentially each time a program is transferred to the computer from a server. Processing may also be performed by a so-called ASP-type service, which does not transfer programs from the server to the computer, but realizes its function only through execution instructions and result retrieval. "ASP" is an abbreviation for application service provider. A program includes information used for processing by an electronic computer that is equivalent to a program. For example, data that is not a direct instruction to the computer but has the nature of defining the computer's processing falls under "equivalent to a program".

[0026] Some or all of the functions of the in-vehicle device 10 may be implemented by a programmable circuit or a dedicated circuit as the control unit 11. In other words, some or all of the functions of the in-vehicle device 10 may be implemented by hardware.

[0027] The operation of the in-vehicle device 10 according to this embodiment will be described with reference to Figures 2 and 3. The operation in Figure 2 corresponds to the method according to this embodiment. In the following, communication between the in-vehicle device 10 and external devices will be performed via the communication unit 13 and the network.

[0028] In S101 of Figure 2, the control unit 11 of the in-vehicle device 10 acquires video from the camera unit 16. The control unit 11 may acquire the video at the timing when the vehicle door is opened or closed. For example, when the control unit 11 acquires information from the vehicle's control unit indicating that the vehicle is stopping or departing for passenger boarding or alighting, it may acquire video captured by the camera unit 16. For example, the control unit 11 may acquire information indicating the vehicle's acceleration detected by the vehicle's acceleration sensor from the vehicle's control unit, and when the vehicle's acceleration is above a predetermined value, it may acquire video captured by the camera unit 16. Figure 3 shows video I captured by the camera unit 16. Video I shows the floor F inside the vehicle, the door D, and a passenger P standing in front of the door D, along with the luggage B held by passenger P.

[0029] In S102, the control unit 11 sets a first area and a second area in the front of the door inside the vehicle's interior in the video acquired in S101. The first area and the second area are two-dimensional planar regions that demarcate the vehicle's floor surface in the video, and are regions within a predetermined distance from the position of the door's seam when the door is closed. For example, the control unit 11 extracts the position of the door and the door's seam in the acquired video using any image analysis technique, and sets a rectangular region with the vehicle's door as one side, including the door's seam, as the first area and the second area. The first area and the second area are not limited to rectangles and may have any shape, such as a circle. The control unit 11 may set at least a part of the first area A1 and the second area A2 overlapping. The set first area A1 may include the set second area A2. In Figure 3, the first area A1, set by the control unit 11 in the video I, is shown with diagonal lines, and the second area A2 is shown in black. Referring to Figure 3, the first area A1 and the second area A2 overlap, and the area of ​​the second area A2 is smaller than that of the first area A1. Both the first area A1 and the second area A2 are rectangular regions with the door D, including the joint DM of the door D, as one side, and are rectangular regions within a predetermined distance from the joint DM when the door D is closed.

[0030] In S103 of Figure 2, the control unit 11 determines from the video acquired in S101 whether at least one of a passenger in the first area and an object in the second area can be detected. First, the control unit 11 extracts regions from the video that represent at least one of the passenger and the object using any image analysis technique. For example, the control unit 11 extracts regions that represent at least one of the passenger and the object based on the contour feature quantity represented by pixels of a different color from the floor surface in the acquired video. The regions that represent passengers or objects may have shapes such as rectangles, circles, or curved contours. The control unit 11 determines that the presence of a passenger is detected if at least a part of the passenger region or all of the passenger region overlaps with the first area set in S102. The control unit 11 determines that the presence of an object is detected if at least a part of the object region or all of the object region overlaps with the second area set in S102. If at least one of the passenger and the object is detected, the control unit 11 proceeds to S104. If no passengers or objects are detected, the control unit 11 proceeds to S106.

[0031] In S104, the control unit 11 generates a control signal to stop the door opening and closing operation and outputs the control signal to the vehicle's control unit. If the vehicle has multiple doors, the control signal may include door identification information indicating the door that is captured in the video and whose opening and closing operation is to be stopped. The vehicle's control unit receives the control signal and stops the door opening and closing operation based on the control signal. If the control signal includes door identification information, the vehicle's control unit stops the opening and closing operation of the target door based on the door identification information. After that, the operation of the control unit 11 of the in-vehicle device 10 proceeds to S105.

[0032] In S105, the control unit 11 transmits detection information indicating that it has detected at least one of a passenger in the first area and an object in the second area to the terminal device of the user monitoring the vehicle via the communication unit 13. The user monitoring the vehicle includes the vehicle's crew or an administrator who remotely controls or monitors the vehicle. By receiving the detection information on the terminal device, the user can immediately know that a passenger or object has been detected near the vehicle's door. For example, if the monitor acting as the output unit 15 is being monitored by a user acting as a crew member, the control unit 11 may output the detection information to the output unit 15. The user may, for example, transmit warning information to the in-vehicle device 10. In this case, the control unit 11 of the in-vehicle device 10 may receive the information and output a message such as "Please move away from the door" or "Please keep objects away from the door" via the output unit 15, either by voice or image.

[0033] In S106, the control unit 11 outputs a control signal to the vehicle's control unit to perform a door opening and closing operation. If the vehicle has multiple doors, the control signal may include door identification information indicating the door that is captured in the video and is the target of the opening and closing operation. The vehicle's control unit performs the door opening and closing operation based on the control signal. If the control signal includes door identification information, the vehicle's control unit performs the opening and closing operation of the target door based on that door identification information. After that, the operation of the control unit 11 of the in-vehicle device 10 returns to S101.

[0034] (Second Embodiment) Next, a second embodiment of the present disclosure will be described. The configuration of the in-vehicle device 10 according to the second embodiment is the same as that of the first embodiment described above, so its description will be omitted.

[0035] The operation of the in-vehicle device 10 according to this embodiment will be described below with reference to Figures 4 to 5B.

[0036] Since steps S201 to S206 in Figure 4 are the same as steps S101 to S106 in Figure 2 of the first embodiment described above, their explanation will be omitted.

[0037] In S207, the control unit 11 acquires video from the camera unit 16. The control unit 11 may, at the same time as outputting a control signal to the vehicle's control unit to execute the door opening and closing operation in S206, acquire video from the camera unit 16.

[0038] In S208, the control unit 11 sets a third area in front of the door inside the vehicle's interior from the video acquired in S207. The third area, like the first and second areas, is a two-dimensional planar region that demarcates the vehicle's floor surface in the video, and is within a predetermined distance from the position of the door's seam when it is closed. In this embodiment, the control unit 11 extracts the positions of the door and the door's seam in the acquired video using any image analysis technique, and sets a rectangular region with the vehicle's door as one side, including the door's seam, as the third area. The third area is not limited to a rectangle and may have any shape, such as a circle. Not limited to this, the control unit 11 may also set a region as the third area that has the same area as the second area, or an area less than or equal to the area of ​​the second area.

[0039] The control unit 11 may use any image analysis technique to detect the opening and closing operation of the vehicle door from the video acquired in S207 and set a third area whose shape changes according to the opening and closing operation. That is, the area of ​​the third area may change according to the degree to which the vehicle door is open during the opening and closing operation. Figures 5A and 5B are diagrams showing the third area A3 set by the control unit 11 in the video I' acquired in S207, using dot hatching. Figure 5A shows an open door D, and Figure 5B shows a door D closing in the direction of the arrow. As shown in Figures 5A and 5B, the area of ​​the third area A3 decreases as the door D closes. The area of ​​the third area A3 may also change significantly as the door D opens.

[0040] In S209 of Figure 4, the control unit 11 determines from the image acquired in S207 whether or not an object is detected in the third area. First, the control unit 11 extracts the region representing the object from the image using any image analysis technique. For example, the control unit 11 extracts the region representing the object based on the contour feature quantity represented by pixels of a different color from the floor surface in the acquired image. The region representing the object may have a shape such as a rectangle, circle, or a curved contour. The control unit 11 determines that the presence of an object is detected if at least a part or all of the region of the object overlaps the third area. If an object is detected, the control unit 11 proceeds to S210. If no object is detected, the control unit 11 proceeds to S212.

[0041] Since steps S210 to S212 are the same as steps S104 to S106 in Figure 2 of the first embodiment described above, their explanation will be omitted.

[0042] According to this embodiment, after the vehicle door restarts its opening and closing operation, objects in front of the door can be detected separately from passengers. This makes it easier to detect objects, for example, when a passenger has moved away from the door but has unintentionally left luggage in front of it, allowing the door to be safely stopped.

[0043] (modified version) As one modification of the present disclosure, the in-vehicle device 10 may further include a projection unit that projects an image onto the floor surface of the vehicle. The projection unit is, for example, mounted on the ceiling of the vehicle and includes a projector capable of projecting an image onto the floor surface inside the vehicle. In this modification, when the control unit 11 sets a first to third area in S102 of Figure 2, S202 of Figure 4, or S208 above, it sends an instruction to the projection unit to project an image representing the set area onto the actual floor surface of the vehicle. The image includes, for example, an image that highlights the set area with color or lines. Image data representing the first to third areas may be stored in the storage unit 12 in advance. When the projection unit receives the instruction and projects the image onto the floor surface, it becomes easier for a passenger standing in front of the door to recognize the area that should not be obstructed by their body or luggage.

[0044] This disclosure is not limited to the embodiments described above. For example, two or more blocks described in the block diagram may be combined, or one block may be divided. Instead of executing two or more steps described in the flowchart in chronological order as described, they may be executed in parallel or in a different order, depending on the processing capacity of the device performing each step, or as necessary. Other modifications are possible without departing from the spirit of this disclosure.

[0045] Some embodiments of the present disclosure are described below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Note 1] A method performed by an in-vehicle device comprising a camera unit for photographing the interior of a vehicle and a control unit, The control unit, In the video captured by the aforementioned camera unit, a first area and a second area are defined on the front side of the interior door of the vehicle. The process involves extracting passengers and objects from the aforementioned video using image analysis technology, When at least one of the passengers in the first area and the objects in the second area that are captured in the aforementioned video is detected, the opening and closing operation of the door is stopped. Methods that include... [Note 2] The method described in Appendix 1, A method wherein the second area has a smaller area than the first area. [Note 3] The method described in Appendix 1, The in-vehicle device further comprises a communication unit that communicates with a user's terminal device that monitors the vehicle. A method further comprising transmitting detection information to the terminal device via the communication unit, indicating that the control unit has detected at least one of a passenger in the first area and an object in the second area. [Note 4] The method described in Appendix 1, A method further comprising stopping the opening and closing operation of the door again if the control unit detects an object present in the second area during the opening and closing operation of the door. [Note 5] The method described in any one of the appendices 1 to 4, A method wherein the first area includes the second area. [Note 6] The method described in any one of the appendices 1 to 4, The in-vehicle device further includes a projection unit that projects an image onto the floor surface of the vehicle. A method further comprising the control unit projecting an image showing at least one of the first area and the second area onto the floor surface via the projection unit. [Note 7] The method described in any one of the appendices 1 to 4, The control unit, In the aforementioned video, a third area is further set whose area changes according to the degree to which the door is opened during the opening and closing operation, When an object present in the third area that is captured in the aforementioned video is detected using image analysis technology, the opening and closing operation of the door is stopped again. Methods that further include the above. [Note 8] The method described in Appendix 7, The method wherein the third area has an area that decreases as the degree to which the door is opened decreases. [Note 9] An in-vehicle device comprising a camera unit for photographing the interior of a vehicle and a control unit, The control unit, In the video captured by the aforementioned camera unit, a first area and a second area are defined on the front side of the interior door of the vehicle. From the aforementioned video, passengers and objects are extracted using image analysis technology. An in-vehicle device that stops the opening and closing operation of the door when it detects at least one of a passenger in the first area and an object in the second area that is captured in the aforementioned video. [Note 10] The in-vehicle device described in Appendix 9, The second area is an in-vehicle device having a smaller area than the first area. [Note 11] The in-vehicle device described in Appendix 9, The in-vehicle device further comprises a communication unit that communicates with a user's terminal device that monitors the vehicle. An in-vehicle device in which the control unit transmits detection information to the terminal device via the communication unit indicating that it has detected at least one of a passenger in the first area and an object in the second area. [Note 12] The in-vehicle device described in Appendix 9, The control unit is an in-vehicle device that, when it detects an object in the second area during the opening and closing operation of the door, stops the opening and closing operation of the door again. [Note 13] An in-vehicle device described in any one of the appendices 9 to 12, An in-vehicle device in which the first area includes the second area. [Note 14] An in-vehicle device described in any one of the appendices 9 to 12, The in-vehicle device further includes a projection unit that projects an image onto the floor surface of the vehicle. The control unit projects an image showing at least one of the first area and the second area onto the floor surface via the projection unit, in-vehicle device. [Note 15] An in-vehicle device described in any one of the appendices 9 to 12, The control unit, In the aforementioned video, a third area is further set whose area changes according to the degree to which the door is opened during the opening and closing operation. An in-vehicle device that, when it detects an object present in the third area that is captured in the aforementioned video using image analysis technology, stops the opening and closing operation of the door again. [Note 16] The in-vehicle device described in Appendix 15, The third area is an in-vehicle device having an area that decreases as the degree to which the door is opened decreases. [Note 17] A computer as an in-vehicle device, which includes a camera unit for photographing the interior of a vehicle and a control unit, In the video captured by the aforementioned camera unit, a first area and a second area are defined on the front side of the interior door of the vehicle. The process involves extracting passengers and objects from the aforementioned video using image analysis technology, and stopping the opening and closing of the door when at least one of the passengers in the first area and the objects in the second area that are captured in the video is detected. A program that performs an action that includes the following: [Note 18] The program described in Appendix 17, The second area is a program having a smaller area than the first area. [Note 19] The program described in Appendix 17, The in-vehicle device further comprises a communication unit that communicates with a user's terminal device that monitors the vehicle. The program further includes transmitting detection information to the terminal device via the communication unit indicating that at least one of a passenger in the first area and an object in the second area has been detected. [Note 20] The program described in Appendix 17, The program further includes stopping the opening and closing of the door again if it detects an object in the second area during the opening and closing of the door. [Explanation of symbols]

[0046] 10 Onboard equipment 11 Control Unit 12 Storage section 13 Communications Department 14 Input section 15 Output section 16. Photography Department

Claims

1. A method performed by an in-vehicle device comprising a camera unit for photographing the interior of a vehicle and a control unit, The control unit, In the video captured by the aforementioned camera unit, a first area and a second area are defined in the front of the door inside the vehicle. The process involves extracting passengers and objects from the aforementioned video using image analysis technology, When at least one of a passenger in the first area and an object in the second area is detected in the aforementioned video, the opening and closing operation of the door is stopped. Methods that include...

2. The method according to claim 1, A method wherein the second area has a smaller area than the first area.

3. The method according to claim 1, The in-vehicle device further comprises a communication unit that communicates with a user's terminal device that monitors the vehicle. A method further comprising transmitting detection information to the terminal device via the communication unit, indicating that the control unit has detected at least one of a passenger in the first area and an object in the second area.

4. The method according to claim 1, A method further comprising stopping the opening and closing operation of the door again if the control unit detects an object in the second area during the opening and closing operation of the door.

5. A method according to any one of claims 1 to 4, A method wherein the first area includes the second area.

6. A method according to any one of claims 1 to 4, The in-vehicle device further comprises a projection unit that projects an image onto the floor surface of the vehicle. A method further comprising the control unit projecting an image showing at least one of the first area and the second area onto the floor surface via the projection unit.

7. A method according to any one of claims 1 to 4, The control unit, In the aforementioned video, a third area is further set whose area changes according to the degree to which the door is opened during the opening and closing operation, When an object present in the third area that is captured in the aforementioned video is detected using image analysis technology, the opening and closing operation of the door is stopped again. Methods that further include this.

8. The method according to claim 7, The method wherein the third area has an area that decreases as the degree to which the door is opened decreases.

9. An in-vehicle device comprising a camera unit for photographing the interior of a vehicle and a control unit, The control unit, In the video captured by the aforementioned camera unit, a first area and a second area are defined in the front of the door inside the vehicle. From the aforementioned video, passengers and objects are extracted using image analysis technology. An in-vehicle device that stops the opening and closing operation of the door when it detects at least one of a passenger in the first area and an object in the second area that is captured in the aforementioned video.

10. The in-vehicle device according to claim 9, The second area is an in-vehicle device having a smaller area than the first area.

11. The in-vehicle device according to claim 9, The in-vehicle device further comprises a communication unit that communicates with a user's terminal device that monitors the vehicle. An in-vehicle device in which the control unit transmits detection information to the terminal device via the communication unit indicating that it has detected at least one of a passenger in the first area and an object in the second area.

12. The in-vehicle device according to claim 9, The control unit is an in-vehicle device that, when it detects an object in the second area during the opening and closing operation of the door, stops the opening and closing operation of the door again.

13. An in-vehicle device according to any one of claims 9 to 12, An in-vehicle device in which the first area includes the second area.

14. An in-vehicle device according to any one of claims 9 to 12, The in-vehicle device further comprises a projection unit that projects an image onto the floor surface of the vehicle. The control unit projects an image showing at least one of the first area and the second area onto the floor surface via the projection unit, in-vehicle device.

15. An in-vehicle device according to any one of claims 9 to 12, The control unit, In the aforementioned video, a third area is further set whose area changes according to the degree to which the door is opened during the opening and closing operation. An in-vehicle device that, when it detects an object present in the third area that is captured in the aforementioned video using image analysis technology, stops the opening and closing operation of the door again.

16. The in-vehicle device according to claim 15, The third area is an in-vehicle device having an area that decreases as the degree to which the door is opened decreases.

17. A computer as an in-vehicle device, which includes a camera unit for photographing the interior of a vehicle and a control unit, In the video captured by the aforementioned camera unit, a first area and a second area are defined in the front of the door inside the vehicle. The process involves extracting passengers and objects from the aforementioned video using image analysis technology, When at least one of a passenger in the first area and an object in the second area is detected in the aforementioned video, the opening and closing operation of the door is stopped. A program that performs an action that includes the following:

18. The program according to claim 17, The second area is a program having a smaller area than the first area.

19. The program according to claim 17, The in-vehicle device further comprises a communication unit that communicates with a user's terminal device that monitors the vehicle. The program further includes transmitting detection information to the terminal device via the communication unit, indicating that at least one of a passenger in the first area and an object in the second area has been detected.

20. The program according to claim 17, The program further includes stopping the opening and closing of the door again if it detects an object in the second area during the opening and closing operation of the door.

Images